You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A Set delete occasionally leaves a list key leaf (interface/name, subinterface/index, address/ip) present in telemetry. Once that happens, the leaf is permanent for the life of the process: deleting the leaf, deleting its list entry, and deleting the entire parent interface all return OK and change nothing.
The failure is self-sealing, which is what makes it nasty. The cache is updated from a diff of the ygot tree before and after the Set. If a leaf disappears from the tree without a corresponding Delete reaching the cache, every later delete attempt diffs a tree that already lacks the node — producing an empty diff, so nothing is ever sent to the cache to remove it.
Observed at roughly 3% of runs; permanent once triggered.
Two OPENCONFIG / LEMMING nodes, linked eth1–eth1, eth2–eth2
Client
gNMI Set / Subscribe ONCE, origin openconfig
Bug 1 (primary): list key leaves can become undeletable
Steps to reproduce
Port-forward the DUT's gNMI service, then:
ADDR=localhost:9339 # confirm with: kubectl -n <topo> get svc
ENTRY='/interfaces/interface[name=eth1]/subinterfaces/subinterface[index=0]/ipv4/addresses/address[ip=192.0.2.1]'# 1. Configure an address.
gnmic -a $ADDR --insecure -e json_ietf \
--update-path "openconfig:${ENTRY}/config/ip" --update-value 192.0.2.1 \
--update-path "openconfig:${ENTRY}/config/prefix-length" --update-value 31 set# 2. Tear it down as hard as gNMI allows.
gnmic -a $ADDR --insecure --delete "openconfig:${ENTRY}"set
gnmic -a $ADDR --insecure --delete 'openconfig:/interfaces/interface[name=eth1]'set# 3. Read back what survived.
gnmic -a $ADDR --insecure --mode once --path 'openconfig:/interfaces/interface[name=eth1]' subscribe
Repeat the whole cycle in a loop. Most iterations are clean — step 3 returns nothing. Occasionally it returns something like:
Delete each survivor by its own path, delete each enclosing list entry, and delete the whole interface again — individually or all in one SetRequest. Every call returns OK. The leaf count does not change. It stays for the life of the pod; only a restart clears it.
Every survivor is a list key leaf. Non-key leaves always delete cleanly.
Expected
After delete /interfaces/interface[name=eth1] returns OK, a subsequent read of that subtree returns nothing. Failing that, a second delete of the same path should at minimum remove what the first one missed.
Actual
Intermittently, key leaves survive, and no sequence of Set operations can remove them.
Frequency
~3% of runs over a 150-run campaign. Once triggered on a pod, the residue persists across all later runs on that pod — so long-lived pods accumulate it. A pod running 15+ hours had four immortal leaves, including addresses from test runs hours earlier.
Bug 2 (secondary, likely the same code path): deleting an existing container is rejected
gnmic -a $ADDR --insecure \
--delete 'openconfig:/interfaces/interface[name=eth1]/subinterfaces'set
InvalidArgument: no match found
The container exists and has children at the time of the call. Deleting the list entry below it (subinterface[index=0]) and deleting the interface above it both succeed — only the intermediate container is rejected.
The message originates in ytypes/node.go:362 (retrieveNode), reached via ytypes.UnmarshalSetRequest in unmarshalSetRequest (gnmi/gnmi.go:317).
Note
Lower confidence than Bug 1 — this was observed through a client library that stamps origin: openconfig, so please confirm with a raw gnmic call before treating it as a lemming bug rather than a client path-construction issue. Per the gNMI specification a delete of a path that does not exist is a no-op success, so InvalidArgument looks wrong either way.
Impact
Any workflow that restores a device to a known state by deleting what it added cannot converge. It also silently corrupts later runs on the same pod: leftover addresses from a previous test appear in telemetry and are indistinguishable from current configuration.
TL;DR
A
Setdelete occasionally leaves a list key leaf (interface/name,subinterface/index,address/ip) present in telemetry. Once that happens, the leaf is permanent for the life of the process: deleting the leaf, deleting its list entry, and deleting the entire parent interface all returnOKand change nothing.The failure is self-sealing, which is what makes it nasty. The cache is updated from a diff of the ygot tree before and after the Set. If a leaf disappears from the tree without a corresponding
Deletereaching the cache, every later delete attempt diffs a tree that already lacks the node — producing an empty diff, so nothing is ever sent to the cache to remove it.Observed at roughly 3% of runs; permanent once triggered.
Environment
us-west1-docker.pkg.dev/openconfig-lemming/release/lemming:gasha256:d8d3e15fdfd4411f56bbb93635da73b6fefd31f8ef3ce94cccc4115189707ef8kind, operatorv0.2.9OPENCONFIG/LEMMINGnodes, linkedeth1–eth1,eth2–eth2Set/Subscribe ONCE, originopenconfigBug 1 (primary): list key leaves can become undeletable
Steps to reproduce
Port-forward the DUT's gNMI service, then:
Repeat the whole cycle in a loop. Most iterations are clean — step 3 returns nothing. Occasionally it returns something like:
Then confirm it is permanent
Delete each survivor by its own path, delete each enclosing list entry, and delete the whole interface again — individually or all in one
SetRequest. Every call returnsOK. The leaf count does not change. It stays for the life of the pod; only a restart clears it.Expected
After
delete /interfaces/interface[name=eth1]returnsOK, a subsequent read of that subtree returns nothing. Failing that, a second delete of the same path should at minimum remove what the first one missed.Actual
Intermittently, key leaves survive, and no sequence of
Setoperations can remove them.Frequency
~3% of runs over a 150-run campaign. Once triggered on a pod, the residue persists across all later runs on that pod — so long-lived pods accumulate it. A pod running 15+ hours had four immortal leaves, including addresses from test runs hours earlier.
Bug 2 (secondary, likely the same code path): deleting an existing container is rejected
The container exists and has children at the time of the call. Deleting the list entry below it (
subinterface[index=0]) and deleting the interface above it both succeed — only the intermediate container is rejected.The message originates in
ytypes/node.go:362(retrieveNode), reached viaytypes.UnmarshalSetRequestinunmarshalSetRequest(gnmi/gnmi.go:317).Note
Lower confidence than Bug 1 — this was observed through a client library that stamps
origin: openconfig, so please confirm with a rawgnmiccall before treating it as a lemming bug rather than a client path-construction issue. Per the gNMI specification a delete of a path that does not exist is a no-op success, soInvalidArgumentlooks wrong either way.Impact
Any workflow that restores a device to a known state by deleting what it added cannot converge. It also silently corrupts later runs on the same pod: leftover addresses from a previous test appear in telemetry and are indistinguishable from current configuration.