plugins: defer spenderp awaiting-channel recovery

[bittylicious]

Summary

spenderp currently scans CHANNELD_AWAITING_LOCKIN / DUALOPEND_AWAITING_LOCKIN channels during plugin init and queues signpsbt recovery attempts for saved funding PSBTs.

On a large mainnet node, one such signpsbt took about 130 seconds before failing with:

Failed signpsbt for waiting channel ...: {"code":-32602,"message":"Could not add keypaths to PSBT?"}

While that recovery work was in the plugin init path, unrelated important builtin plugins were still waiting to finish startup and hit their init timeout. The plugin named in the fatal log varied depending on which builtins were enabled (autoclean, chanbackup, commando, funder, cln-bwatch, topology), causing lightningd to shut down/restart.

This PR defers the awaiting-channel PSBT recovery scan to a plugin timer that runs after startup. The recovery behaviour is preserved, but slow or failing recovery signing no longer blocks spenderp init or the global plugin startup handshake.

Reproduction context

Observed on CLN v26.06 and v26.06.2 with a SQLite wallet DB that was initially around 9.1 GiB, later vacuumed to around 4.7 GiB. Vacuum/integrity checks and upgrade to v26.06.2 did not change the symptom.

The triggering channel was in CHANNELD_AWAITING_LOCKIN, with:

• funding.withheld=false
• a top-level funding_txid
• a saved funding.psbt

Disabling spenderp stopped the startup crash loop. Re-enabling only the funding-related builtin group reproduced the slow signpsbt and later spenderp failure log.

Live validation

This branch was compiled in a Docker build using the existing CLN build Dockerfile. The build completed successfully, including recompilation/linking of plugins/spender/openchannel.c and plugins/spenderp.

A patched build was then deployed on the affected mainnet node with spenderp enabled again. Startup reached Server started cleanly with the normal plugin set. After the delayed recovery timer fired, spenderp retried the awaiting-lockin channel recovery and still hit the original wallet/PSBT problem:

plugin-spenderp: Failed signpsbt for waiting channel ...: {"code":-32602,"message":"Could not add keypaths to PSBT?"}

Crucially, this no longer happened during plugin init. The daemon did not exit or restart, RPC stayed responsive, and getinfo returned normally after the delayed retry/failure.

Notes

This does not attempt to decide whether that saved PSBT should be retried at all. It only prevents slow recovery signing from being startup-fatal.

A follow-up may still be useful to narrow the retry criteria for saved funding PSBTs that already correspond to a broadcast funding transaction.

Changelog

Changelog-None

[cdecker]

Excellent work @bittylicious, do you have any idea why the signpsbt call might be this slow? I like the workaround, but I'm curious as to the root cause.

[bittylicious]

I don't know the definitive root cause yet, but my strongest suspicion is that the time is being spent in wallet/PSBT keypath matching before signpsbt eventually fails with:

Could not add keypaths to PSBT?

The affected channel already had a funding_txid, but because it was still in CHANNELD_AWAITING_LOCKIN, spenderp retried the saved funding.psbt during startup. My guess is that this saved PSBT no longer maps cleanly to the wallet's derivation data/current wallet state, so CLN does a lot of wallet matching work before concluding it cannot add the keypaths.

The large wallet DB may be a contributor, but probably not the whole explanation: on this node the SQLite wallet DB was about 9.1 GiB, VACUUM reduced it to about 4.7 GiB with clean integrity checks, and the slow signpsbt behaviour remained.

So I think the workaround is still useful because it prevents this recovery path from being startup-fatal, but it does not explain or fix the underlying slow signpsbt. The next things I would look at are:

• compare the saved funding.psbt for the affected channel with the broadcast funding transaction;
• instrument/trace signpsbt, especially the wallet PSBT keypath-matching path;
• check whether this retry should be skipped or guarded once a channel already has a known funding_txid and is merely awaiting lock-in.

Unfortunately I don't really have time to dig deeper into the root cause, but having this guard seems sensible.