Repository navigation
Revisit ldk-node's persistence architecture #1003
Description
During the July 23 developer call, we discussed that persistence in and around ldk-node has become difficult to reason about. State is managed across LDK, ldk-node, and BDK, while storage implementations and persistence abstractions are split across multiple layers.
Even when these components use the same underlying store, related updates are generally written independently rather than committed atomically. A crash or persistence error can therefore leave protocol state, payment records, wallet state, and user-facing events out of sync. Replay and startup reconciliation repair some cases, but can also produce missing records, stale state, or duplicate events.
Addressing these problems individually is adding more special-case recovery logic without giving us a clear overall persistence contract.
We would like to collect ideas for simplifying this architecture and making its consistency and recovery guarantees easier to understand. Please share possible approaches, tradeoffs, or relevant experience with local, remote, and custom storage backends.
Activity
joostjager commentedon Jul 24, 2026
I think we need atomic transactions, as proposed in #970. Ideally LDK would let us commit its writes together with the related ldk-node state.
Otherwise, ldk-node could buffer the writes made through the store passed to LDK, then commit them atomically with its own writes before completing LDK's pending futures. This would only provide stronger guarantees for backends that support atomic transactions.
MaxFangX commentedon Jul 28, 2026
Maybe you can find some inspiration from Lexe's
PaymentsManager: https://github.com/lexe-app/lexe-public/blob/master/lexe-ln/src/payments/manager.rs, which owns idempotency and consistency, and is safe across crashes, restarts, etc.It's not perfect, as to reason about it you still need to remember how the background processor drives the event handler which then calls into the
PaymentsManager. But once you get into thePaymentsManager, we have independent state machines for every payment flow that Lexe supports, and each state machine is easily audited. For example: our state machine for onchain sends: https://github.com/lexe-app/lexe-public/blob/7b00c7ba01890f98322e846c6f82807e24d6f0ce/lexe-ln/src/payments/onchain.rs#L32-L104We don't flatten all fields into a single struct for persistence; instead, we use an enum
Paymentwhich contains variants for each of our flows. This lets us removeOptionin cases where the field always exists for a particular type of payment.joostjager commentedon Jul 28, 2026
Thanks for the pointer. I think encapsulating the payment logic and defining strict states could indeed be helpful in ldk-node too.
That said, when zooming out, payments are intrinsically part of Lightning. To me, much of the complexity still stems from the persistence boundary between ldk and ldk-node, combined with the lack of atomic batch writes.
TheBlueMatt commentedon Jul 28, 2026
Irrespective of what we can blame for the issues, I think the next step here is to brainstorm what the answer should be - ignoring whether ldk-node and LDK are separate, what kind of (outbound) payment store would we design in a monolithic app? Its not entirely clear to me what the answer to that is. A few ideas:
ldk-nodeentirely - don't worry about persisting an outbound payments until they complete/fail, providing storage for any additional metadata related to them as a part oflightning. ldk-node APIs which list payments would have to queryChannelManager::list_recent_paymentsbefore returning to merge the lists. Gets a bit complicated with pagination, but nothing too wild. Basically, this means we split responsibility for outbound payments between "still in channels, query the channel logic to figure out what payments are there" and "not in channels anymore, at rest in the DB".Events or the Lexe state machine update methods) but also can be queried so LDK is responsible for replaying any missing payments. We could flip which write we require first from the payment store to LDK if we're willing to store payment metadata inlightning. If we did want to go in the direction of batched writes this would ultimately be the first step, just dropping the replay logic in favor of a write batch would require having some interface where we can pass around write batches and fill them up with montior + payment data across subsystems.I think we should scope this specifically to the outbound payment store as we don't have anywhere close to the same kind of issues on the inbound side (there's no replay to handle - you just process events and if you dont finish before a crash the event gets replayed). Excluding things that are just in-progress and unrelated to the specific architectural question, the only linked issues that are relevant are: #171, parts of #966 and #969. Most of the linked issues seem to point to totally unrelated things.
joostjager commentedon Jul 29, 2026
Agreed that we should think about this as if we were designing a monolithic app. But I'm not sure if your proposed options follow that line. They are framed mostly around moving things between lightning crate, ldk, ldk-node or entirely outside ldk-node?
I think the first question to answer is whether we'd atomically commit channel state and payment state together, and go from there.
Regarding the linked issues: I'd say that for example also #968 and #970 are not 'totally unrelated'. I thought a list of directly or indirectly related issues can be useful for context, but I removed the list now, because the level of relevance isn't what we should be discussing.
joostjager commentedon Jul 31, 2026
We discussed this issue in the team meeting. For the short term, the pragmatic direction is to avoid changing LDK and continue making ldk-node’s persistence handling idempotent, with reconciliation where the persisted views can diverge.
I do want to re-highlight the
StagingStoreoption discussed in #970 as a possible intermediate step. It would not require an LDK change, but could allow ldk-node to buffer writes made through the store passed to LDK and commit them atomically with related ldk-node state when the backing store supports atomic batches.joostjager commentedon Sep 3, 2026
Following up on the question here of whether outbound payment state should be committed atomically with channel state, I put together a concrete cross-repository sketch and described it in the one-shot persistence issue.
In ldk-node, an operation such as
send_paymentholds an internal guard while theChannelManagerand payment store are updated, then waits for the background processor to commit those writes together with the manager and monitor state. Corresponding messages and events remain gated until the commit succeeds. Transaction coordination stays in the store and background processor, without passing transaction objects through the call graph.