[TheSentinel454]

Problem

Each relay node runs one consumer for events that other nodes publish to Redis. It delivered them one at a time and awaited the access gate (visibility and membership lookups) for each event. Under load it fell more than 4,096 events behind the pub/sub broadcast. Each Lagged(n) then skipped n events for good: nothing replayed them, and plain subscribers were never told.

In a 15-node load test, about 50% of cross-node events were skipped at 10x load and about 75% at 13x. Throughput at 13x was also lower than at 10x. Loadgen didn't show this, because it only times events that arrive.

Change

• Concurrent delivery. A dispatcher reads the broadcast and only routes. It drops local echoes and queues each event on one of 16 workers by (community, topic). Each worker delivers its queue in order, so a channel's events keep their order while different channels deliver concurrently. handlers/multinode_fanout.rs
• Known loss closes the affected connections. An event is lost when its worker's bounded queue (1,024 events) is full, or when it isn't delivered within 2 s of this node reading it from Redis, whether it waited in the broadcast, behind a backlog, or in a stalled delivery. ChannelEvent now carries received_at, stamped by the pub/sub subscriber. For a lost event, the relay sends CLOSED "resync: live delivery lagged" to every sync the event would have reached (NIP-AR Sync rule 5). It also closes every connection the event would have reached, so those clients reconnect and catch up from their cursor. Matching runs before the access gate, so a recipient that would have been filtered out may be closed too. That costs a reconnect, never a leak.
• A lost ephemeral event is only counted. Presence, typing and observer frames are never stored, so no reconnect can replay them. Losing one fences and closes nothing; it is still never delivered past its deadline. Without this, one stale global presence event would close nearly every mobile connection in the community, since every mobile client subscribes to presence.
• Unknown loss closes every subscribed connection. If the dispatcher itself lags the broadcast, it can't know which events it skipped. It resyncs every sync (unchanged) and closes every connection that has a subscription.
• Immediate fence. Each affected connection is fenced as soon as the loss is detected. The live fan-out send path (send_to_text_bytes, local and cross-node) delivers it nothing from then on. The fence is written under the connection map's write guard, and the send path checks it and queues its frame under one read guard, so no frame slips in between the check and the send.
• Why 2 s. The mobile client resumes from the newest created_at it received, minus 5 s of slack. The desktop app (block/buzz-app) resumes 960 s before its connection last heard the relay, which already covers the full 900 s created_at drift. While an event waits, local fan-out and other workers keep moving that cursor. The fence stops the cursor, and the deadline means no withheld event is more than 2 s older than the fence, so a reconnect replays it. Without the deadline, an event stuck behind a long backlog could be withheld past the slack and never replayed.
• Paced closes. Only the close is delayed. One closer task closes at most 50 connections per 10 ms (5,000/s per node). A drop in a hot channel therefore can't become a reconnect storm (Review-Proven Rule 9).
• fan_out_pubsub_event is now split into take_local_echo and deliver_pubsub_event, so the dispatcher can check echoes before queueing. Its behavior is unchanged.
• Limits (mobile): replay works while the 2 s deadline, plus the writer's clock skew and its publish delay, stays under the client's 5 s slack. The relay accepts created_at up to 15 min off, so a writer clock more than about 3 s slow can still put an event behind the cursor. An ordinary reconnect has the same limit today; closing it needs a client rewind instruction, which is out of scope here. The desktop app's 960 s resume window has no such gap. A dispatcher that falls behind without overflowing the broadcast still never delivers a late event, but it detects the loss, and fences, late. Events skipped by dispatcher lag are never seen. Neither case is bounded by the deadline, which is a cutoff for attempting delivery, not a guaranteed fence time.
• What a load-test rerun should watch: under sustained overload the deadline turns backlog into steady closes (up to 5,000/s per node), and reconnecting clients replay from the same database pool. Watch buzz_multinode_fanout_dropped_total{reason="stale"}, buzz_multinode_fanout_lag_closed_total and the reconnect REQ rate next to pool wait. On dispatcher lag, the dispatcher scans the subscription registry and fences each subscribed connection synchronously, so also watch how long it stays behind after a Lagged.

Metrics:

• buzz_multinode_fanout_lag_total (dispatcher lag) and buzz_multinode_fanout_total (delivered) keep their meaning.
• buzz_multinode_fanout_dropped_total{reason} counts lost events: queue_full, or stale for one not delivered by its deadline. A lost ephemeral event counts as ephemeral instead and closes nothing.
• buzz_multinode_fanout_lag_closed_total counts close attempts for a loss, one batch at a time.

Rollout

• Staging first. The load-test rerun at 1x, 10x and 13x happens after merge, with the staging load tests planned once the current relay changes land. At 1x, expect zero dropped_total for stored kinds and zero lag_closed_total. Report delivered %, dropped_total{reason}, lag_closed_total, reconnect REQ rate and pool wait.
• Then one production cluster before the rest.
• Rollback trigger: lag_closed_total climbing steadily, or reconnect REQs pushing pool wait up, at normal load. Rollback is a plain revert and redeploy: there are no schema, config or wire changes.

Ordering

Order is kept within a (community, topic). Events on different topics, for example a channel event and a global event, can now reach one subscriber in either order. Before this change they arrived in Redis arrival order. Nothing in the relay depends on order across topics: the access gate re-reads membership and visibility per event. Clients order by created_at.

What got simpler / known debt

• The fan-out loop in main.rs shrinks to one spawn call, and the lag handling lives next to the delivery code.
• No new config knobs. The worker count and queue size are constants until a load test shows they need tuning.
• Known debt, not in this PR: a Redis pub/sub reconnect can still lose events silently for plain subscribers. Syncs are already resynced in that case (artifact_sync::watch_link), but ordinary subscriptions are not.

Tests

handlers::multinode_fanout::tests, each checked with a mutant on Blox:

• events reach their subscriber through the workers, in publish order
• a full worker queue closes exactly the dropped event's recipients
• a lost ephemeral event (full queue or stale) is counted as ephemeral and neither fences nor closes, while a stored event lost on the same queue still closes
• a local echo on a full queue closes nothing
• dispatcher lag closes every subscribed connection
• after a known loss or dispatcher lag, a recipient gets no newer live event while it waits to be closed
• the fence also withholds relay-local fan-out
• an event that waited out its deadline before dispatch is lost, never delivered
• an event behind a backlog is delivered inside its deadline, and past it is lost (its recipient fenced and queued for close), never delivered late (paused clock)
• a delivery that stalls on a database that never answers is cut off at the deadline and handled as lost
• the closer cancels in paced batches (paused clock), and counts each batch as it closes

🤖 Generated with Claude Code

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is 744f3946f6fe1ff7a649c132638d5af608900dbd...d1ba7ee67781064bbd763eba654d8e7776f7192c.
A new review must complete for this exact range. When manual authorization
is required, a user with write access must comment exactly
@buzz-security-review d1ba7ee67781064bbd763eba654d8e7776f7192c to authorize a new review.
Any previous review applies only to its recorded range.

[TheSentinel454]

🤖 Addressed in d1ba7ee.

• Observer frames and other ephemeral kinds: nothing can replay them, so a lost one is now only counted (buzz_multinode_fanout_dropped_total{reason="ephemeral"}). It no longer fences or closes its recipients. Before this PR they were also lost, silently, on Lagged.
• Desktop archive: the in-repo desktop/ app is no longer used. The shipping desktop client (block/buzz-app) resumes 960 s before its connection last heard the relay, which covers the relay's 900 s created_at drift, so a close is replayed there. No change for this one.