Repository navigation
fix(relay): recover from network loss automatically with one retry owner - #992
Merged
Merged
Conversation
loganj
marked this pull request as ready for review
October 13, 2026 20:57
kalvinnchau
reviewed
Oct 13, 2026
kalvinnchau
left a comment
Contributor
There was a problem hiding this comment.
🤖 Code-only review: three actionable recovery regressions below. Recommend fixing these before merge. Runtime validation was unperformed.
loganj
force-pushed
the
larry/auto-retry-recovery
branch
from
October 13, 2026 22:11
0bfcefa to
1325dfd
Compare
kalvinnchau
reviewed
Oct 13, 2026
kalvinnchau
left a comment
Contributor
There was a problem hiding this comment.
🤖 Follow-up code-only review: preview recovery and notice visibility fixes are addressed. One new terminal-live recovery gap remains below. Runtime validation unperformed.
kalvinnchau
approved these changes
Oct 13, 2026
kalvinnchau
left a comment
Contributor
There was a problem hiding this comment.
🤖 approved. the missing conversation retry after an auth/signing failure can be handled as a non-blocking follow-up; ActivityPanel retry or restarting provides a workaround. code-only review; runtime validation unperformed.
loganj
force-pushed
the
larry/auto-retry-recovery
branch
from
October 14, 2026 00:49
1325dfd to
92efddd
Compare
Failed relay reads now retry with capped backoff (1s to 30s) and never give up. Live streams drop their 5-attempt limit. A wake, online, focus or live reconnect retries at once. Edge 401/403 pages (VPN reconnecting) are treated as unavailable, not as relay denials. The per-panel Retry buttons are replaced by one "Reconnecting to relay…" notice with one Retry now. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
Automatic recovery now owns only failures that a later attempt can fix: network, timeout, cancellation, relay unavailable, rate limits and relay error: closes. A denial, a terminal sign-in failure or a deterministic check (such as the recipe catalog read limit) keeps its own message and Retry, and the connection no longer reconnects on those failures. Suspended public previews whose metadata recheck failed now count as a failed recovery source, so the timer and wake signals recheck them. Access stays suspended until fresh signed metadata succeeds. The notice keeps a separate visibility latch, so Retry now and wake signals restart the delay without hiding the notice mid-retry. When recovery removes a focused Retry now button, focus moves to the composer. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
A live CLOSED reason now needs automatic retry unless it is a known needs-action prefix (restricted:, blocked:, invalid:, auth-required:, unsupported:, pow:, duplicate:) or a deterministic local failure. Relays use other and unprefixed reasons for outages; the previous allowlist left those routes stuck. The browser host's preference decoder now reports a 5xx as an unavailable ReadError, and transient classification follows an error's cause through context wrappers such as "Preference decode:". Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
A rejected live AUTH or failed signing leaves live updates in a terminal error. Automatic recovery correctly skips it, but the conversation had no control to try again. Show a Retry live updates button for terminal and permanent live failures; capacity and partial-roster notices stay passive. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
Main's preview polish (#828) focuses a preview's history after Retry, because a preview has no composer. Recovery's focus handoff now does the same. An empty timeline that failed during an outage no longer shows its own Retry messages next to the recovery notice; it waits for the notice. The preview recovery browser tests now drive Reconnecting to relay… and Retry now instead of the removed per-failure warning. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
loganj
force-pushed
the
larry/auto-retry-recovery
branch
from
October 14, 2026 15:40
92efddd to
1a4fc6b
Compare
… clock Main (#1039) now pauses the page clock before the roster refusal so the ignored Retry clicks land inside the cooldown. Automatic recovery shows its notice with the first retry, 1 s after the refusal, so run the paused clock that far before expecting the notice. Cross the cooldown in the broker's real time first, then on the page clock: the queued automatic retry sends as soon as the page deadline passes, and a second refusal would arm a deadline the paused clock never reaches. Listen for the observer update before crossing, because automatic recovery can read the roster during the crossing. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
The clean pending setup test holds Beta's EOSE through a channel switch and diagnostics. The production broker runs live setup in Node, so its 10 s setup deadline uses real time and the page clock cannot hold it. On a cold WebKit worker in the flake check, the deadline passed: the route failed and recovery showed "Reconnecting to relay…", which the test counts as a warning. Let the broker take the live setup deadline as an option, default unchanged at 10 s, and give this one test 120 s. The quota recovery tests in the same file still use the real deadline. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
johnmatthewtennant
added a commit
that referenced
this pull request
Oct 14, 2026
…-sessions * origin/main: (28 commits) docs: link the Buzz feedback community and @buzzdotxyz on X (#1091) feat(claude-code): save a thinking level in agent settings (#1089) feat(reminders): polish the Reminders page (#1048) fix(messages): keep thread panes beside channels when space allows (#1093) fix(inbox): show matching mentions instead of ordinary replies (#822) fix(relay): recover from network loss automatically with one retry owner (#992) fix(search): treat a full author lookup page as ambiguous (#1043) Show agent sessions in the profile Activity tab (#1040) fix(direct-messages): search again for people added after the directory completed (#1044) fix(claude-code): offer every model Claude lists (#1086) fix(plugins): reload the exact recorded source folder (#1030) test(browser): stop WebKit trace screenshots leaking macOS WindowServer surfaces (#1071) fix: decode broker request bodies as a utf-8 stream (#1038) fix(linux): use slight GTK hinting for clearer font spacing (#1047) Make Me conversations feel like project chats (#1005) fix(codex): deliver custom instructions without codex truncation (#1059) test(browser): fix the notification-reveal and nested-replies WebKit flakes (#1087) fix(messages): keep following after a notification reveals the latest row (#1085) Add native nsec backup HTTP client (#995) feat(buzz-mcp): add channel and member management tools (#1055) ... Signed-off-by: John Tennant <jtennant@block.xyz> # Conflicts: # src/bundled/sessions/SessionsPage.loading.test.tsx # src/bundled/sessions/SessionsPage.tsx
johnmatthewtennant
added a commit
that referenced
this pull request
Oct 14, 2026
…lation * origin/main: (29 commits) docs: link the Buzz feedback community and @buzzdotxyz on X (#1091) feat(claude-code): save a thinking level in agent settings (#1089) feat(reminders): polish the Reminders page (#1048) fix(messages): keep thread panes beside channels when space allows (#1093) fix(inbox): show matching mentions instead of ordinary replies (#822) fix(relay): recover from network loss automatically with one retry owner (#992) fix(search): treat a full author lookup page as ambiguous (#1043) Show agent sessions in the profile Activity tab (#1040) fix(direct-messages): search again for people added after the directory completed (#1044) fix(claude-code): offer every model Claude lists (#1086) fix(plugins): reload the exact recorded source folder (#1030) test(browser): stop WebKit trace screenshots leaking macOS WindowServer surfaces (#1071) fix: decode broker request bodies as a utf-8 stream (#1038) fix(linux): use slight GTK hinting for clearer font spacing (#1047) Make Me conversations feel like project chats (#1005) fix(codex): deliver custom instructions without codex truncation (#1059) test(browser): fix the notification-reveal and nested-replies WebKit flakes (#1087) fix(messages): keep following after a notification reveals the latest row (#1085) Add native nsec backup HTTP client (#995) feat(buzz-mcp): add channel and member management tools (#1055) ... Signed-off-by: John Tennant <jtennant@block.xyz>
3 tasks
wesbillman
added a commit
that referenced
this pull request
Nov 5, 2026
) ## Problem On an account in many large channels, the desktop app keeps showing **"Reconnecting to relay…"** and loads slowly. This reproduces on the Block relay with `desktop-v0.7.10` and with current `main`. A diagnostic build logged each `/query` request and why each background read failed. It showed: - **Roster read.** The full roster scan page `[{"#p":[viewer],"kinds":[39002],"limit":500}]` returns **2.5 MB** (468 rosters). Five channels with 2k–5.9k members account for about 76% of that. The relay sends headers in about 300 ms with a `Content-Length`, then delivers the body at about **140–205 KB/s**. For comparison, the same machine downloads from Cloudflare at about 3 MB/s. - **Deadline.** The reader's ten-second deadline (which includes queue time) cancels that read on every attempt, failing with `Relay read timed out`. - **Retry loop.** Since #992, a transient roster failure is a recovery source that retries forever. The banner therefore stays up, and each retry downloads the same 2.5 MB page again. - **Read-state snapshot.** It is **5 MB** (119 kind-30078 events) and takes about 25 s. It also hits the ten-second deadline and competes for bandwidth. No client change made these reads bigger or slower. The request shape, the ten-second deadline and the uncompressed native client are all unchanged since beta.1. The account's data crossed a threshold that the read budget allows (8 MiB) but the deadline cannot accommodate: an 8 MiB read in ten seconds needs about 800 KB/s. ## Change - **Reader option.** `ReadOptions.deadlineMs` replaces the reader's deadline for new work. A read that joins in-flight work keeps that work's deadline. - **Constant.** `LARGE_READ_DEADLINE_MS = 30_000` matches the native transport's per-request ceiling in `src-tauri/src/relay.rs`. - **Which reads use it.** Full roster discovery reads (the viewer scan, the system scan and omission confirmation) and the read-state snapshot. These are the reads whose results can approach `RELAY_READ_BYTES`. - **Read-state budget.** The read-state owner's refresh and pre-send snapshot used a caller signal with its own 10 s timeout, which cancelled the snapshot first. An operation that reads a complete snapshot now gets the large-read deadline plus its usual 10 s. - **Unchanged.** All other reads keep ten seconds. A stalled roster page still fails at 30 s and enters recovery as before, and so does a link that needs more than 30 s per page. The browser development broker keeps its 20 s upstream cap (documented, not changed). - **Docs.** `docs/relay-queries.md` records the exception to the documented read bounds. **Non-goals.** This is a targeted deadline change, not a permanent fix for large accounts. The 5 MB snapshot already needs about 25 s at the observed rate. The real fixes are listed under follow-ups. ## Validation - **Live app.** I ran native `BUZZ_DEV_VIEWER= just desktop` against the Block relay, with production federated-identity flags, on the affected account. - Unmodified `main` (`bac7a4fcc`) showed the banner. - With this change hot-reloaded, the human confirmed that the channel list loads and the banner clears. - A `desktop-v0.7.10` diagnostic build with a 60 s deadline showed the same result. Its read timeouts dropped from 39 to 0, and the roster finished in 12–17 s. - **New tests.** Each fails without its fix and passes with it: - `session-read-retry.test.ts`: a full roster page held for 15 s still settles, with exactly one request. - `session-read-retry.test.ts`: a stalled page fails at 30 s (not at 29.999 s) as a transient error. - `reader.test.ts`: `deadlineMs` overrides the deadline, and a read that joins in-flight work keeps its deadline. - `read-state.test.ts`: through the real reader, a 25 s snapshot completes on refresh, and before a send when a gap drops the capacity basis while the send waits for the publisher lock. It fails at 10 s with either read-state budget reverted. - **Wider suite.** `vitest run src/features/relay` passes at `6e349a69a` (3,049 tests). `tsc`, biome and the commit hooks pass. - **No browser cases added or removed.** **Deferred checks:** - [ ] Agent review - [ ] Second machine - [ ] Full CI ## Follow-ups (not in this PR) - **Roster size.** Membership discovery downloads every member of every channel. Find a way to get the viewer's channel IDs without full `p` lists. - **Read-state growth.** Read-state slots from abandoned installations are never retired, so the snapshot grows with device history. - **Snapshot cancellation.** The native read-state snapshot (`native.ts` `readStateSnapshot`) does not pass its `signal` to `nativeRelayRequest`. The download keeps running after the reader gives up. - **Compression.** The native reqwest client has no `gzip` feature. Cloudflare compresses when the client asks for it. - **Throughput.** Large relay responses are delivered at about 200 KB/s. That is outside this repo, in the gateway or mesh in front of the relay. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖
Summary
Screenshots
Both runs use the same browser test scenario: open the app, drop the relay connection and answer every relay request with a non-relay 403 page (like a VPN still reconnecting), then restore the network.
Before — network down (channel Alpha has gone from the sidebar, "Relay read cancelled", a manual "Retry live updates"):

Before — 30 seconds after the network returns (unchanged; it stays broken until you click or restart):

After — network down (loaded messages and channels stay; one notice, one Retry now):

After — network returns, no click (notice gone, channels and messages load again):

Details
src/features/relay/recovery.ts). Each background read (live stream and channel list, channel templates, Me placement, sidebar preferences, hidden conversations) reports failed / busy / settled. While any source fails, the owner retries every failed source on one shared backoff. A retry still in flight keeps the notice up, so the delay does not reset into a 1-second loop.restricted:,blocked:,invalid:,auth-required:,unsupported:,pow:,duplicate:). A denial, a terminal sign-in failure or a deterministic check (for example the recipe catalog read limit) keeps its own message and its own Retry button, and the connection does not reconnect on it. Retrying those forever would hide the reason and change nothing (errors.tsisTransientFailure,live-recovery.ts).CLOSED restricted:and its metadata recheck fails, the recheck is now a failed recovery source, so the timer and wake signals retry it. Access stays suspended until fresh signed metadata succeeds.