镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

fix(relay): publish group state from one locked snapshot - #8898

Open
TheSentinel454 wants to merge 4 commits into
mainfrom
elrond/group-metadata-fence
Open

TheSentinel454 wants to merge 4 commits into
mainfrom
elrond/group-metadata-fence

Conversation

@TheSentinel454

@TheSentinel454 TheSentinel454 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The relay builds kind:39000 (group metadata) and kind:39001 (admins) from a channel read that takes no lock, and replaces each one on its own transaction. A concurrent emission can therefore publish stale state with a newer created_at:

  1. A membership emission reads the channel. The admission auto-join from feat(relay): auto-join new community members to default channels #8594 is one source of these.
  2. A kind:9002 edit commits, for example auto_join=false.
  3. The edit's own emission publishes the new metadata.
  4. The first emission then replaces it with the old snapshot. Its created_at is head + 1, so the stale tags win.

Only kind:39002 had a freshness fence. This race made test_nip29_announce_channel_wire fail on main after #8594 (https://github.com/block/buzz/actions/runs/41341005385). The race itself is older: any membership change racing a metadata edit could hit it.

Change

  • lock_member_snapshot now takes the replacement locks of all three group-state coordinates (39000, 39001, 39002), in kind order, before the membership lock. It also reads the live channel row on the same transaction.
  • lock_member_snapshot reads the channel row once: the roster's identity tags (kept for a deleted channel) and the live row behind 39000/39001 come from the same read, so a labels-only edit cannot land between them.
  • replace_member_event becomes replace_group_state_event. It accepts 39000, 39001 or 39002 for the locked channel and validates the roster only for 39002.
  • emit_group_discovery_events builds all three events from that one snapshot, replaces them on its transaction, commits, and then fans out the events that became heads.

Every channel writer commits before it emits. So an emission either finishes before a later one starts, or it reads the later writer's committed change. A capture of older state can no longer replace a newer one.

The replacement keys are the same ones replace_addressable_event takes. During a rolling deploy, a relay that still writes each kind on its own transaction serializes with the new path. No writer holds one of these keys while waiting for another, so taking all three in order cannot deadlock.

What got simpler

  • emit_addressable_discovery_event is deleted, along with its separate unlocked created_at lookup.
  • The three discovery events now share one capture, one transaction and one timestamp rule (store_group_state_event), instead of two code paths with different freshness guarantees.
  • A group-state publication is now atomic: a failure rolls back all three events instead of leaving 39000 published without 39002.

Testing

Verified on Blox at f7d8043:

  • held_group_state_capture_cannot_replace_a_newer_metadata_edit (buzz-relay, Postgres):
    • Setup: hold an emission at a new test hook after it captures auto_join=true, commit auto_join=false, then start the edit's emission.
    • Asserts: the edit's emission waits for the held one, and the final kind:39000 head has no auto_join tag.
    • Mutation: moving the channel read back before the lock (the pre-fix shape) fails the test.
  • locked_member_snapshot_blocks_post_capture_membership_mutation (buzz-db) is extended:
    • Asserts: the held snapshot captures the channel row and holds the 39000 and 39001 replacement keys.
    • Mutation: taking only the 39002 key fails the test.
  • locked_member_snapshot_of_deleted_channel_keeps_identity (buzz-db): a deleted channel's snapshot has no live row but keeps its identity. Mutation: deriving identity from the filtered live row fails it.
  • cargo fmt, clippy (-D warnings), and the full buzz-db (655) and buzz-relay (544) Postgres lanes pass; all three mutations turn their tests RED.
  • E2E on an exact-head relay: 20/20 concurrent join + auto_join=false rounds left 39000/39001/39002 matching SQL, and plain edit/join/leave publish fresh state live.

buzz-review-completed

🤖 Generated with Claude Code

TheSentinel454 and others added 4 commits November 1, 2026 15:27
kind:39000 and kind:39001 were built from an unlocked channel read and
replaced on their own transactions. A membership emission (an admission
auto-join, for example) could read a channel just before a kind:9002
edit committed and replace the edit's own 39000 after it, so the stale
snapshot won with a later created_at. Only 39002 had a freshness fence.

lock_member_snapshot now takes the replacement locks of all three
group-state coordinates before the membership lock and reads the live
channel row on the same transaction, and emit_group_discovery_events
builds and replaces 39000, 39001 and 39002 from that one snapshot.
Every channel writer commits before it emits, so a later emission
either waits for an earlier one or reads the newer state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
The postgres-ci nextest profile only selects tests under a
postgres_tests:: module, so the held-capture regression test was
filtered out of CI. Nest it (and its helper) in one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
lock_member_snapshot read the roster identity and the live channel in
two statements. A labels-only edit does not take the membership lock,
so it could commit between them and one publication would carry
mismatched t tags. Read the row once, keep its identity for a deleted
channel, and expose it as the live channel only when not deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@TheSentinel454
TheSentinel454 marked this pull request as ready for review November 1, 2026 21:09
@TheSentinel454
TheSentinel454 requested a review from a team as a code owner November 1, 2026 21:09
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

  • Exact PR diff: cde6c1bf5b1fa1fde6045de8ea29f6ac1c7e6867...f7d8043d8c3a6d71db1787a063ff502823a14d94
  • Model: gpt-6.1-sol

💡 Click "edited" above to see earlier reviews for this PR.


Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability regressions found in the authorized PR range.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

  • Review used read-only source and history inspection. Builds and tests were not run, as requested.

Generated by Codex Security Review |
Requested by: @TheSentinel454 |
Workflow run

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review at f7d8043: no blocking findings.

This review covered two independent source reads of the diff plus a live run of the PR head against an isolated relay and database.

Lock order. The snapshot now takes the 39000, 39001 and 39002 replacement keys, then the membership key. No writer on this head holds the membership key, the TTL lock or the channel row lock and then waits on a group-state replacement key. Settings writes take membership, TTL, then the row lock. Kick paths take only membership. Migration 0032's trigger adds membership after the 39002 key. replace_addressable_event and the admin backfill hold one coordinate per transaction. So taking all three in kind order cannot invert against any of them, including relays still running the old build during a rollout.

Behavior parity. The old 39000/39001 path read the channel and roster with the same deleted_at IS NULL filters the snapshot uses, so a deleted channel still returns ChannelNotFound and deletion still removes the discovery heads rather than publishing a final set. The locked replace keeps the old path's stale-write check, follow-up hook and mention indexing. Roster validation stays 39002-only. No references to the removed helpers remain, and the new test hook is gated by #[cfg(test)] at both the call site and the module.

Live run. In 24 unforced rounds, a kind:9021 join raced a kind:9002 edit setting auto_join=false. In every round, the final 39000/39001/39002 heads matched SQL, arrived live on a channel subscription, and had no stale auto_join or label tags. A plain edit, a labels-only edit, a join, an admin promotion and a leave each published three fresh heads with strictly increasing timestamps, with correct signatures, identity tags, admin roles and roster.

Note (non-blocking). Sharing replacement keys makes old and new relays serialize, but it does not make old relays fresh: an old relay still reads the channel before locking. Group-state freshness is fully guaranteed only once every relay runs this build. The live run did not exercise a mixed-version deployment.

@bradseiler

Copy link
Copy Markdown
Contributor

🤖 Rollout notes from McKeon, Brad Seiler's review agent, at f7d8043. These are suggestions, not an approval.

This change has no flag: it takes effect when the relay image deploys. The PR body has no rollout section yet, so here's what I'd add:

  1. Mixed-version window. Group-state freshness holds only once every relay runs this build. Old relays still read the channel row before taking any lock. The shared replacement keys make old and new relays serialize with each other, but they don't make old relays' captures fresh. A stale 39000 or 39001 is still possible until the rollout finishes.
  2. What to watch during and after the deploy. 39000 and 39001 now wait behind the membership lock and the 39002 publication, under the 5 s lock_timeout.
    • Replacement and membership advisory-lock wait times (observe_advisory_lock, LockType::Replacement).
    • Lock-timeout errors (SQLSTATE 55P03) on emission.
    • NIP-29 group discovery emission failed warnings. Callers only log these. reconcile_large_channel_member_snapshots repairs 39002 only, so a 39000 or 39001 that times out stays stale until the next change to the channel.
  3. Staging check after deploy. Re-run the concurrent join plus auto_join=false race from the Testing section, and confirm the 39000, 39001 and 39002 heads match SQL. Then do a plain edit, a join and a leave, and confirm each one publishes three fresh heads.
  4. Rollback. A plain revert is safe. There's no schema or migration change, and both builds take the same replacement keys.

@presentyourlove

Copy link
Copy Markdown

Hi @TheSentinel454,

I have resolved the merge conflicts with the latest main branch:

  1. crates/buzz-db/src/store/channel_members.rs: Combined the kind == 39002 check for roster snapshots with the superseded_with_mentions_sql! macro from main (fix(db): drop a superseded head's mention rows with it #8948) so superseded mentions are properly cleaned up across all group state events.
  2. crates/buzz-relay/src/handlers/side_effects.rs: Retained the newly merged channel rename and operator authority helpers (PRIVILEGED_METADATA_TAGS, operator_may_rename) from main while cleanly removing the deprecated un-fenced emit_addressable_discovery_event.

A clean, zero-conflict branch is available at:
https://github.com/presentyourlove/buzz/tree/fix/group-metadata-fence-8898

You can pull it directly into your branch:

git fetch https://github.com/presentyourlove/buzz.git fix/group-metadata-fence-8898
git merge --ff-only FETCH_HEAD

bradseiler added a commit that referenced this pull request Nov 5, 2026
## Summary

Cold login currently downloads full channel rosters to learn the
viewer's memberships. This defines compact discovery in NIP-CL and
reserves relay-only kind 39008 for a signed, request-bound response.
Full kind-39002 rosters remain compatible.

The contract binds a complete snapshot to the community, viewer, NIP-98
request and requested channel types. It defines a primary database cut,
overflow errors, lazy-roster precedence, reconnect repair and safe
fallback across mixed relay versions. The next PR implements both query
modes. This PR does not enable either capability.

A separate `membership_public` capability uses relay-only kind 39009 for
lookups of any subject’s listed-open forum, stream and workflow
memberships. It preserves roles and archived state while excluding
private, unlisted, DM and system channels. Signed answers bind
requester, subject and public scope; they cannot grant membership or
sending permissions. The self-mode contract is unchanged.

### Related issue

[BUZZ-615](https://linear.app/squareup/issue/BUZZ-615). No duplicate
compact-membership implementation found. Related work:
[#8898](#8898) protects full-roster
signing; [#9060](#9060) caches
membership lookups.

### Testing

McKeon independently reviewed the contract and the fixes to request
binding, per-channel race fencing and mixed-version fallback. Content
checks parse every JSON example and resolve relative links. Paul’s
review fixes remove implementation details from the normative text and
move the existing 39009 registration and regression test into this PR.
Protocol behavior is unchanged. Runtime validation belongs to the
implementation PR. This description does not attest human testing.

### Implementation and operator notes

Both kinds are registered in `crates/buzz-core/src/kind.rs`, included in
`ALL_KINDS`, and rejected as client-submitted events by the relay-only
guard. The dependent implementation uses one primary SQL statement’s
MVCC snapshot over active `channel_members` joined to nondeleted
channels, supported by the partial `(community_id, pubkey) WHERE
removed_at IS NULL` index. Public requests share the existing
community/requester `ApiCalls` quota configured by
`human_api_calls_per_min`.

`BUZZ_MEMBERSHIP_SNAPSHOT_ENABLED` and `BUZZ_MEMBERSHIP_PUBLIC_ENABLED`
independently default to false. Deploy support across the fleet with
both flags off, then enable capabilities independently. Disable the
affected capability to roll back; client fallback and back-off behavior
remain specified in NIP-CL. Client opt-in and production-shaped
throttled validation follow separately.

Generated with Codex

---------

Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>
bradseiler added a commit that referenced this pull request Nov 6, 2026
## Summary

Cold login downloads other people's full rosters just to find the
viewer's channels. This adds a compact projection from canonical
memberships through the existing Nostr `/query` bridge. One primary SQL
statement reads the viewer's rows and channel state, then the relay
signs that exact result.

Merged spec PR #9117 owns both relay-only kind registrations. Self mode
(`membership_snapshot`, kind 39008) requires the viewer's own NIP-98
key. Public mode (`membership_public`, kind 39009) lets an admitted
requester look up any subject's listed, open forum/stream/workflow
memberships. It excludes private/unlisted channels, DMs, system channels
and deleted or removed memberships, including shared private channels.
Both modes require payload-bound NIP-98 and return signed request-bound
envelopes; overflow or backend errors fail the whole request. Legacy
kind-39002 reads and signatures stay intact. No schema migration or
persisted per-user list is added.

### Related issue

[BUZZ-615](https://linear.app/squareup/issue/BUZZ-615). Implements the
compact discovery contract merged in #9117. Related:
[#8898](#8898),
[#9060](#9060). Client lazy-loading
follows separately; this does not include
[buzz-app#2627](block/buzz-app#2627).

### Testing

Current head `5c0c35400a` rebases the five implementation patches
unchanged onto main `c39ee507`, then repairs the removed Redis test API.
McKeon independently verified the rebase and reviewed that repair. Full
core/database/relay package tests, PostgreSQL membership tests (9/9),
Clippy and every mandatory publication hook pass at this head. Old-head
CI run 42132655155 failed six jobs on that same removed API; four
summary checks propagated those failures. New-head hosted CI is pending.
Main advanced to `e3a0f3d59d` during publication; a local merge-tree
check found no conflicts. Historical runtime evidence below remains
attributed to its tested heads.

Prior self-mode validation at `6d2ffceeee` (historical evidence, not
validation of the public-mode change):

- `just ci`, pre-push hooks and all 2,221 core/DB/relay package tests
passed.
- All six new Postgres protocol tests passed. All five guard mutations
failed assertions: mandatory auth, flag gate, single-statement cut,
overflow probe and tenant predicate. A/B are older runs against
production files unchanged from `9784df51`; C/D/E were rerun on final
SQL/fixtures. The earlier surviving tenant mutation is superseded by the
final duplicate-ID witness.
- The full isolated Postgres run finished 1,288/1,294 green; six
existing push/media/FI/startup cases failed. All six passed when rerun
serially after the concurrent local gates stopped. The failed full run
is retained and is not represented as a clean full-suite pass. The
preceding correction `1c47dfc9` had passed all 1,294 tests.
- `just test` passed its unit steps but integration setup was blocked by
Docker Desktop organization sign-in enforcement. The waiting wrapper was
stopped with SIGTERM; this is not a complete pass.
- A real local relay rebuilt at this head returned 67,896 bytes for 468
memberships in 33 ms unthrottled. Signatures, request/viewer/tenant
binding, content identity, wrong-viewer refusal, removal and role
changes verified.

The scale fixture adds five large rosters and 20,000 unrelated channels.
Compact bytes stay unchanged as other members grow; EXPLAIN checks 468
indexed viewer rows and bounded channel-ID lookups. A same-UUID
cross-tenant fixture verifies fields and duplicate rejection
independently of row order.

Durable local evidence: `WORK_LOGS/BUZZ_615_20261105/MANIFEST.md` and
`MANIFEST.json` map commands/results to commits or verified mutant-file
hashes and index the exported logs. Concrete human-test steps and the
signed-request helper are included. Human testing remains pending; ready
state follows the author’s explicit request. Native/browser 140/205 KB/s
acceptance belongs to the client PRs.

Public-mode validation at `c6e269170e`: McKeon independently ran all
nine isolated PostgreSQL membership tests and five mutations. Every
mutation failed its expected assertion: public selection predicate,
subject tag, public routing, identity domain and self serialization
order. The full core/DB/relay package run, all nine isolated PostgreSQL
membership tests and the relay build passed at c6. A real signed HTTP
workflow verified signatures, bindings, identity hashes, all visibility
exclusions, archived guest roles, requester-independent content identity
and malformed-request refusals. After a channel became private, the
public list shrank from three entries to two; a role change appeared as
admin. The local test relay was stopped afterward.

The restacked child `8c56f97d6409` has the same code, tests and
configuration as c6; only the spec differs. A new full core/DB/relay
package run passed at that exact child head. The parent `17a481a89716`
separately passed the full core package check. The first child attempt
used the wrong local database and failed five existing tests on a
missing table; its log is retained separately from the successful
configured run. These checks do not attest human testing.

### Rollout and rollback

This PR does not deploy or enable the feature. Both
`BUZZ_MEMBERSHIP_SNAPSHOT_ENABLED` and `BUZZ_MEMBERSHIP_PUBLIC_ENABLED`
default to false and independently gate advertisement and execution. An
authorized operator sets them in the relay Helm environment managed by
`squareup/block-coder-tf-stacks`. First deploy the image to every
staging pod with both flags off; then enable each capability separately
for staging validation. Enabling either mode does not require or enable
the other. Repeat that dark-then-enable order for bb-block after staging
passes. Opt-in client rollout follows the relay contract and the
throttled desktop/browser workflow.

Watch `buzz_membership_snapshot_query_seconds`,
`buzz_membership_snapshot_encode_sign_seconds`,
`buzz_membership_snapshot_response_bytes`, split by
`mode=membership_snapshot` or `mode=membership_public`, plus `/query`
413/503 responses and primary-pool wait time for the Authorization
operation. Confirm zero unexpected full-roster discovery downloads on
supported clients and correct roles/membership against the canonical
cut. Abort enablement for any binding/signature mismatch, access
leakage, incorrect complete result, stale-response regrant, unexpected
overflow for the 468-channel account, sustained 503s or a material
regression in primary-pool wait/foreground reads. Turn the affected flag
off to roll back. Self-mode clients recheck support and start a new
legacy authority generation; public-mode clients recheck support and
back off without treating public absence as membership loss. Keep cached
display/user data.

The first named usable channel target of five seconds at 140 KB/s
remains a client acceptance target, not a result established by this
relay PR. The 119-event approximately 5 MB read-state contention and
native/browser throttled tests are required before broad client rollout.

Generated with Codex

---------

Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>

This branch was previously deployed

1 inactive deployment
codex-review — f7d8043d Deployed Nov 1, 2026 by TheSentinel454 via Run Codex Security Review #11569
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex-security-review-current The posted Codex security review matches its recorded range.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants