[htian-dev]

The pin is a temporary workaround for this week's public release. The SQLx fix is still unmerged.

1. We considered a maintained fork, but repository permissions blocked it.
2. We will move to an official SQLx release once it includes the required fixes and is verified with Buzz. Revisit the fork if needed.

Bug

https://linear.app/squareup/issue/BUZZ-656/relay-writer-pool-connections-stranded-after-a-burst-bb-public-2026-11

A short database interruption can leave the relay unable to handle database requests after traffic returns to normal. All connection slots can remain occupied even though no request is using them and PostgreSQL is idle. New requests then time out waiting for a slot.

Cause

SQLx, our database library, returns connections to the pool in a background task. If a socket stays open but stops answering, the final health check has no timeout. That task keeps its slot, and idle-connection cleanup cannot reclaim it (#4349).

Fix

Pin SQLx to c3735abf from PR #4407. After five seconds, it stops a stalled return operation, discards the connection, and frees the slot so new requests can open a replacement.

Preserve the existing transaction rollback (#4394) and SQLite compatibility (#4402) fixes. Give the reader's own check four seconds so it can record its timeout before the outer deadline.

Critical risk

This fix depends on a SQLx patch that has not been merged. Its five-second limit affects every pool using this revision, so it can also discard slow but healthy connections and cause more reconnections. The local reproduction proves recovery from the injected failure; it does not establish production behavior.

Prevent accidental replacement: require SQLx =0.9.0, disable Renovate updates for all seven crates, and fail required CI if the approved revision, lockfile source, or update restriction changes. These guards prevent version drift; they do not remove the runtime risk of the patch. An upgrade must explicitly update the policy after verifying all required fixes.

Before and after evidence

With real PostgreSQL 17.6 on macOS arm64, a proxy silenced 100 existing connections while allowing new connections. All borrowers dropped their connections; PostgreSQL showed 100 idle sessions.

Observation	Before	After
Pool slots still occupied	100 after 10 seconds	0 after 5.008 seconds
New work	Next checkout timed out	All 100 replacement connections served SELECT 1

Captured before and after output

Before

REPRO burst: 100 dropped borrowers, all release pings held: size=100 idle=0 active=100
REPRO burst DB sessions: [("idle", "ClientRead", 100)]
REPRO burst after 10.000680583s: size=100 idle=0 active=100
REPRO burst: later checkout timed out despite PostgreSQL being healthy

After

REPRO burst: 100 dropped borrowers, all release pings held: size=100 idle=0 active=100
REPRO burst DB sessions: [("idle", "ClientRead", 100)]
REPRO burst after 5.008456917s: size=0 idle=0 active=0
REPRO burst: all 100 replacement slots serve SELECT 1 while old sessions stay silent

Before evidence used Buzz 96af488d968a1ae9d961c2132bea27822735f8da with SQLx f1e94ec3. After evidence used Buzz 4bfe60197e8c390a8701053228cd50c5391f37d3 with the SQLx pin committed in f253760085e24358728fa4f4fef059ccf892157f. These are local fault-injection results: they establish the stranded-slot mechanism and recovery, not what originally interrupted production database traffic.

[github-actions]

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

• Exact PR diff: 31fa53174d5bf7995c6cab18954514cc036125e2...50634902f9e6edcf2079ff182e97b94e9bd8877e
• Model: gpt-6.1-sol

💡 Click "edited" above to see earlier reviews for this PR.

---

Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability findings were identified in the authorized PR range.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

• Review used read-only static inspection; no scripts, builds, or tests were executed.
• The newly pinned SQLx revision was unavailable locally, so its implementation could not be independently inspected under the network restriction.

---

_{Generated by Codex Security Review |
Requested by: @htian-dev |
Workflow run}