镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

ci: give WebKit shards headroom and keep fixture servers' idle connections open - #3074

Merged
matt2e merged 2 commits into
mainfrom
mtoohey/ci-webkit-shard-headroom
Nov 9, 2026
Merged

matt2e merged 2 commits into
mainfrom
mtoohey/ci-webkit-shard-headroom

Conversation

@matt2e

@matt2e matt2e commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Fixes the two CI failures of PR #2609's run 42592827846. Neither is caused by that PR; this branch is cut from main and is independent of it.

Summary

Two test-infrastructure fixes, no product code changes:

  • WebKit browser shard 9/14 hits the 15-minute job deadline with every test passing. WebKit journeys now run on eighteen shards.
  • A WebKit lazy import() fails when the Vite fixture server closes an idle keep-alive connection. Fixture and preview servers no longer close idle connections on their own.

Changes

ci: run WebKit browser journeys on eighteen shards (.github/workflows/ci.yml, docs)

  • Root cause: job 129078698059 "Browser journeys (webkit, 9/14)" was cancelled at the browser job's timeout-minutes: 15 with 87 of 92 tests done, all passing. The shard is structurally at the cap on main, not on one slow runner: its whole-job durations in recent runs were 15.1 min (cancelled, this run), 14.9, 14.9, 14.7, 13.7, 12.1, 10.4, 10.2 and 8.9 min, and shard 10/14 runs 13.8–14.1 min; every other shard is ≤ 12.6 min. Playwright shards by test count at file level, and the heaviest files (nested-replies 348 s, navigation-thread-history 253 s, shortcuts 194 s, right-pane-dismissal 188 s) landed together: about 25 min of summed test time on two workers plus ~1.5 min of setup.

  • Fix: eighteen WebKit shards, the smallest count that leaves clear headroom under the unchanged 15-minute deadline. Chromium keeps nine; the matrix excludes shards 15–18 for it (expanded matrix: 9 Chromium + 18 WebKit = 27 jobs). The explanatory comment in ci.yml and the shard counts in docs/browser-testing.md and docs/contributing.md are updated. scripts/flake-check.mjs sizes its own lanes independently of this matrix, so its constants are unchanged.

  • How the count was chosen: per-test durations from this run's browser-journeys-webkit-* artifacts (ci-report.json; shard 9's own report came from main run 42576311891, since the cancelled job uploaded none), combined with playwright test --config tests/browser/playwright.ci.config.mjs --project webkit --list --shard=K/N for each candidate. Predicted job time uses the flake-check model (105 s outside tests + 1.11 × half the shard's summed test time):

    WebKit shards slowest shard: summed test time predicted job time heaviest files in it
    14 (before) 24.9 min 15.6 min (actual 15.1, cancelled; 14.9 on main) nested-replies, navigation-thread-history, shortcuts
    16 20.3 min 13.0 min unread, navigation-groups, agent-activity
    18 18.2 min 11.9 min thread-unread, unread, agent-activity
    20 17.0 min 11.2 min navigation-groups, agent-activity, initial-position

    Calibration on this run: predicted 14.7 vs actual 13.9 min (shard 10), 13.5 vs 12.4 (shard 5), 11.8 vs 10.8 (shard 3). Going from 18 to 20 shards buys 0.7 predicted minutes for two more runners, so 18 it is.

test(browser): keep fixture servers' idle keep-alive connections open (tests/browser/vite-server.mjs, tests/integration/vite-fixture.test.mjs)

  • Root cause: job 129078806819 "Flake check (webkit, 5/11)": reactions.spec.mjs:8 "reaction plus opens a visible emoji-only picker…" failed repeat 3 of 10 at expect(errors.unexplained()).toEqual([]) with Unhandled Promise Rejection: TypeError: Importing a module script failed.; the other 9 repeats and all 43 other results passed. The trace (artifact flake-browser-webkit-5) has exactly one failed request: GET /src/features/messages/reaction-name.ts on the fixture server, "Connection terminated unexpectedly" after 12 ms with no send/wait/receive timings, i.e. the socket was already dead. The previous requests on that server were 4–6 s earlier. Node's http.Server closes a keep-alive connection after its default keepAliveTimeout of 5 s (measured locally: 6.0 s after the last response, with Node's timer granularity); Vite leaves the default. WebKit on Linux reused the idle connection in that instant, the lazy import died with the socket, and nothing retries a module script. tests/browser/page-errors.mjs is deliberately not touched: the app did leave a rejection unhandled, so this is not an engine report to explain away.
  • Fix: keepIdleConnections sets keepAliveTimeout = 0 (Node: "disable the keep-alive timeout behavior on incoming connections") on every fixture dev server, in the fixture plugin's configureServer next to bindAssignedPort so it survives server.restart(), and on every preview server in configurePreviewServer. headersTimeout stays at its 60 s default. Vite destroys open sockets when a server closes, so nothing leaks. A new integration test asserts the setting on a created server, after a restart, and on a preview server; it fails against the unmodified module.

Verification

  • Matrix: expanded ci.yml's browser matrix with the yaml package: 27 jobs, 9 Chromium (1/9…9/9), 18 WebKit (1/18…18/18). playwright test --list --shard=9/14 with and without --no-deps gives the same 92 WebKit tests, so the local listings above match what CI shards.
  • node --test tests/integration/vite-fixture.test.mjs: 9/9 pass; the new test fails (1/1) with tests/browser/vite-server.mjs reverted to main.
  • biome check on the two changed .mjs files: clean.
  • tests/browser/reactions.spec.mjs in WebKit through the CI config against the fixed server: 4/4 pass (one run, one worker).
  • Mechanism, Node level: a plain http.Server with the default closes an idle keep-alive socket 6.0 s after the last response; with keepAliveTimeout = 0 it is still open after 9 s and the client agent reuses it.
  • Not reproduced in a browser: a probe that loads a page from a fixture server, fetches reaction-name.ts, idles, then import()s it never failed in local WebKit (macOS) before the change, including a 2.5 ms sweep from 5.975 to 6.0325 s around the measured close and coarse gaps of 3–8 s. macOS WebKit uses CFNetwork, which retransmits an idempotent request on a stale connection; the CI failure was Linux WebKit (libsoup), which the probe cannot exercise here. The fix therefore rests on the trace evidence and the Node-level mechanism, not on a browser fail-then-pass. After the change the same probe reports keepAliveTimeout 0 and passes at 6 s and 8 s idle.
  • Not run: the full browser suite or a WebKit run on all 18 shards; CI on this PR provides the actual shard durations.

🤖 Generated with Claude Code

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Star Lord’s automated source review via Wes’s account, with independent Rocket, Mantis and Groot coverage.

Reviewed head 7ab8b403bb43e61ca043e59ffcb572f1ed6e4f97 against base/merge-base 58116b826171d190a7376fc57b0509c32c507dde. No blocking source defects found in the shard matrix, fixture keep-alive lifecycle or regression assertions.

Optional documentation correction: docs/browser-testing.md:142 still says “twenty-three Linux selections.” Update it to “twenty-seven” to match the new 9 Chromium + 18 WebKit matrix and the surrounding documentation.

Hosted evidence inspected: run 42597212482 passed all 27 browser shards. The slowest WebKit job was 1/18, at 10m37s including setup/teardown; its timing artifact reports 66/66 passes, 9m26s test wall time and 17m41s summed test execution. The slowest file was appearance.spec.mjs at 207s summed execution. These results support headroom in this run, not a guarantee across runners.

Overall CI remains failing: Rust and tool integration stopped at dependency policy on libsqlite3-sys / RUSTSEC-2026-0459, before the Node integration tests. This PR leaves Cargo.lock and deny.toml unchanged. The new integration test checks the timeout setting across creation/restart/preview; it does not reproduce the Linux WebKit race. The PR’s local tests, original trace analysis and before-change timing claims remain author-reported. This review executed no code, tests, probes or apps.

matt2e added a commit that referenced this pull request Nov 9, 2026
…3.9.0 (#3077)

## Why

The **Dependency policy** step of the "Rust and tool integration" job
fails on every run since about 2026-11-09 09:30Z, independent of any
PR's changes (main's run 42593548798 at 09:27Z passed; PR #3074's run
42597212482, job 129092678543, failed):

> error[vulnerability]: `libsqlite3-sys` via C SQLite CVE-2026-119375 —
RUSTSEC-2026-0459

libsqlite3-sys 0.38.2 bundles SQLite 3.53.2. SQLite before 3.54.0 can
return uninitialised page-cache memory when a database file is opened
together with a sibling `-wal` file whose page size differs. Path:
`libsqlite3-sys 0.38.2 ← rusqlite 0.40.2 ← Buzz` and `←
mesh-llm-log-store 0.78.1 ← mesh-llm-host-runtime ← buzz-mesh-compute`.

## What

**Upgrade is blocked, so this adds a `deny.toml` exception (option 2).**
The advisory's fix is libsqlite3-sys ≥ 0.39.0, which only rusqlite 0.41
uses. `cargo update -p libsqlite3-sys` is a no-op: rusqlite 0.40.2 (the
latest 0.40) pins `libsqlite3-sys ^0.38`. mesh-llm-log-store at the
pinned bridge rev `5d19ca6` requires `rusqlite ^0.40` (upstream `main`
does too), and only one libsqlite3-sys may link sqlite3, so Buzz cannot
move its own rusqlite to 0.41 without a mesh-llm bump. That is a larger
change than clearing CI and is out of scope here.

The exception follows the file's existing style and records why Buzz is
not exposed, checked against the code:
- Buzz's archive (`src-tauri/src/archive/mod.rs`) opens `events.sqlite3`
under Tauri's `app_data_dir`, a directory it creates itself.
- `src-tauri/src/legacy_communities.rs` opens old Buzz's own WebKit
`localstorage.sqlite3` under `~/Library/WebKit/<bundle id>` read-only.
- mesh-llm's log store opens `log_store.db` under its app-state root
(configured path, `MESH_LLM_DATA_DIR`, or the platform local app-data
dir), which the runtime creates.

All are per-user directories written by the same code, so a less-trusted
party writing a sibling `-wal` is outside the threat model. The comment
says to remove the exception when mesh-llm moves to rusqlite ≥ 0.41.

**Also in this PR, because they fail or warn in the same step:**
- RUSTSEC-2026-0462 appeared after the cited run and fails the step too:
rmcp < 3.9.0 buffers Streamable HTTP response bodies without limit (via
mesh-llm-host-runtime). `cargo update -p rmcp` moves rmcp and
rmcp-macros 3.8.0 → 3.9.0, within range.
- The yanked async-executor 1.14.0 moves to 1.14.1 (`cargo update -p
async-executor`).
- The remaining `Cargo.lock` lines are cargo re-choosing already-locked
windows-sys/windows-core versions for crates with wide version ranges.
The package set is unchanged. Each new edge matches `cargo metadata
--locked`.
- The yanked clipboard-win 5.4.1 and nostr-sdk 0.45.5 warnings predate
this (they are in the cited job log) and are left alone.

Resulting versions: libsqlite3-sys 0.38.2, rusqlite 0.40.2 (unchanged);
rmcp 3.9.0; async-executor 1.14.1.

## Verified

- `./bin/cargo-deny --locked check`: `advisories ok, bans ok, licenses
ok, sources ok` (exit 0). Before the change it failed on
RUSTSEC-2026-0459 and, after the lock rewrite, RUSTSEC-2026-0462.
- `./bin/cargo check --workspace --locked --all-targets` compiles.

## Not verified

- Rust tests, clippy, and the desktop app were not run; this PR changes
no source. CI runs them.
- Pushed with `--no-verify`.

Precedent: aceb6fa "deps: tauri-plugin-updater 2.14.1 moves zip to 9,
clearing RUSTSEC-2026-0429 (#2137)".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
matt2e and others added 2 commits November 10, 2026 01:27
WebKit shard 9/14 hit the browser job's 15-minute deadline with every
test passing (run 42592827846, job 129078698059: 87 of 92 journeys done
when the step was cancelled). The shard is at the cap on main, not on
one slow runner: its recent durations are 14.9, 14.9, 14.7 and 13.7
minutes, and shard 10/14 runs 13.8 to 14.1 minutes. Playwright shards by
test count at file level, and the heaviest files (nested-replies,
navigation-thread-history, shortcuts, right-pane-dismissal) landed in
one shard: about 25 minutes of summed test time on two workers.

Eighteen shards is the smallest count that leaves clear headroom. From
that run's per-test durations and `playwright test --list --shard=K/N`,
the slowest shard's predicted job time (105 s outside tests plus 1.11 x
half the summed test time) is:

    14 shards: 15.6 min (actual 15.1, cancelled)
    16 shards: 13.0 min
    18 shards: 11.9 min
    20 shards: 11.2 min

The matrix excludes the new shard numbers for Chromium, which keeps
nine. The docs name the new counts.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A WebKit journey failed once in ten repeats of the flake check (run
42592827846, job 129078806819, reactions.spec.mjs) with "Importing a
module script failed." Its trace shows one failed request: the lazy
import of src/features/messages/reaction-name.ts on the fixture server,
"Connection terminated unexpectedly" after 12 ms with no send or
receive timing. The previous requests on that server were 4 to 6
seconds earlier. Node's http.Server closes a keep-alive connection
after its 5 s default keepAliveTimeout (measured: 6.0 s after the last
response), and WebKit reused the idle connection in that instant. The
request died with the socket, and nothing retries a module script.

Fixture dev servers and preview servers now set keepAliveTimeout to 0,
so they never close an idle connection on their own; Vite destroys open
sockets when a server closes. The dev server sets it in the fixture
plugin's configureServer, next to bindAssignedPort, so a restarted
server keeps it. The integration test covers a created server, a
restarted one and a preview server.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@matt2e
matt2e force-pushed the mtoohey/ci-webkit-shard-headroom branch from 7ab8b40 to 329ea4d Compare November 9, 2026 14:30

@kazemek21 kazemek21 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kyle reviewed this PR and the Nexus review-tracker notes and approved with no comments.

— Posted by Codex (gpt-6.1-sol) on behalf of kylez

@matt2e
matt2e merged commit 1efdf0b into main Nov 9, 2026
124 of 127 checks passed
@matt2e
matt2e deleted the mtoohey/ci-webkit-shard-headroom branch November 9, 2026 21:46
wpfleger96 added a commit that referenced this pull request Nov 9, 2026
…restricted-communities

* origin/main:
  ci: give WebKit shards headroom and keep fixture servers' idle connections open (#3074)
  fix(messages): offer Retry as soon as a send fails (#3080)
  fix(agents): link Settings from key failures and quote the provider (#3081)
  fix(identity): link Builderlab from Identity sync where its page holds the next step (#3082)
  fix(startup): show a recoverable screen when Buzz fails to open or crashes (#3085)
  fix(attention): ask before deleting a watch (#3092)
  fix(agents): remove the owner-access read tools (#3123)
  fix(channels): keep the side pane mounted under the image viewer (#3111)
  fix(solo): ask before Escape or Close drops a new workspace's typed fields (#3114)

# Conflicts:
#	docs/configuration.md
#	vite.config.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants