Repository navigation
Description
Parent: #1
Goal
Move mid-script parallel collection from a heavily tested internal capability toward a generally supportable policy.
Current baseline
The collector can trace running peers through root-publication generations, directly trace frozen parked peers, and abort without sweeping when publication or heap stability does not converge. Sync property waits, condition waits, and contended locks service publication requests at bounded pump points. Specialized fuzzer programs cover parked waiters, async jobs, ThreadLocal roots, finalization, joins, Workers, teardown, termination, and native-callback closure entry.
zig build midgc-profile now isolates this internal policy and reports attempts, finishing sweeps, publication-timeout versus round-limit aborts, deferred-cell aborts, generations, bounded born-growth extension rounds (ext-rnd), deferred-work rounds, post-abort retry backoff skips (backoff), total and worst-generation publication polls (wait-poll / wait-max), total and worst-attempt finish retries (fin-retry / retry-max), running/parked peer observations, actual peer publications, and collector-side total/maximum pause. These counters and parallel_midscript_gc remain testing/profile implementation details, not stable embedder API.
The current profile evidence says convergence pain is mostly round-limit fallback under sustained allocation, not publication-timeout failure. The internal policy now suppresses immediate retry churn after failed attempts and can extend the base generation cap only while a born-cell progress signal remains bounded and safe. Worst-generation, extension/backoff, and deferred-abort counters keep those decisions visible before changing budgets or wake/pump policy further.
Completed milestones
- Add pause, abort-reason, retry, generation, and peer-participation metrics.
- Add accounting invariants for attempt outcomes, abort classification, generations, pause totals, and max-versus-total telemetry.
- Prove both running-peer root publication and direct parked-peer observation paths execute.
- Add a focused repeatable convergence profile independent of the broad contention matrix.
- Add bounded post-abort retry backoff and born-growth convergence extension telemetry.
- Make the waiter-table reclamation witness deterministic at a truly quiescent point and add arm/GC/thread failure diagnostics.
- Add deferred generator/iterator abort telemetry so fail-closed no-GIL finishes are visible separately from generic round-limit churn.
- Add native-callback closure-root coverage for VM-lowered callback entry during mid-script collection.
Remaining work
- Define the supported collection/convergence policy for sync-wait parked peers.
- Continue reducing reliance on bounded publication polling, wake/pump retries, round-limit aborts, and quiescent fallback using the new backoff/extension/deferred-abort profile evidence.
- Expand nested native callback, host re-entry, deep VM/tree-walker, and multi-context coverage.
- Prove termination/teardown behavior when a collection is active at greater sustained depth.
- Decide when the internal
parallel_midscript_gcpath can become production policy; do not expose the test knob as public API.
Acceptance criteria
- TSan-clean higher-depth mid-GC sweeps and deterministic lifecycle oracles.
- Explicit root-publication and lock-order documentation.
- No hangs, lost cleanup jobs, or collection-time UAFs under termination and sync-wait stress.
- Profile evidence supports any convergence-policy or retry-budget change.
Activity
chrisbbreuer commentedon Jul 10, 2026
Landed
3b17a78 test(threads): measure mid-script GC convergenceonmain.Implemented
zig build midgc-profile, separate fromthreads-profile, so collector convergence remains measurable even when unrelated contention scenarios fail.A representative local profile completed 20 attempts with 11 sweeps, 9 round-limit aborts, 0 publication-timeout aborts, 396 generations, 8,016,071 failed publication polls, 5 finish retries, 1,145 running-peer requests, 396 parked-peer observations, and 1,150 actual publications. This points the next optimization work at generation/poll convergence under sustained allocation rather than weakening finish safety.
Deterministic lifecycle oracle
The clean corpus exposed a real intermittent
cve/mc-dos-waiter-table-storm.js2/3 completion. Arm labels and failure-only GC/waiter/thread diagnostics proved that notify and dead-registrant timeout cleanup completed, but the WeakRef reclamation arm could consume all 2,000 turns before the independent wide-key threads finished native teardown. The witness now gates reclamation on final synchronous joins after verifying async results, preserving its reclamation threshold while requiring a genuinely quiescent full collection. It passed three consecutive isolated runs and the subsequent clean corpus.Verification
zig build testzig build threads-test— 227/227zig build threads-test -Dthreads-parallel-js=true -Dthreads-case=sync/condition-wait-notify.jszig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=5— 225 programs, 0 failureszig build threadfuzz -Dtsan=true -Dfuzz-midgc=true -Dfuzz-iters=2— 90 programs, 0 failureszig build threadfuzz -Dfuzz-verify=true -Dfuzz-iters=300— 300 programs, 0 failureszig build test -Dtsan=true -Dtest-filter=parallel_jszig build midgc-profilebun run docs:buildgit diff --checkThe internal collector remains opt-in/test-only. This milestone adds evidence and determinism; it does not claim production policy graduation.
docs(threads): define mid-GC sync-wait policy (#14)
chrisbbreuer commentedon Jul 10, 2026
Landed
6c25d8e(docs(threads): define mid-GC sync-wait policy (#14)).This defines the current internal convergence policy for sync-wait/condition/contended-lock peers:
gc_parked;gc_root_lockso the owner cannot wake and mutate roots mid-read;Updated:
src/root_handshake.zigsource contract commentdocs/threads/production-readiness.mddocs/threads/limits.mdValidation:
zig fmt src/root_handshake.zigbun run docs:buildgit diff --checkThe follow-up work is still open: reduce reliance on the bounded polling/round-limit fallback using profile evidence, then raise sustained depth only after the TSan/fuzzer data supports it.
chrisbbreuer commentedon Jul 10, 2026
Advanced the convergence-policy evidence in
b856b7a(bench(midgc): track worst convergence retries).What changed:
Contextnow records max failed publication polls for any single root-publication generation (gc_par_publication_wait_iterations_max).Contextnow records max self-checking finish retries for any single elected collection attempt (gc_par_finish_retries_max).midgc-profileprints those aswait-maxandretry-maxnext to the existing aggregatewait-pollandfin-retrytotals.Why this matters for #14: before changing publication wait budgets, wake/pump behavior, or round-limit policy, we need to know whether convergence pain is broad across many generations or concentrated in a small number of worst cases. This gives that signal without exposing the internal
parallel_midscript_gcknob as API.Validation, batched:
zig build midgc-profilezig build test -Dtest-filter='parallel_js (M3): mid-script parallel collector'zig build test -Dtest-filter='parallel_js (M3): sync wait peers publish roots'bun run docs:buildgit diff --checkLocal
midgc-profileoutput after the change:Leaving the remaining convergence-policy checkbox open: this is instrumentation needed for the next budget/policy reduction, not yet the reduction itself.
chrisbbreuer commentedon Jul 10, 2026
Landed another mid-script GC maturity slice in 4bf4e35, with dependency support in zig-gc f33d48c.
What changed:
Validation:
Profiler evidence from the latest local run: born-grow was non-zero and deferred was 0, so this workload is currently limited by continued allocation churn rather than deferred tracing.
chrisbbreuer commentedon Jul 10, 2026
From the failed main run on a0375cb, the midgc jobs are now pointing at termination/reaction quiescence rather than a TSan data race:\n\n- plain threadfuzz-midgc failed deterministic Worker/Thread termination oracles (missed asyncJoin rejection counts/scores and thread-teardown settlements).\n- tsan-threadfuzz-midgc hit a per-seed timeout/leak report in the weak-collection stressor, with no TSan data-race summary in the log I inspected.\n\nFocused local repros on the current worktree:\n- ./zig-out/bin/threadfuzz midgcworkertermcondasync 1 11 -> still fails with bad reject score 0\n- ./zig-out/bin/threadfuzz midgcworkertermwaitasync 1 19 -> still fails with a short reject score\n- ./zig-out/bin/threadfuzz midgcworkertermteardown 1 18 -> still fails with short settled count\n- ./zig-out/bin/threadfuzz midgcmoduletermteardown 1 6 -> still fails with short settled count\n\nI’m keeping this as its own mid-script-GC/termination workstream rather than blending it with the lifecycle binding-lock fix in 87c4c8b.
chrisbbreuer commentedon Jul 10, 2026
Pushed 4d92f18 (fix(threads): preserve teardown reactions during mid-gc).
Root cause addressed: mid-script GC plus top-level failure teardown could lose asyncJoin cleanup reactions in two ways:
Fix:
Focused local validation:
The previous 641c5e9 CI red was tsan-threadfuzz-midgc seed 1 timeout rather than a semantic oracle line; this commit targets the semantic reaction-loss path and may also reduce the timeout if it was caused by aborted cleanup/drain churn, but CI still needs to prove the TSan budget.
chrisbbreuer commentedon Jul 10, 2026
Follow-up pushed in d7c69d6 after CI on 4d92f18 exposed a real TSan midgc race: Context.hasRunningJsThreads was scanning Context.js_threads while a peer Thread constructor appended to the list. The scan now takes the same threading API lock used by Thread construction before walking the list, while still checking each record under join_mutex. Focused local validation: zig build threadfuzz-bin -Dtsan=true && ./zig-out/bin/threadfuzz midgcrestrictlife 2 1 passed.
chrisbbreuer commentedon Jul 10, 2026
Follow-up pushed in c03bdf6 for the d7c69d6 CI result: tsan-threadfuzz-midgc no longer reported an engine race, but the 45-subcase midgc seed exceeded the 300s per-seed watchdog under TSan. The watchdog now refreshes before each independently reproducible midgc subcase, so it remains a per-subcase hang detector instead of an aggregate runtime cap for the whole seed. Focused validation: zig build threadfuzz-bin -Dtsan=true, then zig build threadfuzz-bin && ./zig-out/bin/threadfuzz midgc 1 1 (45 programs, 0 failures).
chrisbbreuer commentedon Jul 10, 2026
Update pushed after the
c03bdf6CI failure:tsan-threadfuzz-midgcin run https://github.com/zig-utils/zig-js/actions/runs/29121977453 failed with a real TSan race: concurrentArray.prototype.pushdense-element growth overlappedzig-gcweak-slot scratch growth during parallel marking. The later “worker terminate/ThreadLocal asyncHold cleanup did not attempt a parallel collection” was in the same failed profile after the TSan warning, so I treated the race as the root signal first.zig-utils/zig-gc@a2b8cb2(fix(gc): reserve weak scratch before marking) by reservingweak_slotsbefore full/minor/parallel concurrent marking, matching the existing mark-stack pre-reservation somarkWeakdoes not allocate/reuse aux pages while mutators grow object backing stores.zig-jsstate plus patchedzig-gc:zig build threadfuzz-bin -Dtsan=true --summary noneTHREADFUZZ_SEED_TIMEOUT_MS=300000 ./zig-out/bin/threadfuzz midgc 1 1→threadfuzz midgc: 45 programs from seed 1, 0 failureszig-js@e2dfea18; new CI run is https://github.com/zig-utils/zig-js/actions/runs/29122992826.chrisbbreuer commentedon Jul 10, 2026
CI follow-up from the 441b2c4 run: threadfuzz-midgc crashed during parallel-GC root tracing in zig-gc Visitor.isManaged(), where a mixed-ownership raw root edge could header-peek a stale/wild pointer before proving it belonged to the heap.
Fixed in dependency commit zig-utils/zig-gc@984ae51 by making isManaged() walk the live-cell list for an exact payload match under the allocation lock in parallel mode, plus a regression covering live GC, stack, and deliberately wild pointer inputs.
Validation so far:
The original workflow is still running, so GitHub has not allowed rerunning the failed job yet. Once it completes, rerun threadfuzz-midgc/failed jobs against the new zig-gc/main.
chrisbbreuer commentedon Jul 10, 2026
CI follow-up: the c2b5e61 push is intended to rerun mid-GC on top of zig-utils/zig-gc@6ed2ca8. The prior 66027de run showed the next unsafe maybe-managed edge in zig-gc writeBarrierFrom()/rememberStrongStore during threadfuzz-midgc root publication. That dependency fix changes strong/owner-aware/weak/incremental barriers to prove exact heap membership before reading headers, with a regression for stack and deliberately wild pointer inputs.
Local validation before the rerun:
chrisbbreuer commentedon Jul 10, 2026
CI follow-up from run 29126983628: threadfuzz-amplified failed because seed 1 exceeded the 120s per-seed watchdog. I reproduced locally with zig build threadfuzz -Dfuzz-amplify=true -Dfuzz-iters=1 -Dfuzz-seed=1. Sampling showed active worker execution, not a parked deadlock; hot time was in zig-gc headerForPayload via broad write barriers after the exact maybe-managed safety change.\n\nDependency fix pushed to zig-gc main as 49c7d8f (perf(gc): index live payload lookups). It keeps exact live-payload membership for wild/stale pointers but avoids walking the full live list on each barrier.\n\nValidation:\n- zig-gc: zig build test --summary none\n- zig-js focused repro: ./zig-out/bin/threadfuzz amplify 1 1 => 0 failures (~19s locally).\n\nNext zig-js push should pick up the updated zig-gc checkout in CI.
chrisbbreuer commentedon Jul 11, 2026
Pushed
0f5a438(gc: back off failed mid-script retry churn) for the convergence-policy item.Profile signal before the change was allocation-growth-bound rather than publication-timeout-bound (
pub-abort=0,deferred=0, round-limit aborts with most generations seeing born growth). The driver now keeps the existing abort-safe behavior, but after a publication-timeout or round-limit abort it sets a tiny internal retry cooldown so later safepoints do not immediately elect another expensive collector attempt under the same sustained allocation burst. Successful finishing sweeps clear the retry marker. This remains internalparallel_midscript_gcpolicy, not public API, and quiescent full GC remains the reclaim fallback.Telemetry/docs update:
gc_par_backoff_skipsandgc_par_retry_after_nsinternal counters/state.zig build midgc-profilenow reportsbackoffskips next to attempts/sweeps/aborts/pause.Focused validation:
zig build test -Dtest-filter='parallel_js (M3): mid-script parallel collector' --summary nonezig build test -Dtest-filter='parallel_js (M3): sync wait peers publish roots' --summary nonezig build midgc-profilebun run docs:buildgit diff --checkLatest local profile after the change:
The important new evidence is
backoff=10: the profile now exposes and exercises suppression of immediate post-abort retry churn. Leaving the broader policy/depth checkboxes open until CI/nightly data supports further budget or production-policy changes.4 remaining items
chrisbbreuer commentedon Jul 12, 2026
Landed another focused #14 coverage slice in
zig-js@6abd3634(test(midgc): cover native callback closure roots).What changed:
Array.prototype.mapcallback entry.Local validation batched before push:
/Users/chris/.local/bin/zig build test -Dtest-filter="native callback VM closures survive mid-script parallel GC" --summary all/Users/chris/.local/bin/zig build test -Dtest-filter="parallel_js (M3): mid-script parallel collector" --summary all/Users/chris/.local/bin/zig build test -Dtest-filter="parallel_js (M3): thrown Thread completion roots" --summary all/Users/chris/.bun/bin/bun run docs:buildgit diff --checkCI for this push is pending: https://github.com/zig-utils/zig-js/actions/runs/29200500880
This advances the “nested native callback / deep VM-tree-walker coverage” part of the remaining #14 work. Multi-context and host re-entry depth are still separate follow-up slices.
chrisbbreuer commentedon Jul 12, 2026
CI routing note: the run for
6abd3634was superseded by the follow-up #36 push. The current main-branch gate that includes the native-callback mid-GC witness is now: https://github.com/zig-utils/zig-js/actions/runs/29200654888chrisbbreuer commentedon Jul 12, 2026
CI routing note: the previous current gate was superseded by the follow-up #36 iterator-helper push. The current main-branch gate that includes the native-callback mid-GC witness is now: https://github.com/zig-utils/zig-js/actions/runs/29200732250
chrisbbreuer commentedon Jul 12, 2026
CI routing note: the previous current gate was superseded by the follow-up #36 async-generator push. The current main-branch gate that includes the native-callback mid-GC witness is now: https://github.com/zig-utils/zig-js/actions/runs/29200819725
chrisbbreuer commentedon Jul 12, 2026
CI routing note: the previous current gate was superseded by the follow-up #36 generator-handler push. The current main-branch gate that includes the native-callback mid-GC witness is now: https://github.com/zig-utils/zig-js/actions/runs/29200885804
chrisbbreuer commentedon Jul 12, 2026
Landed
zig-js@4b20837b(profile(midgc): count deferred aborts) as a #14/#36 observability slice.What changed:
gc_par_deferred_abortstelemetry for elected parallel-GC attempts that abort because deferred generator/iterator cells prevent a safe no-GIL finish.deferred_aborts <= round_abortsis now asserted by focused telemetry coherence.zig build midgc-profilenow prints adef-abortcolumn next topub-abort/rnd-abort.Focused validation:
/Users/chris/.local/bin/zig build test -Dtest-filter="deferred generator handlers" --summary all/Users/chris/.local/bin/zig build midgc-profile --summary all/Users/chris/.bun/bin/bun run docs:buildgit diff --checkLocal profile output included the new column; this allocation-churn profile had
def-abort=0, as expected because it does not root deferred generator/iterator cells:CI for this push is pending: https://github.com/zig-utils/zig-js/actions/runs/29201026809
chrisbbreuer commentedon Jul 12, 2026
Green gate note for
d0092f5e: https://github.com/zig-utils/zig-js/actions/runs/29201642565The deferred-abort telemetry/docs slice and native-callback closure-root coverage are covered by the successful combined gate.
threadfuzz-midgc,threadfuzz-lifecycle, and the TSan threadfuzz variants all completed green.chrisbbreuer commentedon Jul 12, 2026
Green gate note for
c6856a28: https://github.com/zig-utils/zig-js/actions/runs/29203285907The mid-script GC gate is green after widening only the non-TSan
threadfuzz-midgcper-seed watchdog to match the existing 300s TSan mid-GC setting. The failed seed (midgcseed 5) replayed locally in ~23s with zero failures, so the change treats loaded-runner watchdog drift as CI configuration rather than weakening the semantic oracle.chrisbbreuer commentedon Jul 13, 2026
CI observation while validating b09fd1e: the first main run hit the known low-frequency mid-script GC convergence oracle in
threadfuzz-midgc:threadfuzz-midgcseed 8: midgc termination-reaction did not finish a parallel collectionFollow-up:
THREADFUZZ_SEED_TIMEOUT_MS=300000 zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=1 -Dfuzz-seed=8 --summary all→ 45 programs from seed 8, 0 failures.No new issue opened: this maps directly onto this issue’s remaining mid-script parallel-GC convergence/termination-reaction work.
chrisbbreuer commentedon Jul 13, 2026
Landed a small #14 hardening slice in
0f645145(gc: service teardown safepoints around joins).What changed:
teardown_stopand draining asyncJoin rejection cleanup.Why:
while (!rec.exited). If a child had already exited, or exited after the host pump/safepoint but before the host parked, teardown could skip the host publication/drive opportunity that helps an in-flight mid-script parallel collection converge. That matched the rarethreadfuzz-midgcseed-8 CI symptom from the prior run.Focused validation:
zig fmt src/context.zigzig build threadfuzz-bin --summary all./zig-out/bin/threadfuzz midgcterm 5 8→ 5 programs, 0 failuresTHREADFUZZ_SEED_TIMEOUT_MS=300000 zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=1 -Dfuzz-seed=8 --summary all→ 45 programs, 0 failuresgit diff --checkNext #14 pass I’d look at whether teardown should deliberately force/drive one bounded final parallel collection attempt when heap pressure is already over threshold, rather than only opportunistically servicing safepoints.
chrisbbreuer commentedon Jul 13, 2026
Follow-up CI fix landed in
e132c6fe(gc: skip teardown safepoint after joins).CI for
0f645145exposed a deterministic local repro inunit (shard 2/4):C-API: JSValueProtect roots survive mid-script parallel GCtagexpected456, gotNaNRoot cause of the previous slice:
Fix:
hasRunningJsThreads()so they only run while at least one spawned record is still live.Focused validation after the fix:
zig build test -Dtest-filter="C-API: JSValueProtect roots survive mid-script parallel GC" --summary all./zig-out/bin/threadfuzz midgcterm 5 8THREADFUZZ_SEED_TIMEOUT_MS=300000 zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=1 -Dfuzz-seed=8 --summary allgit diff --checkchrisbbreuer commentedon Jul 13, 2026
Pushed
e9df0649(gc: harden parallel publication generations) as a small mid-script parallel-GC handshake hardening slice.What changed:
gc_par_requestis now0while idle, andgc_par_next_requestmints non-zero generations.>=/<comparisons, so a stale high generation cannot satisfy a fresh low generation after counter wrap.parallel_js (M3): publication generations skip idle zero at wrap.Focused validation:
zig build test -Dtest-filter="publication generations skip idle zero" --summary allzig build test -Dtest-filter="parallel_js (M3): mid-script parallel collector reclaims garbage" --summary allzig build test -Dtsan=true -Dtest-filter="publication generations skip idle zero" --summary allTHREADFUZZ_SEED_TIMEOUT_MS=300000 ./zig-out/bin/threadfuzz midgc 1 1→ 45 programs, 0 failuresbun run docs:buildgit diff --checkThe previous combined-head run for
7e0afd2ewas clean at 24 success / 1 skipped / 0 failures before this push superseded it. New CI fore9df0649is pending: https://github.com/zig-utils/zig-js/actions/runs/29216356732chrisbbreuer commentedon Jul 13, 2026
Pushed
31313a8(test: stabilize midgc worker-close oracle) after the prior main run exposed a narrow harness race inthreadfuzz-midgc.Root cause:
asyncJoin()reactions, synchronously joined every worker, then used a singlePromise.resolve().then(...)before checking the async aggregate;join()may return just before its correspondingasyncJoin()reaction runs, so the check could observeasyncSum == 0even though worker completion was correct.Fix:
asyncJoin()promise and gate the final oracle withPromise.all(asyncJoins), matching the established neighboring terminate/finalization pattern.Focused validation:
zig build threadfuzz-bin --summary allTHREADFUZZ_SEED_TIMEOUT_MS=300000 ./zig-out/bin/threadfuzz midgcworkerclose 1 15→ 1 program, 0 failuresTHREADFUZZ_SEED_TIMEOUT_MS=300000 ./zig-out/bin/threadfuzz midgcworkerclose 25 15→ 25 programs, 0 failureszig fmt --check conformance/threadfuzz.ziggit diff --checkFresh combined CI gate: https://github.com/zig-utils/zig-js/actions/runs/29218852375
chrisbbreuer commentedon Jul 13, 2026
Closing this tracker with an explicit support decision.\n\nThe internal mid-script parallel collector now has:\n- running-peer generation-based root publication and direct frozen parked-peer tracing;\n- abort-without-sweep behavior when publication, born-cell stability, or deferred generator/iterator tracing cannot converge;\n- bounded wait pumping, retry backoff, born-growth extension rounds, and detailed abort/pause/peer telemetry;\n- waiter, async job, ThreadLocal, finalization, join, Worker, termination, teardown, native-callback closure, and multi-context stress coverage;\n- deterministic lifecycle/mid-GC oracles with per-seed watchdog/replay, including the final released-waiter join correction in
bdb369e7;\n- documented root-publication and lock-order invariants.\n\nProduction-policy decision:parallel_midscript_gcremains an internal testing/profile capability, not stable embedder API. The supported collector may abort this optimization without sweeping and fall back to quiescent/full collection. Bounded publication polling, round limits, deferred-cell aborts, and backoff are load-bearing safety policy, not correctness defects to remove indiscriminately. Promoting the path to public production policy would require new evidence and should get a dedicated issue.\n\nThe current higher-depth TSan mid-GC, lifecycle, no-GIL corpus, and deterministic gates are green: https://github.com/zig-utils/zig-js/actions/runs/29232877388 — 31 success, 1 expected skip, 0 failures. That meets this tracker acceptance without exposing the test knob or weakening fail-closed behavior.