Repository navigation
Description
Follow-up from #29 after zig-gc b99de49, zig-js 78b553a, and zig-js 45a58ce promoted semantics/oom-one-thread.js and added ArrayBuffer byte-slab recovery.
Current status
The landed recovery paths cover these GC-backed capped-context allocation failures:
- GC cell slab allocation failures retry after
Context.collectForAllocationFailure()when the current thread can safely collect. - no-GIL GC-cell slab recovery has a safepoint-owned current-interpreter path from Runtime: make no-GIL heap-cap recovery safepoint-owned #31; it reuses the abort-safe parallel root-publication collector and retries only after a collection actually sweeps.
- GC-backed ArrayBuffer byte allocation failures retry once after the same recovery hook, including a live no-GIL peer witness for safepoint-owned byte-slab pressure.
- Budgeted runtime allocations now have a guarded one-shot recovery hook (
2d4d3ae0): when the heap cap would be exceeded, the quota allocator can collect safely, reserve again, and continue without scattering ad-hoc retries through object/property code. - GC-backed object side-store garbage can now be reclaimed by allocation-failure recovery under the heap cap in safe single-threaded/GIL-capable contexts (
Context heap_limit_bytes object side-store pressure recovers garbage). 72beb646blocks the generic budget-recovery callback insideGcCellBackingallocator-internal critical sections, so GC-cell chunk OOM fails outward to zig-gc's safe outerHeap.createrecovery path instead of self-deadlocking while backing locks are held.b3c86930roots pre-reserved heap-limit catch environments in the interpreter's precise environment root stack, so emergency recovery cannot collect a catch scope needed to convert reservedOutOfMemoryErrorvalues into JScatch (e)bindings.- Single-threaded and GIL-mode contexts can run emergency collection safely when peers are parked/published under the supported scan policy.
- Arena-backed heap caps remain fail-closed and non-reclaimable by design.
That is enough for the PR-249 OOM recovery witness, GC-backed ArrayBuffer byte pressure, ordinary budgeted GC-backed side-store pressure, and the current no-GIL GC-cell / ArrayBuffer emergency-recovery slices, while keeping collection fail-closed when peers could hold unscanned roots, when the current allocation site holds a trace-sensitive lock, or when deferred generator/iterator cells prevent a safe parallel finish.
Side-store policy covered so far
Object side-store pressure is now covered by focused tests:
- dropped object side stores are reclaimable by explicit collection, and a second allocation wave can then succeed under the same GC-backed cap;
- allocation-failure recovery now also reclaims dropped side stores automatically in safe single-threaded/GIL-capable contexts;
- live side-store pressure still fails closed as host
error.OutOfMemory, proving the cap is enforced when collection cannot free enough budget; - live no-GIL trace-sensitive lock sites are pinned by regression coverage so the runtime does not collect/retry while object/environment/promise/async-generator request or allocator-internal locks could deadlock with tracing.
The recovery hook is intentionally guarded, recursion-safe, and disabled inside allocator-internal backing locks rather than a blanket retry. Many no-GIL side-store allocations can happen while object property/element, promise, environment, async-generator request, iterator/generator, or other trace-sensitive state is held, while GC tracing may need those same structures.
Deferred generator / iterator policy
#36 now tracks the deferred generator/iterator seam separately because it is not just an ordinary trace-sensitive-lock problem. The current fail-closed witnesses cover:
- suspended generator execution storage;
- resumable generator handler buffers;
- pending async-generator request buffers;
- iterator-helper backing state;
- telemetry that splits deferred-cell aborts into
gc_par_deferred_aborts/def-abortwithout weakening the round-limit abort accounting identity.
No positive recoverable generator/iterator class has been proven. #36 is closed with the conservative policy that all current deferred generator/iterator side-store pressure fails closed while those edges remain pending, unless a future site can prove the collector can safely sweep through a sound world-stopped path.
Remaining perfection work
- Extend abort-safe no-GIL /
parallel_jsemergency recovery only for allocation classes that can prove the current interpreter can collect without scanning unpublishing peers, holding trace-sensitive locks, or leaving deferred generator/iterator edges unresolved. - Define finer lock-aware no-GIL collect/retry policies for side stores and other non-cell allocations where the allocation site can prove it does not hold locks or deferred state the tracer needs.
- Keep the closed Runtime: prove generator and iterator side-store heap-cap recovery under no-GIL #36 policy documented: deferred generator/iterator side-store pressure is intentionally all-fail-closed unless a future allocation site proves a sound world-stopped recovery path.
- Add focused tests for every newly recoverable allocation class; keep fail-closed witnesses for allocation classes that remain unsafe to recover generically.
- Keep arena-backed heap caps fail-closed unless a caller explicitly opts into GC-backed recovery.
This should stay separate from #29/#24: the promoted PR-249 witness is covered, but capped allocation recovery is not yet perfect for every allocation class or no-GIL interleaving.
Activity
chrisbbreuer commentedon Jul 11, 2026
Pushed 45a58ce (fix oom: retry arraybuffer bytes after collect) as the next #30 slice.
What changed:
Validation:
I deliberately did not add a generic object side-store allocator retry in this slice. Many object side-store allocations happen while holding property/element locks, while GC tracing takes those same locks. A generic allocator-level recovery hook there could deadlock. That part of #30 needs a lock-aware policy or reserve/headroom design rather than a blind retry wrapper.
chrisbbreuer commentedon Jul 11, 2026
Docs/policy sync landed in
34134b7(docs(oom): clarify capped gc recovery policy).The public docs now describe the current split after
78b553aand45a58ce:Validation:
bun run docs:buildgit diff --checkchrisbbreuer commentedon Jul 11, 2026
Follow-up from the full CI gate on
34134b77: the no-suppression TSan run exposed that allocation-failure recovery was still doing a conservative scan of parked peer native stacks in the serialized/GIL recovery path. That races native wait-frame / park-record publication (semantics/oom-one-thread.js, plus the matching TSan unit witness).Pushed
e179650to split the scan policy: allocation-failure recovery now scans only the allocating thread native stack and fails closed for live no-GIL peers; parked peer stack scanning is opt-in for the older GIL mid-script path. Local validation:zig build test -Dtsan=true -Dtest-filter="heap_limit_bytes lets sibling Threads survive one Thread OOM" --summary allzig build threads-test -Dtsan=true -Dthreads-case="semantics/oom-one-thread.js" --summary allThis keeps #30 open: true no-GIL recovery still needs an abort-safe allocation-failure parallel collection path rather than a conservative parked-stack scan.
chrisbbreuer commentedon Jul 11, 2026
Split out #31 as the first narrow no-GIL emergency-recovery slice.
The key seam is not just "run the parallel collector on OOM": allocation-failure recovery currently has only the
Context, while the abort-safe parallel collector needs the currentInterpreterand safepoint-like root-publication conditions. Some future side-store allocation sites may also hold locks that GC tracing needs, so a generic allocator-level retry remains intentionally out of scope.#31 tracks making GC-cell heap-cap recovery attempt the existing abort/no-sweep parallel collector only when the current interpreter can safely act as collector, and otherwise preserve the current fail-closed OOM behavior.
chrisbbreuer commentedon Jul 11, 2026
#31 slice landed in 8db58b6 (fix oom: recover no-gil cell allocation at safepoints).
Parent status update:
Focused validation included the new heap-cap cell witness under TSan plus the existing ArrayBuffer recovery and mid-script parallel collector witnesses. Keeping #30 open for the remaining lock-aware side-store pressure policy and broader allocation-class coverage.
chrisbbreuer commentedon Jul 11, 2026
Docs sync landed in c972087 (docs oom: sync no-gil heap cap recovery status).
docs/threads/limits.mdnow reflects the current #30/#31 split:chrisbbreuer commentedon Jul 12, 2026
Pushed
8e875900(test: cover capped object side-store pressure) as a coverage/policy slice for the remaining side-store part of #30.What changed:
error.OutOfMemoryrather than pretending to be a safe generic retry/catchable path.This intentionally does not add a generic side-store allocation retry. The fail-closed test documents the current lock-aware policy: side-store allocations can happen while locks that tracing also needs are held, so recovery there must not blindly collect/retry.
Focused validation:
zig build test -Dtest-filter="object side-store pressure fails closed" --summary allzig build test -Dtest-filter="object side stores reclaim after explicit collection" --summary allgit diff --check#30 remains open for a future positive lock-aware recovery mechanism, but the missing side-store pressure/reclaim coverage is now in place.
chrisbbreuer commentedon Jul 12, 2026
Follow-up CI for
8e875900was superseded by the #34 push, and the combined head919f148ba90a3c88b78d16d8447e5945374f71c1passed the full CI gate:https://github.com/zig-utils/zig-js/actions/runs/29175661429
So the #30 side-store pressure coverage slice is green in the full suite.
[-]Runtime: extend heap-cap emergency recovery beyond GIL-safe GC cells[/-][+]Runtime: extend heap-cap emergency recovery beyond current safe allocation classes[/+]chrisbbreuer commentedon Jul 12, 2026
Landed
2d4d3ae0(runtime: recover side-store heap pressure). It adds a recursion-guarded one-shot recovery hook to the heap budget allocator and wires it after Context bootstrap, so GC-backed side-store allocation pressure can collect/retry safely without sprinkling per-call-site retries. Focused validation:zig build test -Dtest-filter="BudgetAllocator" --summary all,zig build test -Dtest-filter="heap_limit_bytes object side-store" --summary all, andgit diff --check. Prior pushed run46245541completed green including all no-GIL TSan corpus shards: https://github.com/zig-utils/zig-js/actions/runs/29178220129. New run for2d4d3ae0: https://github.com/zig-utils/zig-js/actions/runs/29179086875.chrisbbreuer commentedon Jul 12, 2026
Follow-up fix landed in
72beb646(runtime: block recovery inside GC backing locks). Local reproduction ofunit (shard 0/4)exposed the new budget-level recovery callback self-deadlocking when invoked insideGcCellBacking.inner_lock; the fix blocks generic recovery in allocator-internal critical sections and lets zig-gcHeap.createperform the safe outer recovery after locks unwind. Focused validation passed for the no-GIL GC-cell recovery test, heap-cap catchability tests, and side-store heap-limit tests; localunit (shard 0/4)then completed green in 72.8s.chrisbbreuer commentedon Jul 12, 2026
Follow-up from the
72beb646run: CI reproduced a real timeout inunit (shard 2/4)andtsan-unit (shard 2/4)afterContext heap_limit_bytes string allocation pressure does not panic; the next test wasContext heap_limit_bytes lets sibling Threads survive one Thread OOM.Root cause: the new allocation-failure recovery can collect while
evalTryis inside the heap-pressure region. Its pre-reserved catch environment forcatch (e)was only held in a Zig local, not a precise interpreter root, so emergency collection could reclaim/reuse it before the catch clause bound the OOM value. Locally sampling the hang showed the main thread spinning inEnvironment.lockBindingswhile binding the catch parameter.Landed two small commits:
b61c3f98(runtime: test blocked budget recovery) adds a direct BudgetAllocator regression proving generic recovery stays disabled inside allocator-internal blocked sections and resumes afterward.b3c86930(runtime: root reserved heap-limit catch envs) roots the reserved heap-limit catch environment ingc_env_rootsfor the duration ofevalTry, including the initial prebinding allocation.Local validation:
zig build test -Dtest-filter="Context heap_limit_bytes lets sibling Threads survive one Thread OOM" --summary allzig build test -Dunit-shard-index=2 -Dunit-shard-count=4 --summary allzig build test -Dtest-filter="BudgetAllocator" --summary allzig build test -Dtest-filter="heap_limit_bytes" --summary allgit diff --checkNew pushed run for
b3c86930: https://github.com/zig-utils/zig-js/actions/runs/29182632742chrisbbreuer commentedon Jul 12, 2026
The follow-up run for
b3c86930completed green across the full CI gate, including the previously timed-outunit (shard 2/4)andtsan-unit (shard 2/4)jobs plus all no-GIL corpus TSan shards:https://github.com/zig-utils/zig-js/actions/runs/29182632742
chrisbbreuer commentedon Jul 12, 2026
Landed
14f181b4(test: cover no-gil oom trace-lock guard) as a #30 policy/witness slice.What changed:
parallel_jsheap-cap regression proving allocation-failure recovery fails closed while the current site holds a trace-sensitive lock.Validation:
zig build test -Dtest-filter="BudgetAllocator" --summary allzig build test -Dtest-filter="heap_limit_bytes object side-store" --summary allzig build test -Dtest-filter="parallel_js heap_limit_bytes fails closed inside trace-sensitive locks" --summary allgit diff --checkThis does not broaden recoverable no-GIL allocation classes yet; it hardens the fail-closed boundary that future lock-aware side-store recovery has to preserve.
CI is queued/pending here: https://github.com/zig-utils/zig-js/actions/runs/29184764043
24 remaining items
chrisbbreuer commentedon Jul 13, 2026
Root cause hardening pushed as
379f40f4(runtime: widen emergency gc publication window).\n\nThe abort-safe collector previously used the same 25 ms per-generation scheduling floor for opportunistic mid-script GC and an allocation-failure recovery attempt. Under an oversubscribed TSan runner, a live peer could miss that window; the collector correctly aborted without sweeping, but the documented positive ArrayBuffer recovery class then failed outward.\n\nThe collector driver now takes a caller-selected bounded floor: routine opportunistic collection remains 25 ms, while emergency allocation recovery gets 100 ms because allocation cannot otherwise make progress. Both retain the 50,000 mark/poll bound, generation caps, deferred-state checks, and abort-without-sweep safety. Docs now state the split explicitly.\n\nFocused validation:\n- failing live-peer ArrayBuffer recovery: 5/5 TSan after the change (plus 10/10 baseline samples while diagnosing);\n- live-peer GC-cell recovery: TSan green;\n- deferred iterator-helper recovery: still fails closed under TSan;\n- docs build and diff check green.\n\nReplacement full gate: https://github.com/zig-utils/zig-js/actions/runs/29236051962chrisbbreuer commentedon Jul 13, 2026
Final acceptance is green after
379f40f4: https://github.com/zig-utils/zig-js/actions/runs/29236051962 — 31 success, 1 expected nightly-only skip, 0 failures.\n\nThe exact previously failingtsan-unit (shard 1/4)passed, along with all no-GIL corpus shards and mid-GC/lifecycle fuzz. Live-peer GC-cell and ArrayBuffer recovery retain a bounded positive convergence window; opportunistic collection keeps the shorter pause policy; trace-sensitive/deferred/unsafe classes still abort without sweep and fail closed. Closing #30 again with that corrected, tested contract.chrisbbreuer commentedon Jul 14, 2026
Reopening from two consecutive Linux CI reproductions after the recent object/GC layout performance series:
fb6f8d88run: https://github.com/zig-utils/zig-js/actions/runs/29335007220/job/8709202404206fc4f30run: https://github.com/zig-utils/zig-js/actions/runs/29335736544Both fail
enable_gc: heap_limit_bytes retries ArrayBuffer byte allocation after collectionin unit shard 1 witherror.OutOfMemory; the same full suite is green locally on macOS. The regression test hard-codes a 32 MiB cap for a 20 MiB slab, implicitly requiring all platform/context overhead to remain below 12 MiB. That is not an invariant and has become platform-sensitive after the object/GC layout changes.I am changing the witness—not the runtime contract—to bootstrap under a generous cap, allocate the first 20 MiB slab, then set the test-local budget limit relative to measured live usage so the second slab deterministically exceeds the cap before collection and has ample room after the first slab is reclaimed. The existing live-byte assertions continue to prove recovery actually freed the first slab.
test(oom): scale recovery cap to live usage (#30)
chrisbbreuer commentedon Jul 14, 2026
Pushed
b20d30e5(test(oom): scale recovery cap to live usage (#30)) tomain.The ArrayBuffer recovery witness now:
This removes the accidental platform/layout invariant that all non-buffer Context overhead must stay below 12 MiB, while still forcing the allocation-failure recovery path.
Local validation: focused test passed 2/2 (root + witness), 0 skipped, 0 failed, 0 leaked; format and diff checks are clean. Keeping #30 open until the replacement Linux CI unit shard confirms the original failure is fixed.
chrisbbreuer commentedon Jul 14, 2026
Replacement CI: https://github.com/zig-utils/zig-js/actions/runs/29336776910
test(oom): drop buffers from a dead frame (#30)
chrisbbreuer commentedon Jul 14, 2026
The dynamic cap in
b20d30e5removed the platform-overhead assumption but Linux CI still retained the first wrapper and failed. The remaining difference from the existing green no-GIL ArrayBuffer recovery witness was reachability shape: the failing test wrote a fresh wrapper through a global slot and immediately overwrote it; the green witness creates it in a helper whose VM frame is fully popped before recovery.Pushed
97bff723(test(oom): drop buffers from a dead frame (#30)) on currentmain. The single-thread witness now uses the same dead-helper-frame shape, retains the measured dynamic cap, verifies both 20 MiB results, and still asserts exactly one slab remains live after recovery. Focused validation passed 2/2 with zero leaks; format/diff checks are clean. Keeping #30 open for the replacement Linux shard.chrisbbreuer commentedon Jul 14, 2026
Replacement CI: https://github.com/zig-utils/zig-js/actions/runs/29338095020
chrisbbreuer commentedon Jul 14, 2026
The
97bff723run was superseded before unit shard 1 completed. Combined replacement run including the unrelated #80 test-oracle fix: https://github.com/zig-utils/zig-js/actions/runs/29338741409chrisbbreuer commentedon Jul 14, 2026
Verified on Linux in unit shard 1 of main run 29338741409:
enable_gc: heap_limit_bytes retries ArrayBuffer byte allocation after collectionpassed after 97bff72. The shard later failed only on the separate stale structuredClone TypeError oracle tracked in #80.