镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Runtime: extend heap-cap emergency recovery beyond current safe allocation classes #30

Closed

Description

@chrisbbreuer

Follow-up from #29 after zig-gc b99de49, zig-js 78b553a, and zig-js 45a58ce promoted semantics/oom-one-thread.js and added ArrayBuffer byte-slab recovery.

Current status

The landed recovery paths cover these GC-backed capped-context allocation failures:

  • GC cell slab allocation failures retry after Context.collectForAllocationFailure() when the current thread can safely collect.
  • no-GIL GC-cell slab recovery has a safepoint-owned current-interpreter path from Runtime: make no-GIL heap-cap recovery safepoint-owned #31; it reuses the abort-safe parallel root-publication collector and retries only after a collection actually sweeps.
  • GC-backed ArrayBuffer byte allocation failures retry once after the same recovery hook, including a live no-GIL peer witness for safepoint-owned byte-slab pressure.
  • Budgeted runtime allocations now have a guarded one-shot recovery hook (2d4d3ae0): when the heap cap would be exceeded, the quota allocator can collect safely, reserve again, and continue without scattering ad-hoc retries through object/property code.
  • GC-backed object side-store garbage can now be reclaimed by allocation-failure recovery under the heap cap in safe single-threaded/GIL-capable contexts (Context heap_limit_bytes object side-store pressure recovers garbage).
  • 72beb646 blocks the generic budget-recovery callback inside GcCellBacking allocator-internal critical sections, so GC-cell chunk OOM fails outward to zig-gc's safe outer Heap.create recovery path instead of self-deadlocking while backing locks are held.
  • b3c86930 roots pre-reserved heap-limit catch environments in the interpreter's precise environment root stack, so emergency recovery cannot collect a catch scope needed to convert reserved OutOfMemoryError values into JS catch (e) bindings.
  • Single-threaded and GIL-mode contexts can run emergency collection safely when peers are parked/published under the supported scan policy.
  • Arena-backed heap caps remain fail-closed and non-reclaimable by design.

That is enough for the PR-249 OOM recovery witness, GC-backed ArrayBuffer byte pressure, ordinary budgeted GC-backed side-store pressure, and the current no-GIL GC-cell / ArrayBuffer emergency-recovery slices, while keeping collection fail-closed when peers could hold unscanned roots, when the current allocation site holds a trace-sensitive lock, or when deferred generator/iterator cells prevent a safe parallel finish.

Side-store policy covered so far

Object side-store pressure is now covered by focused tests:

  • dropped object side stores are reclaimable by explicit collection, and a second allocation wave can then succeed under the same GC-backed cap;
  • allocation-failure recovery now also reclaims dropped side stores automatically in safe single-threaded/GIL-capable contexts;
  • live side-store pressure still fails closed as host error.OutOfMemory, proving the cap is enforced when collection cannot free enough budget;
  • live no-GIL trace-sensitive lock sites are pinned by regression coverage so the runtime does not collect/retry while object/environment/promise/async-generator request or allocator-internal locks could deadlock with tracing.

The recovery hook is intentionally guarded, recursion-safe, and disabled inside allocator-internal backing locks rather than a blanket retry. Many no-GIL side-store allocations can happen while object property/element, promise, environment, async-generator request, iterator/generator, or other trace-sensitive state is held, while GC tracing may need those same structures.

Deferred generator / iterator policy

#36 now tracks the deferred generator/iterator seam separately because it is not just an ordinary trace-sensitive-lock problem. The current fail-closed witnesses cover:

  • suspended generator execution storage;
  • resumable generator handler buffers;
  • pending async-generator request buffers;
  • iterator-helper backing state;
  • telemetry that splits deferred-cell aborts into gc_par_deferred_aborts / def-abort without weakening the round-limit abort accounting identity.

No positive recoverable generator/iterator class has been proven. #36 is closed with the conservative policy that all current deferred generator/iterator side-store pressure fails closed while those edges remain pending, unless a future site can prove the collector can safely sweep through a sound world-stopped path.

Remaining perfection work

  • Extend abort-safe no-GIL / parallel_js emergency recovery only for allocation classes that can prove the current interpreter can collect without scanning unpublishing peers, holding trace-sensitive locks, or leaving deferred generator/iterator edges unresolved.
  • Define finer lock-aware no-GIL collect/retry policies for side stores and other non-cell allocations where the allocation site can prove it does not hold locks or deferred state the tracer needs.
  • Keep the closed Runtime: prove generator and iterator side-store heap-cap recovery under no-GIL #36 policy documented: deferred generator/iterator side-store pressure is intentionally all-fail-closed unless a future allocation site proves a sound world-stopped recovery path.
  • Add focused tests for every newly recoverable allocation class; keep fail-closed witnesses for allocation classes that remain unsafe to recover generically.
  • Keep arena-backed heap caps fail-closed unless a caller explicitly opts into GC-backed recovery.

This should stay separate from #29/#24: the promoted PR-249 witness is covered, but capped allocation recovery is not yet perfect for every allocation class or no-GIL interleaving.

Activity

  1. chrisbbreuer commented on Jul 11, 2026

    @chrisbbreuer
    MemberAuthor

    Pushed 45a58ce (fix oom: retry arraybuffer bytes after collect) as the next #30 slice.

    What changed:

    • Interpreter now has an explicit allocation-recovery callback from Context.
    • GC-backed ArrayBuffer byte allocation retries once after Context.collectForAllocationFailure() succeeds.
    • Added a focused regression where a 20 MiB ArrayBuffer is dropped under a 32 MiB GC-backed heap cap, then a second 20 MiB ArrayBuffer allocation succeeds only because emergency collection frees the first byte slab.

    Validation:

    • zig build test with filter: heap_limit_bytes retries ArrayBuffer byte allocation
    • zig build test with filter: heap_limit_bytes allocation pressure
    • zig build test with filter: ArrayBuffer bytes release when wrapper is collected
    • zig build threads-test -Dthreads-case=semantics/oom-one-thread.js --summary all
    • git diff --check

    I deliberately did not add a generic object side-store allocator retry in this slice. Many object side-store allocations happen while holding property/element locks, while GC tracing takes those same locks. A generic allocator-level recovery hook there could deadlock. That part of #30 needs a lock-aware policy or reserve/headroom design rather than a blind retry wrapper.

  2. chrisbbreuer commented on Jul 11, 2026

    @chrisbbreuer
    MemberAuthor

    Docs/policy sync landed in 34134b7 (docs(oom): clarify capped gc recovery policy).

    The public docs now describe the current split after 78b553a and 45a58ce:

    • arena-backed heap caps remain fail-closed and non-reclaimable;
    • GC-backed capped contexts can collect and retry at safe GC cell allocation failures and ArrayBuffer byte slab allocation failures;
    • the remaining Runtime: extend heap-cap emergency recovery beyond current safe allocation classes #30 work is explicitly no-GIL root-publication emergency recovery and lock-aware side-store pressure, avoiding a generic side-store retry that could deadlock while object property/element locks are held.

    Validation:

    • bun run docs:build
    • git diff --check
  3. chrisbbreuer commented on Jul 11, 2026

    @chrisbbreuer
    MemberAuthor

    Follow-up from the full CI gate on 34134b77: the no-suppression TSan run exposed that allocation-failure recovery was still doing a conservative scan of parked peer native stacks in the serialized/GIL recovery path. That races native wait-frame / park-record publication (semantics/oom-one-thread.js, plus the matching TSan unit witness).

    Pushed e179650 to split the scan policy: allocation-failure recovery now scans only the allocating thread native stack and fails closed for live no-GIL peers; parked peer stack scanning is opt-in for the older GIL mid-script path. Local validation:

    • zig build test -Dtsan=true -Dtest-filter="heap_limit_bytes lets sibling Threads survive one Thread OOM" --summary all
    • zig build threads-test -Dtsan=true -Dthreads-case="semantics/oom-one-thread.js" --summary all

    This keeps #30 open: true no-GIL recovery still needs an abort-safe allocation-failure parallel collection path rather than a conservative parked-stack scan.

  4. chrisbbreuer commented on Jul 11, 2026

    @chrisbbreuer
    MemberAuthor

    Split out #31 as the first narrow no-GIL emergency-recovery slice.

    The key seam is not just "run the parallel collector on OOM": allocation-failure recovery currently has only the Context, while the abort-safe parallel collector needs the current Interpreter and safepoint-like root-publication conditions. Some future side-store allocation sites may also hold locks that GC tracing needs, so a generic allocator-level retry remains intentionally out of scope.

    #31 tracks making GC-cell heap-cap recovery attempt the existing abort/no-sweep parallel collector only when the current interpreter can safely act as collector, and otherwise preserve the current fail-closed OOM behavior.

  5. chrisbbreuer commented on Jul 11, 2026

    @chrisbbreuer
    MemberAuthor

    #31 slice landed in 8db58b6 (fix oom: recover no-gil cell allocation at safepoints).

    Parent status update:

    • GC cell allocation-failure recovery now has a current-interpreter/safepoint-owned no-GIL path.
    • The path reuses the abort/no-sweep parallel collector and only retries allocation after the collection actually sweeps.
    • Peers publish for an emergency collector even when routine parallel_midscript_gc is disabled.
    • A trace-sensitive lock-depth guard keeps object/environment/promise/iterator side-store locks as fail-closed OOM sites, so this does not introduce the generic side-store retry Runtime: extend heap-cap emergency recovery beyond current safe allocation classes #30 deliberately avoided.

    Focused validation included the new heap-cap cell witness under TSan plus the existing ArrayBuffer recovery and mid-script parallel collector witnesses. Keeping #30 open for the remaining lock-aware side-store pressure policy and broader allocation-class coverage.

  6. chrisbbreuer commented on Jul 11, 2026

    @chrisbbreuer
    MemberAuthor

    Docs sync landed in c972087 (docs oom: sync no-gil heap cap recovery status).

    docs/threads/limits.md now reflects the current #30/#31 split:

    • GC-backed caps can retry safe GC-cell and ArrayBuffer byte allocation failures.
    • no-GIL GC-cell slab recovery is safepoint-owned by the active interpreter and uses the abort-safe parallel root-publication collector.
    • recovery still fails closed outside active JS execution or while holding side-store locks the tracer may need.
    • Runtime: extend heap-cap emergency recovery beyond current safe allocation classes #30 remains open for lock-aware side-store pressure coverage.
  7. chrisbbreuer commented on Jul 12, 2026

    @chrisbbreuer
    MemberAuthor

    Pushed 8e875900 (test: cover capped object side-store pressure) as a coverage/policy slice for the remaining side-store part of #30.

    What changed:

    • Added a capped-context regression showing object side-store pressure currently fails closed as host error.OutOfMemory rather than pretending to be a safe generic retry/catchable path.
    • Added a GC-backed capped-context regression showing object side stores from dropped references are reclaimed by explicit collection, then a second allocation wave succeeds under the same cap.

    This intentionally does not add a generic side-store allocation retry. The fail-closed test documents the current lock-aware policy: side-store allocations can happen while locks that tracing also needs are held, so recovery there must not blindly collect/retry.

    Focused validation:

    • zig build test -Dtest-filter="object side-store pressure fails closed" --summary all
    • zig build test -Dtest-filter="object side stores reclaim after explicit collection" --summary all
    • git diff --check

    #30 remains open for a future positive lock-aware recovery mechanism, but the missing side-store pressure/reclaim coverage is now in place.

  8. chrisbbreuer commented on Jul 12, 2026

    @chrisbbreuer
    MemberAuthor

    Follow-up CI for 8e875900 was superseded by the #34 push, and the combined head 919f148ba90a3c88b78d16d8447e5945374f71c1 passed the full CI gate:

    https://github.com/zig-utils/zig-js/actions/runs/29175661429

    So the #30 side-store pressure coverage slice is green in the full suite.

  9. changed the title [-]Runtime: extend heap-cap emergency recovery beyond GIL-safe GC cells[/-] [+]Runtime: extend heap-cap emergency recovery beyond current safe allocation classes[/+] on Jul 12, 2026
  10. chrisbbreuer commented on Jul 12, 2026

    @chrisbbreuer
    MemberAuthor

    Landed 2d4d3ae0 (runtime: recover side-store heap pressure). It adds a recursion-guarded one-shot recovery hook to the heap budget allocator and wires it after Context bootstrap, so GC-backed side-store allocation pressure can collect/retry safely without sprinkling per-call-site retries. Focused validation: zig build test -Dtest-filter="BudgetAllocator" --summary all, zig build test -Dtest-filter="heap_limit_bytes object side-store" --summary all, and git diff --check. Prior pushed run 46245541 completed green including all no-GIL TSan corpus shards: https://github.com/zig-utils/zig-js/actions/runs/29178220129. New run for 2d4d3ae0: https://github.com/zig-utils/zig-js/actions/runs/29179086875.

  11. chrisbbreuer commented on Jul 12, 2026

    @chrisbbreuer
    MemberAuthor

    Follow-up fix landed in 72beb646 (runtime: block recovery inside GC backing locks). Local reproduction of unit (shard 0/4) exposed the new budget-level recovery callback self-deadlocking when invoked inside GcCellBacking.inner_lock; the fix blocks generic recovery in allocator-internal critical sections and lets zig-gc Heap.create perform the safe outer recovery after locks unwind. Focused validation passed for the no-GIL GC-cell recovery test, heap-cap catchability tests, and side-store heap-limit tests; local unit (shard 0/4) then completed green in 72.8s.

  12. chrisbbreuer commented on Jul 12, 2026

    @chrisbbreuer
    MemberAuthor

    Follow-up from the 72beb646 run: CI reproduced a real timeout in unit (shard 2/4) and tsan-unit (shard 2/4) after Context heap_limit_bytes string allocation pressure does not panic; the next test was Context heap_limit_bytes lets sibling Threads survive one Thread OOM.

    Root cause: the new allocation-failure recovery can collect while evalTry is inside the heap-pressure region. Its pre-reserved catch environment for catch (e) was only held in a Zig local, not a precise interpreter root, so emergency collection could reclaim/reuse it before the catch clause bound the OOM value. Locally sampling the hang showed the main thread spinning in Environment.lockBindings while binding the catch parameter.

    Landed two small commits:

    • b61c3f98 (runtime: test blocked budget recovery) adds a direct BudgetAllocator regression proving generic recovery stays disabled inside allocator-internal blocked sections and resumes afterward.
    • b3c86930 (runtime: root reserved heap-limit catch envs) roots the reserved heap-limit catch environment in gc_env_roots for the duration of evalTry, including the initial prebinding allocation.

    Local validation:

    • zig build test -Dtest-filter="Context heap_limit_bytes lets sibling Threads survive one Thread OOM" --summary all
    • zig build test -Dunit-shard-index=2 -Dunit-shard-count=4 --summary all
    • zig build test -Dtest-filter="BudgetAllocator" --summary all
    • zig build test -Dtest-filter="heap_limit_bytes" --summary all
    • git diff --check

    New pushed run for b3c86930: https://github.com/zig-utils/zig-js/actions/runs/29182632742

  13. chrisbbreuer commented on Jul 12, 2026

    @chrisbbreuer
    MemberAuthor

    The follow-up run for b3c86930 completed green across the full CI gate, including the previously timed-out unit (shard 2/4) and tsan-unit (shard 2/4) jobs plus all no-GIL corpus TSan shards:

    https://github.com/zig-utils/zig-js/actions/runs/29182632742

  14. chrisbbreuer commented on Jul 12, 2026

    @chrisbbreuer
    MemberAuthor

    Landed 14f181b4 (test: cover no-gil oom trace-lock guard) as a #30 policy/witness slice.

    What changed:

    • Added a focused parallel_js heap-cap regression proving allocation-failure recovery fails closed while the current site holds a trace-sensitive lock.
    • The witness keeps the existing positive single-threaded side-store recovery path intact, but pins the no-GIL rule that emergency recovery must not collect/retry from lock sites the tracer may also need.

    Validation:

    • zig build test -Dtest-filter="BudgetAllocator" --summary all
    • zig build test -Dtest-filter="heap_limit_bytes object side-store" --summary all
    • zig build test -Dtest-filter="parallel_js heap_limit_bytes fails closed inside trace-sensitive locks" --summary all
    • git diff --check

    This does not broaden recoverable no-GIL allocation classes yet; it hardens the fail-closed boundary that future lock-aware side-store recovery has to preserve.

    CI is queued/pending here: https://github.com/zig-utils/zig-js/actions/runs/29184764043

  15. 24 remaining items

  16. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Root cause hardening pushed as 379f40f4 (runtime: widen emergency gc publication window).\n\nThe abort-safe collector previously used the same 25 ms per-generation scheduling floor for opportunistic mid-script GC and an allocation-failure recovery attempt. Under an oversubscribed TSan runner, a live peer could miss that window; the collector correctly aborted without sweeping, but the documented positive ArrayBuffer recovery class then failed outward.\n\nThe collector driver now takes a caller-selected bounded floor: routine opportunistic collection remains 25 ms, while emergency allocation recovery gets 100 ms because allocation cannot otherwise make progress. Both retain the 50,000 mark/poll bound, generation caps, deferred-state checks, and abort-without-sweep safety. Docs now state the split explicitly.\n\nFocused validation:\n- failing live-peer ArrayBuffer recovery: 5/5 TSan after the change (plus 10/10 baseline samples while diagnosing);\n- live-peer GC-cell recovery: TSan green;\n- deferred iterator-helper recovery: still fails closed under TSan;\n- docs build and diff check green.\n\nReplacement full gate: https://github.com/zig-utils/zig-js/actions/runs/29236051962

  17. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Final acceptance is green after 379f40f4: https://github.com/zig-utils/zig-js/actions/runs/29236051962 — 31 success, 1 expected nightly-only skip, 0 failures.\n\nThe exact previously failing tsan-unit (shard 1/4) passed, along with all no-GIL corpus shards and mid-GC/lifecycle fuzz. Live-peer GC-cell and ArrayBuffer recovery retain a bounded positive convergence window; opportunistic collection keeps the shorter pause policy; trace-sensitive/deferred/unsafe classes still abort without sweep and fail closed. Closing #30 again with that corrected, tested contract.

  18. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Reopening from two consecutive Linux CI reproductions after the recent object/GC layout performance series:

    Both fail enable_gc: heap_limit_bytes retries ArrayBuffer byte allocation after collection in unit shard 1 with error.OutOfMemory; the same full suite is green locally on macOS. The regression test hard-codes a 32 MiB cap for a 20 MiB slab, implicitly requiring all platform/context overhead to remain below 12 MiB. That is not an invariant and has become platform-sensitive after the object/GC layout changes.

    I am changing the witness—not the runtime contract—to bootstrap under a generous cap, allocate the first 20 MiB slab, then set the test-local budget limit relative to measured live usage so the second slab deterministically exceeds the cap before collection and has ample room after the first slab is reclaimed. The existing live-byte assertions continue to prove recovery actually freed the first slab.

  19. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Pushed b20d30e5 (test(oom): scale recovery cap to live usage (#30)) to main.

    The ArrayBuffer recovery witness now:

    • bootstraps under a generous cap;
    • records actual live budget usage after the first 20 MiB slab;
    • lowers the test-local cap to just under the amount required for a second slab;
    • preserves the live-byte assertions proving the first slab is collected and only the second remains.

    This removes the accidental platform/layout invariant that all non-buffer Context overhead must stay below 12 MiB, while still forcing the allocation-failure recovery path.

    Local validation: focused test passed 2/2 (root + witness), 0 skipped, 0 failed, 0 leaked; format and diff checks are clean. Keeping #30 open until the replacement Linux CI unit shard confirms the original failure is fixed.

  20. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor
  21. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    The dynamic cap in b20d30e5 removed the platform-overhead assumption but Linux CI still retained the first wrapper and failed. The remaining difference from the existing green no-GIL ArrayBuffer recovery witness was reachability shape: the failing test wrote a fresh wrapper through a global slot and immediately overwrote it; the green witness creates it in a helper whose VM frame is fully popped before recovery.

    Pushed 97bff723 (test(oom): drop buffers from a dead frame (#30)) on current main. The single-thread witness now uses the same dead-helper-frame shape, retains the measured dynamic cap, verifies both 20 MiB results, and still asserts exactly one slab remains live after recovery. Focused validation passed 2/2 with zero leaks; format/diff checks are clean. Keeping #30 open for the replacement Linux shard.

  22. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor
  23. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    The 97bff723 run was superseded before unit shard 1 completed. Combined replacement run including the unrelated #80 test-oracle fix: https://github.com/zig-utils/zig-js/actions/runs/29338741409

  24. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Verified on Linux in unit shard 1 of main run 29338741409: enable_gc: heap_limit_bytes retries ArrayBuffer byte allocation after collection passed after 97bff72. The shard later failed only on the separate stale structuredClone TypeError oracle tracked in #80.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions