镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Canonical: JavaScript multithreading support #1

Closed

Description

@chrisbbreuer

Goal

Ship production-quality JavaScript multithreading in zig-js across both supported isolation models:

  • isolated agents/Workers exchanging bounded structured-clone values and retained SharedArrayBuffer storage;
  • shared-realm Thread execution preserving JavaScript object identity on real OS threads.

Pre-stabilization backward compatibility was intentionally not a constraint; the public contract reflects the cleaner final design.

Public contract

// True-parallel shared-realm JavaScript by default.
const parallel = try js.Context.createWith(gpa, .{ .enable_threads = true });

// Supported serialized fallback.
const serialized = try js.Context.createWith(gpa, .{
    .enable_threads = true,
    .gil = true,
});

The C API exposes the same choice through ZJSGlobalContextCreateThreaded(gil). parallel_js, parallel_midscript_gc, and profiling/fuzz controls remain internal and test-only.

Delivered

  • Real OS-thread agents and Workers; versioned structured clone; ArrayBuffer transfer/detach; retained shared storage; typed-array wait/notify/waitAsync.
  • Fallible, nonblocking, resource-bounded Worker channels with checked per-message, queued-byte, and message-count accounting; complete-frame limits; exact SAB ownership; observable Zig/JS/C rejection.
  • Shared-realm Thread, Lock, Condition, ThreadLocal, ConcurrentAccessError, property Atomics, and proposal-aligned Atomics.Mutex / Atomics.Condition.
  • No-GIL execution by default with synchronized shapes, properties/elements, environments, promises/microtasks, waiters, tasks, handles, GC roots, and lifecycle state.
  • GC-managed thread-safe cells, precise VM roots, safe native-stack publication, weak/finalization processing, a non-moving one-cycle nursery, and abort-safe internal mid-script parallel collection.
  • Heap caps and catchable OOM where supported; bounded live-peer emergency recovery for GC cells and ArrayBuffer bytes; deliberate fail-closed policy for trace-sensitive, deferred, arena-backed, or otherwise unprovable sites.
  • A documented JavaScript memory model and suppression-free TSan boundary.
  • Profile-guided contention and GC/lifecycle guidance, including the supported .gil = true fallback for coordination-heavy workloads.
  • Security hardening for path confinement, shared-buffer/handle counts, context affinity, buffer/view races, timeout conversion, clone framing/count/depth validation, and nondeterministic random seeding.

Verification

  • Unit, C-API, test262-parallel, no-GIL witness, GIL corpus, four-way no-GIL TSan corpus, suppression-free TSan, deterministic fuzz, lifecycle fuzz, and mid-GC fuzz gates.
  • WebKit PR-249 audit: 236/259 executable files promoted; all 23 reference-only executables and 5 helpers classified with zero missing or uncategorized entries. Remaining cases require nonportable JSC/JIT/heap/WASM/detached-source premises.
  • Focused threads-profile, midgc-profile, and gc-profile evidence for policy and optimization changes.
  • Final combined gate: https://github.com/zig-utils/zig-js/actions/runs/29236051962 — 31 success, 1 expected nightly-only skip, 0 failures.

Completed trackers

Completion criteria

  • Final Zig and C semantics are documented and free of accidental compatibility shims.
  • No known engine-state race remains in supported paths.
  • GC/lifecycle and no-GIL scaling have measured production guidance.
  • Mid-script collection has an explicit abort-safe internal support policy and is not exposed as stable API.
  • Fuzzer and corpus gates cover the supported lifecycle surface at sustained depth.
  • Every executable PR-249 file is promoted or has a current auditable nonportable/feature blocker.
  • Worker delivery has explicit ownership, failure, and resource-control contracts.

Authoritative docs

  • docs/threads/index.md
  • docs/threads/api.md
  • docs/threads/memory-model.md
  • docs/threads/testing.md
  • docs/threads/production-readiness.md
  • docs/threads/limits.md
  • docs/threads/bindings.md

Activity

  1. chrisbbreuer commented on Jun 10, 2026

    @chrisbbreuer
    MemberAuthor

    Also should keep an eye on this one oven-sh/WebKit#249 - maybe we can reuse quite a bit of it

  2. chrisbbreuer commented on Jun 10, 2026

    @chrisbbreuer
    MemberAuthor

    Progress 2026-06-10: Phases 0–2 are implemented and on main (through d85c080).

    • P0: Context thread-affinity guardrails (debug asserts, C-API thread rules), docs/threads/bindings.md audit (10 globals ruled; found two the plan missed: Math.random's process-global PRNG and the global symbol counter — both fixed in P2), PR-249 corpus vendored under reference/webkit-249/ (240 test files + full docs/threads specs + THREAD.md, head 25375a99).
    • P1: src/shared_buffer.zig — refcounted process-wide SAB storage, in-place CAS grow, RetainList per realm; ArrayBufferData reworked behind bytes() (compiler-forced audit of all ~56 byte-access sites); Atomics RMW/CAS/load/store are single SeqCst hardware ops (raw-bits path also fixed BigInt64 precision); 8-byte-aligned buffers.
    • P2: src/agent.zig — $262.agent on real OS threads with the INTERPRETING.md broadcast rendezvous; Atomics.wait genuinely blocks (FIFO waiter table, per-ticket condition + timeout, [[CanBlock]]); notify really wakes; real sleep/monotonicNow; the cooperative model is deleted. Atomics-subtree workers run one test per process so a deadlock costs one 30s timeout.

    Score: 41,898 → 42,137/47,928 (87.9%); built-ins/Atomics 308 → 349/388 (89.9%), zero host-fails (no hangs in two full runs). End-to-end concurrency is unit-tested (agent parks → broadcast → blocked wait → cross-thread notify → report).

    Remaining Atomics failures are mostly Atomics.waitAsync (101 test files) → that's Phase 3, designed in docs/threads/P2-agents.md (async tickets on the same waiter table + drain-loop integration in Context.evaluate). Also still open from P2: the CanBlockIsFalse runner mode (2 tests).

  3. chrisbbreuer commented on Jun 10, 2026

    @chrisbbreuer
    MemberAuthor

    Phase 3 landed (9999301, 28baab2): Atomics.waitAsync settles for real — async tickets share the blocking waiters' FIFO lists (spec notify ordering across wait/waitAsync), and the realm's drain tail resolves the promises on notify/deadline/teardown, abandoning unsettleable infinite waits instead of hanging. The CanBlockIsFalse runner mode closes the last Phase 2 box (both tests now scored and passing).

    42,230/47,930 (88.1%); built-ins/Atomics 356/390 (91.3%), still zero host-fails. ~34 Atomics failures remain (under diagnosis). Next big block: Phase 4 structured clone.

  4. chrisbbreuer commented on Jun 10, 2026

    @chrisbbreuer
    MemberAuthor

    Phase 4 landed (ba41807): structured clone. src/structured_clone.zig serializes value graphs into a self-contained byte stream (deliberately so — it's the Phase 5 postMessage wire format) and deserializes into any realm's arena. Full type coverage incl. cycles/identity, array holes, SAB storage sharing, and { transfer } (move + detach); DataCloneError for the unclonables. The structuredClone global is installed. test262 unchanged at 42,230/47,930 (88.1%) — additive feature, zero regressions, gate run confirms.

    Still open from Phase 4: ArrayBuffer.prototype.transfer predates this work (already done). Next: Phase 5 — embedder Worker API (spawn = Context-per-thread + this wire format + the agent group's channel machinery), then Phase 6 (GIL Thread API, port the vendored PR-249 corpus).

  5. 485 remaining items

  6. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Closed #13 as complete after bdb369e7 removed the last machine-speed-dependent released-waiter oracle. The deterministic/generated lifecycle, mid-GC, TSan, replay, watchdog, and cleanup coverage meets the tracker acceptance criteria; ongoing depth tuning remains normal CI maintenance. Green gate: https://github.com/zig-utils/zig-js/actions/runs/29232877388

  7. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Closed #16 as complete. The non-moving one-cycle nursery, barriers, fallback, telemetry, scratch trimming, regression gates, and pooling guidance meet its acceptance criteria. Based on the focused profile evidence, multi-age/moving/parallel-minor GC does not currently earn its complexity; any future measured bottleneck should get a narrow issue. Green gate: https://github.com/zig-utils/zig-js/actions/runs/29232877388

  8. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Closed #11 as complete. The PR-249 audit is terminal at 236/259 promoted executable cases with all 23 reference-only executables and 5 helpers classified, zero missing/uncategorized entries, and fresh blocker probes confirming the remaining JSC/JIT/heap/WASM/detached-source premises are nonportable rather than hidden passing cases. Focused probe gate passed; full audit gate: https://github.com/zig-utils/zig-js/actions/runs/29232877388

  9. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Closed #30 as complete under the proven per-allocation-class heap-cap policy. GC cells, ArrayBuffer bytes, and safe budgeted side stores recover; trace-sensitive lock sites, deferred generator/iterator stores, allocator-internal locks, arena caps, and unsafe peer-publication states deliberately fail closed with focused witnesses. Universal retry would violate correctness. Green gate: https://github.com/zig-utils/zig-js/actions/runs/29232877388

  10. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Closed #15 as complete. The contention profiler and focused rows now attribute the planned engine/queue/Worker/GC/allocator sources; measured hot paths received regression-gated optimizations; lock ordering and .gil = true fallback guidance are documented. Additional host sampling is ongoing benchmarking, not missing implementation. Green gate: https://github.com/zig-utils/zig-js/actions/runs/29232877388

  11. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Closed #14 with an explicit support decision: the abort-safe mid-script parallel collector is a deeply tested internal optimization, not stable embedder API. Publication/round/deferred abort limits and quiescent/full fallback remain load-bearing correctness policy; the TSan mid-GC/lifecycle matrix is green at https://github.com/zig-utils/zig-js/actions/runs/29232877388.

  12. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Correction: reopened #30 after final #40 CI exposed a real schedule-sensitive failure in the positive live-peer ArrayBuffer heap-cap recovery witness (tsan-unit shard 1): https://github.com/zig-utils/zig-js/actions/runs/29235127135. The Worker changes are unrelated; #30 remains open until live-peer recovery has a bounded convergence contract and green gate.

  13. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Closing the canonical multithreading umbrella as complete. Every child tracker is closed on either implemented behavior or an explicit conservative support decision; the final body records the stable public contract, resource limits, GC/recovery boundaries, nonportable PR-249 tail, verification matrix, and authoritative docs.\n\nFinal combined acceptance gate: https://github.com/zig-utils/zig-js/actions/runs/29236051962 — 31 success, 1 expected nightly-only skip, 0 failures.\n\nFuture bugs, performance evidence, new WebAssembly/JIT scope, or attempts to promote internal mid-script GC policy should open narrow issues with focused witnesses rather than reopening this historical umbrella.

  14. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    The preserved pre-rewrite marker experiment was not applicable wholesale, but its allocator concern was valid on current main. Tracked as #54 and fixed cleanly in 35dff22; the obsolete GC layout and deleted worktree dependency were not pushed.

  15. chrisbbreuer commented on Jul 16, 2026

    @chrisbbreuer
    MemberAuthor

    Landed 7a061ce (test(gc): exercise full concurrent driver (#1)). The M3 concurrent-marker test now disables the independently enabled nursery before its workload, so old-byte growth deterministically exercises and closes multiple full concurrent cycles. This fixes a stale test setup that reproduced on the pre-change main baseline; the focused gate passes 2/2.

  16. chrisbbreuer commented on Aug 2, 2026

    @chrisbbreuer
    MemberAuthor

    048a4b7e closes a no-GIL race in inherited property cache observation found by the sharded TSan corpus.

    The canonical property read already held property_lock, but the advisory cache record reread the prototype shape and receiver shape afterward as plain fields. gc-stress/watchpoint-storm.js therefore raced a main-thread shape transition against a worker VM read. The engine now returns value + immutable holder shape/slot from one locked snapshot and snapshots the receiver shape through the same publication lock.

    Verification: exact storm case under TSan passes 1/1 with 7 optimizer publications and 4 invalidations; the complete unit suite passes 1,477/1,477 across 10 shards with zero failures, skips, or leaks. Test262 flips: 0.

  17. chrisbbreuer commented on Aug 2, 2026

    @chrisbbreuer
    MemberAuthor

    8b1ae613 fixes a process-wide parallel-sync activation race found by sustained lifecycle seed 35.

    Object.lockElements() previously reread the enable-only global gate independently in unlockElements(). During concurrent Context bootstrap, a call could bypass while the gate was false, then release after another host thread enabled it, aborting on trace-depth underflow and attempting an unmatched mutex unlock. Acquisition now returns a token consumed by every release site across the interpreter, VM, GC, structured clone, C API, and object helpers.

    Verification: focused transition witness passes in Debug and TSan; five sustained seed-35 lifecycle runs cover 280 programs with zero failures; the full suite passes 1,478/1,478 across 10 shards with zero failures, skips, or leaks. Test262 flips: 0.

  18. chrisbbreuer commented on Aug 3, 2026

    @chrisbbreuer
    MemberAuthor

    Landed 235f6124 for the unshared WebAssembly.Memory relocation hole exposed by the ReleaseSafe no-GIL gate. Peer mutators now stop only at complete operation boundaries while relocation detaches the historical ArrayBuffer, with the stop and shared-heap GC publication mutually arbitrated. No hidden GIL, suppression, retry, or baseline exemption was added. Evidence: ReleaseSafe 30/30 amplified plus final 1/1, suppression-free TSan 1/1, full units 1,479/1,479.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions