Repository navigation
Closed
Description
Parents: #474, #482
Related: #480, #493, #885
Problem
Two microtask queue ownership paths can silently discard already-committed jobs under allocator pressure:
transferMicrotasksUnlockedcallsappendPendingSlice(... ) catch {}and then clears the source queue unconditionally. If the destination grow fails, every source job is lost while Thread completion continues.drainJobQueuerestores untouched jobs after an abrupt callback withprependSlice(... ) catch {}. If prepend allocation fails, the original error is returned and the batch is deinitialized, discarding the remaining jobs. Reentrant jobs already in the destination survive, so observable FIFO state becomes a partial suffix.
jobQueueDequeueBatch itself leaves the source unchanged when its initial batch copy fails; the unsound boundaries are the later transfer/restore ownership handoffs.
#885 owns Promise state-to-job publication. This issue owns queue-to-queue and batch-to-queue transfer after jobs already exist.
Required implementation
- Make source-to-destination transfer commit only after the complete pending source suffix is guaranteed publishable, or transfer backing/segments allocation-free with exact ownership.
- Make abrupt restoration preserve every untouched batch job ahead of reentrantly enqueued work even when allocator recovery fails.
- Represent ownership explicitly so exactly one live root owns each queued job throughout transfer, restore, GC tracing, Thread exit, and Context teardown.
- Preserve FIFO across source jobs, destination jobs, reentrant jobs, next ticks, Promise reactions, native jobs, thenable jobs, and private ABI work.
- Keep queue locking ordered for two-queue transfer and compatible with concurrent marking/no-GIL producers.
- Do not swallow OOM, clear a source after failed publication, reverse or drop a suffix, cap queue size, retry callbacks, globally serialize JavaScript, or rely on Context destruction.
Acceptance
- Deterministic allocator-fault witnesses cover empty/nonempty destination transfer and abrupt restoration before/after reentrant enqueue.
- Every failure point proves exact source/destination/batch membership, order, GC rooting, and eventual one-time execution after recovery.
- Thread completion cannot publish
donewhile orphaning its local queue; termination and asyncJoin ordering remain exact. - A throwing next-tick/Promise/native callback preserves all untouched jobs before reentrant work without duplicating the throwing job.
- Moving/concurrent GC, bounded heap, Context teardown, private ABI, relevant threadfuzz, full units, and suppression-free TSan pass.
- Exact Promise/async/queue Test262 coverage has zero regressions and documentation makes no unsupported OOM guarantee.
Security boundary
Job count, callback failure, reentrancy, Thread timing, and allocation pressure are script-controlled. Queue ownership must fail closed rather than silently erase committed work or expose a reordered partial schedule after recovery.
Metadata
Metadata
Assignees
Labels
No labels
Activity
chrisbbreuer commentedon Sep 5, 2026
Design audit found a bounded ownership path that avoids both loss points without returning to per-job locking:
MicrotaskQueueobjects are already arena-allocated and outlive their OS threads until realm teardown. Afterrec.microtasks = nullcloses peer publication, the realm queue can splice the complete source queue object as an intrusive FIFO segment in O(1), rather than copying its jobs. Existing realm jobs precede the segment; later realm appends target the logical tail segment, so they follow it. The destination’s lock owns the detached segment after publication, and its enqueue generation advances by the transferred pending count.jobQueueDequeueBatchswaps the selected segment backing into the local active batch instead of copying it. The emptied segment receives the batch’s previously empty retained backing. On error, the untouched batch backing swaps into that segment’s queue-owned front slot; jobs appended reentrantly remain in its ordinary tail buffer and therefore follow the restored suffix. The next drain always consumes front before tail.current_microtaskandcurrent_microtask_batchremain the precise roots while the detached backing is executing. Once restored, the segment chain becomes the root again.This keeps burst execution outside the queue lock, makes source transfer and restoration allocation-free after jobs exist, and does not cap fanout or serialize JavaScript globally. It needs explicit tests for a tail segment receiving reentrant appends while its active backing is detached, repeated throws from a restored front, segment exhaustion/reuse, concurrent transfer, and moving/concurrent GC. No code is being mixed into the running #875 revision.
chrisbbreuer commentedon Sep 8, 2026
Implementation update: abrupt batch restoration is now using the existing Promise queue reservation invariant from #885. Once the batch copy succeeds, its original occupied queue slots become reserved capacity before the source is cleared. Every producer and nested drain accounts for those reservations. An abrupt callback can therefore restore its untouched suffix ahead of reentrant work without any allocation; successful completion releases the reservation. This also composes with an inner checkpoint that has already restored its own suffix and with outstanding resolving-function reservations.
The earlier two-buffer candidate is not needed for this handoff. A fixed front-buffer slot would additionally need an ownership proof for nested checkpoints restoring into the same queue. The reservation approach preserves the existing queue representation and batch execution model.
The audit also found that nested drains and await-driven single-job execution overwrite the outer checkpoint's precise current-job/batch roots. The change preserves these in a dynamically scoped root chain, including tracing, relocation, and root-scan accounting.
Deterministic witnesses cover exhausted allocation with/without reentrant enqueue, both queue-lock modes, nested restoration alongside resolving reservations, one-time FIFO execution after recovery, and outer-root tracing/relocation. Compilation and runtime validation are in progress; none of these gates is being marked passed yet. The exact-parent baseline covers 1,828 passing Promise/async Test262 cases with zero failures.
Thread-local queue transfer and completion remain outstanding under #886/#887. This update does not claim the whole issue is complete.
fix(microtasks): preserve abrupt batch ownership (#886)
chrisbbreuer commentedon Sep 8, 2026
Landed on
main: fix(microtasks): preserve abrupt batch ownership (#886).The commit makes abrupt batch restoration allocation-free by reserving the detached batch's original queue capacity. Reentrant producers, nested drains, and #885 resolving-function transactions cannot consume that reservation. Untouched jobs restore ahead of reentrant work; successful completion releases the reservation. It also keeps outer current jobs and pending suffixes precisely rooted across nested drain/await checkpoints, including moving relocation and root-scan accounting.
Measured validation on this exact source snapshot:
Keeping #886 open. Thread-local source-to-realm transfer still has its separate fallible-copy/clear handoff, and #887 still owns completion rooting/settlement. Full TSan, the broader lifecycle/mid-GC/private-ABI acceptance matrix, and the transfer-specific fault witnesses are not being claimed complete by this commit. Local raw logs, the Debug shard plan/logs, and source hashes are retained under
/tmp/zig-js-886-evidence/.chrisbbreuer commentedon Sep 8, 2026
The remaining transfer implementation now reserves one realm-queue descriptor before OS-thread spawn. Closing the worker queue moves its existing backing into that arena-lived descriptor without copying or allocating. Later materialization reserves the entire expanded layout before consuming either transferred backing or Promise reactions, so an expansion OOM leaves the complete suffix rooted and retryable. Empty completion/startup failure cancels the unused reservation. Address-ordered queue locking is retained; after transfer the destination lock owns the buffer, so tracing/materialization never reacquires the former source lock.
The descriptor payload shares the existing Promise pointer word under the microtask kind tag; the job-size invariant is tested. A GC insertion barrier publishes transferred job roots when marking is already active.
Related #887 changes root the return/throw value in ThreadRecord.result before any post-call drain, reload it after GC-capable work, and remove redundant best-effort temp-root copies. This also preserves a body exception when a draining callback throws and avoids a stale synchronous-join result copy across its drain. AsyncJoin settlement OOM and completion-state transactions remain outstanding under #887.
Initial focused tests passed: 18 microtask tests (including concurrent producer/consumer transfers, transfer/expansion allocator failure, transferred settlement reactions, exact FIFO ownership, precise relocation, and a real concurrent-mark insertion-barrier witness), plus the dedicated pre-done result-root test with exhausted temp-root allocation. The final source snapshot is now running the combined validation batch. No full acceptance claim or issue closure yet.
fix(threads): preserve completion roots and queue handoff (#886, #887)
chrisbbreuer commentedon Sep 8, 2026
Landed on
origin/main: fix(threads): preserve completion roots and queue handoff (#886, #887).The final worker-to-realm transfer now moves the closed source queue's backing into a descriptor whose destination slot was reserved before OS-thread spawn. Publication does not allocate or copy jobs. Materialization reserves the entire expanded layout before consuming any transferred backing or Promise reaction ownership; OOM leaves a retryable descriptor with the exact suffix and order intact. Ordinary microtask payload size remains one pointer word.
The descriptor owns only the unconsumed source suffix. Its graph is traced/relocated and insertion-barrier shaded when transferred after a concurrent root scan. Cleanup is serialized by the destination queue lock or the GIL, including early startup cleanup.
New witnesses cover failed preparation, empty completion, exhausted destination capacity with competing reservations, transferred settlement batches, eight simultaneous producers with 256 exact FIFO jobs, post-root-scan publication, and moving root updates. A real Thread regression forces two throwing drains and confirms all 14 untouched callbacks run exactly once and the original body exception survives, in both GIL modes. The related provisional result-root work is tracked in #887.
Validation on the committed source:
TSAN_OPTIONS=halt_on_error=1)./tmp/zig-js-thread-transfer-evidence/(local evidence, not a portable checked-in report).This follows the abrupt-drain restoration fix in b7c8eb0a. Keeping #886 open pending its remaining broader acceptance coverage, including full-suite TSan, extended lifecycle/mid-GC profiles, and applicable private ABI/PR-249 coverage. No completion claim is made for #887's asyncJoin settlement transaction.
chrisbbreuer commentedon Sep 8, 2026
Additional integration evidence landed with fix(threads): commit prepared asyncJoin completion (#887), which composes prepared asyncJoin publication and startup registration with this issue's reserved final queue handoff.
All 35 distinct focused tests, including the transfer, nested drain, concurrent producer, and GC visitor witnesses, pass in Debug, ReleaseSafe, and suppression-free TSan. The full Debug suite passes 2,332 tests with 1 skip, 0 failures, 0 leaks. PR-249 async-join/nested-thread cases pass in both GIL modes. Default/verify/broad/mid-GC/lifecycle fuzzing passes 5,040 generated programs (including 900 mid-GC and 3,360 lifecycle subcases). Exact-parent Promise/async Test262 diagnostics remain 1,828 passing, zero flips/regressions across the six measured subtrees.
This adds the extended lifecycle/mid-GC evidence missing from the preceding transfer commit. It is not a full-suite TSan or full private-ABI/corpus acceptance claim; #886 stays open for the remaining checks. Detailed logs and inputs are retained locally in
/tmp/zig-js-887-evidence/.fix(promises): preserve dequeued job completion on OOM (#885)
fix(threads): hand off terminal checkpoint errors with queued jobs (#896
chrisbbreuer commentedon Sep 9, 2026
Final integration verification for fix(threads): preserve completion roots and queue handoff (#886, #887) is complete on the pushed engine source in 0927912. The already-measured transfer/materialization/restoration faults, nested FIFO ownership, no-GIL producers, and root visitors now have the remaining full-suite TSan and ABI coverage. Terminal worker checkpoint failures also travel with the retained queue in da4b0105.
Closing this transaction issue. The earlier local incomplete-sweep observation is preserved and remains under investigation in #907; #896's remaining private exception-box admission audit and #906's finalization cleanup ownership work remain separate open tasks. No timeout, workload, expected failure, or sanitizer suppression was relaxed to obtain this evidence.