Description
Parent: #50
Related: #47, #49, #53, #61, #97, #116, #118, #119, #120, #131
Delivered benchmark
The accepted comparison includes sustained short-lived ordinary-object allocation with a bounded live working set. Each lane:
- allocates a fixed 256-object ring;
- repeatedly reads a displaced object's initialized properties;
- replaces the indexed entry with a fresh three-property ordinary object;
- preserves a deterministic, exactly representable checksum over the displaced reads and bounded live tail;
- uses identical source/invocation bytes and equal jobs per lane in zig-js and JSC;
- avoids host-only GC APIs and engine-specific source recognition.
Harness validation covers checksum/work equality, timing floors, lane/job symmetry, sample counts, runner alternation, report/TSV consistency, and bounded live-set documentation. General allocation/GC optimizations preserved prototypes, initialized properties, indexed replacement, barriers, OOM behavior, and tracing semantics.
Final accepted result
The July 15, 2026 128-byte-slab report measured clean source commit 12aa217c2f66c0fe7fa95e38ec414ea5ec15a993 and was published in 78a3fc5c with 1,540 raw samples:
- direct: 101.799 ms zig-js vs 123.020 ms JSC (1.21x zig-js throughput)
- warmed eight-context: 168.520 vs 185.412 ms (1.10x; zig-js wins)
- cold eight-context: 181.819 vs 190.980 ms (1.05x; zig-js wins)
- shared 1/2/4/8 lanes: 146.979 / 273.948 / 604.686 / 1,671.114 ms
- shared eight-lane scaling/RSD: 0.70x / 7.24%
Ordinary Object is now 96 payload bytes and occupies the 128-byte managed slab. Its lazy storage wrapper owns cold/exotic state and external named/element metadata; a plain object with four or fewer named properties keeps values inline. The current object bucket uses 256 KiB chunks.
- report: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.md
- raw evidence: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.tsv
- exact-parent object-layout A/B: https://github.com/zig-utils/zig-js/blob/main/docs/.data/object-churn-128-byte-slab-ab-2026-07-15.md
Scope split
This issue owns the benchmark workload, fairness contract, validation, baseline, general object-allocation evidence, and publication. Those are complete. Shared-realm scaling is a separate engine-performance problem tracked with current attribution and acceptance criteria in #97; duplicating its >=1.0x target here made two issues represent the same unfinished work. #97 remains open, while this benchmark-delivery issue is complete.
Acceptance criteria
- Sustain fresh ordinary-object allocation and indexed replacement in the timed loop with a bounded live ring.
- Exercise multiple initialized properties, displaced-object reads, and deterministic exactly representable checksums.
- Extend harness validation and document timed boundaries, live-set behavior, and limitations.
- Calibrate equal job counts so every full-run engine/mode row exceeds the publication timing floor.
- Capture and link the initial symmetric baseline before optimization.
- Add allocation/GC evidence sufficient to identify the limiting general paths.
- Optimize only general engine behavior; preserve prototypes, accessors, proxies, barriers, OOM behavior, and GC tracing.
- Beat JSC in every fair direct and independent-context row.
- Move the distinct shared-realm scaling contract and evidence to Perf: remove shared-realm object-churn GC scaling collapse #97.
- Pass focused semantic/GC/no-GIL tests, the comparison harness, and batched full-suite/TSan gates.
- Publish the report/TSV and README summary only after all gates pass.
Activity
bench: add bounded object churn comparison (#62)
chrisbbreuer commentedon Jul 14, 2026
The honest symmetric workload baseline is on
mainin445a79f3(bench: add bounded object churn comparison (#62)). One job initializes a lane-local 256-object ring, then creates 20,000 fresh three-property objects while replacing indexed entries and reading the displaced objects. The live ring stays bounded; no host GC API or engine-specific source is used.Order-balanced direct baseline, ten samples per engine, one job, exact checksum
10195248:Eight-lane steady and cold probes produced the same exact aggregate checksum,
82446720, in both engines. The 100-job production default projects JSC above the 50 ms floor while keeping quick mode practical.The initial shared-realm result also exposes a real scaling problem: one-job medians were about 14.75 ms at one lane and 192.92 ms at eight lanes, only 0.61x effective scaling for eight times the work. No favorable report will be published until both the direct/independent deficit and shared scaling failure are fixed through general engine paths.
perf(vm): fast-path dense array replacement (#62)
chrisbbreuer commentedon Jul 14, 2026
First general engine slice is on
mainin0d918362(perf(vm): fast-path dense array replacement (#62)).A symbolized one-second sample of the unoptimized 100-job run attributed the largest leaf to generic indexed
[[Set]](Interpreter.setMemberResult, roughly 23–34% of sampled stacks), followed by object allocation (newObject, roughly 15–26%) and separate literal-property initialization (defineLiteralDataProp, roughly 7–22%). The profile is preserved locally as/tmp/zig-js-object-churn.sample.txt; these symbols and the baseline numbers determine the optimization order.The retained fast path handles an exact numeric key replacing an existing present dense Array element. It rejects Arguments, Proxies, accessors, descriptor maps, indexed named properties, holes, and out-of-range writes; it enforces
Thread.restrict, performs the GC insertion barrier, and combines the presence check and store under the element lock. Focused tests prove selection and preserve accessor, non-writable descriptor, inherited hole setter, and Proxy behavior.Ten-sample direct median at the same one-job checksum
10195248:445a79f3): 15.585 ms0d918362): 12.840 msJSC remains around 1.22 ms, and shared eight-lane behavior remains both slow and noisy. Next measured targets are batched ordinary-object allocation/literal initialization and shared allocation locking; #62 stays open.
perf(gc): expose exact slab cell ownership (#62)
chrisbbreuer commentedon Jul 14, 2026
Landed the next general GC/allocation slice in
16c8a542, paired with zig-utils/zig-gc@304ef51(zig-utils/zig-gc#4).What changed:
Validation:
object_churn, 100 jobs × 7 samples: median1.468 s, exact checksum1030552128;12.695 msversus the retained pre-slice12.840 msobservation, a small movement that I am not treating as a material win.This closes the profiled payload-index allocation cost structurally, but JSC is still well ahead on this row, so the issue remains open for the next general object-allocation/initialization optimization and the eventual batched publication gates.
perf(object): inline small property storage (#62)
chrisbbreuer commentedon Jul 14, 2026
Object-churn optimization checkpoint pushed to main in three small commits:
ReleaseFast single-context object_churn (100 jobs x 7 samples) now has median 1,141,386,292 ns with exact checksum 1030552128, versus the retained 1.468 s pre-checkpoint median (about 22.3% faster). Same-run JSC median is 120,900,125 ns, leaving about a 9.44x gap.
All 751 zig-js tests passed with no leaks, zig-gc tests passed, and the benchmark harness tests passed 7/7. The JSC-win, shared-scaling, TSan, and final report/README acceptance items correctly remain open.
chrisbbreuer commentedon Jul 14, 2026
Measurement correction and new checkpoint:
I discovered that
zig build -Doptimize=ReleaseFastdoes not rebuild the separately installed comparison runner. Its timestamp predated the three prior checkpoint commits, so the previously posted 1.141 s / 9.44x result must be disregarded as stale-binary evidence. The correctness and test results in that update remain valid.I rebuilt the runner with the dedicated
zig build benchmark-comparison-bintarget and compared cleanabdc30bfagainst the explicit-ownership optimization now pushed in8e861ec2(#64). Order-balanced alternating samples gave 867.854 ms at the clean baseline versus 824.063 ms current, about 5.0% less time. A fresh final 100-job x 7-sample run at current source measured 871.043 ms for zig-js versus 120.567 ms for JSC, exact checksum 1030552128, leaving a 7.22x direct gap.Profile evidence is consistent with the A/B:
headerForPayloadfell from roughly 1,800 samples to 28. Full validation remains 751/751 with no skips/failures/leaks and benchmark-harness validation remains 7/7. Future benchmark evidence will explicitly use the comparison-bin target.chrisbbreuer commentedon Jul 14, 2026
Next measured object-churn slice is on main in
0d118f23(#65). Fixed-nameinit_propsites now reuse the existing four-way bytecode inline cache for immutable literal shape transitions, retain seqlock publication and object locking in shared/concurrent modes, and take a guarded direct-append path for isolated fresh literals. No bytecode-cache storage was added.Clean order-balanced
8e861ec2/current A/B medians were 810.839 ms and 780.171 ms, about 3.8% less elapsed time. The fresh post-gate 100-job x 7-sample comparison measured zig-js at 842.744 ms versus JSC at 122.002 ms, checksum 1030552128, leaving a 6.91x direct gap.defineLiteralDataProp, shape lookup, and transition traversal disappeared from the post-change sampled leaves.Validation: 752/752 tests passed with no skips/failures/leaks, including the new cross-realm isolated/seqlock test with duplicate, accessor, numeric, computed,
__proto__, and inline-to-backed cases; benchmark harness tests passed 7/7. #62 remains open for the next profile-driven engine cost and the final fair comparison publication.chrisbbreuer commentedon Jul 14, 2026
Checkpoint after #65 and #66 landed on main:
#62 remains open: object_churn is still not publication-ready and the next work should come from the new post-#66 profile rather than broad benchmark-specific changes.
chrisbbreuer commentedon Jul 14, 2026
Checkpoint: #67 is on main in 3a8eda3. Number-number bitwise and shift bytecodes now stay in the VM numeric fast path while BigInt and observable coercion retain the generic path.
Clean pinned order-balanced A/B against ecd5bb4 improved object_churn from 669.566 ms to 636.429 ms, a 4.949 percent elapsed reduction and 1.0521x throughput gain. The exact checksum remained 1030552128 in all runs. Interpreter.applyBinary, Interpreter.toPrimitive, and Value.toInt32 disappeared from post-change sampled top leaves at the reporting threshold; young-GC sweeping and cell-slab release are now dominant.
Fresh post-gate 100-job medians are zig-js 693.224 ms versus JSC 126.226 ms, leaving a 5.492x direct gap. Validation is clean at 753/753 unit tests with zero skips/failures/leaks plus 7/7 comparison-harness tests.
#62 remains open. The next optimization should target the now-dominant general nursery sweep/free path, then repeat the pinned A/B before any final report publication.
chrisbbreuer commentedon Jul 14, 2026
Nursery-walk checkpoint: zig-utils/zig-gc#8 is implemented on main in zig-utils/zig-gc@0427c2d.
Minor whitening and sweep now stop at the first old cell in the collector's contiguous young prefix; old-kind tracing begins at that boundary, and nursery disable safely tenures a pending prefix. The change is general collector behavior, not benchmark-specific.
Clean pinned 12-sample object_churn A/B at the exact checksum 1030552128:
Validation is clean: full zig-gc suite, 8/8 focused nursery tests, 2/2 poisoned-boundary tests, and 7/7 downstream zig-js nursery integration tests. The gain is modest but repeatable; #62 remains open because the current direct row still trails JSC and the final full comparison/publication gates have not been met.
chrisbbreuer commentedon Jul 14, 2026
Optimization checkpoint: #70 landed on main in 971f994.
The VM now predecodes immutable call/array quickening candidates and negatively caches unsupported isolated property traces. A clean 12+12 fresh-process, order-balanced object_churn A/B improved median time from 645.740 ms to 616.878 ms (4.47% reduction), with checksum 1030552128 on every sample. Full validation passed: 754/754 zig-js tests and 7/7 benchmark-harness tests.
The measured rejected alternatives remain documented in #68, #69, and zig-utils/zig-gc#9; none of those regressions were retained.
chrisbbreuer commentedon Jul 14, 2026
Checkpoint c4f4682 closes #72 with a general fixed-shape object-allocation loop quickening path.
Pinned A/B improved object_churn from 578.091563 ms to 266.911521 ms median across 12 fresh alternating processes per side, a 53.83% elapsed reduction (about 2.166x throughput), with checksum 1030552128 in every run. Full validation passed: 755/755 unit tests and 7/7 benchmark-harness tests.
A separate fresh 7-process-per-engine run measured zig-js at 296.352 ms versus JavaScriptCore at 128.644 ms, both with the exact checksum in all runs. The remaining direct gap is 2.303x, so #62 intentionally stays open. The expanded documented comparison should wait until the remaining rows are actual wins.
13 remaining items
chrisbbreuer commentedon Jul 14, 2026
Rejected #92 after two equal exact-parent batches. Deferring the remaining four quick object-loop frame writes passed both focused filters (6/6, zero leaks) and preserved checksum
1030552128, but only won 8/14 alternating pairs; both the first and confirmation batches were 4/7 amid large bidirectional host outliers. The approximately -1.06% paired median was not strong enough for the added state/error-path complexity, so the experiment was fully reverted with no commit. #91 remains the retained checkpoint.chrisbbreuer commentedon Jul 14, 2026
Progress: db131e0 hoists the realm-intrinsic Object.prototype lookup from each quickened object allocation into the checkpoint-bounded batch. Two alternating exact-parent A/B batches preserved checksum 1030552128 and favored the candidate in 11/14 pair medians; aggregate medians improved 1.35% and 5.56% in the two batches. Full units (760/760, zero leaks) and comparison-harness tests (7/7) pass. A fresh complete comparison run and README/report refresh are next; the direct JSC gap remains an explicit published result, not something to omit.
docs(bench): publish ten-workload comparison (#62)
chrisbbreuer commentedon Jul 14, 2026
Published the expanded honest baseline in fdb32d8 from measured commit db131e0: 1,540 validated samples, exact checksums, and every row above the 50 ms floor. object_churn remains the only direct loss: 204.392 ms zig-js vs 120.512 ms JSC (0.59x zig-js throughput). At eight warmed independent contexts it is 786.543 ms vs 196.866 ms (0.25x), and shared-realm scaling falls to 0.28x. The README now presents the loss alongside the 9/10 wins. Shared-GC contention is split into a focused child issue; the direct allocation gap remains here.
chrisbbreuer commentedon Jul 14, 2026
Published the fixed-shape allocation follow-up in b8a4973 and refreshed all 1,540 comparison samples/README tables in 6b27f4d.\n\nLatest direct object-churn medians: zig-js 195.331 ms vs JSC 116.085 ms (0.59x zig-js/JSC throughput). This improves the previous saved zig-js median of 204.392 ms, while JSC remains ahead. Exact-parent alternating A/B isolated the implementation gain more clearly at -12.66% median with 6/7 candidate wins. The remaining direct and shared allocation/GC work stays tracked by #97.
chrisbbreuer commentedon Jul 14, 2026
The bounded object-churn workload is refreshed after the batch-publication slice (
84d2f9f4, zig-gc#15). Direct checksum and bounded live-tail semantics remain exact.Latest medians:
The published shared 8-lane median improves 33.0% and scaling rises from 0.21x to 0.31x, but #97 remains open for the remaining header/backing/initialization contention and the >1x acceptance target.
Evidence: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-14.md
chrisbbreuer commentedon Jul 14, 2026
Fresh full publication from clean measured commit
c98a36c4, published bye4b72407:#101 reduced the exact-parent shared8 median by 12.2% and the full publication by 10.3%, with exact checksums. The direct JSC win and >1x shared-scaling criteria remain unmet, so #62 stays open.
chrisbbreuer commentedon Jul 14, 2026
#102 is complete and published in
24656d74. Moving GC slab reuse state out of dead cells improved exact-parentobject_churnby 7.37% direct and 2.96% at the shared 8x100 publication load; the full report now measures 176.398 ms direct and 9203.183 ms shared at 8 lanes. Shared object churn remains the main scaling gap (0.33x), so further work should continue from profiles rather than allocator-link batching.chrisbbreuer commentedon Jul 14, 2026
Latest full comparison evidence shipped in
ab386010from clean implementation commit2dac7aca: all 1,540 raw samples plus the generated report, benchmark guide, and README. zig-js now leads the 10-workload geometric mean by 2.69x direct, 2.71x at eight warmed independent contexts, and 2.77x in the cold eight-lane lifecycle. Object churn remains the only direct loss (148.758 ms vs JSC 114.187 ms) and the shared-realm scaling exception (377.316 ms at one lane vs 9,789.216 ms at eight, 0.31x scaling), tracked under #97/#104.chrisbbreuer commentedon Jul 14, 2026
Latest full comparison evidence is on
mainin409d6a30, generated from clean implementation commit5f891bc8: all 1,540 raw samples, generated report, benchmark guide, and README. zig-js leads the 10-workload geometric mean by 2.64x direct, 2.73x at eight warmed independent contexts, and 2.76x in the cold eight-lane lifecycle. Object churn remains the sole direct loss (150.652 ms vs JSC 114.952 ms), while warmed/cold 8x throughput improved 20.4%/11.1% and shared 8x improved 5.1% over the prior publication.chrisbbreuer commentedon Jul 15, 2026
Progress update: #114 and #115 are complete on main. zig-gc 9d4af0d batches sweep accounting; zig-js 86d0c5b publishes the validated 1,540-sample matrix and 9e832d1 records/dirty-checks dependency revisions. Latest direct object churn is 120.334 ms vs JSC 118.117 ms (1.9% behind); warmed 8 independent is 465.616 vs 200.717 ms. The report was captured on battery/discharging, so small deltas are not compared against the prior AC run. Gates are green: 766/766 normal; TSan 765 pass, 1 expected skip, 0 fail/leak. #62 stays open because beating JSC in object churn and eliminating the multi-context gap remain unmet.
chrisbbreuer commentedon Jul 15, 2026
July 15 progress from #118 is published.
d2e29fa01d76bfa0The precise private quick-loop checkpoint makes object churn a direct win: 114.315 ms for zig-js versus 119.322 ms for JSC (1.04x zig-js throughput). The 10-workload direct matrix is now 10/10 for zig-js.
This issue correctly stays open because object churn remains the only cross-engine loss in multi-context rows. At eight lanes it is 277.124 versus 172.070 ms warmed and 351.305 versus 179.788 ms cold. Shared-realm object churn is 193.942 / 353.325 / 749.455 / 4,751.665 ms at 1/2/4/8 lanes, so eight-lane scaling is still only 0.33x.
The final normal, TSan, publication-guard, and docs gates passed; the dated report/TSV and README are published. The remaining unchecked acceptance items are therefore the independent-context JSC wins and shared eight-lane scaling.
chrisbbreuer commentedon Jul 15, 2026
#119 is complete and published:
ab7b08fb; docs/evidence88bd1b00;mmap/munmapframes versus 76 matching parent entries;The full matrix was recorded at 27% battery while discharging and has elevated unrelated dispersion, so the same-session A/B—not cross-run absolute timing—is the accepted optimization evidence. Multi-context and shared object churn remain open; the issue body now carries the exact current endpoints.
chrisbbreuer commentedon Jul 16, 2026
The clean July 15 publication in
6c71f46f(measured source0a74f7d1) closes the directly comparable object-churn gap:The canonical issue body is updated. Direct/independent completion is checked; #97 remains open for the separate shared-realm scaling contract. Report and all 1,540 raw samples are linked from the body and README.