Repository navigation
Closed
Labels
benchmarksBenchmark design, evidence, and regression gatesBenchmark design, evidence, and regression gatesdependenciesDependency ownership and offline build policyDependency ownership and offline build policyenhancementNew feature or requestNew feature or requestperformanceMeasured runtime, memory, scaling, or build performanceMeasured runtime, memory, scaling, or build performance
Description
Parent: #459
Depends on: #460, #462
Related: #461, #503
Goal
Validate zig-js performance against independent, pinned, out-of-tree benchmark suites and additional engines without making any suite or engine a build/runtime dependency.
Why
#460 defines rigorous repository-owned application-shaped workloads and anti-specialization variants. Broad leadership claims also need independent workloads whose structure was not designed around zig-js, plus controls showing whether a result is specific to system JSC.
Scope
- Inventory candidate language/runtime suites by license, host API requirements, determinism, workload coverage, run duration, maintenance, and compatibility with a non-browser embedder.
- Select pinned script/Wasm subsets that can run unchanged or through a minimal documented host adapter; classify excluded browser/Node APIs rather than silently patching semantics.
- Run suites from explicit temporary or sibling checkouts with exact revision/checksum. Do not vendor them, resolve them during ordinary builds, or make them package dependencies.
- Add optional comparison adapters for system JSC and independently installed V8, SpiderMonkey, and QuickJS where exact programming models/timed boundaries are possible.
- Preserve official scores where valid, plus per-test wall/CPU/RSS/tier/GC data and exact failures/skips.
- Keep engines in separate executables/processes so zig-js's JSC-shaped exports cannot interpose.
- Publish suite-specific results and geometric aggregates only after every included row, exclusion, patch, and host difference is visible.
No-workaround rules
- No editing workload logic to favor zig-js, unsupported test deletion, dependency bundled into zig-js, browser score claimed by a non-browser subset, fake public-API equivalence, or “win” based only on an aggregate hiding losses.
Acceptance
- A machine-readable inventory records suite/engine pins, licenses, checksums, adapters, applicable rows, exclusions, and reasons.
- Acquisition and execution are explicit, offline-after-fetch, checksum-verified, and quarantined from normal build/runtime dependencies by Build: enforce owned local dependencies and offline builds #462.
- Results retain every applicable row, failure, skip, raw sample, dispersion, checksum/output validation, and exact timed boundary.
- Anti-specialization mutations and tier attribution show no suite/source recognizer.
- Broad performance wording distinguishes repository-owned, independent-suite, JSC, and multi-engine evidence.
Foundation
The symmetric cross-engine process/timing protocol is established by #44/#45/#50 and tools/benchmark-comparison.py; its owned replacement is tracked by #497.
Metadata
Metadata
Assignees
Labels
benchmarksBenchmark design, evidence, and regression gatesBenchmark design, evidence, and regression gatesdependenciesDependency ownership and offline build policyDependency ownership and offline build policyenhancementNew feature or requestNew feature or requestperformanceMeasured runtime, memory, scaling, or build performanceMeasured runtime, memory, scaling, or build performance
Activity
chrisbbreuer commentedon Aug 5, 2026
The first independent-suite/engine inventory slice landed on main in
f3767f87.zig-js-independent-suite-inventory-v1now freezes:The new
zig build independent-suite-auditis a dependency-free offline Zig gate with malformed-pin and missing-exclusion tests, and it now runs insidebenchmark-comparison-testin CI.dependency-auditconfirms the change introduced no network-capable repository tool, ordinary-build fetch, package, link, or runtime dependency.This is candidate classification, not execution or performance evidence. #504 remains open for the explicit checksum-verified acquisition path, owned minimal shell adapter, exact-output execution, raw repeated results, anti-specialization/tier checks, and additional-engine corroboration.
chrisbbreuer commentedon Aug 5, 2026
The explicit acquisition/offline-verification boundary followed in
77939f13.After an operator performs the documented exact-repository clone and detached checkout outside zig-js,
zig build independent-suite-audit -Dindependent-suite-id=... -Dindependent-suite-checkout=...now verifies, offline:originis the exact inventoried repository;Live verification passed the exact Octane checkout with 22 pinned files and the sparse JetStream metadata checkout with five pinned files. A wrong-suite/wrong-origin probe and an intentional untracked-file probe both failed closed; the probe file was removed and the exact Octane checkout passed again.
Only the documented operator
git cloneuses the network. The verifier itself never fetches and remains outside ordinary build/runtime dependencies. This advances the explicit, offline-after-fetch, checksum-verified acquisition acceptance. The owned shell adapter and retained execution/result schema remain next.chrisbbreuer commentedon Aug 5, 2026
The owned zig-js Octane shell adapter landed on main in
ee31e4db.zig build independent-suite-zig-jsnow performs the exact out-of-tree checkout audit before starting a separate zig-js process. The runner then:base.jsand selected workload bytes;loadandprintto the normal embedding environment andglobalThis, with host functions allocated as real managed cells;richards,regexp,splay,navier_stokes, andbox2d) and rejects excluded/unknown rows;HEAD, or environment drift fromTZ=UTC,LC_ALL=C, andLANG=C;Scoring and attribution are deliberately separate.
scoreleaves execution/allocation/pause instrumentation disabled and writesnot_measured_scored_pathplusnull, never fake zeros.attributionenables the complete named tier, compilation/deoptimization, allocation, GC, and pause inventory; its timing/score is diagnostic rather than scored evidence.The exact post-commit command passed the 22-file checkout audit and a clean-revision Richards score sample. Development compatibility probes also demonstrated that failures remain rows: Richards, Splay/SplayLatency, and Box2D completed, while RegExp retained
SyntaxError: UnexpectedTokenand NavierStokes retainedRangeError: evaluation step budget exceeded. Those probes are not publication evidence and no source or limit was changed to turn them into scores.Validation passed
benchmark-comparison-test, the two-mode adapter self-test, dependency and independent-suite audits, the 100-file docs link gate, and the 49-page docs build. The change flips 0 test262 cases (benchmark/tooling/docs only).#504 remains open for repeated raw-sample collection/dispersion and durable artifacts, anti-specialization variants, system JSC plus independently installed V8/SpiderMonkey/QuickJS adapters, and publication wording after every row/failure/skip is retained across engines.
chrisbbreuer commentedon Aug 5, 2026
The repeated-sample/durable-artifact slice landed on main in
06e4ad5d.zig build independent-suite-collectnow runs every applicable Octane row in fresh isolated processes, with uninstrumented score rounds and separate attribution samples. The schema-1 collection:The exact post-commit two-score/one-attribution diagnostic passed the 22-file checkout audit and retained all 15 children in
/tmp/zig-js-issue504-collector-06e4ad5d.json(SHA-256a31e017b97bf3de3976550b7bed4c6c920275400d2ad103488a687e9840d8f61). Richards, Splay/SplayLatency, and Box2D each retained two valid score samples and one valid attribution sample. RegExp retained three valid failed children withSyntaxError: UnexpectedToken; NavierStokes retained three valid failed children withRangeError: evaluation step budget exceeded. Both had zero valid score samples, so the artifact explicitly reports the aggregate as unavailable and the command exits nonzero after writing it.Validation also passed
benchmark-comparison-test,dependency-audit(14 classified edges, zero migration targets, 60 repository tools),independent-suite-audit, the 100-file docs link gate, and the 49-page docs build. This flips 0 test262 cases.#504 remains open for the underlying RegExp/NavierStokes compatibility work, anti-specialization variants, system JSC and additional-engine adapters, and publication wording after a complete all-passing evidence matrix exists.
chrisbbreuer commentedon Aug 5, 2026
The required-when-available system-JavaScriptCore adapter landed on main in
ff0b132c.zig build independent-suite-jscnow builds and runs a separate macOS executable linked only to the real platform JavaScriptCore framework. It shares one frozen Octane path/checksum/license/result table with the zig-js adapter, while independently owning context creation,load/printcallbacks, exact source evaluation, protocol capture, output validation, and schema-1 JSON serialization. The executable never links zig-js, so zig-js's JSC-shaped exports cannot interpose on the framework.Each child records the adapter executable path/hash and exact adapter source revision, plus framework bundle version, macOS build, argv/environment, source identities, every upstream/auxiliary output, process wall/CPU/peak RSS, validation, and timing boundaries. This macOS supplies JavaScriptCore code through the dyld shared cache rather than a standalone framework image; the artifact records that boundary and leaves the unavailable binary path/hash null. Exact per-context tier/compilation/deoptimization and GC/allocation/pause counters are likewise
unavailable_public_api, never zero.Exact post-commit diagnostics passed the 22-file checkout audit and all five applicable rows unchanged: Richards, RegExp, Splay/SplayLatency, NavierStokes, and Box2D. All processes reported adapter SHA-256
2f37264e3aaa90adc3e07454754ddcbc0c643c08934ac18990364e83d9837686, system JavaScriptCore framework22625.1.24.11.2, and macOS build26A5388g. These were single-sample compatibility diagnostics, not repeated or publishable performance evidence, so their scores are not promoted.Validation passed both engine-adapter self-tests, the shared frozen-table test, full
benchmark-comparison-test,dependency-audit(14 classified edges, zero migration targets, 60 repository tools),independent-suite-audit, the 100-file docs link gate, and the 49-page docs build. This flips 0 test262 cases.#504 remains open for repeated cross-engine collection, optional independently installed V8/SpiderMonkey/QuickJS adapters, anti-specialization variants, the zig-js RegExp/NavierStokes compatibility gaps, and publication wording after complete repeated evidence exists.
chrisbbreuer commentedon Aug 5, 2026
The first zig-js Octane RegExp blocker is removed by zig-regex
7196733, with the exact ECMAScript boundary locked downstream in zig-jse8432679. The unchanged pinnedregexp.jsnow parses and executes instead of failingUnexpectedToken. A fresh isolated diagnostic runner (b00deb1327ae47f47e94a119c43e9f24abd0aac8a29337f169a4dd8ffdde672d) reaches a distinct runtime failure,TypeError: value is not a function; the row therefore remains failed and no score is publishable. I am continuing with that runtime defect, not masking it or promoting the partial result.chrisbbreuer commentedon Aug 5, 2026
The zig-js RegExp row now passes unchanged in both isolated modes. Failure-stack retention
2e849880plus structured Error stacks9633ca81localized the second failure toExeccalling.execon the first literal. The engine fix72745298separates literal/internal RegExp allocation from constructor new.target, eliminating the wrong-prototype object.A fresh clean-revision runner (SHA-256
9622d7034cc0d61808efe2981ff98e3ec350f9d83ca1ac48e32de2a80756741c) verified the exact pinnedbase.jsandregexp.js, retained the expectedRegExpresult and one final score, and reportedstatus: passedin both uninstrumented score and separately instrumented attribution processes. These are single compatibility diagnostics, not repeated/publication evidence, so no score is promoted. The remaining zig-js row blocker is NavierStokes; #504 also remains open for dependency provenance, repeated cross-engine collection, anti-specialization, optional engines, and publication wording.chrisbbreuer commentedon Aug 5, 2026
The exact unchanged Navier-Stokes row now passes end-to-end in both score and separate attribution modes at zig-js revision 6ac8040.
Root cause was a harness/resource-boundary mismatch: the finite pinned workload reached the ordinary 500,000,000-step embedder runaway guard before producing output. Commit 6ac8040 assigns termination uniformly for every selected row and mode to the collector process timeout, sets the isolated harness evaluation budget to u64 maximum, and retains/validates both facts in schema-1 evidence. Source bytes, inputs, iteration counts, timing logic, and expected outputs remain unchanged.
Validation: full benchmark-comparison-test passed; independent-suite inventory and collector drift gates passed; docs link/build gates passed; score mode produced the required NavierStokes result and one final score; attribution mode produced the same required output contract with tier/GC counters retained. Both runs are single diagnostic samples and their numeric scores are not publication evidence.
chrisbbreuer commentedon Aug 5, 2026
Closed the remaining linked-source provenance gap with 0382848.
The zig-js adapter now rejects dirty or noncanonical local path dependencies and retains the resolved path, canonical repository, exact revision, and clean status for both zig-regex and zig-gc. The collector validates the inventory and requires one stable dependency identity across the collection.
An exact clean Richards diagnostic at 0382848 passed and retained zig-regex 719673318b0c6250898d7ced3603876c148e370b plus zig-gc c89a10fc4454bfba3d28183cfa7fade08fab8aa4. Full benchmark-comparison-test, inventory, collector, docs-link, and docs-build gates passed. The single numeric score remains diagnostic and is not publication evidence.
chrisbbreuer commentedon Aug 5, 2026
The first complete all-passing repeated zig-js collection is now validated at revision 0382848.
External artifact: /tmp/zig-js-issue504-collector-03828485.json (332,271 bytes, SHA-256 e17ffe7d041ca85ed3fd27cdacdb35523f1431de6c9243e2239f863137d22430). All 15 fresh processes passed contract validation: Richards, RegExp, Splay/SplayLatency, NavierStokes, and Box2D each retain two valid uninstrumented score children plus one separate valid attribution child, with no failed/invalid indexes. The artifact retains stable zig-js executable/source identity and exact clean zig-regex 71967331 / zig-gc c89a10fc inputs.
The selected-subset aggregate is structurally available now that every applicable row passes. Publication status remains diagnostic_not_publishable exactly as intended: this run has fewer than seven score samples per row and used a diagnostic host class. No numeric score is promoted.
Remaining #504 acceptance work is anti-specialization/source-recognizer evidence, additional-engine availability/adapters beyond the completed system-JSC path, and final evidence-backed wording.
chrisbbreuer commentedon Aug 6, 2026
Anti-specialization/source-recognizer acceptance is now implemented by 3adb981 and validated on the exact clean revision.
The scored path remains byte-for-byte upstream. The new diagnostic path verifies each pinned source SHA-256 first, then prepends a deterministic block comment so the evaluated source hash and offsets change without changing the AST/workload. A static audit also rejects the Octane revision/tree, all selected source hashes, and distinctive suite identifiers anywhere under src/.
External recognizer artifact: /tmp/zig-js-issue504-recognizer-3adb9817.json (219,564 bytes, SHA-256 6fb7935d89c4fd4cdf1765993aa0dd5051b7d255af8a01caacbee80d72287dee). All ten isolated attribution children passed. Richards, RegExp, Splay, NavierStokes, and Box2D exact/mutated pairs each retained changed evaluated hashes, identical required output contracts, and identical nonzero execution-tier plus bytecode-admission sets. The artifact is explicitly diagnostic_not_performance_evidence; no timing or score is promoted.
Validation also passed the full benchmark-comparison-test, inventory/adapter/collector/recognizer self-tests, docs-link-check, and manifest-checked docs build. Remaining #504 work is optional additional-engine availability/adapters and final scoped wording.
chrisbbreuer commentedon Aug 6, 2026
The first additional independently installed engine control landed in 855601b.
This is labeled node-v8 deliberately: a fresh node:vm context in Node v24.18.0 embedding V8 13.6.233.17-node.50, not a standalone d8 shell and not a browser. Each child retains Node executable path/SHA-256 9fcf926400e3997bc0a3be0c0b00f95df95db9b58b5754d5ca9d02e7fed866f9, adapter path/SHA-256 46310908dcf77b8711ec23302f4d13672ac6d418edc54e2661f00bed0a31d377/revision, exact sources, argv/environment, raw process sample, outputs, validation, and timed boundaries. Exact per-context V8 tier/GC fields are unavailable_public_api rather than zero.
Fresh clean-revision diagnostics passed all five applicable rows unchanged with the required Richards, RegExp, Splay/SplayLatency, NavierStokes, and Box2D outputs and one final score each. These are single compatibility diagnostics; no numeric score is promoted. The optional Node/V8 step does not become an ordinary dependency, and standalone d8, SpiderMonkey, and QuickJS remain distinct planned adapters rather than being misrepresented by this control.
Validation passed the Node/V8 adapter self-test, six-engine inventory audit, docs-link gate, and manifest-checked docs build.
chrisbbreuer commentedon Aug 6, 2026
Acceptance is complete on main. Final slices include all-passing repeated zig-js evidence with complete dependency identity (0382848), exact/source-hash-mutated recognizer equivalence (3adb981), and an explicitly labeled independently installed Node/V8 control (855601b) alongside the separate system-JSC adapter. All scored upstream sources remain unchanged; diagnostic runs are not promoted as publication evidence. Standalone d8, SpiderMonkey, and QuickJS were unavailable on this host and remain distinct optional planned adapters, not silently substituted.
bench(matrix): integrate completed evidence panels (#460)