Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Parent: #44
Related: #45, #53, #62, #97
Goal
Keep performance work measurable and prevent accepted wins from silently regressing without turning ordinary CI into a noisy hardware contest.
Landed controls
- Versioned raw TSV samples and rendered reports retain exact date, host/CPU, OS, Zig, zig-js/zig-gc/zig-regex revisions, system JavaScriptCore version, and power metadata.
- The comparison driver validates the complete symmetric matrix, exact sample indexes/counts, stable checksums, cross-engine equal work, runner alternation, a 50 ms full-run timing floor, and per-row min/max/RSD.
- Publishing raw/report evidence refuses a dirty tracked worktree and captures metadata before generated outputs modify the tree.
zig build benchmark-comparison-testexercises matrix and publication guards in ordinary CI without compiling or running benchmarks.- The manual-only Performance workflow, added in
c7e52fb1, runs the full macOS/JSC matrix with configurable sample/lane counts and retains its artifacts for 90 days. It never runs on pushes or pull requests and never gates ordinary CI. 0ff34466adds the controlled history/publication tool. It reruns the full raw-matrix validation, reproduces the paired Markdown report byte-for-byte, and then generates the marker-delimited README scorecard.- History comparison requires exact host, OS, Zig, zig-gc, zig-regex, JavaScriptCore, matrix, jobs, and sample-count matches. Volatile battery details are normalized while power source and charging state remain part of the identity.
- Every engine row is retained with baseline/current medians, delta, and both RSDs. A zig-js row gates only when median wall time worsens by more than 10% and both RSDs are at most 5%; JSC rows remain visible environmental controls. The exact baseline/current report paths and zig-js revisions are recorded in the artifact.
- The manual workflow always generates a README scorecard artifact and optionally accepts a committed controlled baseline pair to produce the per-row history artifact or reject an incompatible run. Failed validation still uploads available evidence.
Current accepted evidence
- measured source:
12aa217c2f66c0fe7fa95e38ec414ea5ec15a993 - report: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.md
- raw 1,540 samples: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.tsv
- 10/10 direct wins (2.67x geometric-mean zig-js throughput)
- 10/10 eight-lane warmed-independent wins (2.95x; 5.14x zig-js scaling vs 4.77x JSC)
- 10/10 eight-lane cold-lifecycle wins (2.91x; 5.18x vs 4.76x)
- 3.97x shared-realm geometric-mean eight-lane scaling; Perf: remove shared-realm object-churn GC scaling collapse #97 owns the remaining object-churn deficit
The generated calculation corrected the previously hand-copied 2.63x / 2.94x / 2.94x aggregate summary to the raw-derived 2.67x / 2.95x / 2.91x, demonstrating the drift this issue was intended to eliminate.
Acceptance criteria
- Store versioned raw samples and rendered reports with exact commit/environment/power metadata.
- Add deterministic standalone harness/unit checks to ordinary CI.
- Add a manually triggered full performance workflow that does not gate ordinary CI.
- Compare only like-for-like host/runtime configurations and retain explicit historical baselines.
- Define a minimum timing floor, expose repeated-sample dispersion, and apply a conservative noise-aware policy only to controlled runs.
- Fail publication when the tracked worktree is dirty and capture metadata before generated outputs change the tree.
- Link optimization commits and resulting benchmark deltas from their workload issues.
- Keep README headline numbers generated from the latest accepted report rather than manually copied.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
bench: guard benchmark evidence publication
docs: publish stricter JSC benchmark results
chrisbbreuer commentedon Jul 13, 2026
Progress in e70eeff and 6fb3b28: publication now refuses a dirty tracked worktree, captures metadata before writing tracked outputs, and retains the exact source commit plus 224 raw samples. #50 remains open for CI harness tests, controlled performance runners/history, derived regression thresholds, and generated README tables.
chrisbbreuer commentedon Jul 13, 2026
The publication guard and dated evidence passed the final combined gate (31 success, 0 failure): https://github.com/zig-utils/zig-js/actions/runs/29280193748. Remaining CI/history/generation work stays open.
bench: publish winning JSC comparison evidence (#50)
docs: share the winning JSC benchmark (#50)
chrisbbreuer commentedon Jul 14, 2026
Latest accepted evidence is now published:
42d0ec26ccee884209d543c3The clean full matrix passed every checksum and 50 ms floor guard. zig-js wins all 36 symmetric single/steady/cold rows. README headline values are still manually summarized, so the generated-headline/workflow acceptance items remain open.
ci: honor locked docs dependencies (#50)
chrisbbreuer commentedon Jul 14, 2026
Final validation for the published comparison is green on
mainat889611de:e8f98e49) and honor the committed Bun lockfile for docs (889611de)The measured report (
ccee8842) and README summary (09d543c3) are therefore validated end-to-end. I am leaving #50 open only for its remaining automation work: generate/check the README summary from the committed report so future benchmark updates cannot drift.bench: align comparison allocators (#50)
chrisbbreuer commentedon Jul 14, 2026
Pushed
0e22e4bb(bench: align comparison allocators (#50)). Every measured zig-js context now uses the same process-wide libc allocator, matching JSC's process-scoped cached allocator model instead of mixing the runner allocator into direct/shared rows. The generator and benchmark guide document the current method while preserving the July 13 report as immutable historical evidence.Validation:
23508000000at one lane and188190000000at eight shared lanesThis is a fairness/alignment change, not a claimed speedup. The next accepted full matrix will measure the unified allocator rather than rewriting the saved report.
chrisbbreuer commentedon Jul 14, 2026
Historical evidence update: fdb32d8 publishes the clean db131e0 ten-workload run with a rendered report, exact environment/power metadata, README summaries, and all 1,540 raw samples. The expanded matrix includes polymorphic properties and bounded object churn; README now shows direct, eight-context, and shared-realm tables and explicitly reports the remaining allocation gap.
chrisbbreuer commentedon Jul 14, 2026
Fresh full publication from clean measured commit
c98a36c4, published bye4b72407:The exact-parent A/B for #101 measured 12.2% faster shared8 and 3.4% faster direct with exact checksums. #101 is complete; object churn remains the sole direct JSC loss and shared lanes above 1x remains unmet, so the parent performance issues stay open.
ci(docs): lock parser-valid highlighter (#50)
chrisbbreuer commentedon Jul 15, 2026
Docs CI follow-up landed in
5100a966(ci(docs): lock parser-valid highlighter (#50)). Two consecutive main runs failed only the docs job because.gitignoreexcludedbun.lock, sobun install --frozen-lockfileon clean runners had no lock to freeze and resolved malformedts-syntax-highlighterbundles. Both published 0.2.1 and 0.2.4 fail Bun 1.3.14 parsing with undeclared/exported bindings; 0.2.0 is parser-valid and API-compatible with the current 24-page build.The fix tracks the text
bun.lockand records an exact 0.2.0 override. Clean local reproduction passed: removenode_modules,bun install --frozen-lockfile, parse/bundle the highlighter, andbun run docs:build(24 pages). This makes the existing frozen-lock CI step real rather than nominal. The new main run will be monitored before this slice is handed off.chrisbbreuer commentedon Jul 15, 2026
CI follow-up: the docs job now passes on the pushed lockfile/override fix in commit
5100a966(run: https://github.com/zig-utils/zig-js/actions/runs/29449252934). This confirms the cleanbun install --frozen-lockfilepath and the full 24-page docs build on GitHub Actions. The remainder of the runtime/TSan matrix is still in progress and will be monitored separately.ci(bench): add manual full comparison workflow (#50)
bench: generate controlled benchmark history (#50)
chrisbbreuer commentedon Jul 16, 2026
Completed in
0ff34466. The focused publication gate passed 17/17 checks; the accepted 1,540-row report reproduces exactly from its TSV; README regeneration is idempotent; and an older run with different zig-gc/zig-regex revisions is rejected as non-comparable. The generated README scorecard is now onmain, and the manual workflow retains generated scorecard/history evidence without making hosted timing an ordinary CI gate.