镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Perf: add benchmark regression gates and historical evidence #50

Closed

Description

@chrisbbreuer

Parent: #44
Related: #45, #53, #62, #97

Goal

Keep performance work measurable and prevent accepted wins from silently regressing without turning ordinary CI into a noisy hardware contest.

Landed controls

  • Versioned raw TSV samples and rendered reports retain exact date, host/CPU, OS, Zig, zig-js/zig-gc/zig-regex revisions, system JavaScriptCore version, and power metadata.
  • The comparison driver validates the complete symmetric matrix, exact sample indexes/counts, stable checksums, cross-engine equal work, runner alternation, a 50 ms full-run timing floor, and per-row min/max/RSD.
  • Publishing raw/report evidence refuses a dirty tracked worktree and captures metadata before generated outputs modify the tree.
  • zig build benchmark-comparison-test exercises matrix and publication guards in ordinary CI without compiling or running benchmarks.
  • The manual-only Performance workflow, added in c7e52fb1, runs the full macOS/JSC matrix with configurable sample/lane counts and retains its artifacts for 90 days. It never runs on pushes or pull requests and never gates ordinary CI.
  • 0ff34466 adds the controlled history/publication tool. It reruns the full raw-matrix validation, reproduces the paired Markdown report byte-for-byte, and then generates the marker-delimited README scorecard.
  • History comparison requires exact host, OS, Zig, zig-gc, zig-regex, JavaScriptCore, matrix, jobs, and sample-count matches. Volatile battery details are normalized while power source and charging state remain part of the identity.
  • Every engine row is retained with baseline/current medians, delta, and both RSDs. A zig-js row gates only when median wall time worsens by more than 10% and both RSDs are at most 5%; JSC rows remain visible environmental controls. The exact baseline/current report paths and zig-js revisions are recorded in the artifact.
  • The manual workflow always generates a README scorecard artifact and optionally accepts a committed controlled baseline pair to produce the per-row history artifact or reject an incompatible run. Failed validation still uploads available evidence.

Current accepted evidence

The generated calculation corrected the previously hand-copied 2.63x / 2.94x / 2.94x aggregate summary to the raw-derived 2.67x / 2.95x / 2.91x, demonstrating the drift this issue was intended to eliminate.

Acceptance criteria

  • Store versioned raw samples and rendered reports with exact commit/environment/power metadata.
  • Add deterministic standalone harness/unit checks to ordinary CI.
  • Add a manually triggered full performance workflow that does not gate ordinary CI.
  • Compare only like-for-like host/runtime configurations and retain explicit historical baselines.
  • Define a minimum timing floor, expose repeated-sample dispersion, and apply a conservative noise-aware policy only to controlled runs.
  • Fail publication when the tracked worktree is dirty and capture metadata before generated outputs change the tree.
  • Link optimization commits and resulting benchmark deltas from their workload issues.
  • Keep README headline numbers generated from the latest accepted report rather than manually copied.

Activity

  1. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Progress in e70eeff and 6fb3b28: publication now refuses a dirty tracked worktree, captures metadata before writing tracked outputs, and retains the exact source commit plus 224 raw samples. #50 remains open for CI harness tests, controlled performance runners/history, derived regression thresholds, and generated README tables.

  2. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    The publication guard and dated evidence passed the final combined gate (31 success, 0 failure): https://github.com/zig-utils/zig-js/actions/runs/29280193748. Remaining CI/history/generation work stays open.

  3. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Latest accepted evidence is now published:

    • measured source: 42d0ec26
    • 616-sample TSV + generated report: ccee8842
    • README + benchmark guide: 09d543c3

    The clean full matrix passed every checksum and 50 ms floor guard. zig-js wins all 36 symmetric single/steady/cold rows. README headline values are still manually summarized, so the generated-headline/workflow acceptance items remain open.

  4. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Final validation for the published comparison is green on main at 889611de:

    • CI run: https://github.com/zig-utils/zig-js/actions/runs/29305518043
    • 32/32 runnable jobs passed; the single nightly-only entry was correctly skipped
    • Includes the docs build, benchmark comparison harness, all unit shards, thread/reference/no-GIL checks, thread-fuzz profiles, TSan unit coverage, and all four TSan no-GIL corpus shards
    • The two follow-up CI fixes are included: isolate property quickening mode in its unit test (e8f98e49) and honor the committed Bun lockfile for docs (889611de)

    The measured report (ccee8842) and README summary (09d543c3) are therefore validated end-to-end. I am leaving #50 open only for its remaining automation work: generate/check the README summary from the committed report so future benchmark updates cannot drift.

  5. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Pushed 0e22e4bb (bench: align comparison allocators (#50)). Every measured zig-js context now uses the same process-wide libc allocator, matching JSC's process-scoped cached allocator model instead of mixing the runner allocator into direct/shared rows. The generator and benchmark guide document the current method while preserving the July 13 report as immutable historical evidence.

    Validation:

    • ReleaseFast comparison runners built successfully
    • benchmark harness contract: 7/7 passed
    • exact array checksum stayed 23508000000 at one lane and 188190000000 at eight shared lanes
    • 7-sample medians were neutral within noise: direct arrays ~61.10 -> ~61.30 ms; shared one lane ~63.15 -> ~63.49 ms; shared eight lanes ~157.59 -> ~156.23 ms

    This is a fairness/alignment change, not a claimed speedup. The next accepted full matrix will measure the unified allocator rather than rewriting the saved report.

  6. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Historical evidence update: fdb32d8 publishes the clean db131e0 ten-workload run with a rendered report, exact environment/power metadata, README summaries, and all 1,540 raw samples. The expanded matrix includes polymorphic properties and bounded object churn; README now shows direct, eight-context, and shared-realm tables and explicitly reports the remaining allocation gap.

  7. chrisbbreuer commented on Jul 14, 2026

    @chrisbbreuer
    MemberAuthor

    Fresh full publication from clean measured commit c98a36c4, published by e4b72407:

    The exact-parent A/B for #101 measured 12.2% faster shared8 and 3.4% faster direct with exact checksums. #101 is complete; object churn remains the sole direct JSC loss and shared lanes above 1x remains unmet, so the parent performance issues stay open.

  8. chrisbbreuer commented on Jul 15, 2026

    @chrisbbreuer
    MemberAuthor

    Docs CI follow-up landed in 5100a966 (ci(docs): lock parser-valid highlighter (#50)). Two consecutive main runs failed only the docs job because .gitignore excluded bun.lock, so bun install --frozen-lockfile on clean runners had no lock to freeze and resolved malformed ts-syntax-highlighter bundles. Both published 0.2.1 and 0.2.4 fail Bun 1.3.14 parsing with undeclared/exported bindings; 0.2.0 is parser-valid and API-compatible with the current 24-page build.

    The fix tracks the text bun.lock and records an exact 0.2.0 override. Clean local reproduction passed: remove node_modules, bun install --frozen-lockfile, parse/bundle the highlighter, and bun run docs:build (24 pages). This makes the existing frozen-lock CI step real rather than nominal. The new main run will be monitored before this slice is handed off.

  9. chrisbbreuer commented on Jul 15, 2026

    @chrisbbreuer
    MemberAuthor

    CI follow-up: the docs job now passes on the pushed lockfile/override fix in commit 5100a966 (run: https://github.com/zig-utils/zig-js/actions/runs/29449252934). This confirms the clean bun install --frozen-lockfile path and the full 24-page docs build on GitHub Actions. The remainder of the runtime/TSan matrix is still in progress and will be monitored separately.

  10. chrisbbreuer commented on Jul 16, 2026

    @chrisbbreuer
    MemberAuthor

    Completed in 0ff34466. The focused publication gate passed 17/17 checks; the accepted 1,540-row report reproduces exactly from its TSV; README regeneration is idempotent; and an older run with different zig-gc/zig-regex revisions is rejected as non-comparable. The generated README scorecard is now on main, and the manual workflow retains generated scorecard/history evidence without making hosted timing an ordinary CI gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions