镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Bench: make zig-js and JSC comparison symmetric and statistically robust #45

Closed

Description

@chrisbbreuer

Parent: #44

Completed fairness protocol

The comparison now has three explicit directly comparable boundaries plus a separate shared-realm capability panel:

  • direct single warmed context;
  • independent warmed contexts on persistent creator threads, using identical semaphore dispatch/evaluation/completion timing;
  • independent cold lifecycles, timing thread/context/source setup through context destruction and join;
  • zig-js shared-realm JavaScript Threads, reported separately because the public JSC embedding API has no shared-object-graph equivalent.

Every parallel mode includes its own one-lane baseline, so scaling never mixes incompatible lifecycle costs.

Acceptance criteria

  • Directly compared single-context rows execute identical workload and invocation bytes with equal work and checksum validation.
  • Add zig-js independent warmed contexts on OS threads, matching JSC independent warmed contexts.
  • Report shared-realm zig-js as a separate capability panel, not a direct JSC ratio.
  • Separate cold thread/context lifecycle from steady-state worker throughput.
  • Calibrate equal job counts so the fastest engine has a sufficiently long timed interval; full rows enforce a 50 ms median floor.
  • Alternate runner order deterministically to reduce thermal/order bias.
  • Report median, min/max, and relative standard deviation from retained raw samples.
  • Define steady-state and GC-state policy for both engines.
  • Record power state when available in addition to compiler/runtime/build/commit metadata.
  • Add standalone harness self-tests for missing groups, duplicate rows, checksum mismatch, too-short samples, and dirty-result publication.
  • Re-run one full matrix only after the revised harness passes quick validation.

Evidence

  • Initial fairness protocol: 82207c2
  • Clean-publication guard: e70eeff
  • Symmetric steady/cold runners, mode-local baselines, power metadata, and harness tests: c9f68e3
  • Quick 88-group integration matrix passed locally on 2026-07-13.
  • Full 616-sample report, raw TSV, stable docs, and README: 2ebda1a
  • Source commit measured by the full report: c9f68e3

Activity

  1. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    First fairness slice landed across 82207c2, e70eeff, and 6fb3b28. The revised full matrix validates 224 samples, enforces a 50 ms floor, alternates runner order, publishes min/max and RSD, and removes the invalid shared-vs-isolated parallel ratio. #45 remains open for symmetric zig-js independent contexts, steady-state parallel timing, stronger GC/power policy, and standalone harness tests.

  2. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    The first fairness slice passed the final combined gate (31 success, 0 failure): https://github.com/zig-utils/zig-js/actions/runs/29280193748. Remaining unchecked criteria stay open.

  3. chrisbbreuer commented on Jul 13, 2026

    @chrisbbreuer
    MemberAuthor

    Completed by c9f68e3 and documented with the accepted 616-sample matrix in 2ebda1a. The remaining performance gaps continue under #44 and workload issues #46-#50.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions