镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

CI: keep benchmark harness within the required-check budget #765

Closed

Description

@chrisbbreuer

Parents: #460, #493
Related: #734, #750, #751, #763, #764

Problem

The required benchmark-comparison-harness CI leg runs zig build benchmark-comparison-test under the gate matrix's 60-minute budget. On CI run 32890815195, 52 other jobs completed successfully and the nightly-only job was intentionally skipped, but this leg was automatically cancelled at exactly 60 minutes. Its last visible completion was the instrumentation-overhead self-test at 20:00:34 UTC; GitHub cancelled the still-running operation at 20:51:48 UTC. The preceding run was also cancelled while this same leg was active.

This prevents a truthful full-integration result even when every engine, sanitizer, corpus, docs, ABI, and threading gate has completed successfully.

Scope

  • Attribute the silent tail to the exact benchmark-comparison-test dependency, build, or child process and distinguish useful cold work from a hang.
  • Preserve a machine-checkable inventory of every benchmark harness dependency and its terminal result.
  • Restructure or bound the expensive work so cold Linux CI completes within the existing required-check budget with useful margin.
  • Preserve pinned Home execution, all fail-closed benchmark/evidence validators, independent-suite adapter coverage, and the rule that this harness validates contracts without running scored benchmarks.
  • Keep local and CI entry points equivalent and make a future timeout name the unfinished subcheck directly.

No-workaround rules

  • No timeout increase as the sole fix, removed/skipped validator, continue-on-error, hidden background process, warm-cache dependency, reduced fixture, swallowed child failure, or replacement of execution with source-text matching.
  • No stack, queue, job, Promise, Worker, or Map/Set iterator implementation changes.

Acceptance

Activity

  1. chrisbbreuer commented on Aug 25, 2026

    @chrisbbreuer
    MemberAuthor

    Causal implementation progress:

    • The silent tail was redundant Home TypeScript compilation: benchmark-comparison-test launched 18 overlapping Home self-test module graphs as separate system commands. A focused algorithmic-growth.ts --self-test sample was still inside Home's TypeScript checker before JavaScript execution, matching the prior CI job's long no-output tail.
    • The candidate build graph now compiles one benchmark-harness-self-tests.ts module graph, then runs the exact same 18 exported self-tests sequentially with named START, PASS, and FAIL records. Failures are accumulated so one failed validator does not hide later terminal results.
    • Native Zig independent-suite audits/adapters, system-JSC boundary coverage, and VM quickening checks remain explicit benchmark-comparison-test dependencies.
    • The dependency inventory now records 40 Home /usr/bin/env command nodes instead of 56, 69 TypeScript tools, 63 classified repository tools, and every changed caller edge. zig build dependency-audit passes.
    • A cold local direct run completed all 18/18 Home validators in 1,018.89 s (16m 58.89s), with 992.37 s user CPU, 25.45 s system CPU, and 94,240,768-byte maximum RSS. Runtime output attributed every subcheck; all passed, including the representative matrix's 39/39 assertions.

    This is not closure evidence yet. The full zig build benchmark-comparison-test graph and a cold Linux main CI leg under the existing 60-minute limit still need to complete. I am holding the local full graph while unrelated Home compiler/corpus work is active so its result is not distorted.

  2. chrisbbreuer commented on Aug 25, 2026

    @chrisbbreuer
    MemberAuthor

    Landed as 418687fd. The complete local zig build benchmark-comparison-test graph passed 15/15 steps and 4/4 native tests in 1,125.70 seconds (18m 45.70s): the named Home inventory passed 18/18, the ReleaseFast zig-js adapter compiled in parallel, both zig-js and system-JSC adapter self-tests passed, the independent-suite audit passed, and both VM quickening checks passed. This flips 0 Test262 cases because the change is CI/tooling-only. The cold Linux main run remains the closure gate.

  3. chrisbbreuer commented on Aug 25, 2026

    @chrisbbreuer
    MemberAuthor

    Main CI run 32905159766 completed with 52 required jobs green and the nightly-only job skipped, but benchmark-comparison-harness was still auto-cancelled at the fixed 60-minute limit. The new attribution changes the diagnosis: the Home module graph compiled and began execution in 7m46s (command start 22:16:23 UTC; first START at 22:24:09), and nine validators passed through exact-parent-regression by 22:26:46. algorithmic-growth then remained active for 47m44s until cancellation at 23:14:31. Therefore the compile consolidation is retained as useful progress, but it is insufficient for closure. The next fix targets the real zig-js-backed runtime hotspot inside the complete algorithmic-growth self-test; no timeout increase, skipped/reduced validator, engine switch, or swallowed failure.

  4. chrisbbreuer commented on Aug 26, 2026

    @chrisbbreuer
    MemberAuthor

    Two linked commits are now on main:

    • Engine root fix: 00bfb3e3 (Strings: make known-ASCII UTF-16 indexing constant time #766, now closed). Known-ASCII UTF-16 length and indexed reads are constant-time; non-ASCII WTF-8 semantics remain on the exact iterator path.
    • Harness attribution: 1ab4ba5c. Algorithmic-growth now names every fixture construction/hash and every accepted/zero-allocation/fail-closed phase without changing a fixture or validator.

    Causal evidence: the pre-fix exact pinned reproduction did not finish its first artifact hash in 368.86 s, and main CI previously spent 47m44s in algorithmic-growth before cancellation. The complete unchanged pinned zig-js-backed self-test now passes all phases in 333.72 s total. The focused 16,384-unit sequential String scan improved from 11.079 s at the index-only intermediate state to 268 ms after the repeated length scan was removed. Focused normal and TSan tests pass 5/5 each; Test262 String remains 1,223/1,223 (0 flips); dependency audit is green with 16 classified edges, 0 migration targets, and 63 classified tools.

    The decisive cold Linux closure run is 32923883777. #765 remains open until every required job, especially benchmark-comparison-harness, finishes green under the existing 60-minute budget.

  5. chrisbbreuer commented on Aug 26, 2026

    @chrisbbreuer
    MemberAuthor

    Completed by 418687fd, 00bfb3e3, and 1ab4ba5c.

    The decisive cold Linux main CI run 32923883777 completed successfully: 53 required jobs passed, the one nightly-only job was skipped as configured, and 0 jobs failed or were cancelled. The benchmark-comparison-harness job ran from 02:58:38 to 03:16:55 UTC (18m17s), leaving 41m43s of margin inside the unchanged 60-minute budget. Its inventory reported 18/18 validators passed. The formerly timing-out algorithmic-growth validator ran from 03:12:23.978 to 03:16:48.891 (4m24.913s) and named every fixture hash, accepted report, zero-allocation replay, and fail-closed mutation as PASS.

    No timeout was raised, no fixture or validator was reduced/skipped, and no failure was swallowed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions