镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Bench: reject competing jobs during exact-parent evidence #617

Closed

Description

@chrisbbreuer

Parents: #461, #492
Related: #614, #615, #616

Problem

tools/exact-parent-regression.ts could collect and write a valid-looking exact-parent artifact while an unrelated Zig build/test or Home test executable overlapped a parent/candidate sample. Process-local instruction counters may remain stable, but wall/cycle/energy attribution and order balance are contaminated.

Two live #615 attempts exposed both races:

  • a Home zig test began during a row and remained alive for the after-snapshot;
  • /private/tmp/home-url-final began and exited entirely inside a row, evading both snapshots while inflating a normally millisecond invocation to roughly ten seconds.

Scope

  • Define the exact evidence competitors explicitly from process-table rows.
  • Reject unrelated zig and maker processes plus known repository test/corpus executables before and after every parent/candidate invocation.
  • Exclude the collector and its own runner ancestry/children without PID guessing.
  • Measure complete-process wall and CPU time; reject CPU occupancy below 60% so a transient competitor cannot hide between snapshots.
  • Persist the full-process wall time, occupancy, and threshold in every raw sample and rendered report.
  • Record no artifact when overlap is detected; partial output paths must not masquerade as complete evidence.
  • Add deterministic fixtures for ancestry exclusion, competitor classification, unrelated process tolerance, before/after failure behavior, occupancy calculation, policy drift, and below-threshold failure.

No-workaround rules

  • No sleeps, retries that hide an overlapped sample, process killing, bare name substring matching, or accepting stable counters as proof of an uncontended host.
  • Do not treat browser/system background activity as a named compiler/test competitor; host-class/RSD/thermal policy continues to govern ordinary noise.
  • Never delete or modify the competing user-owned process.

Acceptance

  • Self-tests prove exact competitor classification and own-runner exclusion.
  • Every measured invocation fails closed on a competitor present before or after it and writes no complete artifact.
  • Complete-process CPU occupancy rejects transient overlap that begins and ends between snapshots.
  • The occupancy observation and minimum threshold are validated and persisted in raw and Markdown artifacts.
  • Existing exact-parent identity, checksum, cleanliness, pairing, and schema gates remain green.
  • Compiler: scan pending TDZ references once #615/Compiler: classify captured loop bindings in one AST pass #616 evidence is collected only after this guard lands and passes.

Activity

  1. added
    performanceMeasured runtime, memory, scaling, or build performance
    benchmarksBenchmark design, evidence, and regression gates
    toolingBuild, test, documentation, and developer tooling
    on Aug 16, 2026
  2. chrisbbreuer commented on Aug 16, 2026

    @chrisbbreuer
    MemberAuthor

    Pushed bcf53f57 (fix(bench): reject competing exact-parent jobs (#617)).

    The collector now checks before and after every parent/candidate invocation. Deterministic fixtures cover collector ancestry, owned runner descendants, sibling Zig/maker processes, Home test executables, zig-js unit/corpus runners, another frontend benchmark, and unrelated-process tolerance. Existing exact-parent identity, checksum, pairing, cleanliness, schema, regression, and thermal/counter tests remain green; dependency audit remains 15 edges / 0 migrations / 60 tools.

    A live fixture then exercised the exact observed race: a Home test began between checks. The collector rejected the row after the invocation with PID/command evidence and produced neither the requested JSON nor Markdown output. No process was killed or modified.

    The issue stays open only until #615/#616 artifacts are successfully collected behind this landed guard.

  3. chrisbbreuer commented on Aug 16, 2026

    @chrisbbreuer
    MemberAuthor

    Follow-up pushed in 16cd30f8 (fix(bench): reject transient exact-parent overlap (#617)).

    The first guard correctly caught persistent jobs, but a second live attempt exposed the remaining interval race: /private/tmp/home-url-final began and exited entirely inside one invocation. The before/after snapshots therefore saw no PID even though a normally millisecond row took roughly ten seconds.

    Every invocation now records its complete /usr/bin/time -l wall, user, and system times, computes process CPU occupancy, and fails below 60%. The raw sample and Markdown report preserve the wall time, occupancy, and threshold; artifact-specific validation rejects missing, below-threshold, over-100%, or policy-drifted quality data before either output is written. Snapshots remain in place for exact PID/command attribution when a competitor persists.

    Validation: exact-parent self-test passes; dependency audit remains 15 classified edges / 0 migrations / 60 tools. This flips 0 Test262 cases. #615/#616 evidence still remains intentionally uncollected while the live Home job is active.

  4. chrisbbreuer commented on Aug 16, 2026

    @chrisbbreuer
    MemberAuthor

    The #617 fail-closed process/occupancy policy is now shared with the focused-build collector in d8fc8c97, linked from #614. Exact-parent’s existing self-test remains green after extracting the common classifier and occupancy calculation. This is a policy-preserving refactor for exact-parent evidence and adds the same protection to V2 build-feedback artifacts; the remaining #617 acceptance item is still the guarded #615/#616 collection.

  5. chrisbbreuer commented on Aug 16, 2026

    @chrisbbreuer
    MemberAuthor

    Related fail-closed runner child #620 is complete in 34f5c8c6: CLI mistakes can no longer silently launch or leave a full Test262 driver tree. The final uncontaminated valid-path smoke passed 38/38 with empty before/after process snapshots.

  6. chrisbbreuer commented on Aug 16, 2026

    @chrisbbreuer
    MemberAuthor

    Live fail-closed validation during #621: a seven-pair exact-parent collection began on a clear host, then a new Home maker/TypeScript conformance/home-url-final cycle appeared between invocations. The collector stopped with competing build/test process detected before benchmark invocation; neither requested JSON nor Markdown output existed afterward, and no process was killed or modified. This is additional real-host evidence for the completed guard behavior. The final #615/#616 successful guarded collections remain the only unchecked acceptance item.

  7. chrisbbreuer commented on Aug 23, 2026

    @chrisbbreuer
    MemberAuthor

    The final acceptance item is now satisfied: a complete successful guarded collection ran end-to-end in 0abebd1a. During the hunt, the collector rejected multiple live attempts fail-closed — competing Home `ts_checker`/`maker` builds and transient `home-url-final` processes between before/after snapshots — writing no artifact for any rejected attempt; the three retained widths (7 pairs each, 42 guarded invocations) all passed pre/post competitor checks with >=60% CPU occupancy. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchmarksBenchmark design, evidence, and regression gatesperformanceMeasured runtime, memory, scaling, or build performancetoolingBuild, test, documentation, and developer tooling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions