Repository navigation
Closed
Labels
benchmarksBenchmark design, evidence, and regression gatesBenchmark design, evidence, and regression gatesperformanceMeasured runtime, memory, scaling, or build performanceMeasured runtime, memory, scaling, or build performancetoolingBuild, test, documentation, and developer toolingBuild, test, documentation, and developer tooling
Description
Parents: #461, #492
Related: #614, #615, #616
Problem
tools/exact-parent-regression.ts could collect and write a valid-looking exact-parent artifact while an unrelated Zig build/test or Home test executable overlapped a parent/candidate sample. Process-local instruction counters may remain stable, but wall/cycle/energy attribution and order balance are contaminated.
Two live #615 attempts exposed both races:
- a Home
zig testbegan during a row and remained alive for the after-snapshot; /private/tmp/home-url-finalbegan and exited entirely inside a row, evading both snapshots while inflating a normally millisecond invocation to roughly ten seconds.
Scope
- Define the exact evidence competitors explicitly from process-table rows.
- Reject unrelated
zigandmakerprocesses plus known repository test/corpus executables before and after every parent/candidate invocation. - Exclude the collector and its own runner ancestry/children without PID guessing.
- Measure complete-process wall and CPU time; reject CPU occupancy below 60% so a transient competitor cannot hide between snapshots.
- Persist the full-process wall time, occupancy, and threshold in every raw sample and rendered report.
- Record no artifact when overlap is detected; partial output paths must not masquerade as complete evidence.
- Add deterministic fixtures for ancestry exclusion, competitor classification, unrelated process tolerance, before/after failure behavior, occupancy calculation, policy drift, and below-threshold failure.
No-workaround rules
- No sleeps, retries that hide an overlapped sample, process killing, bare name substring matching, or accepting stable counters as proof of an uncontended host.
- Do not treat browser/system background activity as a named compiler/test competitor; host-class/RSD/thermal policy continues to govern ordinary noise.
- Never delete or modify the competing user-owned process.
Acceptance
- Self-tests prove exact competitor classification and own-runner exclusion.
- Every measured invocation fails closed on a competitor present before or after it and writes no complete artifact.
- Complete-process CPU occupancy rejects transient overlap that begins and ends between snapshots.
- The occupancy observation and minimum threshold are validated and persisted in raw and Markdown artifacts.
- Existing exact-parent identity, checksum, cleanliness, pairing, and schema gates remain green.
- Compiler: scan pending TDZ references once #615/Compiler: classify captured loop bindings in one AST pass #616 evidence is collected only after this guard lands and passes.
Metadata
Metadata
Assignees
Labels
benchmarksBenchmark design, evidence, and regression gatesBenchmark design, evidence, and regression gatesperformanceMeasured runtime, memory, scaling, or build performanceMeasured runtime, memory, scaling, or build performancetoolingBuild, test, documentation, and developer toolingBuild, test, documentation, and developer tooling
Activity
fix(bench): reject competing exact-parent jobs (#617)
chrisbbreuer commentedon Aug 16, 2026
Pushed bcf53f57 (
fix(bench): reject competing exact-parent jobs (#617)).The collector now checks before and after every parent/candidate invocation. Deterministic fixtures cover collector ancestry, owned runner descendants, sibling Zig/maker processes, Home test executables, zig-js unit/corpus runners, another frontend benchmark, and unrelated-process tolerance. Existing exact-parent identity, checksum, pairing, cleanliness, schema, regression, and thermal/counter tests remain green; dependency audit remains 15 edges / 0 migrations / 60 tools.
A live fixture then exercised the exact observed race: a Home test began between checks. The collector rejected the row after the invocation with PID/command evidence and produced neither the requested JSON nor Markdown output. No process was killed or modified.
The issue stays open only until #615/#616 artifacts are successfully collected behind this landed guard.
fix(bench): reject transient exact-parent overlap (#617)
chrisbbreuer commentedon Aug 16, 2026
Follow-up pushed in 16cd30f8 (
fix(bench): reject transient exact-parent overlap (#617)).The first guard correctly caught persistent jobs, but a second live attempt exposed the remaining interval race:
/private/tmp/home-url-finalbegan and exited entirely inside one invocation. The before/after snapshots therefore saw no PID even though a normally millisecond row took roughly ten seconds.Every invocation now records its complete
/usr/bin/time -lwall, user, and system times, computes process CPU occupancy, and fails below 60%. The raw sample and Markdown report preserve the wall time, occupancy, and threshold; artifact-specific validation rejects missing, below-threshold, over-100%, or policy-drifted quality data before either output is written. Snapshots remain in place for exact PID/command attribution when a competitor persists.Validation: exact-parent self-test passes; dependency audit remains 15 classified edges / 0 migrations / 60 tools. This flips 0 Test262 cases. #615/#616 evidence still remains intentionally uncollected while the live Home job is active.
chrisbbreuer commentedon Aug 16, 2026
The #617 fail-closed process/occupancy policy is now shared with the focused-build collector in d8fc8c97, linked from #614. Exact-parent’s existing self-test remains green after extracting the common classifier and occupancy calculation. This is a policy-preserving refactor for exact-parent evidence and adds the same protection to V2 build-feedback artifacts; the remaining #617 acceptance item is still the guarded #615/#616 collection.
chrisbbreuer commentedon Aug 16, 2026
Related fail-closed runner child #620 is complete in 34f5c8c6: CLI mistakes can no longer silently launch or leave a full Test262 driver tree. The final uncontaminated valid-path smoke passed 38/38 with empty before/after process snapshots.
chrisbbreuer commentedon Aug 16, 2026
Live fail-closed validation during #621: a seven-pair exact-parent collection began on a clear host, then a new Home
maker/TypeScript conformance/home-url-finalcycle appeared between invocations. The collector stopped withcompeting build/test process detected before benchmark invocation; neither requested JSON nor Markdown output existed afterward, and no process was killed or modified. This is additional real-host evidence for the completed guard behavior. The final #615/#616 successful guarded collections remain the only unchecked acceptance item.chrisbbreuer commentedon Aug 23, 2026
The final acceptance item is now satisfied: a complete successful guarded collection ran end-to-end in 0abebd1a. During the hunt, the collector rejected multiple live attempts fail-closed — competing Home `ts_checker`/`maker` builds and transient `home-url-final` processes between before/after snapshots — writing no artifact for any rejected attempt; the three retained widths (7 pairs each, 42 guarded invocations) all passed pre/post competitor checks with >=60% CPU occupancy. Closing.