镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

CI: qualify TSan unit shard completion under the existing budget #1129

Open
0 of 1 issue completed
Open

Description

@chrisbbreuer

The exact-source CI run for f7025b0 completed57 jobs successfully, skipped the two conditional nightly jobs, and cancelled TSan unit shard1 at the existing60-minute job boundary.

No later test-start line survived in the downloaded log. The runner currently writes the start prefix without a newline, so this evidence does not establish which subsequent test was unfinished or whether the cost was evaluation, allocation, GC, teardown or a blocking wait. Do not attribute the failure to a particular test from source order alone. There is no passing full-shard result, race report or assertion result after the last complete record.

The original complete log is retained under native-growth-audit/managed-frames-integration/midgc-finalization-progress/hosted-tsan-unit1-f702.log. The independently qualified focused canonical-array TSan run does not replace this whole-shard gate. The separate fuzzer-only corrections for #1125/#1126 pass900 Debug,900 ReleaseSafe and90 unsuppressed TSan programs and do not change production runtime source.

Acceptance:

  • Emit a complete-line start record before each selected unit test without breaking existing completion/timing/discovery records or changing shard membership.
  • Identify and retain the unfinished test and exact compiled binary/source configuration.
  • Reproduce/profile the owning cost or wait, comparing a clean source control where appropriate.
  • Fix the owning engine or test-infrastructure defect while retaining all cases, original workloads/pressure, sanitizer instrumentation and existing guards. No deadline increase, skipped modes, reduced iteration counts, disabled checks or qualification retry-until-green.
  • Pass all four complete TSan unit shards and final exact-source hosted integration.

Blocks final integration in #1118. Related #1127; no causal edge to its array-root fix is asserted.

Activity

  1. chrisbbreuer commented on Nov 12, 2026

    @chrisbbreuer
    MemberAuthor

    A second unchanged integration run retains the same bounded evidence: on source 91d1f6e, TSan unit shard 1/4 started at 10:05:59 UTC and was cancelled at approximately its configured 60-minute limit. The last completed record is case77/721, lexical slot lifetimes, at10:12:17.027 UTC (17377ms). No complete start record for the following case survived. The log does not identify that pending case or establish its cause.

    Witness: https://github.com/zig-utils/zig-js/actions/runs/43195082134/job/131103985036

    A source-ready progress correction (023f904fe4ecdc2808b76fb6649f3ff64a2c90e1, not yet pushed) emits a newline-terminated selected-test start record before setup while retaining discovery, completion, timing, selection and failure guards. The actual custom-runner interruption fixture exits43 and retains that start record; unchanged discovery/completion and the maintained timing-history self-test pass. Both independent review axes cleared the exact patch. This is observability evidence, not a fix for the delay or a completed TSan unit qualification.

  2. chrisbbreuer commented on Nov 12, 2026

    @chrisbbreuer
    MemberAuthor

    The newline start-record change is delivered on main as fd2c7701c8c322dc7004a8ea3544b7cfe90a876f, after independent protocol reviews and the actual Home-run self-test. It preserves case selection, completion records and timing behavior.

    The first hosted shard using it now directly identifies the pending case:

    2026-11-12T12:29:03.0111617Z 77/721 [1/4] context.test.lexical slot lifetimes: abrupt exits and dynamic captures survive compaction...OK (17729 ms)
    2026-11-12T12:29:03.0112900Z UNIT_TEST_STARTED 78/721 1/4 context.test.captured frames: mapped canonical strings survive heap recovery
    

    A subsequent workflow-only main push cancelled this run at 12:37:56Z. It did not reach the configured 60-minute deadline, so this observation identifies its actual unfinished test but does not establish that test as the cause of every earlier timeout. Replacement source CI is 43219641046.

    The original 5,001 callbacks in each execution mode, 64 MiB heap limit and exact canonical-result checks remain unchanged. The unpublished JSON scratch candidate showed no elapsed-time benefit in the matched standalone ReleaseSafe+TSan comparison and is not being delivered. This issue remains open.

  3. chrisbbreuer commented on Nov 12, 2026

    @chrisbbreuer
    MemberAuthor

    The complete-line marker identifies the unfinished case in the terminal 50913c8 run: context.test.captured frames: mapped canonical strings survive heap recovery, case 78/721, shard 1/4. It starts at 14:11:14.930 UTC and the original job reaches its 60-minute boundary at 15:05:20 UTC. The log does not establish whether the owning cost is evaluation, allocation, collection or a wait.

    89e9be1 adds a separate manual diagnostic with engine source pinned to b42c2c8. The full original Debug+TSan shard, compiler, fixtures, pressure, checks and deadline remain unchanged. It records native executable identity and bounded process/thread samples, attempts three bounded all-thread backtraces, uploads early evidence, then awaits the original command status. Main CI is unchanged.

    Harmless-child tooling checks and actionlint pass; independent source reviews are clear. Linux debugger availability and target attribution are pending in the dispatched diagnostic. This is diagnostic work, not passing full-shard qualification. No test selection, timing budget or sanitizer check has been weakened.

  4. chrisbbreuer commented on Nov 12, 2026

    @chrisbbreuer
    MemberAuthor

    The manual run retained its early artifact while the unchanged full shard continues. Case 78 is bound to native PID3137, executable SHA256 b7c42dfacd72da4915be593220a6718f18a03b7d2e0403a19caa505412e3d875, engine b42c2c8, original fixture/compiler/command. The observed compiler invocations include -fsanitize-thread and -fsanitize=thread; the separate sorted nm probe times out at its five-second budget without symbol witnesses.

    The 24 proc samples cover 109.877 seconds between first and last records. The main thread consumes108.64 CPU seconds, remains runnable, and the only other observed thread is sleeping. RSS changes from735,461,376 to760,639,488 bytes. These observations show CPU activity while the case remains pending; they do not locate the owning cost or establish a retained leak.

    GDB is absent on this runner, so all three backtrace records explicitly report unavailable and no native stack was captured. The original early ZIP is retained with SHA256 eb282a478f7a0b977f5a78cbdd98871a1069c7ea1ea0b4fe176dab9adb571929. Debugger setup is being fixed in the separate diagnostic workflow, including an actual attach/detach preflight before starting another native workload. The engine, shard, callbacks, pressure, sanitizer and original limits remain unchanged. This remains diagnostic evidence, not full-shard qualification.

  5. chrisbbreuer commented on Nov 12, 2026

    @chrisbbreuer
    MemberAuthor

    The corrected manual diagnostic retained three complete all-thread GDB captures, each successful, uncapped and under1.7seconds within the unchanged five-second helper budget. GNU GDB17.1 attaches through the preflight-tested noninteractive sudo path; no kernel/ptrace/sanitizer setting changed. Unsorted nm completes in0.667seconds with all three TSan witnesses.

    Engine source remains b42c2c8, native test PID3185, actual executable SHA256 125b378a26ede1540cef08f8575ae1592eacdb38071d0c99a73f09dd3b1e8f32. The observed main-thread stacks pass through: (1) VM/interpreter/arrayMap under sanitizer memory checks, (2) SafeAllocator stack-trace capture while Object.keys allocates object backing, and (3) SafeAllocator free/checksum during GC sweep, allocation recovery and managed string canonicalization. The other observed thread is the sleeping TSan background worker.

    These are three discrete observations of active execution/allocation/collection, not a dominant-cost measurement or an owning defect diagnosis. The24 proc samples span114.269seconds between first/last records with108.41 CPU seconds. Debugger pauses mean this run cannot qualify elapsed performance. The full unchanged shard continues toward its own outcome; no passing full-shard claim is made.

    Original early artifact13961379192 is retained with ZIP SHA256 f723883700fb4b11e46945d317a603d365305c00c52974ab07717bfc42fef4a1. The earlier debugger-less manual run was explicitly cancelled after preserving its partial evidence; it is not a native deadline result. Required main CI was not cancelled.

    Next diagnosis will measure owning allocation/copy counts against a clean source control before changing source. The existing same-allocator ASCII ownership fast path is already present; it must not be proposed as a new fix. Original allocator checks, callbacks, heap/ballast, sanitizer and all guards remain required for final qualification.

  6. chrisbbreuer commented on Nov 12, 2026

    @chrisbbreuer
    MemberAuthor

    The corrected diagnostic run 43255082770 naturally reached the unchanged 60-minute job deadline. Case 78 remained pending; this is not a pass. It was not manually cancelled. The original terminal log is retained with SHA-256 e4db6d6bf628cbea2c0c26addd4330177471628a39b31cc1684d0ff1bcffae3e.

    The retained early artifact has three complete all-thread GDB captures and 24 process samples. They show active VM/arrayMap execution, SafeAllocator allocation stack capture, and GC/string recovery at three instants. They do not establish a dominant cost, leak, or fix. No allocator, sanitizer, heap, ballast, call count or deadline was changed.

    The next diagnostic uses the unchanged production engine and exact canonical fixture in a small external consumer, with the same SafeAllocator canary/write-after-free settings, 64 MiB heap and all 5,001 calls in both JIT modes. It compares existing allocation counters enabled versus a clean control; the counters can establish volume, not elapsed performance or full shard qualification. Its native build passed under the original local 3,072 MiB / 900-second guards (observed peak 2,170,560 KiB), with actual TSan flags verified on all five modules and linked init/read/write witnesses. Execution remains pending resource coordination. No engine optimization has been made.

    All four unchanged TSan unit shards and hosted integration remain required. Detailed terminal evidence, diagnostic source, build/instrumentation receipts and guard records are retained under Code/Verification/settlers-retained-replay/native-growth-audit/managed-frames-integration/tsan-unit-completion/.

  7. chrisbbreuer commented on Nov 12, 2026

    @chrisbbreuer
    MemberAuthor

    The allocation-volume diagnostic is now complete, with no engine source changes. The same verified Safe+TSan production-module executable ran the unchanged fixture for all 5,001 callbacks under both enable_jit settings in separate control and profiled processes. Both passed every canonical result, collection and SafeAllocator leak assertion within the original local 3,072 MiB / 900-second guards.

    The profiled process recorded 30,684,514 Context backing-allocation operations and 25,550,487 logical GC-cell allocations across the two contexts. After subtracting initial load, backing traffic is about 3,066 operations per callback in each setting. These are distinct accounting domains; reused GC cells are not fresh parent allocations, and release counters also include successful shrinks. Each context completed 162 collections.

    This establishes allocation volume, not its dominant call site or elapsed cost. Profiling adds accounting overhead, and this production executable omits builtin.is_test counters, so its durations are not full Debug TSan unit qualification. The next causal measurement will attribute successful parent allocator operations to their original caller PCs while retaining the same underlying SafeAllocator and workload. No optimization, raised guard, suppressed check or green retry is proposed.

    Verified binary SHA-256 8c534d7a1ba7f21751e5c65e64947a4f0b39c583a36d5038093f18d51e937196; complete source, normal/control/profile logs and resource receipts: Code/Verification/settlers-retained-replay/native-growth-audit/managed-frames-integration/tsan-unit-completion/allocation-attribution/completed-allocation-volume-receipt.json. #1131 separately tracks the diagnostic helper's repository-runtime audit failure. #1129 remains open for all original TSan shard/integration requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions