Repository navigation
Description
Parent: #478
Split from: #530
Related: #461, #480, #493
Problem
#530 replaces deep linear lookup and predictable transition hashing with collision-free persistent AVL indexes. Its functional four-lane and suppression-free TSan witnesses pass, but a dedicated shared-realm publication diagnostic shows that total transition throughput stops scaling beyond two lanes even when lanes publish beneath independent parent shapes.
The pre-implementation fixture is intentionally split into:
- partitioned base shapes: each lane publishes unique fanout edges beneath its own prepared parent shape in one shared Context;
- anti-specialization variant: different base/key layout with identical logical work and checksum;
- contended control: every lane publishes beneath one parent, preserving the serialized worst case.
Each invocation uses a fresh base-shape generation and reuses one object per lane, deleting back to the base after every unique publication. This removes per-iteration object construction while preserving real transition publication, exact own-key order, deletion, and deterministic checksums.
Scope
- Attribute 1/2/4/8-lane time to Shape transition locks, Context arena allocation, object property locks, GC, and Thread lifecycle using shared attribution.
- Remove the causal independent-parent bottleneck with bounded Context-owned storage and an explicit synchronization design.
- Consider per-thread arena tranches, batched immutable-node allocation, or optimistic persistent-root publication only when exact profiles justify them.
- Preserve same-name convergence, collision-independent exact keys, stable slots, delete/re-add undo, OOM atomicity, moving GC, and Context teardown.
- Keep the same-parent contended row visible; do not describe intentional serialization as scalable.
No-workaround rules
- No GIL, hidden serialization outside the measured lock, prepublishing scored keys, reusing a result cache, reducing logical work at higher lane counts, unbounded per-thread reserves, benchmark-name/source recognition, or excluding the 4/8-lane rows.
- Do not trade exact same-name convergence or OOM rollback for duplicate visible shapes.
Acceptance
- Benchmark source, base/variant/contended identities, jobs, checksums, warmup, and timed boundary are committed before the runtime fix.
- Clean 1/2/4/8-lane order-balanced evidence reports transitions/second, instructions/transition, lock contention/wait, allocation bytes, GC, peak/retained RSS, and dispersion.
- Partitioned total publication throughput is monotonic across 1/2/4/8 lanes and the anti-specialization variant agrees; every row retains identical per-lane work.
- The contended row remains bounded and exact, with wait/serialization attributed rather than hidden.
- Same-name convergence, disjoint fanout, deep delete/re-add, allocation-failure rollback, 32 MiB bounded heap, moving/parallel GC, threadfuzz, and suppression-free TSan pass.
- Shallow classes, mixed application, dense Array, and Shapes: index deep named-property lookup without breaking transitions #530 deep lookup rows do not regress materially; retained memory remains bounded and published.
Security boundary
Property names, fanout, and thread schedules are attacker-controlled. The fix must improve independent publication without reintroducing predictable collision behavior, races, half-published roots, or unbounded thread-local retention.
Activity
bench(shape): add shared transition publication rows (#531)
chrisbbreuer commentedon Aug 11, 2026
Benchmark-first commit landed on
mainasd9d54b5b:bench(shape): add shared transition publication rows (#531).Frozen rows:
representative_shape_transition_fanout: one fresh prepared base shape per lane;_variant: different base/key layout with identical logical work/checksum;_contended: all lanes deliberately share one parent shape.Each lane reuses one object, publishes a unique edge, validates exact own-key order/value, deletes back to the prepared base, and checks absence. For 1,000 publications/lane the exact 1/2/4/8 checksums are 503,500 / 1,507,500 / 5,017,000 / 18,042,000. The 18/18 benchmark structural gates pass.
No runtime fix is included in this commit. Quick diagnostics confirm the partitioned row stops scaling beyond two lanes; accepted measurements and causal attribution will follow from this frozen parent.
bench(shape): isolate transition publication work (#531)
chrisbbreuer commentedon Aug 11, 2026
Benchmark boundary correction is now pushed in
d30790f.The committed row no longer charges
Reflect.ownKeysor membership scans to each transition publication. Every operation still publishes a unique computed edge, validates the observed value, and validates deletion back to the prepared base shape; exhaustive key-order and absence semantics remain in the production unit suite.For
jobs=1000, the frozen exact checksums are now:5015001503500500900018026000A quick parent diagnostic still loses aggregate throughput beyond two lanes (about 333k, 303k, 255k, and 137k publications/s at 1/2/4/8 lanes), so this commit is benchmark isolation only, not a runtime fix. I am tracing the remaining shared-realm environment/property ownership and synchronization costs before changing the engine.
bench(shape): precompute transition publication keys (#531)
chrisbbreuer commentedon Aug 11, 2026
A second benchmark isolation correction is pushed in
6236f4b.Unique property-name strings are now constructed in the existing untimed
__benchmarkPreparephase, one immutable key array per lane. The scored worker boundary remains one unique Shape-edge publication, exact value observation, and delete-to-base validation per operation. This removes integer/string formatting and its GC/backing allocations from the claimed transition-publication work.The benchmark matrix structural tests pass 18/18, and all 20 quick samples across 1/2/4/8 lanes retain the frozen checksums (
501500,1503500,5009000,18026000). No runtime fix is included in this commit.bench(shape): expose transition publication attribution (#531)
chrisbbreuer commentedon Aug 11, 2026
Shape transition attribution is now first-class in
6ced428. Tier-attribution rows record requests, hits, misses, and writer yields alongside the existing Environment/Object/allocation/worker counters.On the isolated 1,000-publication-per-lane row, the candidate records:
That evidence rules out contended per-Shape writer locks in the partitioned row. It instead exposed persistent-AVL path-copy amplification and the surrounding ordinary-property synchronization costs. The benchmark comparison self-tests and 18/18 representative matrix structural tests pass.
perf(shape): publish transition edges append-only (#531)
chrisbbreuer commentedon Aug 11, 2026
The Shape publication implementation is pushed in
2708a80.The persistent AVL remains the deep-shape property lookup index. Transition fanout now uses a separate append-only concurrent skip list:
Validation:
The partitioned JS row still does not meet the issue’s monotonic 1/2/4/8 throughput acceptance because surrounding ordinary-property synchronization remains dominant. I am keeping #531 open and continuing that work; the Shape writer-lock diagnosis itself is now closed by measured zero yields.
chrisbbreuer commentedon Aug 11, 2026
The measured non-Shape bottleneck is split into #532: ordinary named-property synchronization and its private/publication proof. #531 remains the parent end-to-end scaling issue; #532 owns the lock-amortization work needed to satisfy its remaining monotonic 1/2/4/8 acceptance. This split does not relax or remove any #531 gate.
chrisbbreuer commentedon Aug 11, 2026
The Environment portion of the measured synchronization cost is complete in #533 via 6a6b508a: exact invocation Environment locks fell about 90.4% at both 4 and 8 lanes with unchanged checksums, full units/TSan/threadfuzz/test262 gates green. #531 remains open because end-to-end monotonic 1/2/4/8 throughput is still not proven; ordinary named-property synchronization remains tracked in #532.
chrisbbreuer commentedon Aug 11, 2026
#532 made another measured step in 58d73f42: exact property-lock counts fell about 38.5% at both 4 and 8 lanes with unchanged workload checksums and full semantic/concurrency gates green. This is attribution evidence only. #531 stays open because its monotonic 1/2/4/8 throughput acceptance has not been satisfied.
chrisbbreuer commentedon Aug 11, 2026
Post-
58d73f421558d3918de2e37af0e7ebbb6305f4c3clean-main scaling check (ReleaseFast,representative_shape_transition_fanout, shared realm, 1,000 jobs, 7 samples/lane):The checksums are exact and all seven rows were valid at every lane count. The 1→2 lane result improves, but 4 and 8 lanes collapse, so #531 acceptance is not met and the issue remains open.
The same run shows remaining synchronization acquisition counts scale roughly linearly while recorded contention is low: 4 lanes had 64,230 property-lock and 36,299 environment-lock acquisitions; 8 lanes had 128,422 and 72,531 respectively. System CPU also rose from 16.987 ms at 4 lanes to 103.136 ms at 8 lanes. The aggregate environment metric currently mixes root, captured, other reads, writes, and marker synchronization; the next step is to split that attribution before changing another lock path.
These are diagnostic
/tmpsamples, not publication evidence.bench(environment): attribute binding lock ownership (#531)
perf(environment): elide guarded private writes (#535)
chrisbbreuer commentedon Aug 11, 2026
Environment lock ownership attribution landed in
3577d307, and the private-write protocol landed in2d895edb/ #535.Exact 8-lane ownership before #535 accounted for all 72,531 Environment acquisitions: 16,334 reads + 56,197 writes + 0 traces. Of the writes, 56,191 were uncaptured private activations. After #535, actual Environment acquisitions are 16,340 and those 56,191 private writes are recorded as guarded elisions (-77.47% locks), with checksum 18,026,000 preserved.
Seven-sample ReleaseFast
representative_shape_transition_fanout, 1,000 jobs per lane:Compared with the prior clean-main medians, wall time improved by approximately 6.4% / 7.7% / 31.7% / 36.4% at 1/2/4/8 lanes. The optimization is material, but #531 acceptance is not met: throughput still peaks at two lanes and declines at four and eight. Keeping this issue open for the next measured bottleneck.
chrisbbreuer commentedon Aug 12, 2026
#532 advanced the parent end-to-end row in dc919306. Exact 8-lane property-lock acquisitions fell
128,422 -> 80,384(-37.4%) with checksum18,026,000; Environment/backing/element counts stayed16,340 / 33,048 / 24,073.Final-code seven-sample medians are 3,584,292 / 6,225,166 / 14,681,083 / 30,657,959 ns at 1/2/4/8 lanes, or 278,995.1 / 321,276.6 / 272,459.5 / 260,943.7 aggregate edges/s. Relative to the prior exact parent in this issue, elapsed time changes by
+1.35% / -2.74% / -19.75% / -23.88%.This is a material four/eight-lane improvement, but #531 acceptance remains unmet because aggregate throughput still peaks at two lanes. The exact attribution and semantic/concurrency evidence are in the #532 update. I am keeping both issues open; I am also leaving Thread/Worker/job lifecycle changes to the separately owned #482/#502 work rather than folding them into the Shape/Object change.
chrisbbreuer commentedon Aug 12, 2026
Profiling the current representative fanout workload found that a substantial share of scheduler yields is outside shape publication: direct-mapped statement-location cache aliases repeatedly enter the shared debugger registry. That independent bottleneck is now tracked in #541. I’ll add opt-in counters before changing the cache, then re-run the same #531 attribution matrix so subsequent shape conclusions are based on separated costs.
chrisbbreuer commentedon Aug 12, 2026
#541 is complete in
59b34db6. The 8-lane base invocation now performs 814 debugger-registry acquisitions instead of 440,593, with identical checksum, object-lock counts, and shape counts. Seven exact-parent diagnostic pairs measured wall 0.568×, instructions 0.6706×, cycles 0.5872×, and peak RSS 1.0004×; variance is explicitly too high for publication evidence.The shape attribution is now much cleaner: base/variant still publish 8,008 misses with zero shape-lock yields, while the deliberately contended control isolates writer yields. I’m returning to the independent object-synchronization child #532 before drawing another conclusion about the remaining >2-lane scaling ceiling.
chrisbbreuer commentedon Aug 13, 2026
Post-
d437b5fcdiagnostics confirm that #532 removed the measured property-lock contention, but #531's end-to-end monotonic gate is still not met.Seven clean-main samples per lane, ReleaseFast, 1,000 jobs/lane:
1.000x / 1.400x / 1.709x / 1.186x1.000x / 1.465x / 1.842x / 1.211x1.000x / 1.404x / 1.641x / 1.042xAll checksums remained exact. These are noisy diagnostics, not publication evidence.
Exact invocation attribution on the partitioned 8-lane row records 8,016 transition requests / 8,008 misses / 0 Shape yields, 48,312 property-lock acquisitions with only 2 contentions / 2 spins, and zero backing/element contention. The benchmark's cooperative-GC telemetry records zero scored collections at every lane count, ruling out the sampled post-invocation nursery relocation work as the timed ceiling.
I tested the remaining apparent
Shape.lookupStateleaf separately in #543 by carrying proven immutable lookup state through OrdinarySet and named Delete. Seven order-balanced exact-parent pairs across all three controls showed no causal improvement and an eight-lane contended regression, so the patch was reverted and #543 closed as not planned. I am keeping #531 open, but will not make another Shape/Object change from this profile alone.chrisbbreuer commentedon Aug 14, 2026
A current-main 1/2/4/8 diagnostic still scales through four lanes and falls at eight (about 282k / 391k / 487k / 377k edges/s). An amplified sample now identifies GcCellBacking single-cell allocation/free as the dominant collapsed path rather than Shape publication or named-property contention. I split that causal allocation problem into #559; #531 remains the end-to-end parent and none of its acceptance gates are relaxed.
chrisbbreuer commentedon Aug 14, 2026
Correction to the #559 split above: b3d41abe proved that the apparent GcCellBacking hotspot was generated by fixture preparation still running inside
__benchmarkRunShared, plus preparation-triggered GC deferred into the next evaluation. #559 is now closed with that causal correction.Under the corrected boundary, the 8-lane x 100000-job scored invocation preserves checksum
180002600000, performs zero collections, and records only 82 slab acquisitions across 800000 published edges (41 allocation, 33 publication, 8 ownership, no relocation/unpublication/free). The GC allocator is therefore ruled out for this row.Corrected three-sample medians are 253,732,709 / 279,234,834 / 319,724,375 / 907,345,417 ns at 1/2/4/8 lanes, or approximately 394k / 716k / 1.25M / 882k aggregate edges/s. Scaling is substantially cleaner and remains strong through four lanes, but the four-to-eight fall still violates #531 acceptance. This issue again owns that residual end-to-end shape/publication analysis; no allocator sharding will be carried forward from the invalid profile.
test(c-api): bound randomized VM OOM sampling (#531)
chrisbbreuer commentedon Aug 15, 2026
An unlanded Object key-order candidate establishes a causal path to monotonic 1/2/4/8 partitioned throughput, but it is not ready to land because the required mid-GC lifecycle fuzz gate exposes a differential failure tracked on #482.
Candidate design: creation-order entries borrow immutable Shape operation-name slices owned by the Context arena; accessor/native names retain owned copies. Object and GC teardown free only owned entries. This removes one duplicate/free pair per ordinary add/delete without adding a cache, reserve, or alternate semantic path.
Exact
8034cdfaparent and candidate ReleaseFast binaries were pinned by SHA-256 (0e00bb5f…parent,162bc678…candidate). Seven alternating fresh-process pairs, 100,000 exact publications per lane:Every sample preserved the frozen checksums
5000150000,15000350000,50000900000, and180002600000. Three-sample candidate diagnostics also remained monotonic for the anti-specialization variant (about 376k / 722k / 1.31M / 1.87M edges/s). The deliberately same-parent control remained exact and visibly serialized (about 387k / 676k / 694k / 504k edges/s).Exact eight-lane attribution preserved checksum
180002600000, zero scored collections, 82 slab acquisitions, 800,008 transition requests / 800,000 misses / 0 Shape yields, and the same Object lock counts. Delegated Object-backing allocations/releases fell from1,725,043 / 1,725,016to125,035 / 125,008: 1.6 million fewer of each, or about 92.7%.Green candidate gates:
Blocking differential:
8034cdfa: full mid-GC seed-10 replay 45/45; focusedmidgcmicrotask 1 103/3;TypeError: Thread constructor requires a callable argument; the long mid-GC profile reported the same failure at seeds 10 and 46 before being stopped;The candidate is stashed, not committed or pushed. #531 remains open: the performance result is diagnostic, the full efficiency/RSS/non-regression publication matrix has not been accepted, and the no-workaround lifecycle gate must be green before this optimization can land.
chrisbbreuer commentedon Aug 25, 2026
#764 removes deterministic adversarial shard/bucket placement from shared string interning in 288b4893, while preserving the 16-way lock topology. This is a bounded security/contention hardening slice only; it does not satisfy #531's independent shape-transition scaling acceptance.
chrisbbreuer commentedon Aug 26, 2026
Acceptance metadata reconciliation only. The benchmark-first boundary is proven by
d9d54b5b,d30790f2, and6236f4be: all three identities, equal work, warmup/preparation boundary, and frozen 1/2/4/8 checksums are committed. The same-parent control is also exact and visibly serialized in every retained diagnostic, including the candidate matrix recorded above, so that acceptance row is proven. I checked only those two boxes. Clean publication-grade 1/2/4/8 evidence, landed monotonic partitioned scaling, the complete current-code lifecycle matrix, and published control/RSS evidence remain open; the causal key-order candidate is still correctly gated by #482.perf(object): borrow immutable shape order keys (#531, #532)
chrisbbreuer commentedon Sep 1, 2026
The previously blocked borrowed-key implementation is now landed on
mainas88cde689:perf(object): borrow immutable shape order keys (#531, #532).Design and safety boundary:
The historical #482 seed-10 blocker is resolved on the current candidate: full mid-GC is 45/45 normally and under TSan, plus three focused seed-10 reproductions. Full units are 2,236 passed / 1 expected skip / 0 failed / 0 leaked. Default, amplified, broad, verify, and lifecycle threadfuzz profiles pass; the focused named-delete/OOM/moving-GC TSan group is clean. Exact Test262 remains Object 3,411 -> 3,411, Array 3,081 -> 3,081, Proxy 311 -> 311, with zero failures and zero flips.
Current exact-parent diagnostics (100,000 publications/lane, frozen checksums) show the expected causal separation:
At eight lanes, exact invocation attribution for both independent fixtures changes backing allocation/release calls from 1,725,044 / 1,725,008 to 125,036 / 125,000 and removes 41,422,408 bytes of matched allocation/free traffic. Net live backing-byte change, transition work, property-lock acquisitions, and scored GC collections remain unchanged; the serialized control is wall-neutral at scale. Shallow classes and the mixed application row are neutral-to-better; the dense-Array control is +0.92% wall with neutral-to-better instructions/cycles/energy/RSS.
These are clean diagnostics, not publication evidence. #531 remains open until the seven-pair quiet-reference 1/2/4/8 base/variant/control artifacts, exact attribution/RSS report, and retained deep-lookup control are checked in. No public performance claim is made from these
/tmpresults.chrisbbreuer commentedon Sep 1, 2026
The standard publication path can now carry this issue exact Shape evidence: #844 landed as
a6feac57, with supporting fail-closed runtime-profiler repairs #845 (928d5797) and #846 (61ec0d95). A complete V33 quick collection accepted 510/510 snapshots and rendered request/hit/miss/writer-yield tables without relaxing allocation, synchronization, GC, RSS, tier, or checksum gates. This issue remains open for the quiet-reference publication artifact and retained deep-lookup row.