镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Frontend: optimize lexing, parsing, bytecode, and module compilation #496

Open

Description

@chrisbbreuer

Parent: #459
Depends on: #460, #461, #465, #479, #481

Goal

Optimize lexing, parsing, AST/bytecode generation, source evaluation, and module compilation for cold and dynamic code without hiding compilation outside timed boundaries.

Scope

  • Add source-size/syntax-mix benchmarks for scripts, modules, functions/constructors, direct eval, dynamic import, JSON-like literals, classes/private fields, async/generators, Unicode identifiers/strings/templates, regex literals, and invalid syntax diagnostics.
  • Attribute lex/token storage, AST allocation/layout, scope analysis, bytecode emission, constant/name tables, source maps/locations, debugger registration, optimizer metadata, and allocator/GC pressure.
  • Reduce passes/allocations and improve locality; consider direct bytecode emission or compact transient IR only with exact diagnostics, early errors, source ranges, and tree-walker availability.
  • Reuse immutable parse/compile artifacts only with exact source/options/realm/dynamic-scope keys and bounded lifecycle.
  • Optimize streaming/module compilation incrementally while preserving failure atomicity and exact final bytes.

No-workaround rules

  • No precompiled benchmark source, omitted source locations/debug metadata, delayed syntax error, reduced Unicode/Annex B/module behavior, hash-only cache collision risk, or third-party parser/compiler.

Acceptance

Foundation

Current pipeline: src/lexer.zig → src/parser.zig/ast.zig → src/compiler.zig/bytecode.zig; module streaming work under #406–#410.

Activity

  1. added
    enhancementNew feature or request
    runtimeJavaScript runtime and builtin implementation
    performanceMeasured runtime, memory, scaling, or build performance
    on Aug 2, 2026
  2. chrisbbreuer commented on Aug 9, 2026

    @chrisbbreuer
    MemberAuthor

    The frontend optimization work now has a fast semantic/rejection loop from tools(test): add focused frontend suite (#494).

    The production-module artifact currently covers precedence, ASI, nested destructuring, templates, private fields, optional chaining/nullish coalescing, regexp literals, and five invalid-source diagnostics. All 12 probes pass in Debug and TSan. This is test-loop infrastructure for the optimization work; it does not satisfy the benchmark or exact-parent acceptance criteria, so #496 remains open.

  3. chrisbbreuer commented on Aug 10, 2026

    @chrisbbreuer
    MemberAuthor

    Closed the strict-parameter frontend complexity slice as #520. Parser duplicate validation now uses an invocation-local exact hash index, and Context/eval/module parsing supplies freeable scratch backing so repeated parses do not retain bucket capacity. At 4,096 unique strict parameters, exact-parent full eval parse-and-compile improves from 12.288 ms to 1.554 ms and from 286.8M to 21.0M instructions. Implementation: a5bb209a and e11bec9c. Evidence: 39c617e4.

  4. chrisbbreuer commented on Aug 12, 2026

    @chrisbbreuer
    MemberAuthor

    #536 is complete on main: 4b5a705b added benchmark-first 1K/2K/4K ordinary-string rows plus an escaped control, and 23df967d made unescaped lexer tokens source-backed while lazily decoding escaped spans.

    The fail-on-first-allocation witness proves the zero-copy path. Across exact-parent diagnostic 1K/2K/4K rows, retired instructions are 0.730x/0.718x/0.720x and median peak RSS is 0.804x/0.676x/0.665x; escaped decoding remains exact. Full units are 1,594 pass / 1 expected skip / 0 failures or leaks, focused TSan passes, and the affected test262 comparison remains 758/758. #496 remains open for the other frontend surfaces and its umbrella acceptance gates.

  5. chrisbbreuer commented on Aug 12, 2026

    @chrisbbreuer
    MemberAuthor

    #537 is complete on main: 09b525bc added benchmark-first 1K/2K/4K ordinary-template rows plus escaped/tagged controls, and 718cb122 made plain cooked quasis source-backed while lazily batch-decoding escaped spans.

    Ordinary exact-parent diagnostic instruction ratios are 0.816x/0.803x/0.800x with median peak-RSS ratios 0.952x/0.905x/0.871x. Escaped/tagged controls retain exact checksums. Source/normalized/decoded pointer-lifetime and OOM witnesses pass; full units are 1,596 pass / 1 expected skip / 0 failures or leaks; template test262 remains 84/84. #496 remains open for its broader frontend gates.

  6. chrisbbreuer commented on Aug 12, 2026

    @chrisbbreuer
    MemberAuthor

    #538 is complete on main: 85dfd44a added benchmark-first 1K/2K/4K escaped-identifier rows with exact decoded-name validation, and 81bef094 removed the redundant raw-name rescan and per-byte append path.

    Escaped exact-parent diagnostic instruction ratios are 0.968x/0.966x/0.966x; the ordinary 4K control is 0.989x. Pointer/allocation, private-name, OOM, forced execution, TSan, full-unit, and 350/350 identifier/reserved-word test262 coverage pass. #496 remains open for its broader frontend gates.

  7. chrisbbreuer commented on Aug 12, 2026

    @chrisbbreuer
    MemberAuthor

    Frontend child #540 is complete in f888b717: private-name duplicate validation now uses a deterministic exact-byte O(N log N) grouping pass over scratch-backed indexes rather than a quadratic nested scan.

    Seven-pair exact-parent medians at 1K/2K/4K private fields fell from 0.874/3.000/13.033 ms to 0.108/0.219/0.450 ms; retired-instruction ratios were 0.163x/0.089x/0.036x. Peak RSS stayed within 0.6%. The affected 2,962 test262 class-element cases and full 1,600-test unit inventory had 0 regressions.

    This unblocks the source-slice allocation work in #539, which is now continuing against f888b717 as its exact parent.

  8. chrisbbreuer commented on Aug 12, 2026

    @chrisbbreuer
    MemberAuthor

    Frontend child #539 is complete in 8cd6034f: unescaped private-name tokens now borrow their complete source slice; escaped names retain owned decoding and OOM semantics.

    Stable exact-parent retired-instruction ratios at 1K/2K/4K plain fields were 0.907x/0.883x/0.870x, with peak-footprint ratios 0.868x/0.811x/0.794x. The escaped 4K control remained 1.0003x instructions with identical median memory. Pointer, zero-memory, decoded-byte, offset, test262, TSan, focused frontend, private runtime, and full unit gates all passed with 0 regressions.

    Together with #540, this leaves private-heavy class parsing bounded and removes proportional arena copies for ordinary private names.

  9. chrisbbreuer commented on Aug 13, 2026

    @chrisbbreuer
    MemberAuthor

    Child #544 is complete and closed.

    4fdff17b replaces incremental numeric-separator normalization with one exact allocation while preserving a borrowed zero-allocation path for plain literals. c7e4f7f7 publishes the exact-parent report and raw evidence.

    Across separated 1K/2K/4K parser workloads, the candidate removes exactly one logical allocation/resize request per literal, reduces cumulative parser allocation bytes by 11.6–14.2%, and reduces stable retired instructions by 5.4–5.6%. The stable 4K wall row is 0.887x parent. The unseparated 4K control preserves allocation requests and bytes exactly, with 1.005x wall and 0.9944x peak RSS. Full units, focused TSan, frontend/lexer tests, and exact-parent numeric/BigInt test262 coverage pass with 0 conformance flips.

  10. chrisbbreuer commented on Aug 13, 2026

    @chrisbbreuer
    MemberAuthor

    Child #545 is complete and closed.

    f91369c6 makes oversized unseparated decimal BigInts borrow their validated immutable source digits and separated forms reuse the exact normalization allocation. e1debe62 publishes the exact-parent report and raw evidence.

    Across unseparated 1K/2K/4K workloads, one logical request is removed per parse and allocation bytes fall 45.2% / 58.1% / 71.5%. The separated same-value 4K control falls 39.0% while retaining its one normalization allocation. Wall medians stay within 2.1% and peak RSS does not regress. Full units, focused TSan/frontend/lexer, and exact-parent BigInt/numeric test262 coverage pass with 0 flips.

  11. chrisbbreuer commented on Aug 13, 2026

    @chrisbbreuer
    MemberAuthor

    Completed child #546 with benchmark-first, implementation, and checked raw evidence on main:

    Oversized binary/octal/hexadecimal BigInt conversion now uses pre-sized base-1e9 scratch limbs and maximal radix chunks instead of byte-per-decimal-digit list growth. The 1K/2K/4K diagnostic rows retain full independent decimal-oracle validation and expose the remaining approximately quadratic growth. Candidate wall ratios are 0.018x/0.017x/0.016x; instruction ratios agree at 0.0229x/0.0184x/0.0164x. Full units, focused TSan, and exact-parent BigInt/numeric test262 comparisons pass without regressions.

    Evidence: docs/.data/exact-parent-frontend-bigint-hex-2026-08-12.md

  12. 40 remaining items

  13. chrisbbreuer commented on Aug 20, 2026

    @chrisbbreuer
    MemberAuthor

    Opened frontend child #668: eval, realm evaluation, ShadowRealm.evaluate, and dynamic Function-family constructors currently pass physical StringCell bytes to the parser. The fix establishes one canonical WTF-8 ingress contract without changing lexer semantics or dynamic-code realm/error ordering.

  14. chrisbbreuer commented on Aug 21, 2026

    @chrisbbreuer
    MemberAuthor

    Frontend child #668 is complete via 355329dc. Eval, realm evalScript, ShadowRealm.evaluate, and Function-family parsing now share a canonical WTF-8 ingress contract without changing lexer/parser semantics or realm/error ordering. Both storage layouts passed focused interpreter/VM coverage; Test262 620/620.

  15. chrisbbreuer commented on Aug 22, 2026

    @chrisbbreuer
    MemberAuthor

    #718 landed in 4740df55, adding activation-local, precisely rooted class-heritage lowering across suspension and moving GC. This closes that slice only; the parent remains open for its broader GC/JIT audit.

  16. chrisbbreuer commented on Aug 23, 2026

    @chrisbbreuer
    MemberAuthor

    Frontend/compiler child #621 is complete in 1a0cf6e2. Exact 1K/2K/4K evidence validates one binding inventory per plain-function compile: linear growth is preserved, retired instructions fall to about 0.94x across all widths, and allocation/byte counts strictly decrease. #496 remains open for the broader frontend optimization roadmap.

  17. chrisbbreuer commented on Aug 25, 2026

    @chrisbbreuer
    MemberAuthor

    Linked security/frontend child #761 owns collision-resistant parser declaration-name membership. The design keeps immutable grammar lookup allocation-free, shares one lazy secure context across a parse root and nested template subparsers, and includes focused source-to-result/growth controls so hardening does not add a per-table entropy syscall.

  18. chrisbbreuer commented on Aug 25, 2026

    @chrisbbreuer
    MemberAuthor

    Annex B frontend/runtime classification child #760 landed in 52a7c387. The invocation-local legacy-name index is securely seeded only on its first eligible candidate; candidate-free/strict/skipped scans remain table- and entropy-free. Exact-parent test/annexB/language stayed 845/845 passing.

  19. chrisbbreuer commented on Aug 25, 2026

    @chrisbbreuer
    MemberAuthor

    Parser declaration/name membership hardening is complete in #761 via fix(parser): seed declaration name membership (#761): every parser source-string table now shares one lazy realm-derived secure context, with failure-atomic entropy/OOM behavior and no syntax regressions.

  20. chrisbbreuer commented on Aug 26, 2026

    @chrisbbreuer
    MemberAuthor

    Compiler binding/capture membership hardening child #770 is complete in a3c39817, with six exact-parent 1K/2K/4K control/adversarial artifacts in 3d249965. Compiler/frontend/full/TSan gates and 9,554 relevant Test262 cases pass with zero flips.

  21. chrisbbreuer commented on Sep 1, 2026

    @chrisbbreuer
    MemberAuthor

    Completed parser child #855 in 8afcf79: cover-grammar node identity remains O(1)-expected and exact, but no longer relies on allocator/ASLR placement as its collision defense. The implementation reuses one lazy parse-root seed across nested template parsers, adds no AST metadata or parse pass, and produced 0 exact-parent Test262 expression flips. Full evidence is on #855.

  22. chrisbbreuer commented on Sep 1, 2026

    @chrisbbreuer
    MemberAuthor

    Completed frontend child #856 in f16ce8a: Annex B and repeated-catch AST identity plans retain O(1)-expected exact lookup without relying on arena/ASLR placement. The compiler reuses one existing lazy root seed, adds no AST fields/pass/runtime state, and produces byte-identical Chunk/Test262 results across seeds and exact parent.

  23. glennmichael123 commented on Sep 12, 2026

    @glennmichael123
    Contributor

    New frontend allocation work is tracked in #913: 261981c reuses scratch token storage across template substitutions without skipping lex/parse work or moving AST payload lifetimes. Correctness gates passed (63 parser units; 1,131 exact-parent test262 cases). Performance publication remains pending #914, which tracks first-class direct-binary provenance so the current collector does not require dummy benchmark-overlay commits. Neither child nor this umbrella is being declared complete prematurely.

  24. glennmichael123 commented on Sep 15, 2026

    @glennmichael123
    Contributor

    The publication previously pending under #913/#914 is now complete in be59a314. Both children are closed; #496 remains open for its broader acceptance gates.

    The seven-pair growth report and embedded raw inputs compare exact parent bff189a with candidate 261981c on the Apple M2 Pro diagnostic host, Zig 0.17.0-dev.1441+d5181a9c9, real GC 0285ef80 and regex c60344de. At 1K/2K/4K substitutions, logical allocation requests per parse fall 2061→1038, 4109→2062, and 8205→4110; allocated bytes fall 354384→125232, 706640→248112, and 1411152→493872. Both parent and candidate retain linear instruction growth across the widths.

    All six rows (three widths plus plain, escaped and tagged controls) preserve raw samples and exact checksums. The escaped control retains identical allocations/bytes and its noisy 1.104x wall ratio (22.09%/26.57% RSD) remains visible, not promoted into either a win or a stable regression.

    This closes the scoped allocation/provenance evidence work, not the quiet-reference throughput/energy gate or the frontend umbrella. README's universal scorecard was not changed. The separate parser correctness series #920–#924 is landed; #925's module-boundary verification is running and is not claimed as performance evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceMeasured runtime, memory, scaling, or build performanceruntimeJavaScript runtime and builtin implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions