镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Frontend: decode escaped identifiers in one span-based pass #538

Closed

Description

@chrisbbreuer

Parent: #496
Related: #493, #460, #461, #536, #537

Problem

Lexer.lexIdentName scans every escaped identifier once to validate Unicode escapes and IdentifierStart/IdentifierPart, then decodeIdent rescans the entire raw identifier and appends every non-escaped byte individually into arena storage. Generated code using escaped bindings/properties, obfuscator output, conformance workloads, and adversarial wide parameter lists therefore pay two full scans plus avoidable per-byte growth/copy work.

Scope

  • Add parser-only 1K/2K/4K escaped-identifier growth rows with frozen source construction, decoded-name structural checks, warmup, and timed parse boundary.
  • Preserve the existing allocation-free source slice for unescaped ASCII/Unicode identifiers.
  • At the first escape, materialize decoded storage once, copy raw spans in batches, decode each Unicode escape during the validating scan, and remove the redundant second scan.
  • Preserve escaped-keyword distinction, IdentifierStart/IdentifierPart validation, Unicode scalar bounds, \\uXXXX/\\u{X} behavior, non-ASCII raw identifiers, diagnostics/source positions, private names, and source lifetime.

Security and correctness

  • Exact decoded bytes, not hashes, determine identifier identity.
  • Invalid escapes and invalid decoded start/continue characters must retain exact SyntaxError classification/location.
  • Decoding must remain OOM-propagating and failure-atomic; no global cache or attacker-predictable intern table may be introduced.
  • Do not special-case benchmark names, loosen Unicode validation, change escaped keyword semantics, or defer early errors.

Acceptance

  • Benchmark source and decoded-name structural checks land before lexer changes.
  • Pointer/allocation witnesses prove unescaped identifiers remain borrowed and escaped identifiers own exact decoded bytes.
  • 1K/2K/4K escaped rows show bounded near-linear work and materially lower instruction/allocation cost versus the exact parent; an ordinary control remains unchanged.
  • Lexer/parser/frontend suites, escaped keyword/private/property/binding cases, malformed Unicode/OOM diagnostics, forced execution paths, full units, and affected test262 identifier subtrees pass with no regressions.

Activity

  1. added
    performanceMeasured runtime, memory, scaling, or build performance
    on Aug 12, 2026
  2. chrisbbreuer commented on Aug 12, 2026

    @chrisbbreuer
    MemberAuthor

    Benchmark-first commit landed on main as 85dfd44a: bench(frontend): add escaped identifier growth rows (#538).

    The parser-only runner now freezes strict 1,024/2,048/4,096 parameter lists spelling every name as paramet\\u0065rN. Source construction is outside warmup and timing. Each parse validates the function declaration, exact parameter count, decoded parameter prefix, and numeric suffix equal to the parameter index before summing all decoded name bytes.

    Frozen one-job checksums:

    • escaped 1,024: 13,233
    • escaped 2,048: 27,569
    • escaped 4,096: 56,241

    The existing unescaped strict-parameter rows remain the control. No lexer behavior or performance claim is included in this commit; the implementation will be compared against this exact parent.

  3. chrisbbreuer commented on Aug 12, 2026

    @chrisbbreuer
    MemberAuthor

    Completed by 81bef094: perf(frontend): decode escaped identifiers in one pass (#538).

    Ordinary identifiers stay on the original allocation-free source-slice loop. The first \\u escape now branches into a validating decode continuation that copies intervening raw source spans in batches and UTF-8-encodes each validated code point during that same scan. This removes the former full decodeIdent rescan and byte-by-byte ordinary append path without adding a cache, hash identity, or interning policy.

    Coverage proves:

    • an ordinary mixed ASCII/Unicode identifier tokenizes with a zero-byte allocator and exact source-pointer identity;
    • an escaped identifier reports OutOfMemory with the same allocator, owns distinct decoded storage when allocation succeeds, preserves raw pos/end, and decodes both \\uXXXX and \\u{X} exactly;
    • escaped private names decode exactly;
    • forced escaped binding, property, and private-name execution returns 17 on parent and candidate;
    • escaped if remains an UnexpectedToken early error.

    Seven order-balanced exact-parent diagnostic pairs, 100 parses per sample, identical frozen decoded-name checksums:

    • 1,024 escaped parameters: 842,565,360 -> 815,533,432 retired instructions (0.968x);
    • 2,048: 1,677,883,753 -> 1,621,251,723 (0.966x);
    • 4,096: 3,375,953,241 -> 3,261,946,145 (0.966x);
    • ordinary 4,096 control: 2,612,326,869 -> 2,584,618,682 (0.989x).

    Candidate normalized escaped instructions per parameter vary by only 0.001x from 1,024 to 4,096, preserving bounded near-linear growth. RSS samples were noisy and are not used for a public claim.

    Validation:

    • lexer units: 11/11
    • parser units: 30/30
    • focused frontend: 12/12
    • lexer TSan: 11/11
    • full sharded units: 1,597 passed, 1 expected skip, 0 failed/leaked
    • exact-parent identifier/reserved/future-reserved test262: 350 -> 350; flips 0

    The benchmark-first requirement is 85dfd44a. All acceptance items are satisfied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceMeasured runtime, memory, scaling, or build performance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions