镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Lexer: consume validated identifier starts once #608

Closed

Description

@chrisbbreuer

Parent: #496
Related: #479, #538, #460, #461, #493

Problem

The lexer validates every IdentifierStart twice. nextRaw first calls identStartHere; after success, lexIdentNameWithOwnedPrefix begins at the same byte and repeats the ASCII classification or the complete non-ASCII UTF-8 length/decode/Unicode ID_Start lookup. Private names do the same after #.

A current-main five-second parser-only profile of representative_frontend_unicode_identifiers_4096 records 508 top-of-stack samples in lexIdentNameWithOwnedPrefix, plus 143 in isIdStartCp, 142 in utf8Decode, and 54 in identStartHere. The existing ASCII, Unicode, escaped, and private-name growth rows already freeze AST content, source ownership, checksums, allocations, and hardware counters.

Scope

  • Have IdentifierStart validation return the exact already-validated byte length (with an explicit escape sentinel).
  • Consume that start exactly once when scanning the rest of IdentifierName.
  • Share the boundary between ordinary and private identifiers.
  • Keep numeric-literal adjacency rejection using the same exact start predicate.
  • Preserve the zero-copy ASCII/Unicode source slice and Frontend: decode escaped identifiers in one span-based pass #538's failure-atomic owned escape decoder.

Constraints

  • No ASCII-only shortcut, Unicode table omission, relaxed IdentifierStart/Part distinction, invalid UTF-8 acceptance, benchmark recognition, token cache, or allocation trade.
  • Do not change generated Unicode data, token spelling, escape flags, source offsets, private-name # ownership, or error classification.
  • Do not touch stacks, queues, jobs, Promise/microtasks, Worker lifecycle, or scheduler code.

Acceptance

  • Each accepted IdentifierStart is classified/decoded once before IdentifierPart scanning.
  • ASCII, Unicode BMP/astral/combining, escaped, private, escaped-private, invalid UTF-8, reserved-word, and numeric-adjacency semantics remain exact.
  • Exact-parent 1K/2K/4K rows report wall, instructions, allocation counts/bytes, RSS, and growth; anti-specialization escaped/private controls remain exact.
  • Lexer/parser focused normal, OOM, and TSan tests, full units, and affected test262 subtrees pass with quantified flips.
  • Retain only if the exact-parent frontend rows show causal CPU-work reduction.

Activity

  1. chrisbbreuer commented on Aug 16, 2026

    @chrisbbreuer
    MemberAuthor

    #608 is complete on main via 4fe004e1.

    identStartLen now returns the exact already-validated ASCII or UTF-8 start length; zero is reserved for the leading Unicode-escape path. Ordinary/private name scanning advances over that known prefix before applying IdentifierPart rules. Numeric-literal adjacency uses the same predicate. The existing borrowed ordinary path and #538 owned/failure-atomic escaped path are unchanged.

    Formal exact-parent results (15 order-balanced pairs, no discards, diagnostic host):

    row wall ratio instruction ratio allocations allocated bytes
    Unicode 1K 0.984x 0.9654x 58 -> 58 784,776 -> 784,776
    Unicode 2K 0.980x 0.9658x 63 -> 63 1,132,824 -> 1,132,824
    Unicode 4K 0.970x 0.9657x 69 -> 69 2,813,232 -> 2,813,232
    ASCII 4K 1.012x 0.9977x 69 -> 69 2,813,232 -> 2,813,232
    escaped 4K 1.011x 0.9981x 8,261 -> 8,261 4,475,968 -> 4,475,968
    private 4K 1.024x 0.9980x 53 -> 53 2,584,328 -> 2,584,328
    escaped-private 4K 0.979x 0.9983x 8,245 -> 8,245 4,210,200 -> 4,210,200

    The Unicode instruction reduction is stable across the growth series. Sub-millisecond control wall medians are noisy/mixed, so no control wall-time claim is made. All identities/checksums matched; median RSS was effectively neutral. Five-second Unicode profiles reduce combined first-start/name-scan leaf samples from about 847 to 716 (~15%).

    Validation:

    • all 21 lexer-focused tests pass
    • focused borrowed/escaped ownership, Unicode 17 start/continue, malformed UTF-8/WTF-8, private-name, OOM, and new ASCII/Unicode/escaped numeric-adjacency cases pass
    • focused TSan passes
    • full sharded units: 1,724 pass, 1 skip, 0 fail/leak
    • test262 identifiers: 268 -> 268
    • statement class elements: 1,534 -> 1,534
    • expression class elements: 1,428 -> 1,428
    • total affected test262: 3,230 -> 3,230, with 0 flips/regressions

    No Unicode table change, token cache, AST change, allocation trade, or jobs/Worker surface is involved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions