Closed
Description
Parent: #496
Related: #479, #538, #460, #461, #493
Problem
The lexer validates every IdentifierStart twice. nextRaw first calls identStartHere; after success, lexIdentNameWithOwnedPrefix begins at the same byte and repeats the ASCII classification or the complete non-ASCII UTF-8 length/decode/Unicode ID_Start lookup. Private names do the same after #.
A current-main five-second parser-only profile of representative_frontend_unicode_identifiers_4096 records 508 top-of-stack samples in lexIdentNameWithOwnedPrefix, plus 143 in isIdStartCp, 142 in utf8Decode, and 54 in identStartHere. The existing ASCII, Unicode, escaped, and private-name growth rows already freeze AST content, source ownership, checksums, allocations, and hardware counters.
Scope
- Have IdentifierStart validation return the exact already-validated byte length (with an explicit escape sentinel).
- Consume that start exactly once when scanning the rest of IdentifierName.
- Share the boundary between ordinary and private identifiers.
- Keep numeric-literal adjacency rejection using the same exact start predicate.
- Preserve the zero-copy ASCII/Unicode source slice and Frontend: decode escaped identifiers in one span-based pass #538's failure-atomic owned escape decoder.
Constraints
- No ASCII-only shortcut, Unicode table omission, relaxed IdentifierStart/Part distinction, invalid UTF-8 acceptance, benchmark recognition, token cache, or allocation trade.
- Do not change generated Unicode data, token spelling, escape flags, source offsets, private-name
#ownership, or error classification. - Do not touch stacks, queues, jobs, Promise/microtasks, Worker lifecycle, or scheduler code.
Acceptance
- Each accepted IdentifierStart is classified/decoded once before IdentifierPart scanning.
- ASCII, Unicode BMP/astral/combining, escaped, private, escaped-private, invalid UTF-8, reserved-word, and numeric-adjacency semantics remain exact.
- Exact-parent 1K/2K/4K rows report wall, instructions, allocation counts/bytes, RSS, and growth; anti-specialization escaped/private controls remain exact.
- Lexer/parser focused normal, OOM, and TSan tests, full units, and affected test262 subtrees pass with quantified flips.
- Retain only if the exact-parent frontend rows show causal CPU-work reduction.
Metadata
Metadata
Assignees
Labels
No labels
Activity
perf(lexer): consume identifier starts once (#608)
chrisbbreuer commentedon Aug 16, 2026
#608 is complete on
mainvia4fe004e1.identStartLennow returns the exact already-validated ASCII or UTF-8 start length; zero is reserved for the leading Unicode-escape path. Ordinary/private name scanning advances over that known prefix before applying IdentifierPart rules. Numeric-literal adjacency uses the same predicate. The existing borrowed ordinary path and #538 owned/failure-atomic escaped path are unchanged.Formal exact-parent results (15 order-balanced pairs, no discards, diagnostic host):
The Unicode instruction reduction is stable across the growth series. Sub-millisecond control wall medians are noisy/mixed, so no control wall-time claim is made. All identities/checksums matched; median RSS was effectively neutral. Five-second Unicode profiles reduce combined first-start/name-scan leaf samples from about 847 to 716 (~15%).
Validation:
No Unicode table change, token cache, AST change, allocation trade, or jobs/Worker surface is involved.