镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Frontend: tokenize modules once under the correct lexical goal #553

Closed

Description

@chrisbbreuer

Parent: #496
Related: #461, #479

Problem

Parser.initWithScratchDiagnostic always tokenized source with Script HTML-comment handling enabled. Parser.parseModule then discarded that complete token stream and tokenized the same source again with HTML-like comments disabled. Every production module parse therefore paid for and retained an unnecessary Script-goal token list before performing the required Module-goal scan.

Scope

  • Freeze parser-only 1K/2K/4K static-module declaration/import/export rows with exact AST, source-range, cardinality, and checksum validation.
  • Freeze Script-vs-Module HTML-like-comment rejection controls and malformed-module diagnostic offsets.
  • Let production module entry points retain exactly one token stream while preserving Module lexical-goal rejection.
  • Preserve the existing Parser.init(...).parseModule() public path.
  • Keep source construction outside warmup/scored boundaries and lexing, parsing, AST construction, early errors, and validation inside.

No-workaround rules

  • No source pre-scan, cached/precompiled benchmark input, hidden warmup tokenization, omitted tokens/source locations, weakened Annex B behavior, delayed errors, or duplicate Script/Module parsers.
  • Do not accept HTML-like comments in modules or reject their Annex B Script forms.
  • Do not touch stacks, queues, jobs, Promise/microtask, or Worker lifecycle code.

Acceptance

  • Benchmark source, 1K/2K/4K identities, jobs, checksums, warmup, timed boundary, and existing Script/Module controls were committed before the runtime change.
  • Production module parsing creates one lexer and one retained token stream while preserving Module lexical-goal rules.
  • Exact AST/module-request/export/source-range checks and diagnostic offsets remain unchanged.
  • Failure paths are allocation-failure atomic and bounded; no discarded Script token arena remains.
  • Focused frontend Debug/TSan, full units, module/Annex B/source-location test262, and forced module execution pass without regressions.
  • Seven-pair exact-parent evidence reports wall, CPU, allocation requests/bytes, peak RSS, instructions/cycles/energy availability, thermal state, and dispersion without discarded samples.

Security boundary

Module source and comment forms are attacker-controlled. The fix removes redundant work without allowing lexical-goal confusion, hiding malformed input, changing diagnostic boundaries, or retaining attacker-proportional discarded storage.

Landed

Activity

  1. added
    enhancementNew feature or request
    performanceMeasured runtime, memory, scaling, or build performance
    runtimeJavaScript runtime and builtin implementation
    on Aug 13, 2026
  2. chrisbbreuer commented on Aug 13, 2026

    @chrisbbreuer
    MemberAuthor

    Benchmark-first commit landed on main as 1240045a: bench(frontend): add module lexical-goal rows (#553).

    The frozen parser-only rows use one wide import list, one matching local export list, and one source-location marker at 1K/2K/4K entries. Source construction remains outside warmup/scored boundaries; the current production Parser.init -> parseModule path, both tokenization passes, parsing, AST construction, early errors, and independent validation remain inside.

    Every sample validates exact module specifier, imported/local/exported names and order, AST kinds, declaration/initializer, entry cardinality, marker source location, and checksum. At 200 jobs the frozen exact checksums are 132427000, 477207800, and 1795915000; all rows pass. No runtime fix is included in this commit.

    The next commit will change only module-goal initialization and preserve the Script compatibility path and HTML-comment rejection controls.

  3. chrisbbreuer commented on Aug 14, 2026

    @chrisbbreuer
    MemberAuthor

    Completed on main in three linked commits:

    • 1240045a freezes the 1K/2K/4K parser-only module rows and exact checksums.
    • 78dbd220 records the first Annex B HTML-like comment in the original Script tokenization and makes Module parsing reject that exact offset while retaining the same token stream.
    • 080f2302 publishes all raw samples and the summary report.

    Seven order-balanced pairs at each size show stable retired-instruction reductions of 33.6-34.4%, allocation-request reductions of 20.6-20.7%, allocated-byte reductions of 31.5-47.9%, and peak-RSS reductions of 9.1-30.3%. The stable 1K wall row improves 35.2%; noisy 2K/4K wall medians remain explicitly diagnostic.

    Validation is green: 1,623 full unit passes plus one expected skip, focused frontend Debug/TSan 12/12, and exact module-code/import/import.meta/Annex B test262 756 -> 756 with zero failures or flips. Script HTML comments remain accepted; Modules reject the first exact open/close comment offset without a second scan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceMeasured runtime, memory, scaling, or build performanceruntimeJavaScript runtime and builtin implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions