镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Dynamic code: canonicalize flat StringData at parser ingress #668

Closed

Description

@chrisbbreuer

Parents: #473, #493, #496
Related: #480

Problem

Dynamic-code entry points currently feed String Values to the lexer through physical asStr() bytes or toStringV() slices. Once prepared flat-Latin-1 storage is enabled, those bytes are one byte per UTF-16 unit rather than canonical WTF-8. Non-ASCII source can then be rejected, mis-tokenized, or trip byte-level assumptions even though its JavaScript String content is valid source text.

Affected shared boundaries include direct/indirect eval, $262.evalScript, ShadowRealm.prototype.evaluate, and Function/GeneratorFunction/AsyncFunction/AsyncGeneratorFunction constructor arguments.

Scope

  • Present canonical WTF-8 to every dynamic parser/lexer ingress.
  • Borrow existing ASCII and canonical WTF-8 StringCell storage with zero conversion allocation.
  • Perform one bounded Latin-1-to-WTF-8 conversion only when physical flat storage requires it.
  • Preserve ToString ordering, constructor-realm SyntaxErrors, direct-eval lexical semantics, source locations/debug registration, lone-surrogate handling, and OOM propagation.
  • Cover ordinary, realm, ShadowRealm, Function, and generator-family parsing with non-ASCII source.

No-workaround rules

  • No representation forcing, replacement character, source escaping, invalid-byte tolerance, parser special case, Test262 exclusion, or hidden permanent canonical copy.
  • No stacks, queues, jobs, Promise reactions, Worker, or Map/Set iterator changes.

Acceptance

  • Every dynamic parser ingress observes exact JavaScript String content under canonical and flat storage.
  • ASCII/canonical input remains borrowed; flat Latin-1 conversion is bounded and OOM-visible.
  • Focused tree-walker/VM/realm units, relevant nonzero Test262 accounting, and the full unit gate pass with zero regressions.
  • Focused witnesses pass with flat_storage_active disabled and enabled.

Implementation and evidence

  • 355329dc establishes canonical WTF-8 at eval, realm evalScript, ShadowRealm.evaluate, and all Function-family parser ingresses.
  • Canonical and ASCII StringCells borrow their existing bytes; only physical flat-Latin-1 source takes the existing bounded conversion.
  • Focused tree-walker/VM witnesses: 2/2 passed under canonical storage and 2/2 with physical flat storage enabled.
  • A fail-first allocator induced OutOfMemory at flat-source conversion before lexing, with 0 successful allocations.
  • Test262 dynamic-code subtrees: 620 passed, 0 failed.
  • Full unit gate: 1,760 passed, 1 skipped, 0 failed, 0 leaked.

Activity

  1. chrisbbreuer commented on Aug 21, 2026

    @chrisbbreuer
    MemberAuthor

    Completed and pushed to main in fix(parser): canonicalize dynamic StringData ingress (#668).

    • Direct/indirect eval, realm evalScript, ShadowRealm.evaluate, and all four Function-family constructors now present canonical WTF-8 to the lexer.
    • Canonical/ASCII source remains borrowed; only physical flat Latin-1 converts.
    • Focused interpreter/VM: 2/2 canonical and 2/2 physical-flat.
    • Flat fail-first allocator: exact OutOfMemory, 0 successful allocations, no parser entry.
    • Test262: 620/620 across all changed dynamic-code subtrees.
    • Full units: 1,760 passed, 1 skipped, 0 failed, 0 leaked.

    No stack, queue, job, Promise-reaction, Worker, or Map/Set iterator implementation changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions