Repository navigation

gh-153569: Simplify tokenizer state - #158124

Merged

pablogsal merged 32 commits into

python:mainfrom

pablogsal:gh-153569-tokenizer-cleanups

Oct 5, 2026

Merged

gh-153569: Simplify tokenizer state#158124
pablogsal merged 32 commits into
python:mainfrom
pablogsal:gh-153569-tokenizer-cleanups

Commits on Sep 29, 2026
- Use diagnostic line for missing-brace errors
  pablogsal
  committed
  597c294
- Keep implicit prepared newline metadata in the reader
  pablogsal
  committed
  9df1aa0
- Reuse parsed formatted string kind in lexer
  pablogsal
  committed
  578b0c5
- Transfer owned tokenizer encoding strings
  pablogsal
  committed
  072439a
- Remove unused tokenizer state and redundant branches
  pablogsal
  committed
  29f8d4a
- Avoid rescanning partial decoded tokenizer lines
  pablogsal
  committed
  c911eb4
- Reuse prepared tokenizer line terminators for CR detection
  pablogsal
  committed
  f0149a8
- Reuse interactive tokenizer chunks without carriage returns
  pablogsal
  committed
  c5465b3
- Release tokenizer constructor resources on allocation failure
  pablogsal
  committed
  02764ba
- Narrow interactive newline normalization wrapper
  pablogsal
  committed
  c04c0bb
- Close duplicated tokenizer descriptors when fdopen fails
  pablogsal
  committed
  b44bde7
- Preserve cached tokenizer lines when newline allocation fails
  pablogsal
  committed
  1e44edc
- Collect cycles involving tokenizer readline callbacks
  pablogsal
  committed
  f995e1a
- Preserve buffered text when decoding source files
  pablogsal
  committed
  6e43c27
- Clear the transferred syntax error line before cleanup
  pablogsal
  committed
  a36c95c
- Propagate tokenizer column conversion failures
  pablogsal
  committed
  ffd9ac3
- Allocate parser repetition buffers on first match
  pablogsal
  committed
  6fefac8
- Allocate type comment storage only when needed
  pablogsal
  committed
  cc4ae2a
- Reuse decoded multiline text when counting token columns
  pablogsal
  committed
  fca5aac
- Skip empty decoded tokenizer chunks
  pablogsal
  committed
  98ced0a
- Check parser array growth before allocating
  pablogsal
  committed
  b8eba43
- Name tokenizer bounds and derive byte lengths from their data
  pablogsal
  committed
  a29e2fd
- Set the tokenizer exception when file input size overflows
  pablogsal
  committed
  3587fb1
- gh-153569: Send exact line endings in the REPL test
  pablogsal
  committed
  39f6d4c
- Share parser repetition buffer growth
  pablogsal
  committed
  ba72635
- Fold Unicode input coverage into the existing REPL test
  pablogsal
  committed
  a768c1d
- Keep parser array growth policy private
  pablogsal
  committed
  7264275
- Drop obsolete bitmap stress from source discard test
  pablogsal
  committed
  e55d8b0
- Preserve input failures while scanning string literals
  pablogsal
  committed
  714d6dd
- fixup! Preserve input failures while scanning string literals
  pablogsal
  committed
  657d92c
- fixup! fixup! Preserve input failures while scanning string literals
  pablogsal
  committed
  39436f7
- gh-153569: Remove assertion-only lexer bookkeeping
  pablogsal
  committed
  452e6fa

Commits

Commits on Sep 29, 2026