镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Design 3.0's language packs and output model (#753, #754) - #940

Merged
derek73 merged 41 commits into
masterfrom
feat/vocabulary-phase4-design
Oct 29, 2026
Merged

derek73 merged 41 commits into
masterfrom
feat/vocabulary-phase4-design

Conversation

@derek73

@derek73 derek73 commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Lands the agreed design for vocabulary phase 3 (language packs) and phase 4 (order, shape and the output model), decided 2026-10-28. It holds no parser change. The record lives in the docs; this description only indexes it.

Phase 3: language packs (#753)

Overview: docs/design/vocabulary-model-rfc.md, Phase 3. Record: decisions/vocabulary-model.md, 2026-10-27 #753 entry. All twelve items were agreed 2026-10-28:

  1. A pack is a choice of languages on Locale.languages, unioned by parser_for. Strict is an argument of the whole choice.
  2. There is one mechanism: the chosen languages select which readings compile. Turning a reading on and preferring one are both derived from the file it sits in, and a shape clash is settled for the chosen language's own reading.
  3. default = false is allowed on name readings. When turned on, they derive markers and never settle a clash (decision 15 unchanged).
  4. Strict waits until name readings record whose names they are, which is sub-phase 3i.
  5. A reading index, loaded for every caller, carries the languages to run time.
  6. A tag selects its own file and its ancestors' files, never its descendants'. No macrolanguage table is read.
  7. A pack is measured three ways: the default does not move, a derived deviation surface, and snapshot.py taking a choice.
  8. Third-party language files are deferred.
  9. Pack-shaped exclusions become default = false readings.
  10. Some things stay out of phase 3: rule data, position rules, and clashes within one language.
  11. The work order is 3a to 3i. 3g's title-run connective remains open.
  12. Phase 3 is 3.0 work, opt-in, and leaves the default's parses unmoved.

Phase 4: the output model (#754)

Overview: the RFC's Phase 4 section. Record: decisions/vocabulary-model.md, 2026-10-28 #754 entry, plus its render patterns and Lexicon surface entries.

  • The seven roles stay. generation/credentials, patronymic and surname2 are views read from a new Token.reading (or from rules/O6's tag).
  • A patronymic is named without emptying family.
  • middle keeps its meaning (Should a two-word given name like María José or Min Ho fill given, instead of given + middle? #704).
  • [rules] compile into a Locale's policy patch.
  • The render defaults follow the order a name was read in, keep its written separators, and keep the title first.
  • render(spec: str | None = None) is a signature break that was decided and waived: it takes no 2.x warning, since no call that works today changes meaning (decisions/3-0-reevaluations, THE RENDER SIGNATURE).
  • Lexicon is configured by reading, and its compiled sets become internal in 3.0. This is a break decided under the freeze, and its 2.x warning was waived on 2026-10-29: 2.8 was the last 2.x feature release, so 3.0's release notes carry the change (decisions/3-0-reevaluations, THE LEXICON SURFACE and THE LEXICON WARNING).
  • Render patterns in CLDR's shape are agreed as a later addition.
  • Reading a title in the positions CLDR writes it is a parser question tracked as Should Wang Xiaoming Senator and Wang Dr Xiaoming read the title, where CLDR writes surname-first names? #938.

Review

A design-docs review over the combined diff led to three fix rounds, each reviewed in turn:

  • 3i counts recounted and dated at c8357e74.
  • A reproduction re-run.
  • A sentence scoped to FAMILY_FIRST.
  • 3-0-reevaluations' Standing lines brought current.
  • The R2 connective case below.

A test now pins a word listed as both a particle and a conjunction where it stands between particles. A mutation that sets aside every interior connective fails it.

R2 amendment and #939

  • rules/R2's Then now matches the code: the base drops every working particle, not only the leading run (decisions/R2, 2026-10-28 Phase 4: order and shapes, the rules a pack can choose, and the output model #754).
  • A connective that is not also particle vocabulary, standing between particles, is set aside when R2 asks whether a part is all particles. So Fritz Freiherr und von Bar should read base 'Freiherr und von Bar' where today it gives 'und'.
  • deviates: #939 markers sit in rules/R2 (family_base), R3 (initials) and R4 (capitalized).
  • rules_budget.json was re-recorded for R2, R3 and R4. corpus_rules.jsonl and the snapshot gain the new examples, and no recorded reading moved.

Merged in

#937 (the facade removal) is merged into this branch. Its test comment quoting R2's old Then is re-quoted to the amended one.

Still open

  • How a call reaches a pack's render pattern: (a) a convention recorded on the name, or (b) through the parser.
  • 3g's connective that counts only inside a title run.
  • Lexicon's own pickle guard learning the ADDED_IN marker.
  • The method names for configuring by reading.

Closes nothing: #753 and #754 stay open for implementation.

Refs #753, #754, #939

🤖 Generated with Claude Code

derek73 and others added 30 commits October 27, 2026 20:32
…e pool (#753)

The RFC's Phase 3 section rewritten and a dated decisions/vocabulary-model
entry, every item marked proposed until Derek decides it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… findings (#753)

Strict cannot be built on today's files (name readings sit where their
count came from), so it waits; the add-only pack contract and the 3-0
Standing line are named as amended; counts, pointers and scope fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estions (#753)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… are views (#754)

A spec draft for review, every item marked proposed. The RFC's Phase 4
section gains a status line, the [rules]-into-policy-patch proposal, the
output model (Token.reading; generation/credentials, patronymic and
surname2 as views; middle unchanged; render defaults following the read
order), an API-change table against the 3.0 freeze, phase 3's Q1
confirmed, and a 4a-4h work order interleaved with 3a-3h.
decisions/vocabulary-model gains the dated, proposed 2026-10-28 #754
entry carrying the measurements from the fielded corpora.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope the render exclusion to word order, so a given-first interpunct
name keeps its dot; re-count the corpus figures over distinct entries
under their declared orders and name the ending tests; point at phase
3's unmerged draft instead of an entry master lacks; say surname2 is
read from O6's tag; leave order out of comparison; list the four
reserved names as without a sub-phase; drop the Đỗ promise Token.reading
cannot keep and the release plan no entry records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rules/R2 says the base is the family without its leading particles; the
code removes every working particle. Each interior one leads its own
name unit, so the per-surname reading and the code agree. Proposed:
amend R2 to the code. Measured over the committed corpus and tse-donors,
in the default policy and with middle_as_family.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…754)

Comma names drop the comma as a separator; the str() change reaches
given-first names with an interpunct; order's exclusion from == is left
open; suffix counts re-measured per word (33/77/64) with the recipe
stated; CLDR samples counted as distinct names; R2's proposal points at
#937's unmerged branch, covers particle runs and trailing particles
(39/505), and names migrate.rst among the sites landing it touches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
family_base removes every working particle wherever it stands, and R2
now says so, its Because calling the base a view the same under every
naming convention rather than a sort key. decisions/R2 records the
Portuguese last-surname and Spanish first-surname filing conventions
with sources (RAE's rules a and e read from the page), and the RFC
leaves open whether a pack supplies a convention's sort key. R2's size
grows by three example lines and reads no new signal; budget
re-recorded. Leaving ParsedName.order out of == is recorded as an
accepted cost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two words are joined by their written separator when they stand one
after the other in the same order in the text and the rendering. A
family comma never qualifies, since the rendering reverses its pair; a
comma before a suffix does, so 'John Smith, Jr.' keeps it (456 of the
1,008 committed entries with a suffix after a family write one).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept separator is whitespace, nothing, an interpunct, a nakaguro or
one comma; delimiters and consumed marker words take the spec's joiner.
Only a family comma keeps a name given-first. The suffix-comma count is
re-taken (214 of 1,003 keep the comma) and the differing render of
'Smith, John, Jr.' is recorded as accepted. R2's entry says where the
base departs from the Spanish key (rule b's La and San), speaks of R2's
old Then in the past, and the RFC keeps R2's all-particle exception.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shape for later: a pack may carry given-first, surname-first and
sorting patterns. The RFC maps Phase 4's views to CLDR's person-name
fields and names the gap (CLDR's prefix and core are the first
surname's), proposes a casefolded key method, and leaves open whether a
pattern is reached through the name or the parser. Two Standing
pointers gain their entry's head words, now that two 2026-10-28 #754
entries exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spec joiner that decorates a field (the ') ' after a maiden name)
stands over a kept separator; a comma may be ASCII or full-width, with
the counts' ASCII scope stated. The CLDR table splits a one-unit
surname (family, today) from a two-unit one, and option (a) names its
cost of a new way to ask for a pattern, render() taking a format spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Title tokens show behaviors, not kinds, and ReadingKind reads every
title as TITLE. The RFC's Phase 4 Left open asks whether Reading gains
an optional vocabulary-kind field, and so whether phase 3's reading
index carries kinds; no title word holds two kinds today (816 checked).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shape question for before the freeze: keep a public field per
behavior set (22 today, 11 added since 2.0.0), or answer word, kind and
property queries from phase 3's reading index and keep the sets
internal. The RFC states the costs on each side: frozen fields, pickles,
reprs, how a caller adds a word, and per-token frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spec joiner that decorates a field now precedes the kept-separator
branch, and a kept comma carries the whitespace written beside it, so
an unspaced full-width comma is kept too. Five of the 728 hold a
full-width family comma; the decisions copy scopes its counts to the
ASCII comma. Left open becomes lists, with the Lexicon fields question
set apart as a shape question; its pickle cost is corrected (Lexicon's
own guard admits no ADDED_IN field) and the index's load cost added.
The title-kind item notes that phase 3's index already holds a kind
per row, and that titles are split by name collision, not kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rule now covers every separator: kept ones (nothing, whitespace,
interpunct, nakaguro, one comma of any kind the parse reads, Arabic
included), the spec's decoration between two fields standing instead,
and the spec's joiner for everything else, in both copies. 36 of the
728 hold a family comma the parse reads (5 full-width, 31 Arabic). The
index load cost carries its no-cache qualifier, and the title-kind item
narrows to whether Reading exposes the vocabulary's kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept separator keeps its kind, not its written whitespace: any
whitespace becomes one space, an interpunct or nakaguro stands alone,
an ASCII or Arabic comma becomes comma-space, and a full-width comma
stands alone. The written whitespace stays in ParsedName.original and
the token spans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The change list no longer counts whitespace-separated names, which the
normalized rule leaves as they are, and the Standing line says the
separators are normalized. Only an interpunct the parse reads as a
separator stands alone; a spaced one is a name word (rules/T3). The
interpunct comma count reads every comma the parse reads, the decisions
copy names directional marks and the Arabic counts, the whitespace note
sits after the rule, and the index-cost comparison names the cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…754)

U+3000 is whitespace like any other, so it renders as one ASCII space;
the full-width comma's own-width reason is scoped to the comma. The
Japanese comma is noted as a parser question outside the design. The
per-word index's import cost now matches decisions/A1's measurement
(0.35 ms of 18.2 ms), where two copies had 0.4 of 19.5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The U+3000 example no longer reads as today's output, which is turned
round; the decisions copy states the full-width comma's reason, says
ASCII space, and carries the Japanese-comma note; the change list
covers only separators between words the rendering keeps side by side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
render(spec=None): None is the order-aware default str() passes, and a
caller's spec renders exactly as written, so 2.x specs keep their
meaning. The family-first layout keeps title first and suffix last with
the name roles in order's sequence, checked against CLDR's surnameFirst
samples. Both marked proposed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The proposed render(spec=None) is named a signature change, not an
addition, in both copies and the API table. A family-comma name keeps a
given-first order, so it renders by today's spec. The CLDR comparison
now says most samples put the title after the family (50 of 70), so
the title's place in the family-first default stays open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in 3a-3h said whose names a name reading's are, which strict
waits on. Record it as open beside the strict paragraph and in the
decisions entry's item 4: an attribute naming each name reading's
bearers' languages, filled word by word, since a box count says where a
word was typed, not whose name it is (do's ORCID family box is led by
Brazil's particle, none of its 7,227 boxes do alone). Proposed as 3i,
after 3a, neutral; nothing decided.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…753)

- The reading index loads for every caller, since Reading.languages
  (4a) is filled from it on every parse; lazy loading would need a
  Reading to carry its word as a hidden key. Cached import measured at
  about 0.45 ms against about 20 ms for import nameparser; the earlier
  11 ms was taken under PYTHONDONTWRITEBYTECODE.
- The Locale/Policy dependency is answered by the 2026-10-28 freeze.
- Work order: 3f and 3h need 4b, 3g needs 4d and 4g, 4f lands with or
  after 3b; RU and TR_AZ wait for 4f.
- The shape-questions-first paragraph is history: the output model was
  agreed 2026-10-28 (#754).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The phase table's rows 3 and 4 say both states: phase 3's design
proposed 2026-10-27 and not agreed, phase 4's output model agreed
2026-10-28. The log keeps both entries in date order. Phase 4's text
now cites phase 3 in this file rather than its branch, mentions 3i in
the interleaving, and drops the stale claim that the reading index
loads only for a caller who chooses a language.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
render(spec: str | None = None) is taken: None is the order-aware
default str() passes. The family-first layout's title place stays open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 and others added 10 commits October 28, 2026 18:14
The family-first default keeps the title first, the one place the parse
reads every title back from; reading titles where CLDR writes them is a
separate parser issue. Vocabulary is configured by reading in 3.0 and
the compiled sets become internal, the first break decided under the
core API freeze; its 2.x warning is open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Items 1, 2, 4, 5, 6 and 8 are accepted as written: the choice of
languages on Locale.languages, one mechanism, strict waiting for 3i,
the reading index loaded for every caller, tag selection by file and
ancestors, and third-party files deferred. Items 3, 7 and 9-12 stay
proposed. The add-only pack contract is amended in 3b's PR, as the
items tie it there. A pack's lexicon fragment and Locale(lexicon=...)'s
successor now point at the Lexicon-surface entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Items 3, 7 and 9-12 are accepted as written, so all twelve are decided
and the entry's head and the RFC's status say agreed. 3g's connective
stays an open design inside the work order. The add-only pack contract
is still amended in 3b's PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R2's example 'da Silva do Amaral de Souza, Miguel' joins
corpus_rules.jsonl and the snapshot as one new entry; no recorded
reading moves, and every agreement record still matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules/R2 gains an Except: a connective between two words of a part is
  set aside when the part is asked whether every word is particle
  vocabulary, so `Fritz Freiherr und von Bar` reads base 'Freiherr und
  von Bar'; today's base 'und' is a deviates: marker for #939. A
  connective ending the part stays a name word (`Juan de y` boundary).
  rules/R3's all-particles Except follows it, with its own marker.
  decisions/R2 records the case, dated 2026-10-28.
- decisions/3-0-reevaluations records the render() signature break as
  waived, taking no 2.x warning, and updates the render-defaults and
  harness Standing lines.
- The 3i counts are recounted at c8357e7 and dated; the Ministro
  reproduction names both titles added; Lim Mei Ling Jessica is scoped
  to FAMILY_FIRST.
- rules_budget re-recorded (R2 and R3 grew by the new clause);
  corpus_rules gains the new example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules/R4's all-particles Except sets a connective between the part's
  words aside as R2 does, with `FRITZ FREIHERR UND VON BAR` marked
  deviates: #939 (today 'Fritz freiherr und von bar').
- rules/R2 and decisions/R2 say a joining connective anchors no base on
  its own but stays in the base view, instead of "is never the base",
  which its own example and R3's `Jon Dough and` contradict.
- decisions/R2 names R4 beside R3, and its unjoined-mark Standing line
  is scoped to a connective ending the part.
- rules_budget re-recorded; corpus_rules and the snapshot gain the new
  example, no recorded reading moved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules/R2, R3 and R4's connective set-aside reaches only a connective
  that is not also particle vocabulary; a word a caller lists as both
  keeps R2's readmission wherever it stands (the readings the #461
  tests and INV2 pin). The default lists no word as both, so this is
  stated in prose with no example line.
- decisions/R2's unjoined-mark Standing line again covers every
  connective and says how it relates to the set-aside; the case-repair
  value moves onto the all-capitals spelling; the Open line names R4.
- decisions/R3 and decisions/R4 each gain a dated #939 pointer.
- R4's new sentence is trimmed to the test; rules_budget re-recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/v2/test_render.py gains a test pinning a word listed as both a
  particle and a conjunction where it stands between particles
  (`Juan de y la` base 'de y la', initials 'J. d. y. l.'; `Fritz
  Freiherr y von Bar` 'F. F. y. v. B.'), guarding R2's scoping of its
  connective set-aside. A mutation setting aside every interior
  connective of an all-particle part fails it, while the neighboring
  #461 test still passes.
- decisions/R2 names the test pinning each reading; the interior ones
  were attributed to tests that never reach them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts resolved: decisions/3-0-reevaluations keeps the removal's
entries beside the render-signature waiver and the Lexicon surface
entry, its render Standing line pointing at the decided defaults and
its harness line under Not standing; decisions/R3 and decisions/R4
keep both same-day entries; rules_budget re-recorded rather than
merged. #937's test comment quoting R2's old Then is re-quoted to the
amended one, and the design branch's prose no longer treats #937 as
pending or points at the retired "clean slate".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 3.0 milestone Oct 29, 2026
@derek73 derek73 added the design label Oct 29, 2026
@derek73 derek73 self-assigned this Oct 29, 2026
decided 2026-10-29: like the render() signature, the Lexicon surface
break takes no 2.x warning. Unlike it, the break does change calls that
work today, but 2.8 was the last 2.x feature release and no deprecation
path ships for the compiled-set fields, so 3.0's release notes carry
the change. decisions/3-0-reevaluations gains THE LEXICON WARNING entry
and its Standing line; decisions/vocabulary-model and the RFC no longer
list the warning as open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 9210f22 into master Oct 29, 2026
12 checks passed
@derek73
derek73 deleted the feat/vocabulary-phase4-design branch October 29, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant