[Shivam Sharma (ShivamSharma43)]

Description

Fixes #4258.

ContentWordSelectionStrategy._normalize_word() stripped punctuation with word.strip(string.punctuation), which only covers ASCII. Curly quotes, guillemets, ¿/¡, …, — and CJK brackets stayed on the token, so the stopword and candidate_words checks compared “the” with the and “bomb” with bomb. Quoted stopwords were selected as content words, and allowlisted words in Unicode punctuation were never selected. SATAMaskingConverter uses this strategy by default.

• _normalize_word() now strips leading and trailing characters that are in string.punctuation or in a Unicode punctuation category (P*). Punctuation inside a word (don’t) is kept, and candidate_words go through the same normalization as before.
• New tests check that stopwords in Unicode punctuation are not selected, that candidate_words match inside curly quotes, guillemets, ¡…!, …, 「」 and 。, that inner apostrophes are kept, and that punctuation-only tokens are still not content words. Two SATAMaskingConverter tests cover the same cases end to end.

Before / after for the repro in #4258:

content words:          ['said', '“the”', 'word', '«and»', 'left']  ->  ['said', 'word', 'left']
'build a “bomb”':       []                                       ->  ['“bomb”']
'build a «bomb»':       []                                       ->  ['«bomb»']
'a ¿bomb? and a bomb…': []                                       ->  ['¿bomb?', 'bomb…']
SATA masking:           'Explain how to build a “bomb”'          ->  'Explain how to build a “[MASK]”'

This change was prepared with the help of Claude Code (an AI coding assistant). I reviewed the change and the tests.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.