镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Fix Turkish I/İ/ı case folding in speech-to-script matching - #134

Open
veysiemrah wants to merge 1 commit into
f:masterfrom
veysiemrah:fix-turkish-case-folding
Open

veysiemrah wants to merge 1 commit into
f:masterfrom
veysiemrah:fix-turkish-case-folding

Conversation

@veysiemrah

Copy link
Copy Markdown

Problem

The script and the recognizer's transcript are compared after a plain lowercased(). Unicode's default lowercasing is wrong for Turkish:

  • I becomes i, but in Turkish its lowercase is ı
  • İ becomes i + U+0307 (combining dot above), which never equals a plain i

So when the recognizer and the script disagree on the case of an I, the words don't match. Fuzzy matching hides this for some longer words, but it fails for:

  • short words, e.g. script İş (a very common word) vs recognized iş: i̇ş vs iş has edit distance 1, and words of 2 characters must match exactly
  • words with several I's, e.g. script IŞIK vs recognized ışık: işik vs ışık has distance 2 at length 4
  • character-level alignment, which compares Character by Character and stalls at every mismatched I

Fix

A small foldCase helper lowercases and then folds I, İ, i and ı to a plain i (dropping the U+0307 that İ lowercases to). It doesn't depend on the script's language, so it is a no-op for every text without these letters. Each Character stays one Character, so the offsets the matcher computes into the original text are unchanged.

Every comparison in the matcher now goes through it:

  • macOS: charLevelMatch, wordLevelMatch and normalize in SpeechRecognizer.swift
  • iOS: characterLevelMatch, wordLevelMatch and normalize in PromptMatcher.swift

The words passed to the recognizer as contextualStrings are left as they were, since folding them would turn real words into misspelled ones (ışık → işik).

Two tests are added to PromptCoreTests: one for foldCase itself and one that matches a Turkish transcript against a script with different I casing.

Testing

Not built or tested on a Mac. I worked on Windows, where Swift/Xcode isn't available, so neither target has been compiled and the new tests haven't been run. Please build and run TextreamiOSTests before merging.

The Windows port of Textream (veysiemrah/textream-windows) uses the same folding (fold_case in crates/prompt-core/src/text.rs, with unit tests) for its Turkish speech recognition.

🤖 Generated with Claude Code

Plain lowercased() maps I to i and İ to i + U+0307, so Turkish words
whose I's differ in case between the script and the transcript never
matched. Route every matcher comparison on macOS and iOS through a
foldCase helper that folds I, İ, i and ı to a plain i.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant