Repository navigation
Fix Turkish I/İ/ı case folding in speech-to-script matching - #134
Open
veysiemrah wants to merge 1 commit into
Open
veysiemrah wants to merge 1 commit into
veysiemrah wants to merge 1 commit into
Conversation
Plain lowercased() maps I to i and İ to i + U+0307, so Turkish words whose I's differ in case between the script and the transcript never matched. Route every matcher comparison on macOS and iOS through a foldCase helper that folds I, İ, i and ı to a plain i. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The script and the recognizer's transcript are compared after a plain
lowercased(). Unicode's default lowercasing is wrong for Turkish:Ibecomesi, but in Turkish its lowercase isıİbecomesi+ U+0307 (combining dot above), which never equals a plainiSo when the recognizer and the script disagree on the case of an I, the words don't match. Fuzzy matching hides this for some longer words, but it fails for:
İş(a very common word) vs recognizediş:i̇şvsişhas edit distance 1, and words of 2 characters must match exactlyIŞIKvs recognizedışık:işikvsışıkhas distance 2 at length 4Fix
A small
foldCasehelper lowercases and then foldsI,İ,iandıto a plaini(dropping the U+0307 thatİlowercases to). It doesn't depend on the script's language, so it is a no-op for every text without these letters. Each Character stays one Character, so the offsets the matcher computes into the original text are unchanged.Every comparison in the matcher now goes through it:
charLevelMatch,wordLevelMatchandnormalizeinSpeechRecognizer.swiftcharacterLevelMatch,wordLevelMatchandnormalizeinPromptMatcher.swiftThe words passed to the recognizer as
contextualStringsare left as they were, since folding them would turn real words into misspelled ones (ışık→işik).Two tests are added to
PromptCoreTests: one forfoldCaseitself and one that matches a Turkish transcript against a script with different I casing.Testing
Not built or tested on a Mac. I worked on Windows, where Swift/Xcode isn't available, so neither target has been compiled and the new tests haven't been run. Please build and run
TextreamiOSTestsbefore merging.The Windows port of Textream (veysiemrah/textream-windows) uses the same folding (
fold_caseincrates/prompt-core/src/text.rs, with unit tests) for its Turkish speech recognition.🤖 Generated with Claude Code