Repository navigation
API request: String Homoglyph.toASCII(String) #6
Description
Please provide a toASCII API which tries to fit the character in ASCII range and returns a string. For example, the following holds true:
Homoglyph homoglyph = HomoglyphBuilder.build();
assertEquals("The quick brown fox jumps over the lazy dog",
homoglyph.toASCII("Τһе ԛυіϲκ Ьгоѡɴ ғох јυⅿрѕ оⅴег τһе ⅼаzу ԁоɡ"));It is useful in the scenarios where we want to run complex REGEX rules on (approximate) ASCII representation. Building complex regex tree equivalent with Homoglyph.search() API is not convenient (at least in certain cases).
Reacted by Nicola Bova, Samuel Philipp, Emil Burzo, viika, Cody Gray and TheEvilSocks
Reacted by Yves M.
Activity
codygray commentedon Dec 11, 2021
This would be extremely useful to me. I was looking for a
canonicalizefunction that would do essentially the same thing. I think that, perhaps,canonicalizeis a better name thantoASCII, since the "base" characters may not be strictly ASCII.codebox commentedon Sep 11, 2022
This change is probably more complicated than it first appears. I don't think a single 'canonical' set of characters could be defined that would make sense for everyone, it would vary depending on the language of the user, and also on the expected content of the text (for example should the digit '1' be replaced with the letter 'l' or left as it is?) I think the library would have to allow the user to specify what they considered canonical. In addition it isn't obvious what the correct behaviour should be if letters within the canonical set are homoglyphs of each other - for example if we just say that the 26 letters of the English alphabet are canonical, do we change the digit 1 to lower-case 'L' or to capital 'I'?
I welcome any suggestions regarding a good way to handle this.
gdude2002 commentedon Mar 14, 2023
Running into this - we'd find it very useful to be able to regex-match including homoglyphs, and normalisation is definitely the only way to handle this.
My suggestion would be to prioritise normalising to letters - the main use-case for a library like this is automated chat moderation; it's unlikely for numbers to be useful matches for problematic content (in my opinion).
Of course, this doesn't solve the latter part of your question - I think the only real solution there is to support generating permutations instead; then they can all be tested.