Repository navigation
Performance issue with --pattern-from #32
Description
Hi Liz,
Another quick fix , when you're using the --pattern-from with regex you're doing a loop with each one of the pattern rather than compiling a single regex with all pattern with | this creates a huge performance penalty because you checking the same file over and over based on the number of line in the input file.
Based on my test on a large CSV 93 MB it takes 294 secs against a 99 regex pattern file vs 84 sec if you convert this as a regex with |.
Can you fix this ?
Activity
lizmat commentedon Nov 24, 2022
Just not quite a quick fix, I'm afraid.
OOC, do any of those patterns consist of just looking for a literal string?
lizmat commentedon Nov 24, 2022
A first stab at merging consecutive regexes into a single regex for multiple patterns
lizmat commentedon Nov 24, 2022
0.2.14 now in the ecosystem. Please let me know how this affected the performance!
Zer0-Tolerance commentedon Nov 25, 2022
Great I'll test again and let you know ? All patterns are regex.
Zer0-Tolerance commentedon Nov 25, 2022
ok with the latest version it's better than before 129 sec now (as opposed to 284 sec) for the list of pattern vs 81 sec with the single regex with |.
This is the cmd I'm using:
time rak --ignore-case --patterns-from=/tmp/file.txt /path/file.csvlizmat commentedon Nov 25, 2022
ok, that's better. By any chance, are one or more of the regexes literal? Consisting of just
/ foo /? Because those are currently converted internally to*.contains("foo"), and thus create multiple regexes, instead of just a single one. Which would explain the remaining differenceZer0-Tolerance commentedon Nov 25, 2022
Sorry I forgot to add the type to the cmd so no pattern is interpreted as literal in my list right ?
time rak --ignore-case --type=regex --patterns-from=/tmp/file.txt /path/file.csvlizmat commentedon Nov 25, 2022
With a literal regex I mean
/ foo /where "foo" consists of only alphanumeric characters. I doubt that if you don't change the file, and then do a--type=regexthat it would produce the right results, or even run at all??Zer0-Tolerance commentedon Nov 25, 2022
my patterns consist of IP addresses , filenames.
lizmat commentedon Nov 25, 2022
Could you give me an example of the IP address and a filename?
Zer0-Tolerance commentedon Nov 25, 2022
sure "8.8.8.8" and "test.exe" for example.
lizmat commentedon Nov 25, 2022
When you have 8.8.8.8 in your regex, you actually mean the string "8.8.8.8", right? But "8a8b8c8" you would not want to match, right?
If that is so, why are you using regexes? Why not just the string "8.8.8.8" (which would do a much cheaper '.contains("8.8.8.8")' on the targets.
Zer0-Tolerance commentedon Nov 26, 2022
yes but I have also tests?.exe in the same list of pattern.
lizmat commentedon Nov 27, 2022
ok, then I suggest grouping the ones with wildcards together, and see how that affects the performance. As the logic now will group together any consecutive regexes into a single regex
lizmat commentedon Jul 29, 2024
FWIW, I'm keeping this one open until it's time to rewrite the expression parsing using RakuAST, and optimize from there
lizmat commentedon Aug 17, 2024
This has now happened with 0.3.4 https://raku.land/zef:lizmat/App::Rak/changes?v=0.3.4 . This now builds a single
Callablefor all of the patterns found using RakuAST, not doing anything special for regexes yet.Please confirm whether this makes things better or worse in your use case.