镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Avoid a redundant String when binding UTF-8 text - #769

Open
thomasklemm wants to merge 1 commit into
sparklemotion:mainfrom
thomasklemm:thomasklemm/skip-utf8-bind-transcode
Open

thomasklemm wants to merge 1 commit into
sparklemotion:mainfrom
thomasklemm:thomasklemm/skip-utf8-bind-transcode

Conversation

@thomasklemm

@thomasklemm thomasklemm commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

While measuring the CPU cost of Ruby's database drivers, I found that binding a UTF-8 string currently allocates a redundant String through rb_str_encode: the condition !UTF8_P(value) || !USASCII_P(value) is always true. A lookup such as where(name: names).pluck(:id) makes one unnecessary String for every name.

Pass UTF-8 strings straight to SQLite, which still takes its own copy. Other encodings keep their existing conversion, including errors for invalid US-ASCII strings.

Objects and Callgrind instructions per 20-name Active Record lookup, Ruby 4.0.9, interpreter:

Before After
Objects 162 142
Instructions 1,092,175 1,083,184

These measurements use SQLite 3.40.1 and do not claim a request-level speedup.

@thomasklemm
thomasklemm force-pushed the thomasklemm/skip-utf8-bind-transcode branch 2 times, most recently from 16c9550 to f1709a4 Compare November 6, 2026 09:59
@thomasklemm
thomasklemm marked this pull request as ready for review November 6, 2026 12:54
@thomasklemm

Copy link
Copy Markdown
Contributor Author

The two failing jobs aren't from this change: sqlcipher (windows, mingw) and the Windows test_replacing_a_collation_releases_the_previous_comparator flake also fail on main. #773 fixes both, and I'll rebase this once it's in.

`bind_param` converts text that is neither UTF-16 nor BLOB to UTF-8
before calling `sqlite3_bind_text`, guarded by

    if (!UTF8_P(value) || !USASCII_P(value))

No string is both UTF-8 and US-ASCII, so the condition is always true,
and every UTF-8 string went through `rb_str_encode`. When the source
and target encodings match and no flags are given, `rb_str_encode`
returns a duplicate without checking the bytes, so the call only
allocated a String that SQLite then copied again under
`SQLITE_TRANSIENT`. A lookup such as `where(name: names).pluck(:id)`
made one of these Strings per name.

Convert only when the string is not UTF-8, as the guard did before
f9005a2 made it always true. UTF-8 strings, including ones with invalid
bytes, reach SQLite with the same bytes as before, and SQLite still
takes its own copy, so the caller's string is neither kept nor
modified. US-ASCII strings keep their conversion, which raises
`Encoding::InvalidByteSequenceError` for bytes above 0x7F; skipping it
too would bind those bytes silently. Other encodings convert as before.

The tests cover what the new path must keep doing. Invalid UTF-8
still binds as TEXT with its exact bytes, so UTF-8 strings must not be
validated or scrubbed on their way to SQLite. Invalid US-ASCII still
raises, so its conversion must not be skipped as well.
@thomasklemm
thomasklemm force-pushed the thomasklemm/skip-utf8-bind-transcode branch from f1709a4 to 64b0190 Compare November 13, 2026 07:14
@flavorjones

Copy link
Copy Markdown
Member

🤖 Thanks, I'll review!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants