Repository navigation
SQLite rowcount is corrupted when combining UPDATE RETURNING w/ table that is dropped and recreated #93421
Description
version info:
$ python
Python 3.10.0 (default, Nov 5 2021, 17:23:47) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.36.0'
we have a test suite that creates a table, runs some SQL, then drops it. if multiple tests run that each perform this task, if the same SQLite connection is used, rowcount starts returning "0". Seems to also require RETURNING to be used. Full demonstration:
import os
import sqlite3
def go():
"""function creates a new table, runs INSERT/UPDATE, drops table,
commits connection.
"""
# create table
cursor = conn.cursor()
cursor.execute(
"""CREATE TABLE some_table (
id INTEGER NOT NULL,
value VARCHAR(40) NOT NULL,
PRIMARY KEY (id)
)
"""
)
cursor.close()
conn.commit()
# run operation
cursor = conn.cursor()
cursor.execute(
"INSERT INTO some_table (id, value) VALUES (1, 'v1')"
)
ident = 1
cursor.execute(
"UPDATE some_table SET value='v2' "
"WHERE id=? RETURNING id",
(ident,),
)
new_ident = cursor.fetchone()[0]
assert ident == new_ident
assert cursor.rowcount == 1, cursor.rowcount
cursor.close()
# drop table
cursor = conn.cursor()
cursor.execute("DROP TABLE some_table")
cursor.close()
conn.commit()
if os.path.exists("file.db"):
os.unlink("file.db")
# passes
conn = sqlite3.connect("file.db")
go()
# run again w/ new connection (same DB), passes
conn = sqlite3.connect("file.db")
go()
print("FAILURE NOW OCCURS")
# run again w/ same connection, fails
go()on the third run, where we ran the "test" on the same connection twice, it fails:
$ python test3.py
FAILURE NOW OCCURS
Traceback (most recent call last):
File "/home/classic/dev/sqlalchemy/test3.py", line 62, in <module>
go()
File "/home/classic/dev/sqlalchemy/test3.py", line 39, in go
assert cursor.rowcount == 1, cursor.rowcount
AssertionError: 0
it would appear there's some internal caching of table state that needs to be cleared.
Metadata
Metadata
Assignees
Projects
- Status
Activity
AA-Turner commentedon Jun 1, 2022
cc: @erlend-aasland
A
erlend-aasland commentedon Jun 2, 2022
Thanks for the report!
Internally, we simply use the function
sqlite3_changesto setrowcount1, implying that we do not calculaterowcountourself; the value comes straight from the underlying SQLite library. Ifsqlite3_changesreturns zero,rowcountwill be zero.I've verified that this only happens with
UPDATE ... RETURNINGstatements, which also implies that the behaviour we are observing comes from the underlying SQLite library. I suggest to report this on the SQLite forum (they do not have a bug tracker).Note: I did some tests where I used
sqlite3_total_changes()in addition tosqlite3_changes(), and they revealed that the number of total changes on the database connection did not increase whithUPDATE ... RETURNINGstatements, only with normalUPDATEstatements.Closing this as not-a-bug.
See also:
Footnotes
Assuming that you've supplied a valid statement, and that no database errors were raised by SQLite. In case of errors,
rowcountwill normally be set to-1. ↩25 remaining items
gh-93421: Update sqlite3 cursor.rowcount only after SQLITE_DONE (#93526)
pythongh-93421: Update sqlite3 cursor.rowcount only after SQLITE_DONE (…
[3.10] pythongh-93421: Update sqlite3 cursor.rowcount only after SQLI…
erlend-aasland commentedon Jun 8, 2022
Fixed by:
erlend-aasland commentedon Jun 8, 2022
Thanks for the report, and for pushing this through, Mike. Thanks for reviewing, Ma Lin!
gh-93421: Update sqlite3 cursor.rowcount only after SQLITE_DONE (GH-9…
[3.10] gh-93421: Update sqlite3 cursor.rowcount only after SQLITE_DONE (
erlend-aasland commentedon Jun 8, 2022
We'd still only update
.rowcountafter SQLITE_DONE, so when the statement is stepped through, you'll have the correct row count, caching or not.That is a separate issue (and it has always been). It is not very large or complex fix; we just need to step through a statement for each loop iteration in
_pysqlite_query_executeifmultipleis true, kinda like we do inexecutescript. It is a bug, though, so it would be quite ok to backport it through 3.10.However, as you point out, such code is probably rare; it does not make sense with
UPDATE ... RETURNINGinexecutemany.Revert 30935 defer bpo45162 to 312 (#94285)
rt121212121 commentedon Dec 1, 2022
As a data point, I have a MacOS user who is using 3.10.8, and getting the
rowcount0 problem. From what I understand the 3.10 backport was merged in June and 3.10.8 was tagged as of October, so should have the fix from what I understand.This is an
UPDATE ... RETURNINGexecute that assertsrowcount == 1, which fails because it is unexpectedly 0.erlend-aasland commentedon Dec 2, 2022
Would you mind posting a minimal reproducer?
Python 3.10.8 should have the fix, yes.
rt121212121 commentedon Dec 5, 2022
The reproducer from the top does not work for him. It'll happen in the course of operation of the larger application repeatedly however. No idea how to reproduce it outside of that.
rt121212121 commentedon Jan 18, 2023
Finally had the problem happen myself. It appears to be a different problem than tables being dropped or recreated. Repro on the above linked issue.