Repository navigation
Conversation
Four bugs in the IVF index could corrupt data or return wrong results: - Slot allocation was append-only (slot = n_vectors), so after any delete left an interior hole, a later insert overwrote a live vector — corrupting its data and dropping the row from KNN results. Allocate the first free slot from the validity bitmap instead. - assign-vectors walked the centroid array with a D-byte stride instead of D*sizeof(float), so every centroid past index 0 was garbage, and it mapped the nearest-centroid array index directly to a centroid id (wrong for the non-contiguous ids that set-centroid allows). It also treated quantized cell blobs as float32. Now operates entirely in quantized space with the correct stride and an id lookup array. - clear-centroids re-inserted raw float32 bytes into quantized cells without re-quantizing, corrupting every vector on a quantized index. - set-centroid stored full-precision centroids that mismatched the quantized cell vector size, breaking subsequent inserts. Now stores the quantized representation. Also make ivf_distance dispatch int8 to L1/cosine/L2 by the column's distance metric (was always L2), and remove a dead undefined-behavior ivf_ensure_stmt call in ivf_cell_find_or_create. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tests/test-unit.c had been mangled by a bad rebase — function bodies were interleaved and the file never compiled, so its IVF/rescore/DiskANN unit tests were never actually run. Reconstruct it from git history. While fixing it: - tests/sqlite-vec-internal.h re-declares struct VectorColumnDefinition for the test translation unit; its Vec0RescoreConfig was missing the oversample_search field present in sqlite-vec.c. The size mismatch shifted the ivf/diskann fields and made vec0_parse_vector_column scribble past the caller's struct (stack smash). Add the missing field. - The test-unit target now defines SQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 so the IVF unit tests link and run. - The rescore int8 quantizer is a fixed [-1,1] -> [-128,127] map, not the adaptive min/max the historical test assumed; update those assertions. - Add the #included index sources (sqlite-vec-ivf*.c, sqlite-vec-rescore.c, sqlite-vec-diskann.c) as prerequisites of the loadable/static targets so edits to them trigger a rebuild. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regression tests covering each IVF bug fix: - test-ivf-mutations.py: insert-after-delete slot reuse (no overwrite), randomized insert/delete churn consistency, and assign-vectors correctness including non-contiguous centroid ids. - test-ivf-quantization.py: clear-centroids / set-centroid / assign-vectors on int8 and binary quantized indexes preserve vectors and recall. New test-ann-embeddings.py exercises IVF, DiskANN, and rescore end to end with real text embeddings from a local llama-server endpoint (localhost:2235), checking recall against exact brute-force results. The whole module skips when the server is unreachable, so it is safe to run in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an "Approximate nearest neighbor (ANN) indexes" section describing the rescore, ivf, and diskann index types, their compile flags, and usage. Include a latency/speedup/recall benchmark table at both 256 and 1024 dimensions and a discussion of how the trade-offs shift as dimensionality grows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Oct 23, 2026
MayCXC
added a commit
to MayCXC/sqlite-vec
that referenced
this pull request
Oct 23, 2026
A build compiled with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 can declare a
float vector column `indexed by ivf()`, which clusters its rows around nlist
centroids (128 by default) with k-means and keeps them in cells of up to 64
rows, each cell holding rows of one centroid. A KNN query reads the cells of
the nprobe centroids nearest its vector (10 by default) and the cells of the
rows no clustering has placed yet, so until compute-centroids runs it reads
every row. With quantizer=int8 or quantizer=binary, the cells and centroids
hold the vectors quantized, the float32 vectors are kept by rowid besides, and
a query ranks the k * oversample rows nearest by quantized distance by their
exact distances. The commands compute-centroids, compute-centroids:{...},
set-centroid:N, assign-vectors, clear-centroids and nprobe=N build and tune
the index. Its tables are _ivf_centroidsNN, _ivf_cellsNN, _ivf_rowid_mapNN
and _ivf_vectorsNN, with the ivf_trained_N row of _info, named as upstream
names them. The prebuilt binaries are built without it.
Ported from asg017's sqlite-vec-ivf.c and sqlite-vec-ivf-kmeans.c as its main
has them at 04d28bd (merged in e2c38f3), with the refusal of a binary
quantizer on dimensions not divisible by 8 (4bee883), the refusal of a vector
UPDATE (9df59b4), auxiliary columns (b7fc459), the command column (6e2c4c6),
the rename of its shadow tables (8b81f40), the finalizing of every cached
statement on commit (fe94171), and the fixes of the unmerged asg017#313: an
insert takes the first free slot of a cell, where upstream takes the slot
after its count and overwrites a live row once a delete leaves a hole;
assign-vectors reads the centroids at the right stride and by their ids, which
set-centroid can leave apart; clear-centroids and set-centroid keep a
quantized column's cells and centroids quantized; and the int8 quantizer ranks
by the column's metric, where upstream ranks by L2.
Where this port differs from upstream:
- ivf_knn() takes rescore_knn()'s arguments and results: a rowid filter
ranks the rows it names by their exact distances, distance constraints
filter the rows the query read, and MMR reranks them. Upstream reads only
the query vector and k, so it returns rows outside a rowid filter or a
distance constraint.
- A quantized column's point query and full scan read its float32 vectors
from _ivf_vectorsNN. Upstream returns the quantized bytes, and its point
query reads a float32 vector's length past them.
- Whether a column has centroids is read from _info at each use. Upstream
caches it per connection, so a connection that queried before another one
ran compute-centroids reads only the cells of no centroid, which are gone,
and finds no rows.
- The cells are found by centroid through the index of a
UNIQUE (centroid_id, cell_id) constraint on a cell_id INTEGER PRIMARY KEY,
which ALTER TABLE RENAME renames along with its table. Upstream creates an
index named after the table, which keeps its name through a rename, so a
table created later under the old name fails to create its own; nothing
inside xRename can drop an index, since OP_Destroy refuses while the ALTER
TABLE statement reads. Every statement on the cells names its columns and
addresses a cell by rowid, so either implementation opens the other's
tables.
- Only float columns can be indexed by ivf(). Upstream accepts int8 and bit
columns and sizes their cells for float32 vectors, so it reads past an
int8 vector and ranks a row's own vector at a distance such as 6.6e10.
- Errors reach the statement, so an interrupt stays an interrupt. Upstream
ignores the results of its writes to the cells and of its blob reads and
writes, and runs compute-centroids in a SAVEPOINT it opens inside xUpdate,
which SQLite refuses while the statement writes. An interrupted command
fails with "interrupted" and the statement's rollback undoes it; upstream
reports "SQL logic error", and the connection's KNN queries then find no
rows. A cell or centroid blob whose size the column's options do not give
fails with a message naming it.
- A cached statement is reset as soon as it has run, so VACUUM works after a
KNN query.
- The commands apply to every IVF column of the table; upstream applies them
to the first. compute-centroids:{...} is read with SQLite's json_each(),
and its keys nlist, max_iterations and seed take integers in range; upstream
finds them with strstr() and reads them with atoi(). The commands take
digits only, and CREATE options and their values must be spelled in full;
upstream accepts any prefix, as in `quant=int8`.
- k-means indexes its arrays with size_t; upstream multiplies int indexes,
which overflow past 2^31 floats.
- optimize, which upstream lacks, leaves an IVF column's rows, which are
keyed by rowid, as they are.
The IVF code shares the DiskANN code's helpers, which take names for both:
diskann_stmt() and diskann_exec_rowid() become vec0_cached_stmt() and
vec0_exec_i64(), and vec0_parse_diskann_int() vec0_parse_option_int().
The tests are asg017#313's test-ivf.py, test-ivf-mutations.py,
test-ivf-quantization.py and test-ivf-rename.py, upstream's IVF fuzz targets
with their dictionary, built with the IVF index compiled in (upstream's fuzz
Makefile builds them without it, so they create no IVF table), and the IVF
cases of asg017#313's test-unit.c, which call the functions through
tests/sqlite-vec-internal.h. asg017#313's test-ann-embeddings.py needs an embedding
server on localhost and is not ported. New tests check that a query reading
every cell answers as a flat column does, constraint for constraint, the
float32 vectors of a quantized column, a second connection after
compute-centroids, a rename followed by a table under the old name, the
refused columns and options, an interrupted command, the commands on every
IVF column, the command errors, the compute-centroids options and seed,
corrupt cells, INSERT OR REPLACE, VACUUM after a KNN query, optimize beside a
flat column, and k-means in the unit tests. conftest.py skips the test-ivf
files when vec_debug() names no ivf build flag, as upstream's does; make
test-unit passes CPPFLAGS to the compiler, and a CI job builds the IVF index
in and runs every suite.
MayCXC
added a commit
to MayCXC/sqlite-vec
that referenced
this pull request
Oct 24, 2026
A build compiled with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 can declare a
float vector column `indexed by ivf()`, which clusters its rows around nlist
centroids (128 by default) with k-means and keeps them in cells of up to 64
rows, each cell holding rows of one centroid. A KNN query reads the cells of
the nprobe centroids nearest its vector (10 by default) and the cells of the
rows no clustering has placed yet, so until compute-centroids runs it reads
every row. With quantizer=int8 or quantizer=binary, the cells and centroids
hold the vectors quantized, the float32 vectors are kept by rowid besides, and
a query ranks the k * oversample rows nearest by quantized distance by their
exact distances. The commands compute-centroids, compute-centroids:{...},
set-centroid:N, assign-vectors, clear-centroids and nprobe=N build and tune
the index. Its tables are _ivf_centroidsNN, _ivf_cellsNN, _ivf_rowid_mapNN
and _ivf_vectorsNN, with the ivf_trained_N row of _info, named as upstream
names them. The prebuilt binaries are built without it.
Ported from asg017's sqlite-vec-ivf.c and sqlite-vec-ivf-kmeans.c as its main
has them at 04d28bd (merged in e2c38f3), with the refusal of a binary
quantizer on dimensions not divisible by 8 (4bee883), the refusal of a vector
UPDATE (9df59b4), auxiliary columns (b7fc459), the command column (6e2c4c6),
the rename of its shadow tables (8b81f40), the finalizing of every cached
statement on commit (fe94171), and the fixes of the unmerged asg017#313: an
insert takes the first free slot of a cell, where upstream takes the slot
after its count and overwrites a live row once a delete leaves a hole;
assign-vectors reads the centroids at the right stride and by their ids, which
set-centroid can leave apart; clear-centroids and set-centroid keep a
quantized column's cells and centroids quantized; and the int8 quantizer ranks
by the column's metric, where upstream ranks by L2.
Where this port differs from upstream:
- ivf_knn() takes rescore_knn()'s arguments and results: a rowid filter
ranks the rows it names by their exact distances, distance constraints
filter the rows the query read, and MMR reranks them. Upstream reads only
the query vector and k, so it returns rows outside a rowid filter or a
distance constraint.
- A quantized column's point query and full scan read its float32 vectors
from _ivf_vectorsNN. Upstream returns the quantized bytes, and its point
query reads a float32 vector's length past them.
- Whether a column has centroids is read from _info at each use. Upstream
caches it per connection, so a connection that queried before another one
ran compute-centroids reads only the cells of no centroid, which are gone,
and finds no rows.
- The cells are found by centroid through the index of a
UNIQUE (centroid_id, cell_id) constraint on a cell_id INTEGER PRIMARY KEY,
which ALTER TABLE RENAME renames along with its table. Upstream creates an
index named after the table, which keeps its name through a rename, so a
table created later under the old name fails to create its own; nothing
inside xRename can drop an index, since OP_Destroy refuses while the ALTER
TABLE statement reads. Every statement on the cells names its columns and
addresses a cell by rowid, so either implementation opens the other's
tables.
- Only float columns can be indexed by ivf(). Upstream accepts int8 and bit
columns and sizes their cells for float32 vectors, so it reads past an
int8 vector and ranks a row's own vector at a distance such as 6.6e10.
- Errors reach the statement, so an interrupt stays an interrupt. Upstream
ignores the results of its writes to the cells and of its blob reads and
writes, and runs compute-centroids in a SAVEPOINT it opens inside xUpdate,
which SQLite refuses while the statement writes. An interrupted command
fails with "interrupted" and the statement's rollback undoes it; upstream
reports "SQL logic error", and the connection's KNN queries then find no
rows. A cell or centroid blob whose size the column's options do not give
fails with a message naming it.
- A cached statement is reset as soon as it has run, so VACUUM works after a
KNN query.
- The commands apply to every IVF column of the table; upstream applies them
to the first. compute-centroids:{...} is read with SQLite's json_each(),
and its keys nlist, max_iterations and seed take integers in range; upstream
finds them with strstr() and reads them with atoi(). The commands take
digits only, and CREATE options and their values must be spelled in full;
upstream accepts any prefix, as in `quant=int8`.
- k-means indexes its arrays with size_t; upstream multiplies int indexes,
which overflow past 2^31 floats.
- optimize, which upstream lacks, leaves an IVF column's rows, which are
keyed by rowid, as they are.
The IVF code shares the DiskANN code's helpers, which take names for both:
diskann_stmt() and diskann_exec_rowid() become vec0_cached_stmt() and
vec0_exec_i64(), and vec0_parse_diskann_int() vec0_parse_option_int().
The tests are asg017#313's test-ivf.py, test-ivf-mutations.py,
test-ivf-quantization.py and test-ivf-rename.py, upstream's IVF fuzz targets
with their dictionary, built with the IVF index compiled in (upstream's fuzz
Makefile builds them without it, so they create no IVF table), and the IVF
cases of asg017#313's test-unit.c, which call the functions through
tests/sqlite-vec-internal.h. asg017#313's test-ann-embeddings.py needs an embedding
server on localhost and is not ported. New tests check that a query reading
every cell answers as a flat column does, constraint for constraint, the
float32 vectors of a quantized column, a second connection after
compute-centroids, a rename followed by a table under the old name, the
refused columns and options, an interrupted command, the commands on every
IVF column, the command errors, the compute-centroids options and seed,
corrupt cells, INSERT OR REPLACE, VACUUM after a KNN query, optimize beside a
flat column, and k-means in the unit tests. conftest.py skips the test-ivf
files when vec_debug() names no ivf build flag, as upstream's does; make
test-unit passes CPPFLAGS to the compiler, and a CI job builds the IVF index
in and runs every suite.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains fixes to the vector indexes code, worked out with Fable and Opus 4.8, and adds a chapter on using the indexes to the README file. It contains commits of different stages of the work, so you can cherry pick those that you're comfortable with.