[diskann-garnet] Implement XVCREATE, XVIMPORT by metajack · Pull Request #1455 · microsoft/DiskANN

Jack Moffitt (metajack) · 2026-09-30T16:21:52Z

This PR adds direct import of externally stored diskann graphs to Garnet. This can be useful for interoperating with indices created with other DiskANN systems.

It adds two new commands, XVCREATE to create a vector set without doing a VADD, and XVIMPORT to directly import index terms to a vector set. To make quantizer state compatible, XVCREATE allows directly specifying the quantizer state.

Finalization of import checks that all required terms are present and have sensical sizes, but does not verify quantize vectors match unquantized vectors or that the neighbor lists make sense. It is expected that the graph is correctly constructed in the external system.

This currently requires that the internal ID 0 not be used in the graph since it will get used by the start point once the graph is imported.

Copilot

Copilot review overview

🟡 Changes recommended

Arbitrary-ID imports, malformed neighbor counts, reserved-ID mappings, and quantizer-state replacement can currently corrupt or fail imported indexes.

Review effort: Balanced
Findings: 4 · 2

Open (6)

Expand all intermediate blocks for arbitrary imported IDs · New
Reject quantizer changes after index initialization · New
Reject neighbor counts exceeding max_degree · New
Reject external-ID mappings to reserved ID zero · New
Document all operations that disable importing · New
Remove stray Add prefix from field label · New

What changed in this PR

Adds bulk import support for externally generated DiskANN graphs through new Garnet FFI operations.

Changes:

Adds quantizer-state injection and term import/finalization APIs.
Persists import eligibility and validates imported graph terms.
Adds import tests, documentation, and a patch version bump.

File	Description
`Cargo.lock`	Updates the package version.
`diskann-garnet/Cargo.toml`	Bumps the crate version.
`diskann-garnet/diskann-garnet.nuspec`	Bumps the NuGet version.
`diskann-garnet/README.md`	Documents bulk import usage.
`diskann-garnet/docs/ffi-design.rs`	Documents import FFI APIs.
`diskann-garnet/docs/data-design.md`	Documents imported-term storage.
`diskann-garnet/src/dyn_index.rs`	Exposes import operations through `DynIndex`.
`diskann-garnet/src/ffi_tests.rs`	Adds import and FFI validation tests.
`diskann-garnet/src/fsm.rs`	Adds explicit internal-ID claiming.
`diskann-garnet/src/garnet.rs`	Adds term conversion and existence checks.
`diskann-garnet/src/lib.rs`	Exposes import-related C APIs.
`diskann-garnet/src/provider.rs`	Implements import gating, validation, and finalization.
`diskann-garnet/src/quantization.rs`	Supports externally supplied quantizer state.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Codecov Comments Bot (codecov-commenter) · 2026-09-30T17:09:27Z

Codecov Report

❌ Patch coverage is 91.23476% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.97%. Comparing base (18c9cce) to head (7a2e72c).

Files with missing lines	Patch %	Lines
diskann-garnet/src/provider.rs	91.29%	66 Missing ⚠️
diskann-garnet/src/dyn_index.rs	71.60%	23 Missing ⚠️
diskann-garnet/src/lib.rs	80.70%	22 Missing ⚠️
diskann-garnet/src/quantization.rs	91.11%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1455      +/-   ##
==========================================
+ Coverage   90.94%   90.97%   +0.02%     
==========================================
  Files         586      586              
  Lines      118391   119512    +1121     
==========================================
+ Hits       107675   108726    +1051     
- Misses      10716    10786      +70

Flag	Coverage Δ
miri	`90.97% <91.23%> (+0.02%)`	⬆️
unittests	`90.67% <91.23%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-garnet/src/fsm.rs	`98.25% <100.00%> (+3.73%)`	⬆️
diskann-garnet/src/test_utils.rs	`98.69% <100.00%> (+0.07%)`	⬆️
diskann-garnet/src/quantization.rs	`96.83% <91.11%> (-0.09%)`	⬇️
diskann-garnet/src/lib.rs	`93.77% <80.70%> (-0.63%)`	⬇️
diskann-garnet/src/dyn_index.rs	`79.42% <71.60%> (-4.16%)`	⬇️
diskann-garnet/src/provider.rs	`88.17% <91.29%> (+1.07%)`	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Mark Hildebrand (hildebrandmw)

My biggest concern is about the dynamic disabling of imports. Adding/requiring that dynamic check on all function seems somewhat hard to maintain - seems like it could race and actually successfully complete before any call to finish_import.

Harsha Simhadri (harsha-simhadri)

It would be nice to follow up with more description and testing of workflows that use this. Creation, tear down from partial import, making sure reimport does not break things, documenting what happens when you try to modify after importing etc.

@arrayka

# DiskANN v0.61.0 ## Highlights - **Quantization:** Added 3-bit spherical quantization with AVX2 acceleration, faster spherical encoding, and NEON float32 MinMax kernels. - **In-memory indexing (experimental):** Added product quantization and 1-8-bit scalar quantization, unified quantized representations, and AArch64 prefetching. - **Multi-vector search:** Added `i8 x i8` and MinMax8 x MinMax4 MaxSim kernels. - **Fixes:** Preserve virtual start-point out-edges when writing disk indexes, correctly round-trip scalar quantization configurations without standard deviation, and support zero-column mutable matrix iterators. ## Breaking changes and upgrading Changes are concentrated in quantization APIs and experimental/tooling crates. * PQ training (#1467) - Training now returns `BasicTable`. `SimplePivots` was removed. Replace `flatten()` with `view_pivots().as_slice()` and use `pivots_for(chunk)` for per-chunk access. * PQ distance tables (#1458) - Change `ProcessInto` bounds to `ProcessInto<f32>` and calls from `process_into::<Metric>(...)` to `process_into::<Metric, f32>(...)`. * MaxSim kernels (#1429) - Generic kernel/erasure code now requires `T: MaxSimElement` and score buffers of `T::Score` (`f32` for floats, `i32` for `i8`). Update `build_max_sim` error handling from `NotSupported` to `BuildMaxSimError`. * `diskann-inmem` (#1485) - Replace `repr::Spherical` with `repr::Quantized`. Wrap the quantizer in `repr::quantized::spherical::Spherical::new(...)` when configuring it. Configuration and `Rerank` types now live under `repr::quantized`. * `diskann-benchmark-runner` (#1475) - `input::Registered` and public `Registry::input` were removed. Use the concrete `Input` implementation's `tag()` and `example()` methods instead. ## All Changes * Fix serialization round trip for QuantizationType::SQ without standard deviation by @arrayka in #1461 * i8 x i8 MaxSim Kernels by @suri-kumkaran in #1429 * Adding `subview_mut` to `MatrixMut` by @arkrishn94 in #1463 * Add InsertResult::SuccessUpdate by @metajack in #1449 * [diskann-quantization] PQ Infrastructure for `diskann-inmem` by @hildebrandmw in #1458 * [diskann-quantization] Teach PQ training to return a `BasicTable`. by @hildebrandmw in #1467 * [deps] Run `cargo-update` by @hildebrandmw in #1452 * Support zero-column mutable matrix iterators by @arrayka in #1462 * [diskann-quantization] Use `f32::NAN` instead of `core::f32::NAN`. by @hildebrandmw in #1471 * [diskann-wide] Add `V4::new_checked_miri_uncached`. by @hildebrandmw in #1480 * [diskann-wide] Add maddubs intrinsic wrappers to V3 and V4 by @partychen in #1479 * Bump the github-actions group across 1 directory with 5 updates by @dependabot[bot] in #1482 * [benchmark-runner] Simplify Input representation by @hildebrandmw in #1475 * [diskann-quantization] Optimize spherical encoding scale search by @xinyuwen2 in #1478 * [diskann-garnet] Implement XVCREATE, XVIMPORT by @metajack in #1455 * [diskann-garnet] Support configurable start points and allow importing start points by @metajack in #1464 * [diskann-garnet] Add spherical 2-bit and 4-bit quantization by @metajack in #1465 * [deps] Address depandabot issue 32 by @hildebrandmw in #1496 * [diskann-disk] Preserve virtual start point out-edges when writing the disk index by @doliawu in #1483 * [diskann-quantization] Add Neon F32 Minmax Kernels by @pfoxARM in #1431 * [diskann-garnet] Add macOS arm64 support to NuGet publishing by @metajack in #1495 * [diskann-disk] Require start points at index initialization. by @hildebrandmw in #1497 * Expose metric-aware quantizer centroid distance by @cherryrhea in #1395 * [diskann-inmem] Add support for Product Quantization by @hildebrandmw in #1469 * Adds aarch64 prefetching into diskann-vector and diskann-inmem by @pfoxARM in #1408 * [diskann-inmem] Add scalar quantization by @hildebrandmw in #1484 * Bump the github-actions group with 3 updates by @dependabot[bot] in #1505 * [diskann-inmem] Spherical Quantization Round 2 by @hildebrandmw in #1485 * [diskann-quantization] Add SphericalQ 3-bit support and AVX2 kernel by @xinyuwen2 in #1474 * [muilt-vetor] Add standalone MinMax8 x MinMax4 MaxSim kernel by @partychen in #1494 ## New Contributors * @cherryrhea made their first contribution in #1395 **Full Changelog**: v0.60.0...v0.61.0 Co-authored-by: Mark Hildebrand <mhildebrand@microsoft.com>

Jack Moffitt (metajack) requested review from a team and a balanced review from Copilot September 30, 2026 16:21

Copilot started reviewing on behalf of Jack Moffitt (metajack) September 30, 2026 16:22

View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

github-advanced-security Bot found potential problems Sep 30, 2026

View reviewed changes

Jack Moffitt (metajack) force-pushed the push-ttwwvqlmswws branch from 50b606d to 84d419c Compare September 30, 2026 16:51

Jack Moffitt (metajack) added this pull request to stack #1466 October 6, 2026 19:44

Jack Moffitt (metajack) force-pushed the push-ttwwvqlmswws branch 2 times, most recently from d99ff9a to dd739d1 Compare October 9, 2026 16:25

Mark Hildebrand (hildebrandmw) reviewed Oct 12, 2026

View reviewed changes

Jack Moffitt (metajack) force-pushed the push-ttwwvqlmswws branch from dd739d1 to 44a1107 Compare October 12, 2026 19:37

github-advanced-security Bot found potential problems Oct 14, 2026

View reviewed changes

Comment thread diskann-garnet/src/ffi_tests.rs Dismissed

Mark Hildebrand (hildebrandmw) approved these changes Oct 14, 2026

View reviewed changes

Harsha Simhadri (harsha-simhadri) approved these changes Oct 15, 2026

View reviewed changes

Jack Moffitt (metajack) force-pushed the push-ttwwvqlmswws branch from 255fb96 to e93ae05 Compare October 15, 2026 14:27

Jack Moffitt added 2 commits October 15, 2026 14:54

[diskann-garnet] Implement XVCREATE, XVIMPORT

20ddae3

address review feedback

7a2e72c

Jack Moffitt (metajack) force-pushed the push-ttwwvqlmswws branch from e93ae05 to 7a2e72c Compare October 15, 2026 14:54

Mark Hildebrand (hildebrandmw) approved these changes Oct 15, 2026

View reviewed changes

Jack Moffitt (metajack) merged commit 7880822 into main Oct 15, 2026
29 checks passed

Jack Moffitt (metajack) deleted the push-ttwwvqlmswws branch October 15, 2026 17:19

Mark Hildebrand (hildebrandmw) mentioned this pull request Oct 26, 2026

[version] Set to version 0.61.0 #1511

Merged

Conversation

Jack Moffitt (metajack) commented Sep 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Codecov Comments Bot (codecov-commenter) commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Mark Hildebrand (hildebrandmw) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Harsha Simhadri (harsha-simhadri) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Codecov Comments Bot (codecov-commenter) commented Sep 30, 2026 •

edited

Loading