[Zhiyue Li (lizhiyuell)]

• Does this PR have a descriptive title that could go in our release notes?
• Does this PR add any new dependencies?
• Does this PR modify any existing APIs?
• Is the change to the API backwards compatible?
• Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

None.

What does this implement/fix? Briefly explain your changes.

This PR introduces PathSeer, an efficient filtered graph search strategy for DiskANN's in-memory index, based on our research.
Building upon our research, we further adapt and improve PathSeer to work natively with DiskANN's existing graph index, without requiring any modifications to the index structure.

We evaluate PathSeer using the same two workloads as DiskANN's official filtered search benchmarks. PathSeer consistently achieves a better recall–throughput trade-off than existing methods, achieving 1.11–2.18× throughput (1.46× on average) at a target recall of 0.90 and 1.12–2.15× throughput (1.42× on average) at a target recall of 0.95, compared with the best-performing baseline for each workload.
PathSeer has also demonstrated its effectiveness in vector databases, as highlighted in Weaviate's blog.

The core idea of PathSeer is to selectively compute distances for vectors that do not satisfy the filter predicate, allowing them to serve as bridges during graph traversal, while opportunistically applying filtering to reduce unnecessary distance computations.
PathSeer employs a two-stage search strategy:

Stage 1: Compute distances for all direct neighbors while exploring a bounded number of second-hop neighbors using pre-filtering to discover more valid candidates.

Stage 2: Once the filtered result heap is full, stop second-hop exploration and apply parent-aware filtering, where a neighbor participates in distance computation if either the neighbor or its parent satisfies the filter predicate.

Any other comments?

Experimental setup

We follow DiskANN's official evaluation and use two publicly available datasets.

• Caselaw: We directly use the official Low (0.5%–1%), Medium (1%–10%), and High (10%–50%) filter-selectivity query sets.
• YFCC: Following the official evaluation, we construct three query sets based on filter selectivity: Low (0.01%–0.1%), Medium (0.5%–3.7%), and High (11.4%–33.8%).

We vary the search parameter L to evaluate the recall–throughput trade-off of PathSeer and existing filtered search methods. For each query, we precompute an allow_ids bit vector and extend diskann-benchmark to accept these bit vectors. The reported throughput measures in-memory vector search only, excluding the preprocessing time for constructing filter bit vectors. All experiments use Top-k = 10, consistent with the official benchmark.

Experimental results

The results demonstrate that PathSeer consistently achieves comparable or better performance than existing methods. Compared with the best-performing baseline for each workload, PathSeer achieves 1.11–2.18× throughput (1.46× on average) at a target recall of 0.90 and 1.12–2.15× throughput (1.42× on average) at a target recall of 0.95.

[Zhiyue Li (lizhiyuell)]

@microsoft-github-policy-service agree

[Copilot]

🟡 Changes recommended

Accepted start points can suppress valid results, and unannotated unwrap calls violate the CI lint configuration.

3 open issues

• Replace heap unwraps with lint-compliant presence checks · New
• Start-point matches consume result slots before postprocessing · New
• Recheck heap fullness before phase-two exploration · New

What changed in this PR

Adds PathSeer as a filtered-search strategy for DiskANN’s in-memory graph index, without changing the stored graph structure.

Changes:

• Implements two-phase traversal with bounded second-hop exploration and parent-aware filtering.
• Adds a provider interface for synchronous adjacency access and implements it for in-memory graphs.
• Exposes the strategy through the search API and adds focused algorithm and integration tests.

File	Description
diskann/​src/​provider.rs	Defines the PathSeerGraph adjacency interface.
diskann/​src/​graph/​test/​provider.rs	Implements adjacency access for the test provider.
diskann/​src/​graph/​test/​cases/​pathseer.rs	Tests second-hop discovery, empty matches, and phase-two filtering.
diskann/​src/​graph/​test/​cases/​mod.rs	Registers PathSeer tests.
diskann/​src/​graph/​search/​pathseer.rs	Implements PathSeer traversal and result collection.
diskann/​src/​graph/​search/​mod.rs	Exports and documents the strategy.
diskann/​src/​graph/​index.rs	Adds PathSeer to the search API documentation.
diskann-providers/​src/​model/​graph/​provider/​async_/​inmem/​provider.rs	Implements the interface for the in-memory provider.
diskann-providers/​src/​index/​diskann_async.rs	Adds a full-precision in-memory integration test.

_{Review effort: Balanced}

[Mark Hildebrand (hildebrandmw)]

Thanks for the contribution Zhiyue Li (@lizhiyuell), the results look promising! Magdalen Dobson Manohar (@magdalendobson) - do you have any thoughts here?

[Magdalen Dobson Manohar (magdalendobson)]

Thanks for this PR, Zhiyue Li (@lizhiyuell)!

I agree that the algorithm is interesting and the results look promising. Thanks for using the datasets that we use in our previous evaluations and for submitting reasonably high-quality code that is easy to follow. We are interested in working towards including at least some of your contribution on our main branch.

I looked over the code and algorithm descriptions, and from what I can tell there are a few separate ideas going on:

1. Explore both matching and non-matching candidates in graph traversal, and use the furthest distance from the Lth matching candidate for the termination criteria.
2. Explore two-hop candidates when L predicate-satisfying candidates have not yet been found.
3. When two-hop candidates are explored, limit exploration to a small number of candidates (here the budget appears set to max_degree_with_slack.
4. Once L predicate-satisfying candidates have been found, tweak traversal by only traversing edges where at least one endpoint is filter-satisfying. This essentially means changing the algorithm so that the out-neighbors of a non-predicate-satisfying node may only be visited if they are predicate-satisfying.

I think it would be helpful to see if you agree with this characterization, and if so, could we work on benchmarking these ideas separately? Since some of these ideas, especially the phase differentiation, add a lot of code complexity, I think the next step for our understanding would be to benchmark these ideas separately and understand how much they contribute to the speedup. I'm not sure if you have looked at our diskann-benchmark crate, but adding your algorithm there and using it to do those experiments would be a smooth way to do this. We would want your algorithm included in that crate for any final versions of this PR anyway. Furthermore, since I don't know the details of the machine you used for benchmarking, including distance comparisons as a machine-agnostic measure would be really helpful. I would also like to know the parameters you used to build the graph.

A separate note. Since so many of these transitions and stopping criteria rely on finding L predicate-satisfying candidates, I do worry a bit that there could be some bad edge-case behavior when L is very high. It's a bit hard to read the graphs but it looks like in your experiments L never reaches higher than around 300. Have you tested higher values of L and/or included any safeguards I missed here?

Thanks again, and let me know if you are open to the exploration that I propose.