镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

[Bug] In-memory union and intersection fail to combine equal scan rows #3396

Closed

Description

@imbajin

Bug Type

Logic / incorrect query results.

Environment

Development source reviewed in PR #3395, Java 17. Reproduced with the production Store iterators in a standalone JVM; no running Store cluster or native RocksDB instance was used.

Expected and actual behavior

BackendColumn.equals() compares byte-array contents, but the class has no matching hashCode(). Real RocksDB scans allocate a new BackendColumn on each next() call. The in-memory union/intersection maps therefore do not reliably combine content-equal rows from independent scans.

With 100 keys, each appearing twice as separately allocated, byte-equal columns:

Iterator Expected rows Observed rows
Intersection, size = 2 100 0
Union, below its sort threshold 100 200

A separate probe also finds two equal newly allocated columns occupying two entries in a HashSet.

Reproduction

  1. Generate 100 keys as four-byte integer arrays; use the same value bytes for each occurrence of a key. Generate two occurrences per key, calling BackendColumn.of() with new arrays for every occurrence.
  2. Populate an IntersectionWrapper from one pass, with a collision-free key extractor for this dataset, and call proc().
  3. Create a fresh second pass of those rows. Read IntersectionFilterIterator(..., wrapper, 2) to exhaustion; repeat using UnionFilterIterator with the existing BackendColumn serializer and name comparator.
  4. Compare the result counts with the table above. Reusing the same Java object for repeated rows hides the defect.

Relevant implementation

Focused follow-up

Fix the stable map-key/equality contract and add regressions with a fresh object per scanned row. Explicitly preserve the intended identity rule: current BackendColumn equality compares name and value, while alternate union and sorted paths group by name. Adding a content hash alone must not be treated as proof that those grouping rules are equivalent.

This predates PR #3395 and is tracked separately so its iterator probe/cleanup fixes need not grow in scope. Original review discussion.

Issue search performed for BackendColumn/hashCode and UnionFilterIterator; no matching existing issue was found.

Activity

  1. vaidikgarg-bit commented on Oct 23, 2026

    @vaidikgarg-bit
    Contributor

    Hi @imbajin, I'd like to work on this.

    My plan is to add a hashCode() to RocksDBSession.BackendColumn that matches its equals(), hashing both the name and value bytes. I'll also add regression tests for IntersectionFilterIterator and for UnionFilterIterator below its sort threshold. Like your reproduction, the tests will build a fresh BackendColumn for every scanned row.

    I'd leave the grouping question (equality on name + value vs. grouping by name on the other union paths) and the SortShuffle path from #3397 out of this PR. Does that scope sound right?

  2. added a commit that references this issue on Oct 25, 2026
    6e71d9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions