You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] In-memory union and intersection fail to combine equal scan rows #3396
Development source reviewed in PR #3395, Java 17. Reproduced with the production Store iterators in a standalone JVM; no running Store cluster or native RocksDB instance was used.
Expected and actual behavior
BackendColumn.equals() compares byte-array contents, but the class has no matching hashCode(). Real RocksDB scans allocate a new BackendColumn on each next() call. The in-memory union/intersection maps therefore do not reliably combine content-equal rows from independent scans.
With 100 keys, each appearing twice as separately allocated, byte-equal columns:
Iterator
Expected rows
Observed rows
Intersection, size = 2
100
0
Union, below its sort threshold
100
200
A separate probe also finds two equal newly allocated columns occupying two entries in a HashSet.
Reproduction
Generate 100 keys as four-byte integer arrays; use the same value bytes for each occurrence of a key. Generate two occurrences per key, calling BackendColumn.of() with new arrays for every occurrence.
Populate an IntersectionWrapper from one pass, with a collision-free key extractor for this dataset, and call proc().
Create a fresh second pass of those rows. Read IntersectionFilterIterator(..., wrapper, 2) to exhaustion; repeat using UnionFilterIterator with the existing BackendColumn serializer and name comparator.
Compare the result counts with the table above. Reusing the same Java object for repeated rows hides the defect.
Fix the stable map-key/equality contract and add regressions with a fresh object per scanned row. Explicitly preserve the intended identity rule: current BackendColumn equality compares name and value, while alternate union and sorted paths group by name. Adding a content hash alone must not be treated as proof that those grouping rules are equivalent.
This predates PR #3395 and is tracked separately so its iterator probe/cleanup fixes need not grow in scope. Original review discussion.
Issue search performed for BackendColumn/hashCode and UnionFilterIterator; no matching existing issue was found.
My plan is to add a hashCode() to RocksDBSession.BackendColumn that matches its equals(), hashing both the name and value bytes. I'll also add regression tests for IntersectionFilterIterator and for UnionFilterIterator below its sort threshold. Like your reproduction, the tests will build a fresh BackendColumn for every scanned row.
I'd leave the grouping question (equality on name + value vs. grouping by name on the other union paths) and the SortShuffle path from #3397 out of this PR. Does that scope sound right?
Activity
vaidikgarg-bit commentedon Oct 23, 2026
Hi @imbajin, I'd like to work on this.
My plan is to add a
hashCode()toRocksDBSession.BackendColumnthat matches itsequals(), hashing both the name and value bytes. I'll also add regression tests forIntersectionFilterIteratorand forUnionFilterIteratorbelow its sort threshold. Like your reproduction, the tests will build a freshBackendColumnfor every scanned row.I'd leave the grouping question (equality on name + value vs. grouping by name on the other union paths) and the SortShuffle path from #3397 out of this PR. Does that scope sound right?
fix(store): hash BackendColumn by its name and value (apache#3404)