Repository navigation
Conversation
… deletes - LLAP reports the file row of the first row of each batch: ORC files with position deletes or virtual columns read through LLAP IO - The non-LLAP vectorized ORC reader forces orc.sarg.to.filter off: the row-level filter drops rows and whole batches without reporting their positions; the SARG still prunes stripes and row groups (HIVE-30110) - Impact: correctness/routing; ORC files with deletes now read through LLAP IO (not timed)
- llap_iceberg_read_orc_deletes.q: V2 position deletes and V3 deletion vectors on an ORC file with three row groups, read through LLAP IO (LLAP IO counters in the golden) - The row-mode ORC reader forces orc.sarg.to.filter off too: Iceberg's OrcIterable takes the batch's file position before ORC can skip filtered batches - LlapRecordReader.getRowNumber returns Long.MIN_VALUE (unknown) instead of throwing - OrcEncodedDataConsumer: drop the redundant noIndex check - TestOrcEncodedDataConsumerRowPositions: drop redundant tests and asserts; a read without columns replaces the faked ALL_RGS case - HiveVectorizedReader comments
…tadata cache - Impact (cluster, 1TRC part2 V2, 40 x r5d): 711.9 s without the cache vs 394.3 s with it; hit rate 81-84%; local miss-path decode 235 -> 20 ns per record
- Compare hive.llap.io.cache.deletes case-insensitively, as HiveConf validates it (CachingDeleteLoader and the ACID reader) - The best-effort metadata cache put no longer waits for memory: it evicts while that frees memory, then fails; the cache tag is computed inside the best-effort block - TestCachingDeleteLoader: NONE level; the put that cannot reserve memory returns without an interrupt; drop the duplicated DV counters test and a duplicated assert
- Impact (local): 19.0 -> 1.32 ns/row; cluster (1TRC part2 V2, 40 x r5d, together with a cheaper delete-cache miss path): delete overhead 179.0 -> 29.4 s, part2 394.3 -> 263.1 s
- HiveVectorizedReader.reader: drop the unused readSchema parameter and the equality deletes TODO - HiveDeleteFilter: drop the empty-batch guard (HiveBatchIterator ends the scan at an empty batch) - TestHiveDeleteFilter: drop the loader tests, which TestCachingDeleteLoader covers - testHiveDeleteFilterWithOrcRowFiltering runs on every format version - DeletionVectors: keep the HIVE-30108 javadoc
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Optimization. Local: 19.0 -> 1.32 ns/row at 7% scattered deletes (a local measurement of the delete filter; the harness is not part of this PR). Cluster, 1TRC part2 V2, 40 x r5d: these figures combine this PR with HIVE-30108 (and its cheaper delete-cache miss path): delete overhead 179.0 -> 29.4 s, part2 394.3 -> 263.1 s
Does this PR introduce any user-facing change?
No
How was this patch tested?