Repository navigation
Conversation
…tream to the partition - A spill opens a codec stream per partition: SnappyCodec and Lz4Codec parse the buffer size from the Configuration and allocate and zero a 256 KiB buffer for each, and IFile.Writer a 64 KiB write buffer, while a partition holds a few KB (Hive LLAP, TPC-DS 10 TB q75 / q64: writer setup in spills is 3.2 / 3.5 % of LLAP CPU, Integer.parseInt 1.5 / 1.5 % and these buffers 1.0 / 1.4 % of it) - CodecUtils reads the buffer size once per output and builds these codecs' BlockCompressorStream itself: a stream known to fit one block gets a block of its length and a buffer of its compression bound, which writes the same bytes; IFile.Writer sizes its write buffer to the stream too - The spill passes each partition's raw length bound from the buffers' record and byte counts, the final merge the sum of the spilled raw lengths; other codecs and writers keep codec.createOutputStream - JMH, one spill of 1,009 partitions (Snappy, 22 B / 2.2 KB / 22 KB per partition): -76 % / -48 % / -10 %, 336 MB allocated -> 5 / 11 / 62 MB; LZ4 -59 % / -41 % / -12 %
|
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A spill of
UnorderedPartitionedKVWriteropens anIFile.Writerper partition, and for Snappy and LZ4 each one goesthrough
codec.createOutputStream: the codec parses its buffer size from theConfiguration(under the codec's conflock), and
BlockCompressorStreamallocates and zeroes a 256 KB buffer plus the compressed buffer, while a partitionoften holds a few KB. With 1,009 partitions that is ~336 MB allocated per spill. In Hive on Tez (LLAP), TPC-DS 10 TB,
the part of this setup that the patch removes is 2.5 % (q75) and 2.8 % (q64) of LLAP CPU (profile in TEZ-4762).
The patch sizes each partition's stream to its data:
CodecUtils.getOutputBufferSize(codec)reads the buffer size once per output (-1 for codecs other than Snappy andLZ4), and
CodecUtils.createOutputStream(codec, out, compressor, bufferSize, dataLength)builds theirBlockCompressorStreamdirectly, with noConfigurationlookup and no lock. A stream known to fit in one block gets ablock of its length and a buffer of its compression bound, and writes the same bytes as the configured stream;
otherwise the configured stream is used.
IFile.Writertakes the codec buffer size and an upper bound of its raw length (Writer.maxRawLength(records, keyValueBytes)), and sizes its write buffer to the stream too.UnorderedPartitionedKVWriterpasses each partition's bound from the buffers' record and byte counts on a spill, andthe sum of the spilled raw lengths on the final merge. Other codecs and other writers keep
codec.createOutputStream.Microbenchmark (JMH, one spill of 1,009 partitions, 22 B records, JDK 21, Apple M-series, avgt of 10; before = the
configured stream, as on master):
Tests:
TestCodecUtils(a sized stream writes the same bytes as the configured one and round-trips, also when thelength is underestimated),
TestIFile(a sized writer writes the bytes of an unsized one),TestUnorderedPartitionedKVWriter(a spill's streams are sized to their partitions, bounded allocation, and everypartition is written as the unsized path would); with the existing tests of the three classes, 401 run, 0 failures.
Nine mutations of the sizing (no sizing, buffer without the compression bound, block one byte short, bound without the
vints, spill unsized, spill reading the conf, merge ignoring the spills, write buffer unsized, spill bound from the first
buffer) each fail at least one of them.