ZSTD compression #1101

Activity

1. zhicwu

   added

   enhancement

   on Oct 3, 2022
2. zhicwu

   added this to the 0.3.3 release milestone on Oct 3, 2022
3. subkanthi commented on Oct 9, 2022

   

   subkanthi

   on Oct 9, 2022

   Contributor

   More actions

   Hi @zhicwu , I was looking at this , I guess the change needs to be done for request and response , would this be a good library to use - https://github.com/luben/zstd-jni

       static ClickHouseOutputStream wrap(ClickHouseFile file, OutputStream output, int bufferSize,
               Runnable postCloseAction, ClickHouseCompression compression, int compressionLevel) {
           final ClickHouseOutputStream chOutput;
           if (compression == null || compression == ClickHouseCompression.NONE) {
               chOutput = new WrappedOutputStream(file, output, bufferSize, postCloseAction);
           } else {
               switch (compression) {
                   case GZIP:
                       try {
                           chOutput = new WrappedOutputStream(file, new GZIPOutputStream(output), bufferSize,
                                   postCloseAction);
                       } catch (IOException e) {
                           throw new IllegalArgumentException("Failed to wrap input stream", e);
                       }
                       break;
                   case LZ4:
                       chOutput = new Lz4OutputStream(file, output, bufferSize, postCloseAction);
                       break;
                   default:
                       throw new UnsupportedOperationException("Unsupported compression algorithm: " + compression);
               }
           }
           return chOutput;
   
4. zhicwu commented on Oct 9, 2022

   

   zhicwu

   on Oct 9, 2022

   Contributor

   More actions

   Yes, it's supposed to has better performance compare to https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/zstd. We probably should support both and take the chance to fix compression/decompression issue in gRPC as well.

   👍React with 👍1Reacted by Kanthi
5. zhicwu

   linked a pull request that will close this issueintroduce ClickHousePassThruStream and enable broken gRPC request chunking and compression #1189on Jan 10, 2023
6. zhicwu

   closed this as completedin #1189on Jan 18, 2023
7. zhicwu

   linked a pull request that will close this issueextract clickhouse-data from clickhouse-client #1197on Jan 18, 2023