Repository navigation
Issues with the benchmark results #656
Description
The benchmark results feel more and more like a mess due to a number of issues in the way the benchmark is organized and how the results are shown.
-
As the title says,
Cold Runresults make no sense with in-memory and remote storage, so ideally they should not be shown on the generated html page. -
The benchmark rules mention the following:
run.sh: a loop for running the queries; every query is run three times; if it's a database with local on-disk storage, the first query should be run after dropping the page cache;
This rule is not applied to some of the contestants that run on the local disk like Arc which produces wrong results in Combined and Cold Run categories. For instance, Arc uses DuckDB as its query engine: it outperforms DuckDB+parquet in Cold Run by 3.08x, but it's slower than DuckDB in Hot Run by 2.7x - that's quite puzzling. More than that, many if the Arc's Cold Run query execution times are worse than the corresponding Hot Run times.
- The benchmark rules should specify the disk/volume type and its settings. The
machinefield used to havegp2 500GBin the text - now it doesn't, so it looks like contestants can use any volume type, e.g. io2, making the Cold Run results questionable.
Activity
[-]Cold run makes little sense for databases with in-memory and remote storage[/-][+]Issues with the benchmark results[/+]puzpuzpuz commentedon Oct 24, 2025
@alexey-milovidov is it on purpose that the benchmark turned into цирк с конями? It used to be a decent, trustworthy benchmark, but that's no longer the case.
alexey-milovidov commentedon Oct 24, 2025
Yes, it will make sense to hide in-memory systems from Cold and Combined results.
Did you find an error in Arc's benchmark scripts? They use a really big / complicated Python script, but at the first glance everything was alright. Let's figure it out.
This is a wrong accusation. Here you see how the benchmark is run: https://github.com/ClickHouse/ClickBench/blob/main/run-benchmark.sh. The runs are automated to put every contestant in a fair position (for self-hosted entries). As far as I know, none of the systems used
io2disks. No reason to share speculations.There is another phrase, "плохому танцору яйца мешают".
puzpuzpuz commentedon Oct 25, 2025
To me, the same should be done for Cloud systems as Cold/Hot run is undefined for those.
I saw no traces of page cache flush in their
run.shand the results suggest that it does not occur between the query runs like it's done for other databases.I didn't see the script. In our submissions, I was using results measured on the instance types provisioned with 500GB gp2. The only place where the 500GB gp2 in the readme is mentioned in the Rules and Contribution section is here:
This phrase makes an impression that other instance types may use arbitrary volume. Would be great to make this part of the rules more clear.
Половые органы тут не причем. Речь скорее о том, что части танцоров привязали к ногам пудовые гири. Впрочем, лучшие в данном танцевальном марафоне, с гирями или без, это DuckDB и CedarDB - тут трудно спорить.
Если без шуток, раньше бенчмарк производил лучшее впечатление и был более удобоварим для рядового пользователя СУБД и это не только мое мнение. Здесь я попытался изложить проблемы, которые я вижу - что делать с этим фидбеком и делать ли что-то вообще, решать вам.
xe-nvdk commentedon Oct 26, 2025
@puzpuzpuz Alexey itself tested this in all the systems that they have, and I don't think that the storage was different there. In our test, we also comply with using GP2 500GB, we don't have any interest of cheating in any way. Test can't be reproduced cause before the launch of the first release, we normalized our api endpoints and stopped to work.
Is quite interesting read your jokes in russian. I had you guys (QuestDB team) as a more serious people, but keep the good mood ;)
puzpuzpuz commentedon Oct 26, 2025
@xe-nvdk
No need to mention QuestDB team, I created this issue as an individual who's keen to learn optimizations met in OSS databases and compare their impact on performance, so, I guess, you should blame me. The issue is not focused on Arc: I simply listed the problems accumulated in the benchmark for the past few years, as I see them, and mentioned Arc as a recent example where I didn't find OS page cache flush before each query iteration, a step which is important to reproduce the scenario. Keeping the page cache around is a critical factor since 500GB gp2 is limited with 250MB/s and 3000 IOPS in burst mode, so if the query needs to read a "heavy" varchar column, the volume immediately appears to the bottleneck. This effect is more pronounced on larger boxes like c6a.metal since it has plenty of RAM to keep the data files in the page cache and, in general, is more prone to contention when warming up the pages (when handling major page faults).
When benchmarking various databases, it's important, yet challenging, to put them in the same conditions. This is the goal of any decent benchmark unless the real goal is benchmarketing. Other than that, it appears that this bug in scripts was present in a few databases (#659), not only Arc. To me, it's good to have all the contestants aligned. Needless to say, that I'm sorry if my GH issue led to any drama, that wasn't the goal although Alexey and I had a fruitful eastern-European conversation.
puzpuzpuz commentedon Jan 14, 2026
I know it's tough to put everyone under the same conditions, but I've noticed that some of the contestants don't bother flushing page cache (or flushing buffers, if applicable). I may be wrong, but it looks like Polars is not doing that. There may be others, but I haven't checked. @rschu1ze am I missing something?
nwoolmer commentedon Jan 14, 2026
For reference, against those results, DuckDB comes out a hair faster overall for hot queries, yet 8x slower for the cold run.