Migrate from lukewarm to true cold runs #793

Activity

1. xe-nvdk commented on Feb 18, 2026

   

   xe-nvdk

   on Feb 18, 2026

   Contributor

   More actions

   I can take a few, or do you prefer vendors taking theirs.
2. alexey-milovidov

   added

   close in a month if not active

   on May 5, 2026
3. rschu1ze

   removed

   close in a month if not active

   on May 9, 2026
4. alexey-milovidov commented on May 11, 2026

   

   alexey-milovidov

   on May 11, 2026

   Member

   More actions

   I want to refactor ClickBench scripts so that parts related to system's installation, start/stop, and running a query, are put in separate scripts and can be reused. This will be changed for the main ClickBench (not the 'hardware' or 'versions' variants). This will affect only systems that are installed locally (not cloud (managed) systems).

   For every system:

   Add install script. It will do all the work related to the environment preparation and installation of the system on a fresh VM. Does not have to include startup or health check.

   Add start, check, and stop scripts. start will start up the system (it could be empty if the system is stateless), stop stops it, check runs a trivial query to check if the system is responding, and returns success exit code if it does.

   Add load script. It will expect dataset files in a particular location and perform the load. It should delete the source files after loading, then do sync.

   Add query script. It accepts a query on stdin and runs it, returns query results in any format to stdout and query runtime in fractional seconds (e.g., 0.123) to stderr, it returns non-zero exit code on any error and prints error to stderr. The query runtime can be measured by the server if possible or inside the script. If there is already an old file named query doing other thing, it has to be renamed.

   Note about dataframe-like and embedded systems that are run inside a single process. For these systems, we have to turn them into an HTTP server (e.g., with a python module), so we can start it up and run queries separately.

   The benchmark.sh script should be unified: it will install the system, start it up, then download the dataset, unpack it if necessary, run the load script and measure its total time, then for each query: flush caches, restart the system, (wait for it to stop and wait for it to start), then run the query three times, collecting time or errors. As it will be hard to restart in-memory or dataframe systems, we can skip restart on each query for them.

   The forward-looking goal of this refactoring will be to allow changing the datasets or the sets of queries or benchmark parameters easily, and also provide an online service to run queries on different systems.

   After done with the refactoring, test that the benchmark still works for all affected systems.
5. alexey-milovidov commented on May 11, 2026

   

   alexey-milovidov

   on May 11, 2026

   Member

   More actions

   Implemented in #860.
6. alexey-milovidov

   self-assigned this

   on May 11, 2026
7. alexey-milovidov

   closed this as completedin #860on May 11, 2026