Scale Apache DataFusion across a cluster — without leaving the DataFusion you already know.
Note
This project is not part of Apache DataFusion.
DataFusion Distributed is a toolkit that extends Apache DataFusion with distributed query execution. It aims for a developer experience as close as possible to vanilla DataFusion while staying unopinionated about your networking stack.
It's not an out-of-the-box distributed engine — it's a library for building one, with sane defaults for the common case of file-based data sources. A distributed plan is a normal DataFusion physical plan, with a few extra nodes that stream Arrow data between machines, so you can take an existing single-node DataFusion system and add distributed execution with minimal changes.
Apache DataFusion Ballista is a distributed query engine with a scheduler, executors, clients, and operational tooling. Its components can be customized to build specialized engines. It supports staged execution and experimental pipelined execution for latency-sensitive queries.
DataFusion Distributed is a library for embedding distribution into an existing DataFusion application, leaving worker management, networking, and deployment to that application. It does not persist intermediate stage results for recovery; failed queries must restart from scratch.
Consider Ballista if:
- You want to adopt an existing distributed engine as your foundation.
- You are comfortable adopting its deployment and networking model.
- Your workload benefits from materialized intermediate results and the task recovery provided by its staged execution model.
Going distributed takes exactly three things:
- Enable the distributed planner on your session.
- Tell it where your workers are.
- Run the worker gRPC servers.
The Quick start
walks through all three in a few minutes. For everything beyond that — resolving
workers dynamically, distributing your own custom ExecutionPlans, collecting
runtime metrics, and more — see the
full documentation.
The following results compare specific versions and configurations on the workloads and hardware described below. They are a historical snapshot, rather than a general performance ranking. The chart shows relative query times, normalized to DataFusion Distributed (lower is better):
Per-dataset totals
| Dataset | df-dist | Spark | Trino | Queries compared |
|---|---|---|---|---|
| TPC-H SF1 | 7s | 30s | 18s | 22 |
| TPC-H SF10 | 10s | 51s | 33s | 22 |
| TPC-H SF100 | 42s | 261s | 93s | 19 |
| TPC-DS SF1 | 29s | 101s | 85s | 67 |
Conditions. All engines ran on the same cluster: 12 AWS EC2 c5n.2xlarge
instances (8 vCPUs and 21 GiB of memory each, with up to 25 Gbps networking)
reading Parquet files stored in Amazon S3. Each engine's total is the sum of
per-query median (p50) latencies over the queries that all compared engines
completed successfully; lower is better.
The benchmarking code is public and open for anyone to easily reproduce. It uses AWS CDK for automating the creation of the benchmarking cluster so that anyone can reproduce the same results in their own AWS account. The code can be found in the datafusion-distributed-dev-tools repository.
- Be as close as possible to vanilla DataFusion, providing a seamless integration with existing DataFusion systems and a familiar API for building applications.
- Unopinionated about networking. This crate does not take any opinion about the networking stack, and users are expected to leverage their own infrastructure for hosting DataFusion nodes.
- No coordinator-worker split. To keep infrastructure simple, any node can act as a coordinator or a worker.
- A library, not an engine. The goal is to provide the tools for people to build distributed engines, not being one.