Introducing Ember
In our first post we argued that agents running exploration loops need a data layer that treats the whole loop as one slowly evolving computation and pays only for what changes between rounds. Ember is the engine at the center of that layer. This post explains what it is, what sets it apart, and how it does on its first benchmarks.
What Ember is
Ember is a distributed DataFrame engine written in C++20, built for processing large datasets on a cluster. Programs are written against a DataFrame API in Python or C++. Ember uses Axiom for query optimization and distributed planning, and Velox for query execution. A driver schedules the stages, retries tasks, and recomputes lost outputs from their lineage. Results come back as Arrow tables.
from ember import Session
from ember.functions import avg, col, countAll
ember = Session.builder.master("local[*]").getOrCreate()
runs = ember.read.parquet("runs.parquet")
(runs.filter(col("loss") < 0.8)
.groupBy("model")
.agg(countAll().alias("runs"), avg("loss"))
.show())
Ember’s main target is a cluster: many workers on Kubernetes, processing datasets far larger than one machine. It also runs on a single node. In local mode the driver and an executor live inside the caller’s process, so a small query from an agent, or a quick test on a laptop, needs no cluster at all. The same program moves from one to the other without changes.
Pay for the deltas
Most jobs in an exploration loop are small variations of jobs that ran a minute earlier: the same scoring pass with a different threshold, the same corpus with a slightly different filter. Ember is built to notice.
Every stage of a plan gets a content key: a hash of its plan, everything
upstream of it, the files it reads, the configuration that can change its
result, and the program’s build. When a later job in the same session contains
a stage with the same key, that stage does not run again; its consumers read
the output stored the first time. explain() shows the keys and marks which
stages were reused.
For a threshold sweep this means the expensive scoring stage runs once, and
each new threshold only re-runs the cheap filter on top of it. No cache()
calls are needed, although they are available. When shuffle data lives in
Kiln, stage outputs are also
recorded in Kiln’s catalog, so a later run of the same program build can reuse what
an earlier one computed.
Model calls as expressions
Agents filter, label and grade large experiment outputs with models all the time. In Ember a model call is an ordinary DataFrame expression:
from ember import ai
from ember.functions import col, concat, lit
judged = reviews.withColumn("judgment", ai.generate(
concat(lit("Rate the sentiment of this review from 0 to 1: "), col("text")),
endpoint="small",
output={"positive": "boolean", "sentiment": "double"}))
ai.generate sends one request per row to any OpenAI-compatible endpoint, such
as vLLM, SGLang or a hosted API, and returns a typed result. Three properties
make it fit an agentic loop:
- Calls only where needed. Calls run asynchronously on Velox’s RPC operator, and only on rows that pass the filters before them.
- Failures are data. A failed row gets a status instead of failing the job, so one bad request does not cost a thousand good ones.
- Reuse covers model calls too. A stage with model calls is reused like any
other, so rerunning an unchanged program makes no calls at all.
session.stats()reports calls, failures and latency.
Shuffle that outlives its workers
Data moves between stages through a shuffle. By default map outputs are files on the workers that wrote them. With Kiln as the shuffle store, each map output is an immutable chunk in Kiln’s DRAM or local-disk tier, read with one-sided RDMA where the network supports it. Outputs then survive the loss of the worker that wrote them, which matters on the preemptible capacity that long-running loops run on. When an output is lost anyway, the driver recomputes it from its inputs.
First numbers
We started with a 20 GB sort: simple, shuffle-heavy, and easy to validate. We compared Ember with Spark 3.5 (DataFrame), Spark with Gluten 1.7 (Velox native) and Ray Data 2.60, each reading and writing Parquet, at equal memory budgets. The cluster is four Kubernetes nodes with four cores each, running on a single 16-vCPU Google Cloud VM.
Ember finished first in every cell: 1.9x faster than the fastest alternative in memory, and 1.23x and 1.36x faster when the sort spills to disk. When spilling, every system is bound by the VM’s disk, and Ember wins mostly by moving fewer bytes.
We read these runs as correctness validation and a performance sanity check, not as a performance claim. All four nodes share one machine’s cores and disk, so they exercise the distributed code paths but not scaling across hosts. There is no real RDMA network and no GPU in this setup, and a sort is not an exploration loop. We are now running cluster-wide benchmarks of agentic loop workloads (many concurrent, near-identical jobs with reuse), which is where we expect Ember’s design to matter most, and we will share those numbers soon.
What’s next
Ember is early. Next on the list:
- GPU execution through Velox’s cuDF backend.
- RDMA on real networks, and runs across many hosts.
- Reuse across sessions and agents, through a persistent driver and catalog.
- Differential and progressive execution: comparing a candidate with its baseline from the point where they diverge, and answers whose confidence intervals tighten as more data is read.
The code is on GitHub under the Apache 2.0 license, including a Linux wheel for Python and a local mode that needs no cluster. In the next posts we will introduce Kiln and Stoke.
