<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://agentic-data-infra.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://agentic-data-infra.github.io/" rel="alternate" type="text/html" /><updated>2026-11-17T10:59:26-08:00</updated><id>https://agentic-data-infra.github.io/feed.xml</id><title type="html">Agentic Data Infra</title><subtitle>Open data infrastructure for agentic and scientific workflows.</subtitle><entry><title type="html">Introducing Ember</title><link href="https://agentic-data-infra.github.io/blog/2026/11/17/introducing-ember/" rel="alternate" type="text/html" title="Introducing Ember" /><published>2026-11-17T10:30:00-08:00</published><updated>2026-11-17T10:30:00-08:00</updated><id>https://agentic-data-infra.github.io/blog/2026/11/17/introducing-ember</id><content type="html" xml:base="https://agentic-data-infra.github.io/blog/2026/11/17/introducing-ember/"><![CDATA[<p>In our <a href="/blog/2026/11/17/data-infrastructure-for-the-agentic-loop/">first post</a>
we argued that agents running exploration loops need a data layer that treats
the whole loop as one slowly evolving computation and pays only for what
changes between rounds. <a href="https://github.com/agentic-data-infra/ember">Ember</a>
is the engine at the center of that layer. This post explains what it is, what
sets it apart, and how it does on its first benchmarks.</p>

<h2 id="what-ember-is">What Ember is</h2>

<p>Ember is a distributed DataFrame engine written in C++20, built for processing
large datasets on a cluster. Programs are written
against a DataFrame API in Python or C++. Ember uses
<a href="https://github.com/facebookincubator/axiom">Axiom</a> for query optimization and
distributed planning,
and <a href="https://github.com/facebookincubator/velox">Velox</a> for query execution. A
driver schedules the stages, retries tasks, and recomputes lost outputs from
their lineage. Results come back as Arrow tables.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">ember</span> <span class="kn">import</span> <span class="n">Session</span>
<span class="kn">from</span> <span class="nn">ember.functions</span> <span class="kn">import</span> <span class="n">avg</span><span class="p">,</span> <span class="n">col</span><span class="p">,</span> <span class="n">countAll</span>

<span class="n">ember</span> <span class="o">=</span> <span class="n">Session</span><span class="p">.</span><span class="n">builder</span><span class="p">.</span><span class="n">master</span><span class="p">(</span><span class="s">"local[*]"</span><span class="p">).</span><span class="n">getOrCreate</span><span class="p">()</span>
<span class="n">runs</span> <span class="o">=</span> <span class="n">ember</span><span class="p">.</span><span class="n">read</span><span class="p">.</span><span class="n">parquet</span><span class="p">(</span><span class="s">"runs.parquet"</span><span class="p">)</span>
<span class="p">(</span><span class="n">runs</span><span class="p">.</span><span class="nb">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s">"loss"</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mf">0.8</span><span class="p">)</span>
     <span class="p">.</span><span class="n">groupBy</span><span class="p">(</span><span class="s">"model"</span><span class="p">)</span>
     <span class="p">.</span><span class="n">agg</span><span class="p">(</span><span class="n">countAll</span><span class="p">().</span><span class="n">alias</span><span class="p">(</span><span class="s">"runs"</span><span class="p">),</span> <span class="n">avg</span><span class="p">(</span><span class="s">"loss"</span><span class="p">))</span>
     <span class="p">.</span><span class="n">show</span><span class="p">())</span>
</code></pre></div></div>

<p>Ember’s main target is a cluster: many workers on Kubernetes, processing
datasets far larger than one machine. It also runs on a single node. In local
mode the driver and an executor live inside the caller’s process, so a small
query from an agent, or a quick test on a laptop, needs no cluster at all. The
same program moves from one to the other without changes.</p>

<h2 id="pay-for-the-deltas">Pay for the deltas</h2>

<p>Most jobs in an exploration loop are small variations of jobs that ran a minute
earlier: the same scoring pass with a different threshold, the same corpus with
a slightly different filter. Ember is built to notice.</p>

<p>Every stage of a plan gets a content key: a hash of its plan, everything
upstream of it, the files it reads, the configuration that can change its
result, and the program’s build. When a later job in the same session contains
a stage with the same key, that stage does not run again; its consumers read
the output stored the first time. <code class="language-plaintext highlighter-rouge">explain()</code> shows the keys and marks which
stages were reused.</p>

<p>For a threshold sweep this means the expensive scoring stage runs once, and
each new threshold only re-runs the cheap filter on top of it. No <code class="language-plaintext highlighter-rouge">cache()</code>
calls are needed, although they are available. When shuffle data lives in
<a href="https://github.com/agentic-data-infra/kiln">Kiln</a>, stage outputs are also
recorded in Kiln’s catalog, so a later run of the same program build can reuse what
an earlier one computed.</p>

<h2 id="model-calls-as-expressions">Model calls as expressions</h2>

<p>Agents filter, label and grade large experiment outputs with models all the
time. In Ember a model call is an ordinary DataFrame expression:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">ember</span> <span class="kn">import</span> <span class="n">ai</span>
<span class="kn">from</span> <span class="nn">ember.functions</span> <span class="kn">import</span> <span class="n">col</span><span class="p">,</span> <span class="n">concat</span><span class="p">,</span> <span class="n">lit</span>

<span class="n">judged</span> <span class="o">=</span> <span class="n">reviews</span><span class="p">.</span><span class="n">withColumn</span><span class="p">(</span><span class="s">"judgment"</span><span class="p">,</span> <span class="n">ai</span><span class="p">.</span><span class="n">generate</span><span class="p">(</span>
    <span class="n">concat</span><span class="p">(</span><span class="n">lit</span><span class="p">(</span><span class="s">"Rate the sentiment of this review from 0 to 1: "</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s">"text"</span><span class="p">)),</span>
    <span class="n">endpoint</span><span class="o">=</span><span class="s">"small"</span><span class="p">,</span>
    <span class="n">output</span><span class="o">=</span><span class="p">{</span><span class="s">"positive"</span><span class="p">:</span> <span class="s">"boolean"</span><span class="p">,</span> <span class="s">"sentiment"</span><span class="p">:</span> <span class="s">"double"</span><span class="p">}))</span>
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">ai.generate</code> sends one request per row to any OpenAI-compatible endpoint, such
as vLLM, SGLang or a hosted API, and returns a typed result. Three properties
make it fit an agentic loop:</p>

<ul>
  <li><strong>Calls only where needed.</strong> Calls run asynchronously on Velox’s RPC
operator, and only on rows that pass the filters before them.</li>
  <li><strong>Failures are data.</strong> A failed row gets a status instead of failing the job,
so one bad request does not cost a thousand good ones.</li>
  <li><strong>Reuse covers model calls too.</strong> A stage with model calls is reused like any
other, so rerunning an unchanged program makes no calls at all.
<code class="language-plaintext highlighter-rouge">session.stats()</code> reports calls, failures and latency.</li>
</ul>

<h2 id="shuffle-that-outlives-its-workers">Shuffle that outlives its workers</h2>

<p>Data moves between stages through a shuffle. By default map outputs are files
on the workers that wrote them. With Kiln as the shuffle store, each map output
is an immutable chunk in Kiln’s DRAM or local-disk tier, read with one-sided
RDMA where the network supports it. Outputs then survive the loss of the worker
that wrote them, which matters on the preemptible capacity that long-running
loops run on. When an output is lost anyway, the driver recomputes it from its
inputs.</p>

<h2 id="first-numbers">First numbers</h2>

<p>We started with a 20 GB sort: simple, shuffle-heavy, and easy to validate. We
compared Ember with Spark 3.5 (DataFrame), Spark with Gluten 1.7 (Velox
native) and Ray Data 2.60, each reading and writing Parquet, at equal memory
budgets. The cluster is four Kubernetes nodes with four cores each, running
on a single 16-vCPU Google Cloud VM.</p>

<figure>
  <img src="/assets/images/posts/ember-sort-kind.svg" alt="Bar chart of 20 GB sort job times on four Kubernetes nodes: in memory Ember 76 s, Spark with Gluten 146 s, Spark DataFrame 167 s, Ray Data 207 s; spilling with 16 partitions Ember 230 s, Gluten 282 s, Spark 315 s, Ray did not complete; spilling with 128 partitions Ember 155 s, Gluten 211 s, Spark 240 s, Ray did not complete." />
  <figcaption>Figure 1. 20 GB sort, Parquet to Parquet, on four Kubernetes nodes sharing one 16-vCPU VM. Faster of two runs; every output validated. Ray Data's processes were killed for running out of memory in both spilling cells.</figcaption>
</figure>

<p>Ember finished first in every cell: 1.9x faster than the fastest alternative in
memory, and 1.23x and 1.36x faster when the sort spills to disk. When spilling,
every system is bound by the VM’s disk, and Ember wins mostly by moving fewer
bytes.</p>

<p>We read these runs as correctness validation and a performance sanity check,
not as a performance claim. All four nodes share one machine’s cores and disk,
so they exercise the distributed code paths but not scaling across hosts. There
is no real RDMA network and no GPU in this setup, and a sort is not an
exploration loop. We are now running cluster-wide benchmarks of agentic loop
workloads (many concurrent, near-identical jobs with reuse), which is where we
expect Ember’s design to matter most, and we will share those numbers soon.</p>

<h2 id="whats-next">What’s next</h2>

<p>Ember is early. Next on the list:</p>

<ul>
  <li><strong>GPU execution</strong> through Velox’s cuDF backend.</li>
  <li><strong>RDMA on real networks</strong>, and runs across many hosts.</li>
  <li><strong>Reuse across sessions and agents</strong>, through a persistent driver and
catalog.</li>
  <li><strong>Differential and progressive execution</strong>: comparing a candidate with its
baseline from the point where they diverge, and answers whose confidence
intervals tighten as more data is read.</li>
</ul>

<p>The code is on <a href="https://github.com/agentic-data-infra/ember">GitHub</a> under the
Apache 2.0 license, including a Linux wheel for Python and a local mode that
needs no cluster. In the next posts we will introduce Kiln and Stoke.</p>]]></content><author><name>Patrick Stuedi</name></author><summary type="html"><![CDATA[A distributed DataFrame engine for agents: content-hashed stages, model calls as expressions, and shuffle that outlives its workers.]]></summary></entry><entry><title type="html">Data infrastructure for the agentic loop</title><link href="https://agentic-data-infra.github.io/blog/2026/11/17/data-infrastructure-for-the-agentic-loop/" rel="alternate" type="text/html" title="Data infrastructure for the agentic loop" /><published>2026-11-17T00:00:00-08:00</published><updated>2026-11-17T00:00:00-08:00</updated><id>https://agentic-data-infra.github.io/blog/2026/11/17/data-infrastructure-for-the-agentic-loop</id><content type="html" xml:base="https://agentic-data-infra.github.io/blog/2026/11/17/data-infrastructure-for-the-agentic-loop/"><![CDATA[<p>Data systems were built around a human loop. A person writes a query, waits,
reads the result and decides what to do next. That loop has shaped almost
every design decision in today’s engines, from how jobs are scheduled to how
errors are reported.</p>

<p>That loop is changing. More and more jobs are now issued by AI agents, and
their results are read by agents too. The most demanding version of this is
what we call <em>continuous exploration</em>: AI systems that propose a hypothesis,
run an experiment, evaluate it, refine the hypothesis and repeat, thousands of
times in parallel, to attack hard problems in science and engineering.</p>

<p>We started an initiative to build open-source data infrastructure for exactly
this setting. This post explains why we think it needs new infrastructure, and
what we are building.</p>

<h2 id="the-exploration-loop">The exploration loop</h2>

<p>An exploration loop is the scientific method, automated. Scientists set the
objective and approve the expensive steps. Agents do the rest, round after
round:</p>

<ol>
  <li><strong>Propose.</strong> Agents form new hypotheses, often by mining the results of
earlier rounds.</li>
  <li><strong>Implement.</strong> Agents turn each hypothesis into a change to code or data,
for example a candidate training dataset.</li>
  <li><strong>Run.</strong> The candidates are trained, simulated or sent to a lab, small runs
first, with only the best moving on to larger scales.</li>
  <li><strong>Evaluate.</strong> Every candidate is scored against the current baseline, per
slice and across seeds, and the results go into a ledger the next round
learns from.</li>
</ol>

<figure>
  <img src="/assets/images/posts/exploration-loop.svg" alt="An exploration loop: propose, implement, run and evaluate, each step reading and writing data through one shared data layer." />
  <figcaption>Figure 1. An exploration loop. Scientists set the objective; agents run each step as hundreds of parallel variants. Every step processes data (blue), all through one shared data layer.</figcaption>
</figure>

<p>The loop has the same shape whether the experiment is an ML training recipe, a
genome-wide analysis or a materials simulation. What changes is the runner and
the evaluator. What stays the same is that every step reads or writes large
amounts of data.</p>

<h2 id="why-the-data-layer-sets-the-pace">Why the data layer sets the pace</h2>

<p>Once agents propose and implement hypotheses, the speed of the loop is bounded
by everything else: preparing inputs, feeding training and judging outcomes.
In many domains those steps are dominated by large-scale data processing.</p>

<p>This is especially true in data-centric ML, where many hypotheses <em>are</em> data
changes: a new quality filter, a different dedup threshold, a reweighted
mixture. If the data layer treats every one of those experiments as an
independent full pass over the corpus, the whole loop runs at the speed of the
data layer.</p>

<h2 id="what-changes-when-agents-are-the-users">What changes when agents are the users</h2>

<p>When the client of a data system changes from a person to an agent, almost
every assumption behind its design changes with it.</p>

<table class="compare">
  <thead>
    <tr>
      <th> </th>
      <th>Human-driven</th>
      <th>Agent-driven</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Who issues jobs</td>
      <td>Engineers and analysts</td>
      <td>Planner, implementer and analyst agents</td>
    </tr>
    <tr>
      <td>Job rate</td>
      <td>Tens to hundreds per day per team</td>
      <td>Thousands to millions per day per loop</td>
    </tr>
    <tr>
      <td>Job shape</td>
      <td>Mostly independent</td>
      <td>Mostly small variations of earlier jobs</td>
    </tr>
    <tr>
      <td>Tolerated latency</td>
      <td>Minutes to hours</td>
      <td>Every second of waiting is an idle reasoning step</td>
    </tr>
    <tr>
      <td>Result consumer</td>
      <td>Dashboards and notebooks</td>
      <td>Models reading compact, structured results</td>
    </tr>
    <tr>
      <td>Failure handling</td>
      <td>A human reads the error</td>
      <td>Errors must be actionable by a model</td>
    </tr>
  </tbody>
</table>

<p>Five properties of this workload drive our design:</p>

<ul>
  <li><strong>Massive, redundant concurrency.</strong> A proposer fans out hundreds of variants
at once, and most differ from each other by one threshold or a few mixture
weights. An engine should pay for the differences, not the totals.</li>
  <li><strong>Latency stalls reasoning.</strong> Time to first result matters as much as
throughput, so fixed overheads and stragglers matter as much as raw
bandwidth.</li>
  <li><strong>Long-running loops.</strong> A loop runs for days and keeps coming back to the
same baselines, scores and embeddings, so results must outlive a single job.</li>
  <li><strong>Mixed CPU and GPU compute.</strong> Joins, dedup and aggregation run on CPUs,
interleaved with classifiers, embedders and model-based graders on GPUs.</li>
  <li><strong>Two ways to consume data.</strong> The same data is analysed in batch and
streamed into trainers at accelerator speed.</li>
</ul>

<p>Today these needs are spread across systems that share neither an optimizer
nor a cache: batch engines with little reuse across jobs, warehouses that are
awkward for model-in-the-loop transforms, and training dataloaders that live
apart from analytics and evaluation.</p>

<h2 id="what-we-want-the-data-layer-to-be">What we want the data layer to be</h2>

<p>We think the right design treats a whole exploration loop as one large, slowly
evolving computation, and pays only for what changes between rounds. Concretely:</p>

<ul>
  <li><strong>One engine for the whole loop.</strong> Data preparation, model inference,
evaluation and training feeds share one plan representation and one cache,
so work done for one experiment is reused by every later one.</li>
  <li><strong>Built for agents.</strong> Fast enough to keep up with an agent’s reasoning step,
safe under thousands of concurrent and occasionally wrong clients, with
results and errors a model can act on.</li>
  <li><strong>At the scale of science.</strong> The same loop that tunes an ML data recipe also
drives genome-wide analyses, climate scoring, compound screening and
particle-physics event selection, on CPUs, GPUs and RDMA fabrics, elastic on
preemptible capacity.</li>
  <li><strong>Open and portable.</strong> Apache 2.0, built on open engines
(<a href="https://github.com/facebookincubator/velox">Velox</a> and
<a href="https://github.com/facebookincubator/axiom">Axiom</a>), and deployable on any
Kubernetes cluster.</li>
</ul>

<h2 id="the-first-building-blocks">The first building blocks</h2>

<p>We have built the first three pieces of this layer, all open source:</p>

<ul>
  <li><strong><a href="https://github.com/agentic-data-infra/ember">Ember</a></strong> is a distributed C++
DataFrame engine on Velox and Axiom. Every stage of a job gets a content
hash, so a stage that an earlier job already computed is reused instead of
recomputed, and calls to language models are ordinary DataFrame expressions.</li>
  <li><strong><a href="https://github.com/agentic-data-infra/kiln">Kiln</a></strong> is a tiered store for
ephemeral data: shuffle data, cached results and training-feed chunks. Chunks
are immutable and content-addressed, so two agents that produce the same
result share one copy, and clients read and write them with one-sided RDMA.</li>
  <li><strong><a href="https://github.com/agentic-data-infra/stoke">Stoke</a></strong> is a distributed
PyTorch dataloader. Preprocessing runs as an Ember job, and Stoke streams its
output to trainers through Kiln while the job is still running.</li>
</ul>

<p>An agent submits DataFrame jobs to Ember. Every intermediate result that should
outlive a task lives in Kiln, so it survives worker loss and later jobs can
reuse it. Stoke turns Ember’s output into training batches.</p>

<figure>
  <img src="/assets/images/posts/architecture.svg" alt="Architecture: agents submit DataFrame jobs to Ember, which calls model endpoints and passes chunk ids to Stoke; Stoke feeds trainers; Kiln holds every intermediate result in DRAM, local disk and object-store tiers." />
  <figcaption>Figure 2. How the pieces fit together. Blue boxes are the systems we built.</figcaption>
</figure>

<p>These are early systems, tested so far on a single host. Next we are taking
them to clusters with GPUs and real RDMA networks.</p>

<h2 id="whats-next">What’s next</h2>

<p>In the next post we will introduce Ember in more detail: how it reuses work
across near-identical jobs, how it runs model calls inside a query, and how it
compares with established engines. If you are building agentic or scientific
workflows and want to try any of this, the code is on
<a href="https://github.com/agentic-data-infra">GitHub</a>, and we would love to hear from
you.</p>]]></content><author><name>Patrick Stuedi</name></author><summary type="html"><![CDATA[Why agents that run experiments need a new kind of data layer, and what we are building for it.]]></summary></entry></feed>