Data infrastructure for the agentic loop

November 17, 2026 · Patrick Stuedi

Data systems were built around a human loop. A person writes a query, waits, reads the result and decides what to do next. That loop has shaped almost every design decision in today’s engines, from how jobs are scheduled to how errors are reported.

That loop is changing. More and more jobs are now issued by AI agents, and their results are read by agents too. The most demanding version of this is what we call continuous exploration: AI systems that propose a hypothesis, run an experiment, evaluate it, refine the hypothesis and repeat, thousands of times in parallel, to attack hard problems in science and engineering.

We started an initiative to build open-source data infrastructure for exactly this setting. This post explains why we think it needs new infrastructure, and what we are building.

The exploration loop

An exploration loop is the scientific method, automated. Scientists set the objective and approve the expensive steps. Agents do the rest, round after round:

  1. Propose. Agents form new hypotheses, often by mining the results of earlier rounds.
  2. Implement. Agents turn each hypothesis into a change to code or data, for example a candidate training dataset.
  3. Run. The candidates are trained, simulated or sent to a lab, small runs first, with only the best moving on to larger scales.
  4. Evaluate. Every candidate is scored against the current baseline, per slice and across seeds, and the results go into a ledger the next round learns from.
An exploration loop: propose, implement, run and evaluate, each step reading and writing data through one shared data layer.
Figure 1. An exploration loop. Scientists set the objective; agents run each step as hundreds of parallel variants. Every step processes data (blue), all through one shared data layer.

The loop has the same shape whether the experiment is an ML training recipe, a genome-wide analysis or a materials simulation. What changes is the runner and the evaluator. What stays the same is that every step reads or writes large amounts of data.

Why the data layer sets the pace

Once agents propose and implement hypotheses, the speed of the loop is bounded by everything else: preparing inputs, feeding training and judging outcomes. In many domains those steps are dominated by large-scale data processing.

This is especially true in data-centric ML, where many hypotheses are data changes: a new quality filter, a different dedup threshold, a reweighted mixture. If the data layer treats every one of those experiments as an independent full pass over the corpus, the whole loop runs at the speed of the data layer.

What changes when agents are the users

When the client of a data system changes from a person to an agent, almost every assumption behind its design changes with it.

  Human-driven Agent-driven
Who issues jobs Engineers and analysts Planner, implementer and analyst agents
Job rate Tens to hundreds per day per team Thousands to millions per day per loop
Job shape Mostly independent Mostly small variations of earlier jobs
Tolerated latency Minutes to hours Every second of waiting is an idle reasoning step
Result consumer Dashboards and notebooks Models reading compact, structured results
Failure handling A human reads the error Errors must be actionable by a model

Five properties of this workload drive our design:

Today these needs are spread across systems that share neither an optimizer nor a cache: batch engines with little reuse across jobs, warehouses that are awkward for model-in-the-loop transforms, and training dataloaders that live apart from analytics and evaluation.

What we want the data layer to be

We think the right design treats a whole exploration loop as one large, slowly evolving computation, and pays only for what changes between rounds. Concretely:

The first building blocks

We have built the first three pieces of this layer, all open source:

An agent submits DataFrame jobs to Ember. Every intermediate result that should outlive a task lives in Kiln, so it survives worker loss and later jobs can reuse it. Stoke turns Ember’s output into training batches.

Architecture: agents submit DataFrame jobs to Ember, which calls model endpoints and passes chunk ids to Stoke; Stoke feeds trainers; Kiln holds every intermediate result in DRAM, local disk and object-store tiers.
Figure 2. How the pieces fit together. Blue boxes are the systems we built.

These are early systems, tested so far on a single host. Next we are taking them to clusters with GPUs and real RDMA networks.

What’s next

In the next post we will introduce Ember in more detail: how it reuses work across near-identical jobs, how it runs model calls inside a query, and how it compares with established engines. If you are building agentic or scientific workflows and want to try any of this, the code is on GitHub, and we would love to hear from you.