Alchemist: Agentic ML Experimentation

We present Alchemist, a human-in-the-loop agentic platform for ML experimentation at Snapchat. ML Engineers set the objectives agents pursue, refine hypotheses based on past experience, and evaluate model performance to guide future experiments. With its first recipe, feature selection, we achieved a 50% reduction in cycle time while achieving a 30% reduction in feature cost. This unlocks more time for engineers to innovate and more budget for teams to innovate on model impacting features.

Snapchat engineers run thousands of model experiments a month on Bento, our ML platform, training hundreds of models a day and consuming significant GPU and compute capacity. Each experiment starts with an engineer reading a new paper, applying learnings from a conference workshop, or creating novel solutions to Snapchat-scale problems.

Historically such experiments took weeks to achieve meaningful results, with much of that time spent outside the core modelling work. Instead time is spent tuning several independent configuration files, waiting on GPU availability, and handcrafting performance reports. And that doesn’t include the various failure modes an engineer is expected to debug, with training runs sometimes failing because of configuration or submission issues that can often be caught earlier in the process. And after going through all of this, a successful experiment has proven just one hypothesis out of the many worth trying.

Figure 1. Where time goes in one experiment. Gray is machine time and yellow is engineer attention.

We were inspired by this previous CodePal post on what changed once AI made writing code fast. The same shift has reached ML experimentation: as implementation gets faster, the bottleneck moves to systematically evaluating a larger queue of ideas and measuring their impact. In our experience agents are good at writing model code, editing configurations, and reading logs, and worse at two things: running a repeated, deterministic set of tasks, and making the directional choices that follow.

The Alchemist platform defines a new ML experimentation loop that uses agents where they are useful while constraining where agent judgment is applied.

Figure 2. Alchemist on Bento. The harness is deterministic. The agent works only at its two edges, proposing hypotheses at the start and recommending next steps at the end, and nothing reaches the training infrastructure without the engineer's approval.

An experimental loop starts with one prompt, naming the production model and a goal:

The agent then identifies a number of candidate hypotheses, each one associated with a concrete configuration change and stated expectation. The configuration is grounded either in the baseline model the engineer supplies, or in earlier loop iterations of the current experiment.

Engineers curate the hypotheses, dropping, editing, re-ranking, or adding through a conversation with the agent. After explicit approval, the agent translates the surviving hypotheses into a single typed plan for the round. The plan lists the concrete steps the harness will run for each hypothesis, an estimated cost, and links to supporting ML artifacts (training datasets, existing training workflows, model code) required for the experiment. The plan passes a local smoke test that checks required inputs against defined constraints before the harness drives it to a terminal success or failure state.

The harness never retries a failed step on its own. Failures are pruned immediately, and the failure details (including debug recommendations) are included in the iteration result for an engineer to choose a followup action for the next iteration.

When the iteration finishes, the agent assembles a comparison across all hypotheses and the baseline and suggests possible next directions. The human steps in to approve a new hypothesis, and the loop continues.

Figure 3. Who can say yes to what. The engineer approves the plan by its fingerprint and decides every retry. An edited plan loops back unapproved, and anyone can stop submission with a HALT file.

An Alchemist loop will run in agent sessions engineers are already working with today. The deterministic harness used by the agent is what orchestrates the experiment loop, with all state maintained as concrete artifacts, and key actions gated through human approval. That determinism is deliberate: validating and submitting plans, polling runs, collecting results, and enforcing budgets are code the agent calls, not behavior it improvises, because those are the steps that must not vary between runs.

At the center of the harness is a “recipe”, a typed description of one experimental goal such as feature selection, an architecture sweep, or a data-window study. It specifies how the engineer’s intent becomes a validated plan, how this plan becomes executable within an experiment loop, and which runbook, the written step-by-step procedure, the loop follows during execution. Most recipes began as a procedure an engineer already ran by hand, and each is promoted into Alchemist only after it has produced repeatable output and passed the same code review as the rest of the platform. Alchemist does not bypass existing model review or production release processes.

Figure 4. One round of the harness, on disk. Each stage reads what earlier stages wrote and adds one file. The approval carries the plan's fingerprint, and every hypothesis keeps its own folder, so one failure never touches another.

The harness itself is a state machine composed of explicit stage transitions built on persisted artifacts:

Every plan, approval, execution submission, and result is an artifact isolated to its specific hypothesis. This isolation reduces unintended context carryover across experiments and loop iterations, and provides nonfunctional benefits including auditability and repeatability. The harness guardrails are also enforced through code rather than prompt text, constraining how many times an experiment can run versus what it may try within a run.

Feature selection is a repeatable process used by Snapchat ML engineers to manage compute cost while maintaining or improving model performance. Today an engineer trains the model with every candidate feature, scores each one, writes a new configuration that keeps the useful ones, retrains, and checks quality against the baseline. Then they repeat until the set fits a budget. Each pass means editing a config by hand, submitting a workflow, waiting on it, and reading the failures. Thus, most of the calendar time is spent on orchestration and failure recovery, not on the model. It was a perfect candidate for the first Alchemist recipe.

The recipe came out of one study on a Story ads model, run prior to the introduction of Alchemist. Over four weeks and 130 workflow submissions, one engineer worked manually to remove 30% of the model's features with no regression to evaluation metrics.

Figure 5. The feature-selection recipe. Every stage writes an immutable feature config and runs as machine time. The engineer touches the run at two flags: approving the plan, and choosing the next round.

Extracted from the case study, the recipe encodes a process for selecting features two different ways. Permutation importance shuffles one feature at a time and measures how sensitive the model is. Separately, a learned stochastic gate on every feature is penalized during training, so that features which do not contribute are eliminated.

The recipe is much more than two scoring methods. It is a durable knowledge base of an entire process, and it only grows as new techniques are learned or flaws are fixed. Gaps a run exposes get closed once, in code; every run after it starts from everything learned before. The same procedure today is one prompt and one approved plan, reducing cycle time from 4 to 2 weeks.

Building the Alchemist platform has provided much insight into how to build with and develop tools on frontier agents. We have found that a properly-constructed deterministic harness can help agents rather than constraining them. With the harness owning the steps that must not vary, the more of the loop it handles, the less the workflow depends on unverified agent output and the less mistakes or loss of context can affect experimental results. We have (re-)learned the utility of runbooks, where human judgement and experience can seed critical context that the agent can reference and prevent wasted effort across multiple experiment runs. And we have found that keeping humans in the loop at critical approval gates has supported adoption and confidence in the workflow.

Feature selection is Alchemist's first recipe, not its last. On the horizon we envision more idea exploration, faster and cheaper feedback, and extending each recipe's knowledge base across Snap's ML community so what one team learns reaches the others sooner.

If you'd like to work on systems like this, we're hiring. Apply here.