SEMA: Letting an Agent Run the Slow Ends of the Model-Improvement Loop

SEMA connects research discovery, model development, and live evaluation through a shared experiment journal. Each cycle draws on earlier outcomes to decide what deserves another training run. As results accumulate, the coordinator updates its gain estimates, selection weights, and confidence in related strategies.

Six modules carry an idea through this loop. They exchange versioned artifacts and maintain a shared tracker, so engineers can inspect the evidence behind each decision.

SEMA in action. Engineers use the interface to guide each iteration and review the agent’s decisions.

Figure 2. The SEMA iteration loop. Six modules connect research discovery, candidate selection, offline evaluation, and guarded online experimentation. Online outcomes feed back into future candidate selection, while engineers retain control over each cycle and launch.

Human-in-the-Loop Interface. Engineers open a cycle, specify the research topic, pin promising papers, and review proposals and results. The interface makes the agent’s decisions visible throughout the workflow. Engineers retain control over which changes reach full traffic.

Search and Community Crawl. An initial crawl builds a pool of research papers together with practitioner reports about reproductions, baseline comparisons, and serving constraints. These reports help identify ideas whose operating assumptions fit the target model before training begins. Source links remain available for inspection, and later cycles re-rank the same candidate pool.

Strategy Coordinator. The coordinator matches candidate ideas to the model’s current weaknesses. It combines research evidence, predicted gain, implementation and serving cost, and outcomes from the journal to choose which candidates deserve the limited training budget. Each selected idea becomes a bounded configuration change with a hypothesis, a success criterion, and an abort criterion.

Offline Trainer and Evaluator. This module applies and validates the proposed change, then trains with timeouts and divergence checks. It compares the candidate with the baseline across ranking quality, predictive loss, calibration, and available regression checks. The outcome is promote, reject, or inconclusive. Failed runs and rejected candidates enter the journal alongside successful ones, preserving evidence for future selection.

Online Deployer. Candidates that pass the offline gate enter guarded A/B experiments against a control. The deployer continuously monitors regression thresholds and rolls back a candidate if a guarded metric crosses its limit. This stage tests whether an offline improvement translates into value under live conditions.

Online Evaluator. The evaluator compares observed business outcomes with the offline prediction and records the gap. Those outcomes feed back into gain estimates, candidate-selection weights, and strategy-family priors. It also prepares a launch document with the hypothesis, configuration change, results, and risks. An engineer reviews the recommendation; an approved improvement becomes the next baseline.

Suppose a candidate improves offline AUC but produces no useful online lift. The experiment still contributes evidence: its prediction error enters the journal and informs future estimates and metric weighting. Related strategies may receive lower priority, with explicit conditions for reconsideration. The next cycle can then choose from the remaining pool with a better understanding of what has—and has not—worked for this model.

That feedback is what makes the workflow self-evolving: each experiment supplies evidence about a candidate and helps refine the judgment behind the next choice.

SEMA evolves both the target model and the strategy used to improve it. An accepted change becomes the next baseline. Experiment outcomes update the coordinator’s expectations, the signals it trusts, and the ideas it prioritizes. Each cycle begins with evidence accumulated from the cycles before it.

An evolving selection policy. Guarded online A/B experiments provide feedback for both launch decisions and future candidate selection. SEMA compares observed outcomes with offline predictions to recalibrate its gain estimates. It uses Bayesian optimization to refine the weights assigned to offline metrics such as AUC, AUPRC, and calibration, assessing proposed weight combinations by replaying historical candidate selection against outcomes recorded in the journal. Evidence-factor weights are also recalibrated from online results. The next cycle therefore ranks candidates using updated estimates and weights that reflect experience with the target model.

Memory that transfers across related strategies. The experiment journal connects individual results to broader strategy families, such as user-behavior-sequence models. Each family carries a prior shared by its members, so one strategy’s outcome can change the ranking of its untried siblings. A confirmed win raises confidence in related approaches. A loss lowers their priority while preserving the reason and an explicit condition for reconsideration—for example, a change in data scale. This lets SEMA carry lessons across experiments and revisit earlier judgments when their underlying conditions change.

Community evidence that guides exploration. Before committing to a training run, SEMA gathers practitioner reports about failed reproductions, omitted baselines, and serving costs. The coordinator ranks candidates using six evidence factors: citation count, relevance to the model’s weakness, discussion volume, community sentiment, reported reproduction outcomes, and the strategy-family prior. Reports of incompatible operating conditions can eliminate a candidate before training, with the reason preserved in the journal. Later cycles re-rank the existing candidate pool using updated weights and priors, connecting external research evidence with experience accumulated on the target model.

Human control over the learning loop. Engineers open each cycle and retain authority over launches. Every proposed change is a bounded configuration diff with a hypothesis, success criterion, and abort criterion. Online experiments run behind regression thresholds with automatic rollback. Before a change reaches full traffic, an engineer reviews its launch document and approves the release. SEMA can adapt its experimentation strategy while accountable engineers control how those decisions affect production.

Inside Snap. SEMA is used internally to improve Snap’s ads-ranking models. Our engineering-effort study covered 46 engineers and more than 600 completed end-to-end model iterations. With SEMA, the median active engineering time fell from 51.33 to 2.38 hours per iteration, a 21.6x reduction. The 90th percentile fell from 95.22 to 3.31 hours, a 28.8x reduction. While AI-assisted coding helps reduce implementation time, SEMA also saves time in finding promising ideas and running experiments. For example, median active time for paper retrieval fell from 2.67 to 0.08 hours, and online-test operation fell from 6.33 to 0.39 hours.

Table 1. Active engineer-hours per iteration by stage, excluding time spent waiting for training or online experiments. Percentiles are taken per iteration, so stage values do not sum to the total.

In the open, on TencentGR-1M. To demonstrate how SEMA can be applied beyond Snap with different datasets and models, we ran the full workflow on the public TencentGR-1M dataset, starting from a simple wide-and-deep style CTR model. Since this dataset contains logged interactions rather than live traffic, we used held-out data to simulate online evaluation. One partition supplied the offline metrics used for candidate selection, while the average AUC across three additional partitions served as the simulated-online launch metric and provided feedback for later iterations. The user specified the target model and launch metric, then started the run with a single prompt: “find ads-ranking papers from the past decade.” Across the nine accepted launch events shown below, simulated-online AUC increased from 0.6226 to 0.6658, an absolute gain of 0.0432.

Table 2. Accepted launches on TencentGR-1M. Simulated-online AUC is averaged over three held-out partitions.

The internal efficiency and conversion numbers are observational comparisons across periods with and without SEMA, not a controlled longitudinal study. The public demonstration uses a deliberately simple baseline. The system improves one human-specified model per deployment, with the dataset and the evaluation contract fixed in advance. Community sourcing is fragile to platform access, since public discussion APIs are increasingly gated, and the comment-informativeness ranking is a hand-designed heuristic. Discussion volume is not adjusted for paper age, so recent papers are slightly disadvantaged. The economics of the loop, compute spent per validated idea against engineer time saved, still need accounting across several quarters of operation.

SEMA brings research discovery, experimentation, and launch review into a repeatable workflow. We see an opportunity to help engineers explore more promising ideas and use the results of each experiment to make better decisions about what to try next.

We’re excited to keep building tools that help engineers turn research into better experiences for Snapchatters. If you’re interested in machine learning, recommendation systems, and the agents that help improve them, explore opportunities at Snap and come build with us.