SEMA: Letting an Agent Run the Slow Ends of the Model-Improvement Loop

We present SEMA, a Self-evolving multi-agent system that decides which ideas Snap’s ads-ranking models should try next, and proves them in live experiments.

Every experiment should make the next experiment decision smarter.

A successful test raises confidence in related approaches. A failed reproduction helps the system avoid repeating an unproductive direction. A gap between offline gains and online outcomes changes which signals it trusts. SEMA carries these lessons forward by recalibrating gain estimates, reweighting selection signals, and updating strategy-family priors. Failed ideas retain explicit conditions for reconsideration as data scale or deployment constraints change.

SEMA searches research papers and practitioner discussions, then combines that evidence with its own experiment history to decide where to invest training computation.

Engineers open each cycle and approve every launch to full traffic. Within those boundaries, SEMA connects research, execution, and evaluation into a learning loop: accepted improvements become the next model baseline, and accumulated outcomes inform the next decision.

In our internal study, average active engineer time fell from 57.0 to 2.7 hours per model iteration (21×), while implementation-to-launch conversion rose from 13% to 37% (approximately 3×). SEMA also won the Big Brain Energy award at Snap’s 2026 internal AI Hackathon.

Figure 1. Active engineer time and implementation-to-launch conversion, without and with SEMA, across completed model iterations.

SEMA connects research discovery, model development, and live evaluation through a shared experiment journal. Each cycle draws on earlier outcomes to decide what deserves another training run. As results accumulate, the coordinator updates its gain estimates, selection weights, and confidence in related strategies.

Six modules carry an idea through this loop. They exchange versioned artifacts and maintain a shared tracker, so engineers can inspect the evidence behind each decision.

SEMA in action. Engineers use the interface to guide each iteration and review the agent’s decisions.

Figure 2. The SEMA iteration loop. Six modules connect research discovery, candidate selection, offline evaluation, and guarded online experimentation. Online outcomes feed back into future candidate selection, while engineers retain control over each cycle and launch.

Human-in-the-Loop Interface. Engineers open a cycle, specify the research topic, pin promising papers, and review proposals and results. The interface makes the agent’s decisions visible throughout the workflow. Engineers retain control over which changes reach full traffic.

Search and Community Crawl. An initial crawl builds a pool of research papers together with practitioner reports about reproductions, baseline comparisons, and serving constraints. These reports help identify ideas whose operating assumptions fit the target model before training begins. Source links remain available for inspection, and later cycles re-rank the same candidate pool.

Strategy Coordinator. The coordinator matches candidate ideas to the model’s current weaknesses. It combines research evidence, predicted gain, implementation and serving cost, and outcomes from the journal to choose which candidates deserve the limited training budget. Each selected idea becomes a bounded configuration change with a hypothesis, a success criterion, and an abort criterion.

Offline Trainer and Evaluator. This module applies and validates the proposed change, then trains with timeouts and divergence checks. It compares the candidate with the baseline across ranking quality, predictive loss, calibration, and available regression checks. The outcome is promote, reject, or inconclusive. Failed runs and rejected candidates enter the journal alongside successful ones, preserving evidence for future selection.

Online Deployer. Candidates that pass the offline gate enter guarded A/B experiments against a control. The deployer continuously monitors regression thresholds and rolls back a candidate if a guarded metric crosses its limit. This stage tests whether an offline improvement translates into value under live conditions.

Online Evaluator. The evaluator compares observed business outcomes with the offline prediction and records the gap. Those outcomes feed back into gain estimates, candidate-selection weights, and strategy-family priors. It also prepares a launch document with the hypothesis, configuration change, results, and risks. An engineer reviews the recommendation; an approved improvement becomes the next baseline.

Suppose a candidate improves offline AUC but produces no useful online lift. The experiment still contributes evidence: its prediction error enters the journal and informs future estimates and metric weighting. Related strategies may receive lower priority, with explicit conditions for reconsideration. The next cycle can then choose from the remaining pool with a better understanding of what has—and has not—worked for this model.

That feedback is what makes the workflow self-evolving: each experiment supplies evidence about a candidate and helps refine the judgment behind the next choice.

SEMA evolves both the target model and the strategy used to improve it. An accepted change becomes the next baseline. Experiment outcomes update the coordinator’s expectations, the signals it trusts, and the ideas it prioritizes. Each cycle begins with evidence accumulated from the cycles before it.

An evolving selection policy. Guarded online A/B experiments provide feedback for both launch decisions and future candidate selection. SEMA compares observed outcomes with offline predictions to recalibrate its gain estimates. It uses Bayesian optimization to refine the weights assigned to offline metrics such as AUC, AUPRC, and calibration, assessing proposed weight combinations by replaying historical candidate selection against outcomes recorded in the journal. Evidence-factor weights are also recalibrated from online results. The next cycle therefore ranks candidates using updated estimates and weights that reflect experience with the target model.

Memory that transfers across related strategies. The experiment journal connects individual results to broader strategy families, such as user-behavior-sequence models. Each family carries a prior shared by its members, so one strategy’s outcome can change the ranking of its untried siblings. A confirmed win raises confidence in related approaches. A loss lowers their priority while preserving the reason and an explicit condition for reconsideration—for example, a change in data scale. This lets SEMA carry lessons across experiments and revisit earlier judgments when their underlying conditions change.

Community evidence that guides exploration. Before committing to a training run, SEMA gathers practitioner reports about failed reproductions, omitted baselines, and serving costs. The coordinator ranks candidates using six evidence factors: citation count, relevance to the model’s weakness, discussion volume, community sentiment, reported reproduction outcomes, and the strategy-family prior. Reports of incompatible operating conditions can eliminate a candidate before training, with the reason preserved in the journal. Later cycles re-rank the existing candidate pool using updated weights and priors, connecting external research evidence with experience accumulated on the target model.

Human control over the learning loop. Engineers open each cycle and retain authority over launches. Every proposed change is a bounded configuration diff with a hypothesis, success criterion, and abort criterion. Online experiments run behind regression thresholds with automatic rollback. Before a change reaches full traffic, an engineer reviews its launch document and approves the release. SEMA can adapt its experimentation strategy while accountable engineers control how those decisions affect production.

The internal efficiency and conversion numbers are observational comparisons across periods with and without SEMA, not a controlled longitudinal study. The public demonstration uses a deliberately simple baseline. The system improves one human-specified model per deployment, with the dataset and the evaluation contract fixed in advance. Community sourcing is fragile to platform access, since public discussion APIs are increasingly gated, and the comment-informativeness ranking is a hand-designed heuristic. Discussion volume is not adjusted for paper age, so recent papers are slightly disadvantaged. The economics of the loop, compute spent per validated idea against engineer time saved, still need accounting across several quarters of operation.

SEMA brings research discovery, experimentation, and launch review into a repeatable workflow. We see an opportunity to help engineers explore more promising ideas and use the results of each experiment to make better decisions about what to try next.

We’re excited to keep building tools that help engineers turn research into better experiences for Snapchatters. If you’re interested in machine learning, recommendation systems, and the agents that help improve them, explore opportunities at Snap and come build with us.