Apache Spark has long been a backbone of Snap’s analytical data infrastructure, powering robust, scalable pipelines for reporting, dashboarding, and batch data processing. However, as our machine learning initiatives have expanded, we’ve found that Spark alone isn’t enough to meet the evolving needs of ML development. ML data pipelines introduce a fundamentally different set of challenges:
Iterative Experimentation at Scale: Unlike traditional analytics, ML development is inherently iterative. Modelers constantly experiment with new ideas, trying different data slices, feature definitions, or preprocessing logic. Each experiment may require regenerating training datasets from massive raw inputs over and over again. This loop of experimentation pushes data infrastructure to sustain high-throughput workloads repeatedly, not just on a fixed schedule.
Flexibility during Development, Stability in Production: Unlike more stable analytics pipelines, ML workflows often linger in an extended experimentation phase. Early ML development thrives on freedom, including ad hoc data access, fast turnaround, and flexible schemas. But once a model moves to production, the same pipeline must become efficient, scalable, and observable. Supporting both phases smoothly requires tooling that evolves with the workflow.
Diverse Data Formats: ML teams use what’s best for the job: TFRecord for TensorFlow, Protobuf for gPRC-based serving, JSON for lightweight experimentation, etc. Forcing everything through a single format (like Parquet) would create unnecessary friction and bottlenecks. Tooling needs to meet users where they are.
Spark is powerful and battle-tested for large-scale data processing, especially well-suited for SQL-based transformations and stable batch workflows. While it serves many use cases well, it can be less intuitive for ML practitioners. As ML data processing becomes more complex and bespoke, Spark’s steep learning curve and lack of beginner-friendly abstractions can slow teams down. This is especially true during the early stages of experimentation when flexibility, speed, and ease of use matter most.
Consider the life of an ML engineer without supportive abstractions:
They must understand distributed systems' internals to write efficient Spark jobs.
They often reimplement boilerplate data prep logic already solved elsewhere.
They manage infrastructure: clusters, dependencies, upgrades, on their own.
Debugging distributed job failures becomes a time sink with little payoff.
All of this is in addition to the tools they already need to master: TensorFlow or PyTorch, Kubeflow, notebooks, experiment tracking, and more. Spark is essential for scale, but it’s not the heart of their job. ML engineers want to focus on experiments not infrastructure.
At Snap, we have built an ML-optimized data platform that bridges the above gap. Our goal is to abstract away infrastructure, standardize common patterns, and integrate tightly with Snap’s internal ecosystem, so ML engineers can:
Define what data they need, not how to compute it
Iterate faster with less glue code and boilerplate
Reuse robust, secure, and scalable data processing patterns
Focus on model innovation and business logic
With this foundation, ML development becomes faster, more consistent, and more enjoyable, freeing engineers to do what they do best: build great models that drive impact.
