Bringing Real People into Your World: Animated Gaussian Splatting on Specs AR Glasses
One of the most compelling promises of AR glasses is presence: the feeling that someone, or something, is truly there with you. A dancer performing in your living room, a coach demonstrating a golf swing at the driving range, a musician playing at your kitchen table. For these experiences to work, the virtual person has to hold up to the scrutiny of a viewer who can walk around them, lean in, and look from any angle, all while the real world remains visible right behind them.
This is much harder to get right than on a phone. On a phone, a flat video or a stylized avatar is often good enough, because the viewer's perspective is fixed by the device they hold. On SPECS AR glasses, the viewer's head is the camera. Every step to the side reveals a new view of the subject, and any shortcut in how that subject was captured or represented is immediately visible.
In this post, we describe how we use animated Gaussian splatting to bring photorealistic, volumetric captures of real people to SPECS AR glasses. We cover why this representation is a natural fit for wearable AR, the challenges of extending it from static scenes to motion, and what it takes to render the result on a device that sits on your face.
Putting a realistic human into an AR scene is not a new problem, and several approaches are in use today. Each comes with trade-offs that become more pronounced on head-worn displays.
Video billboards. The simplest approach is to place a flat video of a person, with the background removed, into the scene. This looks great from exactly one viewpoint, and breaks down as soon as the viewer moves. On AR glasses, where the viewer moves constantly, the illusion collapses within seconds.
Rigged avatars. A 3D character model driven by a skeleton can be viewed from any angle and is efficient to render. However, achieving true photorealism with rigged avatars remains difficult: clothing, hair, skin, and subtle facial expression are hard to author, and the result often reads as a video game character rather than a real person.
Mesh-based volumetric video. Volumetric capture studios have produced textured mesh sequences for years. These are viewable from any angle and render efficiently, but meshes struggle with fine and semi-transparent structure such as hair, loose fabric, and motion blur, and the texture maps bake in lighting in a way that can look flat up close.
Neural radiance fields (NeRFs). NeRFs brought a leap in photorealism by learning a continuous volumetric representation of a scene. Unfortunately, rendering a NeRF requires evaluating a neural network many times along every pixel's ray, which is far beyond the compute and power budget of a wearable device, especially for a sequence that changes every frame.
Gaussian splatting. 3D Gaussian Splatting (3DGS) represents a scene as a large collection of small, semi-transparent, coloured ellipsoids, or "splats." Each splat has a position, an orientation and scale, an opacity, and a color that can vary with viewing direction. Splats are optimized from a set of calibrated photographs until their rendered views match the input images. Crucially, rendering them is a rasterization problem rather than a ray-marching one: splats are projected to the screen and blended, much like conventional graphics primitives. The result combines NeRF-level photorealism, including soft, fuzzy, and view-dependent detail, with performance that can run in real time.
Most work on Gaussian splatting has focused on static scenes. People, however, move.
Most Gaussian splatting today captures static scenes: buildings, rooms, and objects. Our focus is people in motion. By a performance, we mean a person moving, such as a dancer, an athlete swinging a golf club, or someone demonstrating an exercise, recorded from all sides at the same time. Our goal is to take such a performance captured by a set of synchronized cameras with the performer's permission, and turn it into a sequence of splats that can be played back on SPECS AR glasses, viewable from any angle and placed anywhere in the user's environment. At a high level, the system has three stages: capture, reconstruction, and on-device playback.

Figure 1: From multi-camera capture to playback on Specs AR glasses
Every animated splat sequence starts with a performance recorded simultaneously by many cameras surrounding the subject. We have worked with captures ranging from professional volumetric stages with dozens of cameras to small portable rigs with only a handful of cameras.
Capture quality sets the ceiling for everything that follows, and we learned that the "boring" parts of capture matter as much as the reconstruction itself:
Synchronization. At 30 frames per second, a fast-moving hand can travel several centimeters between frames. If even one camera is a single frame out of sync, the reconstruction must reconcile views that show the subject in two different poses, which produces ghosted or duplicated limbs. The most reliable solution is hardware synchronization (genlock), which triggers every camera at the same instant. When that isn't available, recording at a high frame rate, such as 120 fps, and selecting the closest-matching frames from each camera reduces the worst-case offset to a few milliseconds, at the cost of more light and more data. We also found that common automatic synchronization signals can report a confident answer that is simply wrong, so we always verify synchronization visually on fast motion.
Geometric Calibration. Reconstruction relies on knowing exactly where every camera is and how its lens behaves. Small calibration errors that would be invisible in a single photograph show up as blur or doubled edges once dozens of views are combined.
Color consistency. Different cameras, even of the same model, rarely agree on exposure and white balance, and these disagreements turn into color shifts as the viewer walks around the subject. We lock exposure and white balance on every camera before capture, so nothing drifts during a take. If cameras still disagree, we correct each camera’s remaining color offset by comparing how different cameras see the same surfaces, bringing all of them in line with their shared consensus.
Fine detail and motion. Thin structures such as fingers and hair cover only a small number of pixels in each camera, and fast motion smears them with blur. This makes them the most fragile part of a capture, and the place where reconstruction errors tend to start. We counter this at capture time with short exposures that freeze motion, bright and even lighting to keep those short exposures clean, and camera placement and resolution that put as many pixels as possible on the performer. Because a fast-moving hand changes position between even slightly offset exposures, we use genlock where possible so every camera sees the same instant, and a high frame rate where it isn’t.
Reconstruction turns the synchronized, calibrated images into splats. A naive approach is to treat every frame of the performance as an independent static scene and fit a separate set of splats to each. This works surprisingly well for any single frame, but plays back poorly: since each frame is optimized in isolation, splats appear, vanish, and shift from one frame to the next, and the subject visibly shimmers even when standing still.
The research community has explored several directions for representing motion with Gaussians, including per-frame reconstruction, a canonical set of splats deformed over time by a learned motion field, and keyframes combined with changes between them. Each makes a different trade-off between visual fidelity, temporal stability, robustness to fast and complex motion, and storage size.
Our pipeline treats a performance as a sequence rather than a collection of unrelated frames, sharing information across time so that the subject stays stable where it should be stable and moves where it actually moves. It builds on open-source foundations: COLMAP for camera calibration, and Nerfstudio’s splatfacto together with the Gsplat rasterizer for training. On top of these, we built our own sequence trainer. Its main components are:
Sequence-aware training. Rather than reconstructing each frame from scratch, every frame builds on the result of the previous one, so the splats carry over and follow the performer through the performance, and training encourages neighboring splats to move together, the way the surface they represent does.
Targeted refinement. A second pass adds detail only where the carried-over splats fall short, for example in areas that come into view as the performer turns.
A fixed budget per frame. Every frame uses the same number of splats, which keeps playback and streaming cost predictable.
Capture preparation. Isolating the performer from the background, sharpening, harmonizing color across cameras, and paying special attention to the regions where viewers are most sensitive to errors, especially faces.
Temporal clean-up. Smoothing splat motion and reducing any remaining flicker across the exported sequence.
Two artifacts deserve particular mention because they are characteristic of splat-based reconstruction:
Floaters. Stray splats can end up suspended in empty space, often in regions that only a few cameras can see. In a static scene they are a minor annoyance; in an animated sequence they flicker in and out and draw the eye. We tackle them at three points. Before training, every camera image gets a mask that separates the performer from the background, and training is required to keep everything outside the performer transparent, so splats have no reason to fill empty space. During training, splats that contribute little are removed or moved to where they are needed. After training, we check each splat against the masks from all cameras: a splat that falls outside the performer’s silhouette in most views is removed.
Temporal flicker. Even when every frame looks correct on its own, small frame-to-frame inconsistencies in color, opacity, or splat placement add up to visible shimmer. A sequence has to be judged as a sequence, not frame by frame. Sequence-aware training removes most of this at the source: because each frame starts from the previous one, the same splats persist from frame to frame instead of being rearranged every time, and because neighboring splats are encouraged to move together, surfaces stay coherent as the performer moves. Targeted refinement then changes only the parts of a frame that need it, rather than reshuffling the whole frame. The input matters too: consistent synchronization and color across cameras keep the training from chasing differences that aren’t really there.

[Figure 2: Side-by-side — independent per-frame reconstruction vs. temporally consistent reconstruction, shown as a short looping clip]
Rendering on SPECS AR glasses introduces constraints that desktop viewers never face.
Rendering twice, on a power budget. SPECS AR glasses render a separate view for each eye, at a high frame rate and low latency, to keep virtual content locked to the world and comfortable to view. All of this must happen within the power and thermal envelope of a device you wear on your face: the power available for rendering is just a few 100mW, roughly 1,000x less than a high-end desktop GPU can draw. A splat sequence that renders comfortably on a desktop GPU can easily exceed that budget, so the number of splats becomes a first-class design constraint during reconstruction, not an afterthought. In practice, we target 10K to 15K splats per frame, a small fraction of the hundreds of thousands to millions typical of static splat scenes. Sequences range from a few seconds up to a few minutes, depending on the experience and how much motion it contains.
Blending order. Because splats are semi-transparent, correct blending traditionally requires drawing them in back-to-front order, which means re-sorting them whenever the viewpoint changes. On AR glasses, the viewpoint changes every frame. Managing this cost is one of the central engineering challenges of splat rendering on mobile hardware, and an active area of work across the industry.
Data size. A single splat reconstruction of a person is already sizable, and a performance needs many of them every second. The straightforward approach is to compress each frame on its own: a general-purpose tool such as Draco brings a splat down to around 11 bytes, and SPZ, a popular splat format, to about 13 bytes when its positions are stored at the same precision as ours. Both treat every frame as if it were the first. Because consecutive frames of a performance are highly similar, there is a lot of redundancy left to exploit. Our own format shares data between frames rather than storing each one from scratch, reaching 1.5 to 5.5 bytes per splat depending on how much changes from frame to frame: about 2 to 7 times smaller than Draco and 2.5 to 9 times smaller than SPZ, while preserving the detail that makes a performance feel real. At 15,000 splats per frame and 30 frames per second, that is roughly 0.7 to 2.5 MB per second of performance, compared with about 5 MB per second for Draco and 6 MB per second for SPZ.
Optical see-through displays. SPECS AR glasses use see-through displays that add light to the real world rather than replacing it. This has a subtle but important consequence: dark colors appear transparent. A black t-shirt that looks perfect on a monitor can practically vanish when rendered on SPECS AR glasses, letting the room show through the subject. This affects choices all the way back to capture, such as wardrobe, lighting, and color grading, and is a good example of why content for AR glasses cannot simply be ported from other platforms.

[Figure 3: The same splat frame rendered on a monitor vs. photographed through SPECS AR glasses, illustrating dark-color transparency]
We have applied this pipeline to a range of performances, including dance, martial arts and movement practice, and sports such as golf and football. Some of the observations that stood out:
Walking around a performer is the killer feature. The moment a viewer takes a step to the side and the performer remains visually consistent from the new angle, the experience changes from "watching a video" to feeling immersed in a three-dimensional performance. Splats' view-dependent appearance, such as the sheen of fabric or the soft edge of hair, is a large part of what sells this.
Motion is where quality is won or lost. Static frames of splats are now relatively easy to make look good. The perceived quality of an animated sequence is dominated by temporal stability: viewers forgive a slightly soft frame far more readily than a shimmering one.
Faces carry the experience. Viewers are significantly more sensitive to errors in faces than anywhere else, with hands a close second. Investing reconstruction effort in these regions pays off more than uniform improvements elsewhere.
Capture discipline pays compounding dividends. Nearly every artifact we chased in reconstruction could be traced back, at least partly, to synchronization, calibration, or color inconsistency in the capture. A careful capture saves far more time than any amount of downstream fixing.
As with many perceptual systems, quantifying quality precisely is challenging. Standard image metrics measure how closely individual rendered frames match held-out photographs, but they do not capture temporal stability, and they say little about how a sequence feels when viewed through see-through AR glasses in a real environment. For now, we rely heavily on side-by-side viewing on device as our primary measure of progress.
[Video: Kpop Dancer animated Gaussian splat performance]
Interactive link to the animated splat model
[Video: Salsa Dancer animated Gaussian splat performance]
Interactive link to the animated splat model
Animated Gaussian splatting brings a new level of realism to people and performances on SPECS AR glasses. By combining photorealistic, view-dependent appearance with a representation that renders through rasterization, it makes it possible to place a real performer in your world and let you walk around them as if they were there.
Getting there requires attention across the entire pipeline: careful multi-camera capture, reconstruction that treats a performance as a coherent sequence, and playback designed around the realities of a wearable device, from its power budget to its see-through displays.
We see this as a foundation for potential future AR experiences: performances and entertainment, coaching and instruction where you can study a movement from any angle, and ultimately, new ways for people to share authorized, immersive performances and experiences. Looking ahead, we are excited about progress across the field on compressing and streaming dynamic splats, relighting captured people to match the environment around them, and reducing the number of cameras needed to capture a high-quality performance.