World-scale Outdoor AR Tracking with GNSS

Introduction

Augmented reality (AR) has the power to transform the world around us, and its true potential is unlocked when digital content can be reliably anchored to specific, real-world locations. Imagine placing a digital object using geographic coordinates and seeing it there in 3D, or having navigational cues seamlessly overlaid onto your path, no matter when or where you look. This seamless integration requires a highly sophisticated understanding of the device's position in the world.

Our goal is to achieve geo-referenced 6DoF (Six Degrees of Freedom) tracking. This means knowing two crucial things at all times: the device's absolute location on Earth (latitude, longitude, and altitude) and its precise orientation relative to true north and vertical - both updated at full display update rate. This capability is fundamental because it allows AR content creators to specify virtual objects using standard geographic coordinates, which the Spectacles can then render at the correct position and perspective in the user's view.

Rendering of virtual objects specified using geographic coordinates.

Heading-from-motion by fusing relative motion (VIO) with GNSS positions.

A fast, accurate initialization is vital for a positive product experience. Users expect AR content to appear in the right place right away. Therefore, the GNSS tracker leverages the magnetometer to provide an “instant” north-aligned orientation immediately upon startup. While this orientation can be temporarily inaccurate due to local interference, it provides a serviceable starting point until a GNSS fix has been acquired and the more robust heading-from-motion calculation becomes informative enough.

To explain how the system works, we need to introduce first some coordinates: the GNSS data provides the position in LLA (Latitude, Longitude, Altitude) or ECEF (Earth-Centered, Earth-Fixed) coordinates. However, AR rendering requires a simple, Euclidean coordinate system, which is achieved by defining a Local Tangent Plane (LTP). This LTP is essentially a local frame set up at the content’s or device's current location, aligned with both north and vertical.

The high-level processing flow is as follows:

Data Acquisition: Gather GNSS data (absolute position and velocity), high-rate motion data from VIO (relative motion and orientation), and magnetometer (magnetic field).
Coordinate Conversion: Convert the incoming GNSS LLA/ECEF coordinates into the established local LTP frame.
Sensor Fusion: The core of the tracker uses graph based optimization techniques, an advanced sensor fusion method, to combine the noisy, absolute GNSS signals with the precise, relative VIO and magnetometer data. This fusion process yields the optimal 6DoF pose—position and orientation—at the high rate required for rendering.
Rendering: The system then renders the AR content, which was originally specified in geographic coordinates, into the device’s view based on the determined pose in the LTP.

By fusing all these sensor inputs, the system gets the best attributes of each sensor, providing stable, render-rate 6DoF poses that are firmly anchored to the real world.

VIO, while quite robust, is not always available. For instance, in low-light conditions, the VIO system can fail. Also, inside vehicles, Spectacles’ so-called 'travel mode' gets activated where VIO is reconfigured to track against the interior of the vehicle instead of the outside world, rendering it useless in the context of the GNSS tracker. To ensure continuity, the system includes an alternative mode that instead of full VIO uses only the Inertial Measurement Units (IMUs)—the device's accelerometers and gyroscopes—to generate relative poses. This IMU approach, when combined with GNSS (and magnetometers), maintains geo-referenced tracking, although the fused output is slightly less robust to GNSS noise artifacts than the full VIO-GNSS fusion. The ability to smoothly transition between VIO-assisted and IMU-only modes ensures a persistent and reliable user experience.

Results

We tested the GNSS tracker with Spectacles, both on recorded datasets as well as live. The new tracker turned out to be particularly effective for on-device authoring of world-locked content: It enables the user to drop pins on a 2D map view, and subsequently render those dropped pins in the real world. It is easy to see how this—marking a geo-referenced point of interest and using AR to help you find it—could quickly be expanded into a powerful navigation experience.

Displaying of on-device authored content: dropped pins

Quantifying our system's performance precisely is challenging due to the difficulty of acquiring absolute ground truth data. Without a reliable, high-precision reference, we cannot provide definitive numerical accuracy metrics. Consequently, our performance assessments are based on qualitative observations from the use-case above rather than absolute error statistics.

Since the GNSS receiver is the sole provider of absolute position in our tracker, its behavior will ultimately determine the performance of our tracker. For consumer grade GNSS receivers, while stationary, and with an unobstructed view of the sky, one typically gets the following behavior: The Time-To-First-Fix (TTFF) is around half a minute or more in a cold-start scenario. This is governed by the time it takes to download the ephemeris data from SVs. When downloading this data from the internet instead, it is possible to speed up the TTFF to roughly 10s.

Stationary GNSS behavior under clear sky, Time-To-First-Fix and Circular Error Probability

Conclusion

Our Spectacles 2024 still are among the few AR glasses that work well indoors and outdoors and the new GNSS tracker represents a significant step forward in our commitment to delivering augmented reality reliably anchored to specific, real-world locations. We learned that no single sensor—not VIO, not GNSS, and not the magnetometer—is sufficient on its own. Instead, only by combining the local, smooth tracking of VIO with the global, absolute positioning of GNSS can we create a practical, high-performance geo-referenced AR system. This unlocks a wide range of compelling use cases, including:

Precise outdoor navigation: Providing intuitive, overlaid directions while walking
Outdoor activities: E.g. golf or hiking
Points of Interest: Tagging real-world locations with digital information
Friend or Ride Finding: Anchoring virtual markers to real-time locations
Scavenger Hunts: Creating large-scale location-based games
Field markup and inspection: For construction and agriculture
Educational experiences: Historical reconstruction aligned to the actual ruins, art installation anchored to a plaza, etc.

This opens the door to truly location-aware outdoor AR experiences across the real world, fulfilling the promise of Spectacles.

The high-level processing flow is as follows:

Data Acquisition: Gather GNSS data (absolute position and velocity), high-rate motion data from VIO (relative motion and orientation), and magnetometer (magnetic field).
Coordinate Conversion: Convert the incoming GNSS LLA/ECEF coordinates into the established local LTP frame.
Sensor Fusion: The core of the tracker uses graph based optimization techniques, an advanced sensor fusion method, to combine the noisy, absolute GNSS signals with the precise, relative VIO and magnetometer data. This fusion process yields the optimal 6DoF pose—position and orientation—at the high rate required for rendering.
Rendering: The system then renders the AR content, which was originally specified in geographic coordinates, into the device’s view based on the determined pose in the LTP.

By fusing all these sensor inputs, the system gets the best attributes of each sensor, providing stable, render-rate 6DoF poses that are firmly anchored to the real world.

Results