World-scale Outdoor AR Tracking with GNSS

Augmented reality (AR) has the power to transform the world around us, and its true potential is unlocked when digital content can be reliably anchored to specific, real-world locations. Imagine placing a digital object using geographic coordinates and seeing it there in 3D, or having navigational cues seamlessly overlaid onto your path, no matter when or where you look. This seamless integration requires a highly sophisticated understanding of the device's position in the world.

Our goal is to achieve geo-referenced 6DoF (Six Degrees of Freedom) tracking. This means knowing two crucial things at all times: the device's absolute location on Earth (latitude, longitude, and altitude) and its precise orientation relative to true north and vertical - both updated at full display update rate. This capability is fundamental because it allows AR content creators to specify virtual objects using standard geographic coordinates, which the Spectacles can then render at the correct position and perspective in the user's view.

The integration of Global Navigation Satellite System (GNSS) technology—GPS, Galileo, GLONAS, BeiDou, etc.—is therefore critical for Spectacles' core mission: to deliver seamless outdoor connectivity and enable true digital-physical convergence. This blog post tells the story of how we built a robust system to bridge the gap between high-precision local device tracking and world-scale, absolute positioning.

To understand why a dedicated GNSS tracker is necessary, we must first look at the limitations of state-of-the-art tracking methods and sensors.

Spectacles, like many AR devices, rely heavily on Visual-Inertial Odometry (VIO). VIO provides smooth, high-frequency relative tracking—it tells the system precisely how the device has moved from its previous position. In doing so, it slowly accumulates drift over time. However, VIO's coordinate system is inherently arbitrary; it is initialized the moment the system starts and is only accurate relative to that starting point. It does not know where "north" is or where the device is on Earth.

This is where the GNSS comes in. GNSS receivers provide absolute position and velocity on a world scale, leveraging signals from orbiting satellites. Crucially, however, GNSS data is too slow and inherently noisy for the demands of high-fidelity AR rendering. Furthermore, it does not directly provide a full, reliable 6DoF pose since orientation is missing.

Another sensor often considered is the magnetometer or compass. This can provide a rough, instant estimate of heading (orientation relative to magnetic north). Unfortunately, magnetometers are extremely susceptible to local magnetic field disturbances—anything from metal structures in a building to an urban environment's electrical interference—making them unreliable as a standalone solution for precise heading. Also, the noise on the raw magnetometer data results in about 10-20 degrees heading error, which is an order of magnitude away from the needs of an AR system. Like GNSS, the magnetometer does not provide a full pose. In short, a reliable, world-anchored AR experience requires more than any one of these sensors can provide alone.

Visual positioning systems (VPS)—often called map relocalization—are an alternative technology that provides high-precision positioning by matching camera images against massive, pre-mapped databases, making them ideal for dense, complex urban, or indoor environments. However, their availability is inherently restricted to selected, pre-mapped locations and often relies on cloud-based lookups.

GNSS-based tracking complements this by offering global, ubiquitous coverage that works anywhere a satellite fix is possible, including urban, rural, and remote areas. Furthermore, GNSS is attractive from a privacy as well as a power perspective because it enables on-device processing without the need to transmit visual data for server-side matching. By leveraging GNSS alongside VPS, Spectacles can ensure a consistent and reliable world-anchoring experience that scales with the outdoor and on-the-go use-cases of the device.

Our solution is a fusion system designed to harness the strengths of multiple sensors while mitigating their individual weaknesses. The system operates based on a clear conceptual division of labor:

• VIO supplies the high-frequency, smooth relative motion information.

• GNSS acts as absolute position and velocity anchors, periodically correcting the VIO drift and tethering the system to the Earth's coordinates.

This setup allows one to accurately determine the device’s geo-referenced 6DoF pose (position and orientation, including heading), provided it undergoes sufficient displacement or velocity. This movement, however, takes a bit of time, especially for pedestrian use-cases.

A fast, accurate initialization is vital for a positive product experience. Users expect AR content to appear in the right place right away. Therefore, the GNSS tracker leverages the magnetometer to provide an “instant” north-aligned orientation immediately upon startup. While this orientation can be temporarily inaccurate due to local interference, it provides a serviceable starting point until a GNSS fix has been acquired and the more robust heading-from-motion calculation becomes informative enough. 

To explain how the system works, we need to introduce first some coordinates: the GNSS data provides the position in LLA (Latitude, Longitude, Altitude) or ECEF (Earth-Centered, Earth-Fixed) coordinates. However, AR rendering requires a simple, Euclidean coordinate system, which is achieved by defining a Local Tangent Plane (LTP). This LTP is essentially a local frame set up at the content’s or device's current location, aligned with both north and vertical.

The high-level processing flow is as follows:

• Data Acquisition: Gather GNSS data (absolute position and velocity), high-rate motion data from VIO (relative motion and orientation), and magnetometer (magnetic field).

• Coordinate Conversion: Convert the incoming GNSS LLA/ECEF coordinates into the established local LTP frame.

• Sensor Fusion: The core of the tracker uses graph based optimization techniques, an advanced sensor fusion method, to combine the noisy, absolute GNSS signals with the precise, relative VIO and magnetometer data. This fusion process yields the optimal 6DoF pose—position and orientation—at the high rate required for rendering.

• Rendering: The system then renders the AR content, which was originally specified in geographic coordinates, into the device’s view based on the determined pose in the LTP.

By fusing all these sensor inputs, the system gets the best attributes of each sensor, providing stable, render-rate 6DoF poses that are firmly anchored to the real world.

VIO, while quite robust, is not always available. For instance, in low-light conditions, the VIO system can fail. Also, inside vehicles, Spectacles’ so-called 'travel mode' gets activated where VIO is reconfigured to track against the interior of the vehicle instead of the outside world, rendering it useless in the context of the GNSS tracker. To ensure continuity, the system includes an alternative mode that instead of full VIO uses only the Inertial Measurement Units (IMUs)—the device's accelerometers and gyroscopes—to generate relative poses. This IMU approach, when combined with GNSS (and magnetometers), maintains geo-referenced tracking, although the fused output is slightly less robust to GNSS noise artifacts than the full VIO-GNSS fusion. The ability to smoothly transition between VIO-assisted and IMU-only modes ensures a persistent and reliable user experience.

We tested the GNSS tracker with Spectacles, both on recorded datasets as well as live. The new tracker turned out to be particularly effective for on-device authoring of world-locked content: It enables the user to drop pins on a 2D map view, and subsequently render those dropped pins in the real world. It is easy to see how this—marking a geo-referenced point of interest and using AR to help you find it—could quickly be expanded into a powerful navigation experience.

Quantifying our system's performance precisely is challenging due to the difficulty of acquiring absolute ground truth data. Without a reliable, high-precision reference, we cannot provide definitive numerical accuracy metrics. Consequently, our performance assessments are based on qualitative observations from the use-case above rather than absolute error statistics.

Since the GNSS receiver is the sole provider of absolute position in our tracker, its behavior will ultimately determine the performance of our tracker. For consumer grade GNSS receivers, while stationary, and with an unobstructed view of the sky, one typically gets the following behavior: The Time-To-First-Fix (TTFF) is around half a minute or more in a cold-start scenario. This is governed by the time it takes to download the ephemeris data from SVs. When downloading this data from the internet instead, it is possible to speed up the TTFF to roughly 10s.

Our Spectacles 2024 still are among the few AR glasses that work well indoors and outdoors and the new GNSS tracker represents a significant step forward in our commitment to delivering augmented reality reliably anchored to specific, real-world locations. We learned that no single sensor—not VIO, not GNSS, and not the magnetometer—is sufficient on its own. Instead, only by combining the local, smooth tracking of VIO with the global, absolute positioning of GNSS can we create a practical, high-performance geo-referenced AR system. This unlocks a wide range of compelling use cases, including:

• Precise outdoor navigation: Providing intuitive, overlaid directions while walking

• Outdoor activities: E.g. golf or hiking

• Points of Interest: Tagging real-world locations with digital information

• Friend or Ride Finding: Anchoring virtual markers to real-time locations

• Scavenger Hunts: Creating large-scale location-based games

• Field markup and inspection: For construction and agriculture

• Educational experiences: Historical reconstruction aligned to the actual ruins, art installation anchored to a plaza, etc.

This opens the door to truly location-aware outdoor AR experiences across the real world, fulfilling the promise of Spectacles.