Building the Spatial Interaction and Interface Frameworks for Specs

Spatial interaction and interfaces are full of subtle and difficult problems. Here are a few we've spent time digging deep into, and the engineering that went into solving them.

The problem: Even with accurate raycasts, raycasting can break down when your hand is very close to a UI element. Small hand movements translate to large angular changes. But, there's an even harder problem: when your finger is touching a button, how does the system know if you want to poke it or pinch it? Both gestures happen at close range, and getting this wrong means frustrated users.

The solution: We built the Interaction Plane, a zone in front of elements that triggers a different interaction mode. When your hand enters this zone, SIK switches from raycasting to physical targeting. Your fingertip becomes the cursor.

Each of these hand positions would interact with the panel differently. Position A would use far-field raycasts, using larger arm motions to guide the targeting. Position B would use near-field raycasts, using small index and wrist adjustments to guide the targeting. Position C would fade the raycast to guide the user to physically touch the buttons instead.

Here's a look at some of the code that makes this work:

For gesture disambiguation between poke and pinch gestures, we use bistable thresholding in the PhysicalInteractionProvider. When you're already poking, the system uses a larger detection radius to maintain contact through hand tracking noise:

The unified state machine also enables seamless Poke → Pinch transitions. If you poke a button and then pinch without lifting your finger, the system recognizes this as a gesture transformation, not two separate events. So, you can poke to select, then pinch to drag, without ever losing your target.

Finally, buttons also enforce poke directionality, so they only trigger when pressed from the front. No accidental activations from the side or behind.

The result: Near-field interaction that feels like touching a real surface. Improving the user's comfort and confidence in the near-field interaction zone is always a priority of ours. We plan to continue improving and stabilizing targeting intent prediction in upcoming versions.

If objects are out of arm's reach, you need raycasting to target them. But, a lot of users lack experience with this style of input. So, this method requires a lot of thought and iteration to get it to feel intuitive for users. Early versions of our cursor would "stick" to the depth of the first object you targeted. This felt unresponsive and unintuitive. If you targeted a button 30cm away, then wanted to select something 2 meters out, the cursor stubbornly hovered at 30cm until you explicitly "acquired" the distant target. Moving between targets at different depths was a confusing tug-of-war between intention and hidden information.

The solution: The latest version of our cursor system required a complete rewrite, and it paid off. Instead of committing to one object's depth along the targeting ray, we score every interactable within a targeting cone based on proximity to the ray:

The cursor depth along the targeting ray becomes a weighted average, not a binary commitment. As you sweep your aim from a near button toward a distant object, the cursor smoothly transitions through depth. This makes targeting feel forgiving without sacrificing precision.

The cursor also uses context-aware depth smoothing with two spring systems that have different response curves:

It feels fast for 3D objects in the world, slow and precise for dense UI panels. The cursor also fades automatically as your hand approaches an InteractionPlane, seamlessly handing off to fingertip targeting.

The result: A cursor that feels like a natural extension intent, not a sluggish pointer fighting against you.

The problem: Spectacles are wearable computers with strict compute, power and thermal constraints. So, optimization is incredibly important. If the interaction layer is slow, your entire experience suffers. The performance of real-time interaction and interface systems is an ever-present concern as the form factor of each generation of wearable computers gets more streamlined.

The solution: Treat performance as an ongoing discipline, not as one-off optimization efforts. With each release of SIK and UIKit, we deliver new features and systems while also optimizing the existing ones.

Our recent version releases include a bunch of great optimizations. SIK v0.16.4 introduced a new FrameCache system that eliminates redundant per-frame calculations, refactored hand visual proximity checks to use time-slicing and processing colliders in batches across multiple frames, and conditional spherecasting for poke detection that only fires when hands are near colliders. UIKit v0.1.4 rebuilt the RoundedRectangle mesh to move per-fragment shader work into vertex data, reducing GPU power draw by ~40-50% compared to previous versions, as well as baking dynamic graphics into textures for reduced shader complexity, and reduced redundant material updates during initialization.

You can learn about new features, improvements, and optimizations with each version in the release notes for SIK and UIKit. This is the result of continuous profiling, iteration, and a team that treats “fast and efficient” as a starting point. When you build with SIK and UIKit, you inherit all of these improvements with each version release.

The earliest versions of SIK had cursors that could jitter depending on the object it was targeting, especially small objects with uneven surfaces: a common problem with hand-tracking based raycast targeting. When targeting, the cursor visibly popped on and off the surface. In extreme cases, targeting felt delicate and unreliable.

An opportunity for improvement: Our raycasting system used an iterative-approximation algorithm (a GJK-based approach) that converges on a "close-enough" result rather than computing an exact intersection. This works well for medium to large sized objects, but accuracy degrades significantly at smaller scales. Exactly the scale of UI and interactive elements in AR, which are often just a few centimeters.

The fix: We replaced the iterative algorithm with direct geometric ray-primitive intersections for common shapes (boxes, spheres, capsules, cylinders), while improving the performance. Instead of iterating toward an estimated intersection, we now calculate the exact hit point and normal analytically.

The result:

• Eliminated jitter. Cursors now sit precisely on surfaces, even small ones.

• Up to 45% faster. Geometric solutions outperform iterative approximation.

Scale-independent accuracy. Works reliably whether your UI is 10cm or 10m away.

UI components sound simple. A button is just a thing you click, right? But in spatial computing, a button needs: a physics collider for hit detection, an Interactable component that tracks hover/trigger/drag states, a state machine managing transitions between idle/hovered/pressed/toggled, different visuals per state/style/theme, and feedback (animations, sounds, effects) that communicate state changes. Building this from scratch for every component is tedious and error-prone.

The architecture: UIKit uses a modular Element + Visual = VisualElement pattern. The Element handles logic (colliders, state machines, events). The Visual handles appearance (meshes, materials, animations). The VisualElement orchestrates state changes into visual effects. This separation means you can swap visuals without rewriting logic, or extend base classes to create custom components while still leveraging everything UIKit provides.

Some of the details we handle for you:

Responsive sizing. You can't just scale a button—that skews corners and curves. UIKit uses a custom Size parameter that controls bespoke meshes with custom vertex topology. The material uses a vertex shader to resize the mesh and recalculate UVs from scratch. This creates custom visuals with correct borders, dynamic gradients, and scalable textures at any size.

Scrolling physics. ScrollWindow implements momentum scrolling with edge “bounce back” using a full spring-damper physics system. The scroll feels like it has weight and inertia, decelerating smoothly when you release, then bouncing elastically at the edges.

Nested event handling. What happens when you drag a Slider inside a ScrollWindow? A naive implementation would accidentally scroll the window. UIKit's event propagation system lets parent containers intercept or pass through child events. Drag a slider, and the ScrollWindow knows to stay put.

Spectacles Interaction Kit and Spectacles UI Kit are the result of collaboration across multiple teams at Specs Inc. We'd like to thank all of the engineers, designers, and researchers who contributed to these frameworks, and the broader Spectacles developer community whose feedback continues to shape our work!