XR server

Purpose

XRServer is the engine's XR (VR/AR/MR) abstraction. It owns an XR origin transform, a list of XRTracker instances, the active XRInterface (the actual VR/AR runtime), and a per-frame "world scale". Built-in nodes like XROrigin3D, XRCamera3D, XRController3D, XRAnchor3D, and XRHandTracker plug into this layer.

Directory layout

servers/xr/
├── xr_server.{cpp,h}                 The XRServer singleton + interface registry
├── xr_interface.{cpp,h}              Abstract XRInterface (one per backend)
├── xr_interface_extension.{cpp,h}    Extension hook for module-supplied interfaces
├── xr_pose.{cpp,h}                   Pose data with confidence + linear/angular velocity
├── xr_tracker.{cpp,h}                Base XRTracker class
├── xr_positional_tracker.{cpp,h}     Tracker for things with a position (controller, headset)
├── xr_controller_tracker.h           Subclass adding controller-specific input
├── xr_body_tracker.{cpp,h}           Whole-body tracking (IK retargeting source)
├── xr_face_tracker.{cpp,h}           Facial expression tracking
├── xr_hand_tracker.{cpp,h}           Hand tracking with finger joints
└── xr_vrs.{cpp,h}                    Variable-rate-shading helper for foveated rendering

modules/openxr/                       Primary XR backend (OpenXR runtime)
modules/mobile_vr/                    Phone-based VR (Cardboard-style stereo)
modules/webxr/                        WebXR Device API (web platform only)

scene/3d/xr/                          XROrigin3D, XRCamera3D, XRNode3D, XRController3D,
                                      XRAnchor3D, XRBodyModifier3D, XRFaceModifier3D,
                                      XRHandModifier3D

Concepts

• XRServer — singleton registered as XRServer. Hosts the interface registry, an active XRInterface, and the shared world transform (set_world_origin, set_world_scale).
• XRInterface — one per backend (OpenXR, MobileVR, WebXR). Implements initialize, process (called each frame), get_camera_transform, get_projection_for_view, pre_draw_viewport/post_draw_viewport, and tracker enumeration.
• XRTracker — represents a tracked thing (headset, controller, hand, body). Subclasses add specifics (controller buttons, hand joints, body bones).
• XRPose — a single pose with a transform, confidence enum, and linear/angular velocity. Trackers expose multiple poses (e.g., "default", "aim", "grip" for controllers).
• XR_VRS — Variable-Rate Shading helper that fills a VRS attachment so foveated rendering can sample at lower rates outside the gaze region.

How a frame proceeds

sequenceDiagram
    participant Tree as SceneTree
    participant XRS as XRServer
    participant Iface as XRInterface (OpenXR/MobileVR/WebXR)
    participant Origin as XROrigin3D
    participant Camera as XRCamera3D
    participant RS as RenderingServer

    Tree->>XRS: pre_render
    XRS->>Iface: process()
    Iface->>XRS: tracker poses + world_origin updates
    Tree->>Origin: NOTIFICATION_INTERNAL_PROCESS
    Origin->>Camera: child transforms updated from XRPose
    Tree->>RS: viewport_set_use_xr(true) + draw
    RS->>Iface: pre_draw_viewport (acquire swapchain)
    RS->>Iface: render per-eye via stereo or multiview
    RS->>Iface: post_draw_viewport (submit frame to compositor)

The XR origin is the global anchor — when the player physically moves, controllers and HMD nodes track relative to it. The world scale lets the engine treat scenes as larger or smaller than they appear in physical space (1 m = 1 unit by default).

Backends

OpenXR (modules/openxr)

The recommended modern backend. Implements an XRInterface over the OpenXR runtime (Khronos standard). Highlights:

• Action sets / interaction profiles (declarative input mapping with bindings selected at runtime per controller type).
• Composition layers — passthrough, quad layers, equirect layers in addition to projection layers.
• Hand and body tracking via OpenXR extensions (XR_FB_face_tracking, XR_EXT_hand_tracking, XR_FB_body_tracking).
• VRS attachment via XR_VARJO_foveated_rendering or XR_FB_foveation.
• Spatial anchors via XR_FB_spatial_entity.

The module ships its own action map editor under modules/openxr/editor/ so users can configure interaction profiles per project.

Mobile VR (modules/mobile_vr)

Cardboard-style stereo rendering on phones (no positional tracking, just rotation from device sensors). Limited but useful for casual mobile VR.

WebXR (modules/webxr)

Implements XRInterface against the WebXR Device API. Compiled only on the web target; the runtime depends on the browser providing WebXR.

Tracker types

XRTracker subclasses model different tracked devices:

Tracker	File	Holds
XRPositionalTracker	xr_positional_tracker.h	Multiple named poses (default, aim, grip), positional confidence
XRControllerTracker	xr_controller_tracker.h	Adds button/axis input: button_pressed, input_value, input_vector2
XRHandTracker	xr_hand_tracker.h	26-joint hand skeleton with confidence
XRFaceTracker	xr_face_tracker.h	ARKit / OVRLipSync blend shape weights
XRBodyTracker	xr_body_tracker.h	Body skeleton joints for IK retargeting

The scene-tree XRNode3D family wraps these by name: XRController3D follows the tracker named "left_hand" by default; XRHandModifier3D consumes XRHandTracker data and applies it to a Skeleton3D for hand mesh animation.

Variable-rate shading (foveated rendering)

XRVRS (servers/xr/xr_vrs.cpp) builds a 2D texture that the rendering server uses as a VRS attachment. The texture's pixel values describe the shading rate per tile, with the lowest rates farthest from the eye gaze (or a fixed centre on devices without gaze tracking). The renderer then runs fewer fragment shader invocations in the periphery.

Key abstractions

Abstraction	File	Role
XRServer	servers/xr_server.h	Singleton, interface registry, world transform
XRInterface	servers/xr/xr_interface.h	Per-backend interface
XRPose	servers/xr/xr_pose.h	One transform + confidence + velocities
XRTracker family	servers/xr/xr_*_tracker.h	Tracked devices
XROrigin3D	scene/3d/xr/xr_nodes.cpp	World anchor for XR-tracked subtrees
XRCamera3D	scene/3d/xr/xr_nodes.cpp	Stereo HMD camera

Integration points

• The active Viewport opts into XR via viewport_set_use_xr(true). The rendering server then routes per-eye projection through XRInterface::get_projection_for_view and submits frames through XRInterface::commit_views.
• XRServer exposes add_tracker / remove_tracker so backends and (via GDExtension) third-party trackers can plug in.
• XRHandModifier3D and XRBodyModifier3D are SkeletonModifier3D subclasses (under scene/3d/xr/) that retarget tracker data onto user skeletons.

Entry points for modification

• Adding a new XR runtime → subclass XRInterfaceExtension and XRTracker from a module; register in register_module_types.
• Custom controller bindings → use the action map editor in modules/openxr/editor/openxr_action_map_editor.cpp.
• Foveated rendering tuning → XRVRS::make_vrs_texture and viewport_set_vrs_* on the rendering server.