Realitykit

RealityKit

Build AR experiences on iOS using RealityKit for rendering and ARKit for world tracking. Covers RealityView, entity management, raycasting, scene understanding, and gesture-based interactions. Targets Swift 6.3 / iOS 26+.

Contents

• Setup
• RealityView Basics
• Loading and Creating Entities
• Anchoring and Placement
• Raycasting
• Gestures and Interaction
• Scene Understanding
• Common Mistakes
• Review Checklist
• References

Setup

Project Configuration

1. Add NSCameraUsageDescription to Info.plist
2. On iOS, RealityViewCameraContent displays an AR camera view by default (iOS 18+, macOS 15+); use .virtual camera mode for explicit non-AR fallback
3. No entitlement is required for basic AR. If AR is core to the app, add the arkit required-device capability; otherwise gate AR UI with isSupported.

Device Requirements

Check the exact AR configuration's isSupported value before presenting AR UI.

import ARKit

guard ARWorldTrackingConfiguration.isSupported else {
    showUnsupportedDeviceMessage()
    return
}

Key Types

RealityView Basics

RealityView is the SwiftUI entry point for RealityKit. RealityViewCameraContent is the iOS/macOS content type. On iOS, it uses an AR camera view by default and can use content.camera = .virtual for non-AR mode when requested or when AR/camera access is unavailable.

import ARKit
import SwiftUI
import RealityKit

struct ARExperienceView: View {
    var body: some View {
        RealityView { (content: RealityViewCameraContent) in
            if !ARWorldTrackingConfiguration.isSupported {
                content.camera = .virtual
            }

            let sphere = ModelEntity(
                mesh: .generateSphere(radius: 0.05),
                materials: [SimpleMaterial(
                    color: .blue,
                    isMetallic: true
                )]
            )
            sphere.position = [0, 0, -0.5]  // 50cm in front of camera
            content.add(sphere)
        }
    }
}

Make and Update Pattern

Use the update closure to respond to SwiftUI state changes:

struct PlacementView: View {
    @State private var modelColor: UIColor = .red

    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(
                    color: .red,
                    isMetallic: false
                )]
            )
            box.name = "colorBox"
            box.position = [0, 0, -0.5]
            content.add(box)
        } update: { content in
            if let box = content.entities.first(
                where: { $0.name == "colorBox" }
            ) as? ModelEntity {
                box.model?.materials = [SimpleMaterial(
                    color: modelColor,
                    isMetallic: false
                )]
            }
        }

        Button("Change Color") {
            modelColor = modelColor == .red ? .green : .red
        }
    }
}

Loading and Creating Entities

Loading from USDZ Files

Load 3D models asynchronously to avoid blocking the main thread:

RealityView { content in
    if let robot = try? await ModelEntity(named: "robot") {
        robot.position = [0, -0.2, -0.8]
        robot.scale = [0.01, 0.01, 0.01]
        content.add(robot)
    }
}

Adding Components

Entities use an ECS (Entity Component System) architecture. Add components to give entities behavior:

let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)

// Make it respond to physics
box.components.set(PhysicsBodyComponent(
    massProperties: .default,
    material: .default,
    mode: .dynamic
))

// Add collision shape for interaction
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))

// Enable input targeting for gestures
box.components.set(InputTargetComponent())

Anchoring and Placement

AnchorEntity

Use AnchorEntity to anchor content to detected surfaces or world positions:

RealityView { content in
    // Anchor to a horizontal surface
    let floorAnchor = AnchorEntity(.plane(
        .horizontal,
        classification: .floor,
        minimumBounds: [0.2, 0.2]
    ))

    let model = ModelEntity(
        mesh: .generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .orange, isMetallic: false)]
    )
    floorAnchor.addChild(model)
    content.add(floorAnchor)
}

Anchor Targets

Raycasting

Keep RealityKit scene queries separate from ARKit real-world raycasts:

• RealityViewCameraContent.ray(through:in:to:) returns a camera ray in RealityKit coordinate spaces. It projects a screen point into the virtual scene; it is not proof of a detected physical surface.
• RealityViewCameraContent.hitTest(point:in:query:mask:) hits virtual entities made hittable by CollisionComponent shapes. Use those shapes for entity picking and targeted gestures, not ARKit plane detection.
• Use AnchorEntity(.plane(...)) for simple placement on detected planes.
• Use ARKit ARRaycastQuery plus ARSession.raycast(_:) when the task needs a one-shot intersection with real-world surfaces, then anchor with AnchorEntity(raycastResult:).

let results = session.raycast(query)
if let result = results.first {
    let anchor = AnchorEntity(raycastResult: result)
    anchor.addChild(model)
    content.add(anchor)
}

Do not treat entity hit tests as substitutes for ARKit surface raycasts.

Gestures and Interaction

For gesture-based entity interaction, add CollisionComponent for the hittable shape and InputTargetComponent for input targeting.

Drag Gesture on Entities

struct DraggableARView: View {
    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: true)]
            )
            box.position = [0, 0, -0.5]
            box.components.set(CollisionComponent(
                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
            ))
            box.components.set(InputTargetComponent())
            box.name = "draggable"
            content.add(box)
        }
        .gesture(
            DragGesture()
                .targetedToAnyEntity()
                .onChanged { value in
                    let entity = value.entity
                    guard let parent = entity.parent else { return }
                    entity.position = value.convert(
                        value.location3D,
                        from: .local,
                        to: parent
                    )
                }
        )
    }
}

Scene Understanding

Per-Frame Updates

Subscribe to SceneEvents.Update for continuous scene work rather than driving RealityKit with a SwiftUI timer. Keep the subscription alive and use