Vision Framework

Vision Framework

Detect text, faces, barcodes, objects, and body poses in images and video using on-device computer vision. Prefer the modern iOS 18+ request APIs and load the legacy reference only when the deployment target requires it.

See references/vision-requests.md for complete code patterns and references/visionkit-scanner.md for DataScannerViewController integration.

Contents

• Two API Generations
• Request Pattern (Modern API)
• Text Recognition (OCR)
• Face Detection
• Barcode Detection
• Document Scanning (iOS 26+)
• Image Segmentation
• Object Tracking
• Other Request Types
• Core ML Integration
• VisionKit: DataScannerViewController
• Common Mistakes
• Review Checklist
• References

Two API Generations

Vision has two distinct API layers. Prefer the modern API for new code: Swift-native request types plus try await request.perform(on:). Keep VN*, VNImageRequestHandler, VNSequenceRequestHandler, completion handlers, and legacy CGRect helpers inside explicit legacy fallback sections or files.

The modern API uses the ImageProcessingRequest protocol. Each request type has a perform(on:orientation:) method that accepts CGImage, CIImage, CVPixelBuffer, CMSampleBuffer, Data, or URL. Most requests are structs; stateful requests such as GeneratePersonSegmentationRequest, TrackObjectRequest, TrackRectangleRequest, and DetectTrajectoriesRequest are final classes.

Request Pattern (Modern API)

All modern Vision requests follow the same pattern: create a request, call perform(on:), and handle the typed result.

import Vision

func recognizeText(in image: CGImage) async throws -> [String] {
    var request = RecognizeTextRequest()
    request.recognitionLevel = .accurate
    request.recognitionLanguages = [Locale.Language(identifier: "en-US")]

    let observations = try await request.perform(on: image)
    return observations.compactMap { observation in
        observation.topCandidates(1).first?.string
    }
}

Legacy Pattern (Pre-iOS 18)

For pre-iOS 18 targets, use the corresponding VNRequest with VNImageRequestHandler or VNSequenceRequestHandler. Load references/vision-requests.md for complete legacy request and handler patterns.

Text Recognition (OCR)

Modern: RecognizeTextRequest (iOS 18+)

var request = RecognizeTextRequest()
request.recognitionLevel = .accurate       // .fast for real-time
request.recognitionLanguages = [
    Locale.Language(identifier: "en-US"),
    Locale.Language(identifier: "fr-FR"),
]
request.usesLanguageCorrection = true
request.customWords = ["SwiftUI", "Xcode"] // domain-specific terms

let observations = try await request.perform(on: cgImage)
for observation in observations {
    guard let candidate = observation.topCandidates(1).first else { continue }
    let text = candidate.string
    let confidence = candidate.confidence  // 0.0 ... 1.0
    let bounds = observation.boundingBox   // NormalizedRect
}

Legacy: VNRecognizeTextRequest

The legacy request uses string language identifiers and the handler pattern in the reference; both generations support accurate and fast recognition levels.

Face Detection

Detect face rectangles, landmarks (eyes, nose, mouth), and capture quality.

// Modern API
let faceRequest = DetectFaceRectanglesRequest()
let faces = try await faceRequest.perform(on: cgImage)

for face in faces {
    let boundingBox = face.boundingBox   // NormalizedRect
    let roll = face.roll                 // Measurement<UnitAngle>
    let yaw = face.yaw                  // Measurement<UnitAngle>
}

// Landmarks (eyes, nose, mouth contours)
var landmarkRequest = DetectFaceLandmarksRequest()
let landmarkFaces = try await landmarkRequest.perform(on: cgImage)
for face in landmarkFaces {
    let landmarks = face.landmarks
    let leftEye = landmarks?.leftEye.points
    let nose = landmarks?.nose.points
}

Coordinate System

Vision uses a normalized coordinate system with origin at the bottom-left. Convert to UIKit (top-left origin) before display:

import Vision

func imageRectForDisplay(_ rect: NormalizedRect, imageSize: CGSize) -> CGRect {
    rect.toImageCoordinates(imageSize, origin: .upperLeft)
}

Barcode Detection

Detect 1D and 2D barcodes including QR codes.

var request = DetectBarcodesRequest()
let symbologies: [BarcodeSymbology] = [.qr, .ean13, .code128, .pdf417]
request.symbologies = symbologies

let barcodes = try await request.perform(on: cgImage)
for barcode in barcodes {
    let payload = barcode.payloadString          // decoded content
    let symbology = barcode.symbology            // .qr, .ean13, etc.
    let bounds = barcode.boundingBox             // NormalizedRect
}

Type annotate local values first, then assign request properties separately.

Document Scanning (iOS 26+)

RecognizeDocumentsRequest provides structured document reading with layout understanding beyond basic OCR. Returns DocumentObservation objects with a nested Container structure for paragraphs, tables, lists, and barcodes. Currently, Vision returns one document observation for each image.

var request = RecognizeDocumentsRequest()
let documents = try await request.perform(on: cgImage)

for observation in documents {
    let container = observation.document

    // Full text content
    let fullText = container.text

    // Structured access to paragraphs
    for paragraph in container.paragraphs {
        let paragraphText = paragraph.text
    }

    // Tables and lists
    for table in container.tables { /* structured table data */ }
    for list in container.lists { /* structured list data */ }

    // Embedded barcodes detected within the document
    for barcode in container.barcodes { /* barcode data */ }

    // Document title if detected
    if let title = container.title { print(title) }
}

For simpler document camera scanning, use VisionKit's VNDocumentCameraViewController which provides a full-screen camera UI with auto-capture, perspective correction, and multi-page scanning.

Image Segmentation

Modern: GeneratePersonSegmentationRequest (iOS 18+)

var request = GeneratePersonSegmentationRequest()
request.qualityLevel = .accurate  // .balanced, .fast

let mask = try await request.perform(on: cgImage)
// mask is a PixelBufferObservation with a pixelBuffer property
let maskBuffer = mask.pixelBuffer
// Apply mask using Core Image: CIFilter.blendWithMask()

Legacy: VNGeneratePersonSegmentationRequest

For older targets, VNGeneratePersonSegmentationRequest exposes its mask through the first pixel-buffer observation; use the reference's handler and mask-composition recipe.

Quality levels:

• .accurate -- best quality, slowest (~1s), full resolution
• .balanced -- good quality, moderate speed (~100ms), 960x540
• .fast -- lowest quality, fastest (~10ms), 256x144, suitable for real-time

Instance Segmentation (iOS 18+)