Apple On Device AI

On-Device AI for Apple Platforms

Guide for selecting, deploying, and optimizing on-device ML models. Covers Apple Foundation Models, Core ML, MLX Swift, and llama.cpp.

Contents

• Framework Selection Router
• Apple Foundation Models Overview
• Core ML Overview
• MLX Swift Overview
• Multi-Backend Architecture
• Performance Best Practices
• Common Mistakes
• Review Checklist
• References

Framework Selection Router

Use this decision tree to pick the right framework for your use case.

Apple Foundation Models

When to use: Text generation, summarization, entity extraction, structured output, and short dialog on iOS 26+ / macOS 26+ devices with Apple Intelligence enabled. No app-managed API key, network round trip, or model hosting; still handle system model asset readiness.

Best for:

• Generating text or structured data with @Generable types
• Summarization, classification, content tagging
• Tool-augmented generation with the Tool protocol
• Apps that need guaranteed on-device privacy

Not suited for: Complex math, code generation, factual accuracy tasks, or apps targeting pre-iOS 26 devices.

Core ML

When to use: Deploying custom trained models (vision, NLP, audio) across all Apple platforms. Converting models from PyTorch, TensorFlow, or scikit-learn with coremltools.

Best for:

• Image classification, object detection, segmentation
• Custom NLP classifiers, sentiment analysis models
• Audio/speech models via SoundAnalysis integration
• Any scenario needing Neural Engine optimization
• Models requiring quantization, palettization, or pruning

MLX Swift

When to use: Running specific open-source LLMs (Llama, Mistral, Qwen, Gemma) on Apple Silicon with maximum throughput. Research and prototyping.

Best for:

• Highest sustained token generation on Apple Silicon
• Running Hugging Face models from mlx-community
• Research requiring automatic differentiation
• Fine-tuning workflows on Mac

llama.cpp

When to use: Cross-platform LLM inference using GGUF model format. Production deployments needing broad device support.

Best for:

• GGUF quantized models (Q4_K_M, Q5_K_M, Q8_0)
• Cross-platform apps (iOS + Android + desktop)
• Maximum compatibility with open-source model ecosystem

Quick Reference

Apple Foundation Models Overview

Use the system language model for short generation, summarization, tagging, structured output, and tool-augmented tasks on Apple Intelligence devices. Gate every entry point before creating a session:

import FoundationModels

switch SystemLanguageModel.default.availability {
case .available:
    guard SystemLanguageModel.default.supportsLocale(Locale.current) else {
        // Use locale fallback before generating
        break
    }
    // Proceed with model usage
case .unavailable(.appleIntelligenceNotEnabled):
    // Guide user to enable Apple Intelligence in Settings
case .unavailable(.modelNotReady):
    // System model assets are not ready; show loading state
case .unavailable(.deviceNotEligible):
    // Device cannot run Apple Intelligence; use fallback
case .unavailable(let reason):
    // Unknown or future unavailable reason; use fallback and log reason
}

Then create a session and keep its shared context budget small:

let session = LanguageModelSession {
    "You are a helpful cooking assistant."
}
session.prewarm()
let response = try await session.respond(to: "Suggest a quick pasta recipe")

Required guardrails:

• Sessions are stateful and accept one request at a time; serialize access and check isResponding before issuing another response.
• Instructions, tools, schemas, prompts, transcripts, and output share the context window. Register only necessary tools and keep schemas compact.
• Resolve the locale with supportsLocale(_:); do not raw-match language lists.
• Keep untrusted user content in prompts, never instructions. System guardrails remain active, so handle refusal and other generation errors with fallback UI.

Load the Foundation Models reference when the task needs @Generable, @Guide, streaming, tool definitions, transcripts, generation options, custom adapters, prompt design, or detailed error handling.

Core ML Overview

Apple's framework for deploying trained models. Automatically dispatches to the optimal compute unit (CPU, GPU, or Neural Engine).

Model Formats

Always use mlprogram (.mlpackage) for new work.

Conversion Pipeline (coremltools)

import coremltools as ct

# PyTorch conversion (torch.jit.trace)
model.eval()  # CRITICAL: always call eval() before tracing
traced = torch.jit.trace(model, example_input)
mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(shape=(1, 3, 224, 224), name="image")],
    minimum_deployment_target=ct.target.iOS18,
    convert_to='mlprogram',
)
mlmodel.save("Model.mlpackage")

Validate, Fix, and Reconvert

1. Freeze representative source-model fixtures and acceptable output/task tolerances before conversion.
2. Convert, then run the same fixtures through the source and Core ML models.
3. If output parity or task metrics miss tolerance, inspect shapes, operators, precision, and preprocessing; fix the conversion and rerun the fixtures.
4. Compress only after the uncompressed model passes. Revalidate accuracy after each compression change and undo or tune changes that miss the threshold.
5. Profile the passing model on physical target devices, then repeat until correctness, latency, memory, and package-size targets all pass.

Boundary with coreml

This skill owns Python-side conversion, compression, profiling, and framework selection. Use the sibling coreml skill for Swift app integration, prediction APIs, runtime configuration, Vision request wiring, and detailed model loading.

See references/coreml-conversion.md for the full conversion pipeline and references/coreml-optimization.md for optimization techniques.

MLX Swift Overview

Apple's ML framework for Swift. Highest sustained generation throughput on Apple Silicon via unified memory architecture.

Loading and Running LLMs

import MLX
import MLXLLM
import MLXLMCommon
import MLXLMHFAPI

let container = try await LLMModelFactory.shared.loadContainer(
    from: HubClient.default,
    using: TokenizersLoader(),
    configuration: .init(id: "mlx-community/Qwen3-4B-4bit")
)
let session = ChatSession(container)
print(try await session.respond(to: "Hello"))

Model Selection by Device