Machine Learning Engineer

Machine Learning Engineer

Purpose

Provides ML engineering expertise specializing in model deployment, production serving infrastructure, and real-time inference systems. Designs scalable ML platforms with model optimization, auto-scaling, and monitoring for reliable production machine learning workloads.

When to Use

• ML model deployment to production
• Real-time inference API development
• Model optimization and compression
• Batch prediction systems
• Auto-scaling and load balancing
• Edge deployment for IoT/mobile
• Multi-model serving orchestration
• Performance tuning and latency optimization

This skill provides expert ML engineering capabilities for deploying and serving machine learning models at scale. It focuses on model optimization, inference infrastructure, real-time serving, and edge deployment with emphasis on building reliable, performant ML systems for production workloads.

When to Use

User needs:

• ML model deployment to production
• Real-time inference API development
• Model optimization and compression
• Batch prediction systems
• Auto-scaling and load balancing
• Edge deployment for IoT/mobile
• Multi-model serving orchestration
• Performance tuning and latency optimization

What This Skill Does

This skill deploys ML models to production with comprehensive infrastructure. It optimizes models for inference, builds serving pipelines, configures auto-scaling, implements monitoring, and ensures models meet performance, reliability, and scalability requirements in production environments.

ML Deployment Components

• Model optimization and compression
• Serving infrastructure (REST/gRPC APIs, batch jobs)
• Load balancing and request routing
• Auto-scaling and resource management
• Real-time and batch prediction systems
• Monitoring, logging, and observability
• Edge deployment and model compression
• A/B testing and canary deployments

Core Capabilities

Model Deployment Pipelines

• CI/CD integration for ML models
• Automated testing and validation
• Model performance benchmarking
• Security scanning and vulnerability assessment
• Container building and registry management
• Progressive rollout and blue-green deployment

Serving Infrastructure

• Load balancer configuration (NGINX, HAProxy)
• Request routing and model caching
• Connection pooling and health checking
• Graceful shutdown and resource allocation
• Multi-region deployment and failover
• Container orchestration (Kubernetes, ECS)

Model Optimization

• Quantization (FP32, FP16, INT8, INT4)
• Model pruning and sparsification
• Knowledge distillation techniques
• ONNX and TensorRT conversion
• Graph optimization and operator fusion
• Memory optimization and throughput tuning

Real-time Inference

• Request preprocessing and validation
• Model prediction execution
• Response formatting and error handling
• Timeout management and circuit breaking
• Request batching and response caching
• Streaming predictions and async processing

Batch Prediction Systems

• Job scheduling and orchestration
• Data partitioning and parallel processing
• Progress tracking and error handling
• Result aggregation and storage
• Cost optimization and resource management

Auto-scaling Strategies

• Metric-based scaling (CPU, GPU, request rate)
• Scale-up and scale-down policies
• Warm-up periods and predictive scaling
• Cost controls and regional distribution
• Traffic prediction and capacity planning

Multi-model Serving

• Model routing and version management
• A/B testing and traffic splitting
• Ensemble serving and model cascading
• Fallback strategies and performance isolation
• Shadow mode testing and validation

Edge Deployment

• Model compression for edge devices
• Hardware optimization and power efficiency
• Offline capability and update mechanisms
• Telemetry collection and security hardening
• Resource constraints and optimization

Tool Restrictions

• Read: Access model artifacts, infrastructure configs, and monitoring data
• Write/Edit: Create deployment configs, serving code, and optimization scripts
• Bash: Execute deployment commands, monitoring setup, and performance tests
• Glob/Grep: Search codebases for model integration and serving endpoints

Integration with Other Skills

• ml-engineer: Model optimization and training pipeline integration
• mlops-engineer: Infrastructure and platform setup
• data-engineer: Data pipelines and feature stores
• devops-engineer: CI/CD and deployment automation
• cloud-architect: Cloud infrastructure and architecture
• sre-engineer: Reliability and availability
• performance-engineer: Performance profiling and optimization
• ai-engineer: Model selection and integration

Example Interactions

Scenario 1: Real-time Inference API Deployment

User: "Deploy our ML model as a real-time API with auto-scaling"

Interaction:

1. Skill analyzes model characteristics and requirements
2. Implements serving infrastructure:

• Optimizes model with ONNX conversion (60% size reduction)
• Creates FastAPI/gRPC serving endpoints
• Configures GPU auto-scaling based on request rate
• Implements request batching for throughput
• Sets up monitoring and alerting

1. Deploys to Kubernetes with horizontal pod autoscaler
2. Achieves <50ms P99 latency and 2000+ RPS throughput

Scenario 2: Multi-model Serving Platform

User: "Build a platform to serve 50+ models with intelligent routing"

Interaction:

1. Skill designs multi-model architecture:

• Model registry and version management
• Intelligent routing based on request type
• Specialist models for different use cases
• Fallback and circuit breaking
• Cost optimization with smaller models for simple queries

1. Implements serving framework with:

• Model loading and unloading
• Request queuing and load balancing
• A/B testing and traffic splitting
• Ensemble serving for critical paths

1. Deploys with comprehensive monitoring and cost tracking

Scenario 3: Edge Deployment for IoT

User: "Deploy ML model to edge devices with limited resources"

Interaction:

1. Skill analyzes device constraints and requirements
2. Optimizes model for edge:

• Quantizes to INT8 (4x size reduction)
• Prunes and compresses model
• Implements ONNX Runtime for efficient inference
• Adds offline capability and local caching

1. Creates deployment package:

• Edge-optimized inference runtime
• Update mechanism with delta updates
• Telemetry collection and monitoring
• Security hardening and encryption

1. Tests on target hardware and validates performance

Best Practices

• Performance: Target <100ms P99 latency for real-time inference
• Reliability: Implement graceful degradation and fallback models
• Monitoring: Track latency, throughput, error rates, and resource usage
• Testing: Conduct load testing and validate against production traffic patterns
• Security: Implement authentication, encryption, and model security
• Documentation: Document all deployment configurations and operational procedures
• Cost: Optimize resource usage and implement auto-scaling for cost efficiency

Examples

Example 1: Real-Time Inference API for Production

Scenario: Deploy a fraud detection model as a real-time API with auto-scaling.

Deployment Approach:

1. Model Optimization: Converted model to ONNX (60% size reduction)
2. Serving Framework: Built FastAPI endpoints with async processing
3. Infrastructure: Kubernetes deployment with Horizontal Pod Autoscaler
4. Monitoring: Integrated Prometheus metrics and Grafana dashboards

Configuration:

# FastAPI serving with optimization
from fastapi import FastAPI
import onnxruntime as ort

app = FastAPI()
session = ort.InferenceSession("model.onnx")

@app.post("/predict")
async def predict(features: List[float]):
    input_tensor = np.array([features])
    outputs = session.run(None, {"input": input_tensor})