Flash

Runpod Flash

Write code locally, iterate with flash dev — it runs your functions on remote Runpod GPUs/CPUs with hot-reload and live worker logs — then flash deploy to ship. Endpoint handles provisioning.

runpod-flash releases on its own cadence, so `flash --help` and `flash <command> --help` are authoritative for the command surface — this skill is the mental model, the decision rules, and the gotchas that help output does not carry. Confirm the installed version with pip show runpod-flash before concluding a subcommand or flag is unavailable.

Worked examples first for anything multi-step. Flash appears in verified end-to-end paths — 03 variant B (whisper endpoint via flash) and 08 (fine-tune → serve); the full index is runpod/golden-paths/README.md. Open the matching path before planning a deploy — it carries the ordering and the cost cleanup this skill only summarizes.

Load on demand — this skill keeps the mental model + gotchas inline; details live in [`reference/`](reference/):

Quick start: uv tool install runpod-flash → flash login (or export RUNPOD_API_KEY=...) → flash init my-project → flash dev. Details in reference/setup-and-cli.md.

Dev vs Deploy

• flash dev — iterate. Local server at :8888, but your decorated functions execute on remote GPU/CPU workers. Hot-reloads on save and streams the worker's logs live to the terminal. No build/upload/deploy wait — use this the whole time you develop.
• flash deploy — ship. Builds an artifact and deploys a stable endpoint. Slow (build + upload + provision); only do this once the code works under flash dev.

flash dev ships only the function body to the worker, so a NameError for a module-level name surfaces immediately here. flash deploy imports the whole module and can mask that bug (see Gotcha #1). Develop against flash dev and you catch it first.

Autonomous Dev Loop

flash dev is a long-running server. Three rules:

• Run it in the background — don't block on it.
• Capture its output to a log file.
• Drive it over HTTP.

The captured log is the remote worker's live stream (cold start, model load, prints, tracebacks) — read it to debug.

flash dev > /tmp/flash-dev.log 2>&1 &                          # background; never run it blocking
for i in $(seq 1 60); do grep -q "flash dev  localhost:" /tmp/flash-dev.log && break; sleep 2; done  # bounded ~2min; if it never appears, check the log for errors
URL=$(grep -o "localhost:[0-9]*" /tmp/flash-dev.log | head -1)               # actual port (8888 bumps if taken)
curl -s "$URL/main/predict" -d '{"data": {...}}'               # dispatches to the remote worker

• Read the real URL from the log — flash auto-bumps the port if 8888 is in use, and prints ✓ flash dev localhost:<port> plus the route table.
• Routes are namespaced by file: main.py's /predict is served at /main/predict.
• Two route shapes, two body shapes (mismatch → 422 naming the missing field in loc):
• Load-balanced (@api.post("/predict")) → POST /main/predict, body is the arg at top level: a handler def predict(data: dict) wants {"data": {...}} (not the bare object).
• Queue-based (bare @Endpoint decorator) → POST /main/runsync (the local dev server only generates /runsync; production also exposes /run), body is double-wrapped in input: a handler def synthesize(data: dict) wants {"input": {"data": {...}}}. The outer input is the queue envelope; the inner key is the handler's param name.
• Edit a handler and save — hot-reload re-syncs the body; just re-send the request, no redeploy. Add --auto-provision to skip the first-call cold start. kill %1 when done.

Endpoint: Three Modes

Full constructor params and the GPU/CPU enum tables are in reference/api.md.

Mode 1: Your Code (Queue-Based Decorator)

One function = one endpoint with its own workers.

from runpod_flash import Endpoint, GpuGroup

@Endpoint(name="my-worker", gpu=GpuGroup.AMPERE_80, workers=5, dependencies=["torch"])
async def compute(data):
    import torch  # MUST import inside function (cloudpickle)
    return {"sum": torch.tensor(data, device="cuda").sum().item()}

result = await compute([1, 2, 3])

Mode 2: Your Code (Load-Balanced Routes)

Multiple HTTP routes share one pool of workers.

from runpod_flash import Endpoint, GpuGroup

api = Endpoint(name="my-api", gpu=GpuGroup.ADA_24, workers=(1, 5), dependencies=["torch"])

@api.post("/predict")
async def predict(data: list[float]):
    import torch
    return {"result": torch.tensor(data, device="cuda").sum().item()}

@api.get("/health")
async def health():
    return {"status": "ok"}

Mode 3: External Image (Client)

Deploy a pre-built Docker image and call it via HTTP.

from runpod_flash import Endpoint, GpuGroup, PodTemplate

server = Endpoint(
    name="my-server",
    image="my-org/my-image:latest",
    gpu=GpuGroup.AMPERE_80,
    workers=1,
    env={"HF_TOKEN": "xxx"},
    template=PodTemplate(containerDiskInGb=100),
)

# LB-style
result = await server.post("/v1/completions", {"prompt": "hello"})
models = await server.get("/v1/models")

# QB-style
job = await server.run({"prompt": "hello"})        # optional: webhook="https://..." for completion callback
await job.wait()
print(job.output)

Connect to an existing endpoint by ID (no provisioning):

ep = Endpoint(id="abc123")
job = await ep.runsync({"prompt": "hello"})  # runsync wraps this as {"input": {"prompt": "hello"}}
print(job.output)

How Mode Is Determined

The table above is how the mode is picked from params. When to reach for image=:

When to use image= (custom container) vs your own code

Default to writing Python (decorator / routes) — it runs arbitrary code with dependencies=[...]/system_dependencies=[...] and needs no Dockerfile. Even large HuggingFace models stay in decorator mode (weights stream at runtime — see reference/patterns.md → Loading ML models). Reach for image= only when you need:

• a pre-built inference server — vLLM, TensorRT-LLM (image="vllm/vllm-openai:latest", or runpod/worker-vllm, runpod/worker-comfy)
• system-level deps not pip-installable — a specific CUDA/cuDNN, OS libraries
• models baked into the image — to skip the runtime download entirely
• an existing Runpod Serverless worker — you already have a working image

Trade-off: image= mode can't run arbitrary Python (the image owns all logic) and the image must implement a Runpod Serverless handler. Full list + examples: https://docs.runpod.io/flash/custom-docker-images

Gotchas