Modly Image To 3d

Modly Image-to-3D Skill

Skill by ara.so — Daily 2026 Skills collection.

Modly is a local, open-source desktop application (Windows/Linux) that converts photos into 3D mesh models using AI models running entirely on your GPU — no cloud, no API keys required.

Architecture Overview

modly/
├── src/                    # Electron + TypeScript frontend
│   ├── main/               # Electron main process
│   ├── renderer/           # React UI (renderer process)
│   └── preload/            # IPC bridge
├── api/                    # Python FastAPI backend
│   ├── generator.py        # Core generation logic
│   └── requirements.txt
├── resources/
│   └── icons/
├── launcher.bat            # Windows quick-start
├── launcher.sh             # Linux quick-start
└── package.json

The app runs as an Electron shell over a local Python FastAPI server. Extensions are GitHub repos with a manifest.json + generator.py that plug into the extension system.

Installation

Quick start (no build required)

# Windows
launcher.bat

# Linux
chmod +x launcher.sh
./launcher.sh

Development setup

# 1. Clone
git clone https://github.com/lightningpixel/modly
cd modly

# 2. Install JS dependencies
npm install

# 3. Set up Python backend
cd api
python -m venv .venv

# Activate (Windows)
.venv\Scripts\activate

# Activate (Linux/macOS)
source .venv/bin/activate

pip install -r requirements.txt
cd ..

# 4. Run dev mode (starts Electron + Python backend)
npm run dev

Production build

# Build installers for current platform
npm run build

# Output goes to dist/

Key npm Scripts

npm run dev        # Start app in development mode (hot reload)
npm run build      # Package app for distribution
npm run lint       # Run ESLint
npm run typecheck  # TypeScript type checking

Extension System

Extensions are GitHub repositories containing:

• manifest.json — metadata and model variants
• generator.py — generation logic implementing the Modly extension interface

manifest.json structure

{
  "name": "My 3D Extension",
  "id": "my-extension-id",
  "description": "Generates 3D models using XYZ model",
  "version": "1.0.0",
  "author": "Your Name",
  "repository": "https://github.com/yourname/my-modly-extension",
  "variants": [
    {
      "id": "model-small",
      "name": "Small (faster)",
      "description": "Lighter variant for faster generation",
      "size_gb": 4.2,
      "vram_gb": 6,
      "files": [
        {
          "url": "https://huggingface.co/yourorg/yourmodel/resolve/main/weights.safetensors",
          "filename": "weights.safetensors",
          "sha256": "abc123..."
        }
      ]
    }
  ]
}

generator.py interface

# api/extensions/<extension-id>/generator.py
# Required interface every extension must implement

import sys
import json
from pathlib import Path

def generate(
    image_path: str,
    output_path: str,
    variant_id: str,
    models_dir: str,
    **kwargs
) -> dict:
    """
    Required entry point for all Modly extensions.
    
    Args:
        image_path:  Path to input image file
        output_path: Path where output .glb/.obj should be saved
        variant_id:  Which model variant to use
        models_dir:  Directory where downloaded model weights live
    
    Returns:
        dict with keys:
            success (bool)
            output_file (str) — path to generated mesh
            error (str, optional)
    """
    try:
        # Load your model weights
        weights = Path(models_dir) / variant_id / "weights.safetensors"
        
        # Run your inference
        mesh = run_inference(str(weights), image_path)
        
        # Save output
        mesh.export(output_path)
        
        return {
            "success": True,
            "output_file": output_path
        }
    except Exception as e:
        return {
            "success": False,
            "error": str(e)
        }

Installing an extension (UI flow)

1. Open Modly → go to Models page
2. Click Install from GitHub
3. Paste the HTTPS URL, e.g. https://github.com/lightningpixel/modly-hunyuan3d-mini-extension
4. After install, click Download on the desired model variant
5. Select the installed model and upload an image to generate

Official Extensions

Python Backend API (FastAPI)

The backend runs locally. Key endpoints used by the Electron frontend:

# Typical backend route patterns (api/main.py or similar)

# GET /extensions         — list installed extensions
# GET /extensions/{id}    — get extension details + variants
# POST /extensions/install — install extension from GitHub URL
# POST /generate          — trigger 3D generation
# GET /generate/status    — poll generation progress
# GET /models             — list downloaded model variants
# POST /models/download   — download a model variant

Calling the backend from Electron (IPC pattern)

// src/preload/index.ts — exposing backend calls to renderer
import { contextBridge, ipcRenderer } from 'electron'

contextBridge.exposeInMainWorld('modly', {
  generate: (imagePath: string, extensionId: string, variantId: string) =>
    ipcRenderer.invoke('generate', { imagePath, extensionId, variantId }),

  installExtension: (repoUrl: string) =>
    ipcRenderer.invoke('install-extension', { repoUrl }),

  listExtensions: () =>
    ipcRenderer.invoke('list-extensions'),
})

// src/main/ipc-handlers.ts — main process handling
import { ipcMain } from 'electron'

ipcMain.handle('generate', async (_event, { imagePath, extensionId, variantId }) => {
  const response = await fetch('http://localhost:PORT/generate', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ image_path: imagePath, extension_id: extensionId, variant_id: variantId }),
  })
  return response.json()
})

// src/renderer/components/GenerateButton.tsx — UI usage
declare global {
  interface Window {
    modly: {
      generate: (imagePath: string, extensionId: string, variantId: string) => Promise<{ success: boolean; output_file?: string; error?: string }>
      installExtension: (repoUrl: string) => Promise<{ success: boolean }>
      listExtensions: () => Promise<Extension[]>
    }
  }
}

async function handleGenerate(imagePath: string) {
  const result = await window.modly.generate(
    imagePath,
    'modly-hunyuan3d-mini-extension',
    'hunyuan3d-mini-turbo'
  )

  if (result.success) {
    console.log('Mesh saved to:', result.output_file)
  } else {
    console.error('Generation failed:', result.error)
  }
}

Writing a Custom Extension

Minimal extension repository structure

my-modly-extension/
├── manifest.json
└── generator.py

Example: wrapping a HuggingFace diffusion model

# generator.py
import torch
from PIL import Image
from pathlib import Path

def generate(image_path, output_path, variant_id, models_dir, **kwargs):
    device = "cuda" if torch.cuda.is_available() else "cpu"
    weights_dir = Path(models_dir) / variant_id

    try:
        # Load model (example pattern)
        from your_model_lib import ImageTo3DPipeline
        
        pipe = ImageTo3DPipeline.from_pretrained(
            str(weights_dir),
            torch_dtype=torch.float16
        ).to(device)

        image = Image.open(image_path).convert("RGB")
        
        with torch.no_grad():
            mesh = pipe(image).mesh

        mesh.export(output_path)

        return {"success": True, "output_file": output_path}

    except Exception as e:
        return {"success": False, "error": str(e)}

Configuration & Environment