Docker model

Docker Model Runner Skill

Docker Model Runner (DMR) manages and serves AI models through Docker Desktop or Docker Engine, exposing OpenAI-compatible APIs. Models are pulled as OCI artifacts from Docker Hub (ai/ namespace), any OCI registry, or Hugging Face, and stored locally. For Drupal work it provides a free, local, keyless backend for the AI module ecosystem during development.

---

Enabling

• Docker Desktop: Settings → enable Docker Model Runner (Beta features).
• Docker Engine (Linux): supported without Desktop; models are served on the host. GPU support: NVIDIA (CUDA), AMD (ROCm), Vulkan; Apple Silicon on macOS; CPU everywhere.

---

Core CLI

docker model status                      # is the runner active?
docker model pull ai/smollm2             # fetch a model (Docker Hub ai/ namespace)
docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF  # from Hugging Face
docker model list                        # local models
docker model run ai/smollm2 "Hello"      # one-shot prompt
docker model run ai/smollm2              # interactive chat (exit with /bye)
docker model configure --context-size 8192 ai/smollm2   # adjust context window
docker model inspect ai/smollm2          # model metadata
docker model logs                        # runner logs
docker model rm ai/smollm2               # delete local model

Run docker model --help for the full, current command list — the CLI is still evolving.

---

OpenAI-compatible API

Endpoint	Method
/engines/v1/models	GET
/engines/v1/chat/completions	POST
/engines/v1/completions	POST
/engines/v1/embeddings	POST

Base URLs:

• From the host: http://localhost:12434 (default TCP port)
• From containers (Docker Desktop): http://model-runner.docker.internal
• From containers (Docker Engine): http://172.17.0.1:12434

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "ai/smollm2", "messages": [{"role": "user", "content": "Hi"}]}'

Any OpenAI SDK works by pointing base_url at http://localhost:12434/engines/v1 — no API key required.

---

Compose integration — models: top-level element

Declare models next to services; Compose pulls and provisions them:

services:
  app:
    image: my-app
    models:
      - llm                      # short syntax

models:
  llm:
    model: ai/smollm2
    context_size: 4096
    runtime_flags:
      - "--no-prefill-assistant"

Short syntax injects environment variables into the service container, named after the model key: LLM_URL and LLM_MODEL. Long syntax picks your own variable names:

services:
  app:
    image: my-app
    models:
      llm:
        endpoint_var: AI_MODEL_URL
        model_var: AI_MODEL_NAME

---

Using DMR as a Drupal AI backend

The Drupal AI module (drupal/ai) talks to providers over the OpenAI API. Point an OpenAI-compatible provider (e.g. drupal/ai_provider_openai) at the Model Runner endpoint to develop AI features without cloud keys:

• Base URL (Drupal in a container, Docker Desktop): http://model-runner.docker.internal/engines/v1
• Base URL (Drupal on the host): http://localhost:12434/engines/v1
• API key: any non-empty placeholder — DMR does not check it.
• Model name: exactly as listed by docker model list (e.g. ai/smollm2).

This gives local, reproducible AI development for content generation, embeddings/search experiments, and automated tests without external costs.

---

Troubleshooting

Symptom	Fix
docker model: command not found	Enable Model Runner in Docker Desktop settings, or install the plugin on Docker Engine
Connection refused on 12434	Enable host-side TCP support in the Model Runner settings; check docker model status
Container cannot reach model-runner.docker.internal	On Docker Engine use http://172.17.0.1:12434 instead
Responses truncated	Raise the context window: docker model configure --context-size <n> <model>
Model too slow / out of memory	Pull a smaller quantized variant from the ai/ namespace; check GPU is actually used (docker model logs)