Run Jazz with local and air-gapped models

Jazz can run against a self-hosted inference server such as Ollama, llama.cpp, vLLM, or SGLang. Local providers need no API key unless the server requires one. JAZZ_OFFLINE disables Jazz’s public catalog, library, and update requests; it is a convenience switch, not a network sandbox. Use host or container egress controls for an enforced air gap.

Quick setup (Ollama)

1. Install Ollama on the server and pull a tool-capable model:

   ollama pull qwen3

2. Set the environment for the Jazz process:

   export JAZZ_OFFLINE=1
   export OLLAMA_BASE_URL=http://127.0.0.1:11434

   Point OLLAMA_BASE_URL at the machine running Ollama if it lives elsewhere on the internal network. Alternatively use the config file:

   {
     "llm": {
       "ollama": {
         "base_url": "http://ollama.internal:11434"
       }
     }
   }

   Or set it interactively: run jazz, enter /settings, then choose Model providers, pick Ollama, llama.cpp, vLLM, or SGLang, and enter the server address as host:port (or a full URL). Jazz adds the scheme and the provider’s REST path for you, so 192.168.1.50:11434 is enough. This is the same base_url as above and takes precedence over the environment variable.

3. Create an agent and chat. Jazz lists models straight from Ollama’s /api/tags endpoint, so no external catalog is needed:

   jazz agent create
   jazz agent chat <agent-name>

llama.cpp works the same way via LLAMACPP_BASE_URL (default http://127.0.0.1:8080/v1), or the first-use server-URL prompt in jazz agent create; start llama-server with --jinja for tool calling.

For vLLM, choose the vllm provider. It defaults to http://127.0.0.1:8000/v1, accepts VLLM_BASE_URL, and reads the IDs available at /v1/models. Jazz selects a sole ID automatically, or asks you to choose among several. On each run it keeps the saved ID if still served, and otherwise uses the first live ID and its context window. Use a chat-capable model and configure tool calling on the server if the agent uses tools; the model list does not reveal whether the server’s tool parser is enabled. See vLLM setup.

For SGLang, choose sglang. It defaults to http://127.0.0.1:30000/v1, accepts SGLANG_BASE_URL, and uses the same live-model selection pattern. Its /v1/models cards report the served max_model_len for the base model; LoRA adapters inherit that window when their own card omits it. Configure the server’s tool and reasoning parsers for the model you load. See SGLang setup.

Jazz refreshes the served model and context window at the start of each run. For llama.cpp, the loaded model takes precedence over the saved agent model name. Setting llm.numCtx limits Jazz’s context accounting; it does not reconfigure the server.

Offline mode

With JAZZ_OFFLINE=1 (or true), Jazz skips these product-service requests:

• No update check: the periodic npm registry version check is skipped (equivalent to JAZZ_DISABLE_UPDATE_CHECK=1).
• No models.dev fetch: the model catalog (used for cloud-provider model lists and metadata enrichment like context windows and pricing) is not fetched. Jazz uses the configured catalog URL’s on-disk snapshot if one exists from a previous online run (~/.jazz/cache/models-dev.json for the default catalog), and otherwise falls back to provider-reported metadata and defaults.
• No library fetch: jazz persona browse, jazz workflow browse and jazz skill browse read the snapshots at ~/.jazz/cache/persona-registry.json, ~/.jazz/cache/workflow-registry.json and ~/.jazz/cache/skill-registry.json from a previous online run, and error if there is none. Installing from the library is refused offline. Point JAZZ_LIBRARY_URL at an internal library, with JAZZ_OFFLINE unset, to browse and install inside the airgap.

It does not block inference, web_fetch, http_request, remote MCP, OTLP export, or a command the agent runs. In an air-gapped deployment, use a local provider, disable OTLP export, omit network-capable tools and MCP servers, and enforce egress at the OS, container, or firewall boundary.

Model catalog options

Ollama, llama.cpp, vLLM, and SGLang agents work with no catalog at all: model lists come from the local server. Ollama’s /api/show and llama.cpp’s /props provide additional capability and context metadata; vLLM and SGLang use /v1/models for current IDs and context lengths. For an uncatalogued vLLM or SGLang model, Jazz allows tool configuration, but that does not confirm the server is set up to execute tool calls.

If you want catalog metadata (e.g. pricing display for cloud models) inside the airgap, either:

• Seed the snapshot: run Jazz once with network access (or copy ~/.jazz/cache/models-dev.json from another machine). Every successful catalog fetch refreshes this snapshot, and offline mode reads it automatically.
• Mirror internally: host a copy of https://models.dev/api.json on your network and set JAZZ_MODELS_DEV_URL=http://mirror.internal/api.json (leave JAZZ_OFFLINE unset so Jazz fetches from the mirror).

Each mirror keeps its own snapshot at ~/.jazz/cache/models-dev.<sha256-of-url>.json. Keep the same JAZZ_MODELS_DEV_URL when switching to offline mode; Jazz reads only that source’s snapshot and never substitutes a catalog from another URL. A snapshot without a source identifier from an older Jazz version is used only for the default catalog.

Other network surfaces to know about

• Web tools: the web_search tool requires a configured search provider API key and will simply error without one; web_fetch and http_request reach whatever URL the agent targets: inside an airgap they can still hit internal services, which is often desirable. Network enforcement should ultimately live at the firewall.
• MCP servers: stdio servers run locally; HTTP servers connect to the URL you configure.
• Telemetry: NDJSON is always local under ~/.jazz/telemetry; if an OTLP endpoint is configured, Jazz also exports to it. Leave OTLP unconfigured or set telemetry.otlp.enabled to false for a local-only deployment.

Environment variable reference