Configure model providers

An agent pins a provider and model. Changing either does not change its persona, tools, memory scopes, workflow, or deployment surface.

Run jazz agent create for the normal setup path. The wizard discovers configured providers, fetches their current model catalogs, and stores the selected provider and model on the agent.

Supported providers

Where a provider is known by two names, Jazz accepts both. GEMINI_API_KEY is what Google’s own documentation and CLI use, so it works as well as the GOOGLE_GENERATIVE_AI_API_KEY the AI SDK reads; set either. When both are set the canonical one wins.

The provider identifiers below come from AVAILABLE_PROVIDERS in packages/core/src/constants/models.ts. Model names are not maintained in this page: hosted catalogs change frequently, so Jazz resolves them at runtime.

Store credentials

Jazz resolves a provider key in this order:

1. an agent-specific config.llm.apiKeys override;
2. the provider’s environment variable;
3. the stored global provider key (keyring, then config-file value).

The configuration wizard writes secrets to macOS Keychain or libsecret when available. On a host without a keyring it falls back to the protected Jazz secrets file. jazz config show and jazz config get redact resolved secrets, including those read from the keyring or the environment; add --reveal to print them.

Each Jazz home has separate credentials. After changing JAZZ_HOME or --data-dir, configure keys and MCP OAuth again for that home.

To set a key by hand, run jazz config set <provider> (for example jazz config set anthropic) and paste the key when asked, so it never lands in your shell history. Jazz trims a pasted key and, for providers with a free endpoint to ask (OpenAI, Anthropic, Gemini, OpenRouter, xAI, Cerebras, DeepSeek, Fireworks, Groq, Mistral, Together AI), checks it before saving: a key rejected with 401 is asked for again. A 403 saves the key with a warning because a restricted key may allow inference without allowing model listing; the first model request determines whether it can run. A provider pointed at a custom base_url is not checked.

In the interactive /settings → Model providers and Web search menus, a configured key stays masked. Press e to replace it or c to copy it to the system clipboard. Press Esc at the key prompt to keep the saved key and return to the provider list. Jazz never prints the key in the terminal. ChatGPT’s OAuth credential can also be copied from its provider settings. It includes the refresh token. For hosted CI, set the copied JSON bundle as JAZZ_CHATGPT_CREDENTIAL; Jazz removes it from its child-process environment, stores it in its normal private credential store, and refreshes it automatically. A workflow must persist rotated bundles if its runners are ephemeral.

For CI and containers, inject the environment variable from the platform’s secret store. Do not commit provider keys in an agent JSON file merely because llm.apiKeys exists.

ChatGPT subscription

The chatgpt provider runs OpenAI models on a ChatGPT Plus or Pro plan instead of API credits, through the same sign-in the Codex CLI uses. Usage counts against the plan’s limits, so Jazz shows no per-token cost for these models.

Run jazz config set chatgpt, or run jazz, type /settings, then choose Model providers, then ChatGPT, and pick how to sign in:

• Open a browser on this machine opens OpenAI’s sign-in page and receives the result on localhost:1455. Finish or cancel any Codex CLI sign-in first, because it uses the same port.
• Enter a code on another device shows a code to enter at auth.openai.com/codex/device from any phone or laptop. Use this over SSH and on servers without a browser.

Choosing ChatGPT for an agent in jazz agent create starts the same sign-in when you are not signed in yet.

The tokens are stored in the keyring (or the Jazz secrets file on hosts without one), and config.json records only the account ID and plan. Jazz refreshes the token on its own. The model list comes from your plan, so it only shows models the plan can use. Web search uses OpenAI’s built-in search unless you have chosen an external search provider.

For hosted CI, set JAZZ_CHATGPT_CREDENTIAL to the JSON bundle copied from /settings → Model providers → ChatGPT (c). Jazz reads it only when no stored ChatGPT credential exists and removes the variable before launching child processes. Refresh tokens rotate, so ephemeral workflows must persist a newly refreshed bundle before the next run; serialize all workflows that use the same credential. See CI runs for unattended setup and the pull-request reviewer for a working workflow.

If a request fails with Not signed in to ChatGPT, OpenAI has rejected the stored refresh token. Choose ChatGPT under /settings → Model providers and press r. Jazz tries to refresh the token, and when OpenAI rejects it, Jazz starts a new sign-in. Press e there to sign in again or sign out.

OpenRouter for model portability

OpenRouter is useful when the workflow should stay stable while the underlying hosted model changes. openrouter/free routes to an available free model; it is useful for experiments but not a reliability guarantee. openrouter/auto is also a router rather than a fixed model, so exact capabilities and pricing depend on the selected upstream model. Jazz requests OpenRouter usage accounting and uses its per-call usage.cost when returned, falling back to the catalog estimate otherwise. See OpenRouter’s usage accounting guide.

Create separate agents when you need a pinned production model and an experimental router. That keeps evaluation and cost attribution honest.

export OPENROUTER_API_KEY="..."
jazz agent create

The CI reviewer shows the same workflow running through OpenRouter or a self-hosted model.

Vercel AI Gateway

Vercel AI Gateway can select among upstream providers and reports the per-call amount in providerMetadata.gateway.cost. Jazz uses that amount when available. See Vercel’s AI Gateway pricing guide.

OrcaRouter

OrcaRouter exposes an OpenAI-compatible catalog of provider-prefixed model IDs, plus routed models such as orcarouter/auto. Jazz requests OrcaRouter’s per-call cost annotation and uses its reported charge in the footer when present; otherwise the footer falls back to the catalog-based estimate. The router’s automatically selected upstream can differ between calls, so its reported charge is more accurate than estimating from the configured router model. See OrcaRouter’s per-request cost documentation.

export ORCAROUTER_API_KEY="..."
jazz agent create

NVIDIA NIM

The nvidia provider talks to NVIDIA’s hosted NIM API at https://integrate.api.nvidia.com/v1. Get a key from NVIDIA Build; new accounts receive free inference credits. Jazz lists the models your key can call from NIM’s /v1/models, leaving out embedding, reranking, guardrail and document-parsing models, and fills in context windows and tool support from the models.dev catalog. A model the catalog lists only under another host keeps that entry’s context window and tool support but shows no price, since the price belongs to that host. A model the catalog does not list at all gets a 128,000-token window and is assumed to call tools, like any cloud model nothing describes. If it cannot, NIM rejects the request; set "supportsTools": false for it under llm.capabilityOverrides. Setting "supportsTools": true there also makes jazz agent create offer tool selection for it.

export NVIDIA_API_KEY="nvapi-..."
jazz agent create

Jazz sends no reasoning control to NIM by default, so each model reasons the way its deployment is configured and an agent’s reasoning setting has no effect. NIM rejects request fields a model’s schema does not declare, and models differ in which reasoning field they accept, so a guessed control would fail the request. To control reasoning for a model you have tested, declare it under llm.capabilityOverrides. Check which field the model’s NIM page documents before adding an entry. This one declares DeepSeek V4.1 Flash’s reasoning_effort ladder: NIM accepts low, high, max and none for it and rejects medium, so Jazz runs a medium agent at low:

{
  "llm": {
    "capabilityOverrides": {
      "nvidia": {
        "deepseek-ai/deepseek-v4.1-flash": {
          "reasoning": {
            "kind": "effort",
            "transport": "openai-compatible.chat.reasoning-effort",
            "efforts": ["low", "high", "max"],
            "canDisableReasoning": true
          }
        }
      }
    }
  }
}

Ollama

Ollama agents require no key when the server is local and unprotected. Jazz discovers models from the running server.

ollama pull qwen3-coder
ollama serve
jazz agent create

The default API base URL is http://127.0.0.1:11434/api. The first time jazz agent create uses Ollama, it asks for the server URL and saves it; override it later with llm.ollama.base_url, OLLAMA_BASE_URL, or /settings → Model providers in the jazz home screen, which accepts a bare host:port and fills in the scheme and REST path; saved configuration wins over the environment.

{
  "llm": {
    "ollama": {
      "base_url": "http://ollama.internal:11434/api",
      "keep_alive": "30m"
    }
  }
}

During agent creation Jazz asks for llm.numCtx, the context window sent to Ollama as num_ctx. Set it to a value the model and host can actually sustain. Without it, Jazz reads the window Ollama is running the model at from /api/ps once the model is loaded, which reflects a server-level OLLAMA_CONTEXT_LENGTH. Before the model is loaded, or when the server does not answer, it falls back to the estimate and may compact later than the server truncates.

To point one agent at a different Ollama server than the global llm.ollama.base_url, set llm.baseUrls in its agent file. The same field works for llamacpp, vllm and sglang; the agent’s URL wins over the global setting and the provider’s environment variable.

{
  "llm": {
    "provider": "ollama",
    "model": "saluki-27b",
    "baseUrls": { "ollama": "http://ollama.internal:11434" }
  }
}

keep_alive controls how long Ollama keeps the model loaded. -1 keeps it resident; an omitted value uses Ollama’s default.

Models tagged :cloud or -cloud execute through Ollama Cloud and need OLLAMA_API_KEY or an authenticated local Ollama installation. They are not local merely because the provider ID is ollama.

llama.cpp

Jazz connects to llama-server through the llamacpp provider. The default base URL is http://127.0.0.1:8080/v1; the first time jazz agent create uses it, Jazz asks for the server URL and saves it. You can also set llm.llamacpp.base_url, LLAMACPP_BASE_URL, or use /settings → Model providers in the jazz home screen (a bare host:port is enough). llama.cpp needs no API key unless the server runs with --api-key; when it answers 401, jazz agent create asks for the key, and /settings → Model providers can set it.

llama-server -m /models/model.gguf --jinja --port 8080
jazz agent create

Use --jinja when the model should call tools. Jazz reads /props for context and chat-template metadata when the server provides it. For reasoning models, Jazz maps the agent’s reasoning effort string to llama.cpp’s supported thinking controls; behavior depends on a recent server and a compatible template.

vLLM

Use the separate vllm provider for a vLLM server. Its default base URL is http://127.0.0.1:8000/v1. On first use, jazz agent create asks for the address and saves llm.vllm.base_url; you can also set VLLM_BASE_URL or use /settings → Model providers in the jazz home screen. Jazz reads /v1/models: it selects the sole served ID automatically, or asks you to choose when the server lists several. On each run, Jazz refreshes the model list and uses the saved ID while it is still listed; if it disappears, Jazz uses the first live ID and its context window. Configure llm.vllm.api_key or VLLM_API_KEY if the server uses --api-key; an authorization error during agent creation prompts for the key.

vllm serve <model> --port 8000
jazz agent create

Jazz uses vLLM’s OpenAI-compatible chat API. Tool calls require a compatible model and server configuration, commonly --enable-auto-tool-choice with the matching --tool-call-parser; /v1/models alone cannot prove those flags are enabled. Follow vLLM’s tool-calling guide for the model-specific setup. For a server reachable beyond the host, protect the deployment at the network boundary: vLLM documents that --api-key does not authenticate every endpoint.

SGLang

Use the sglang provider for an SGLang server. The default base URL is http://127.0.0.1:30000/v1. On first use, jazz agent create asks for the address and saves llm.sglang.base_url; SGLANG_BASE_URL and /settings → Model providers in the jazz home screen also work. Jazz discovers current IDs from /v1/models, selecting the sole model automatically or offering a picker for multiple IDs such as loaded LoRA adapters. At the start of each run it keeps the saved ID if still served, otherwise uses the first current ID. It also reads the live max_model_len for context accounting; a LoRA card without a length inherits its listed base model’s length.

python -m sglang.launch_server --model-path <model> --port 30000
jazz agent create

Set llm.sglang.api_key or SGLANG_API_KEY if the server requires bearer authentication. Jazz prompts for a key when model listing returns 401 or 403. SGLang uses the OpenAI-compatible chat API. Tool calls need a compatible model and the matching --tool-call-parser; reasoning output needs an appropriate --reasoning-parser for the model. /v1/models does not report parser configuration, so test those features against your server. See SGLang’s reasoning guide and server argument reference. llm.numCtx limits Jazz’s accounting; it does not change the server’s context length.

Model capability overrides

Models.dev supplies broad metadata such as context length, tool support, and whether a model reasons. It does not describe every provider or local-template reasoning control. Jazz applies its own exact provider + model capability profiles after catalog metadata. For a private model or a self-hosted template, an operator can provide a strict local correction under llm.capabilityOverrides; overrides use only Jazz-defined transports and cannot inject arbitrary provider request fields.

{
  "llm": {
    "capabilityOverrides": {
      "llamacpp": {
        "Qwen3-32B-Instruct": {
          "reasoning": {
            "kind": "budget",
            "transport": "openai-compatible.chat.template-thinking-budget",
            "minimumBudgetTokens": 256,
            "maximumBudgetTokens": 32768,
            "canDisableReasoning": true
          },
          "supportsTools": true
        }
      }
    }
  }
}

Keys are exact server-facing model IDs. To set one field from the command line, quote a model ID that contains dots: jazz config set 'llm.capabilityOverrides.nvidia."deepseek-ai/deepseek-v4.1-flash".supportsTools' true. An override also corrects the model lists jazz agent create and jazz agent edit show, so their tool and reasoning steps match what requests do. Resolution is operator override, live local-server metadata, Jazz’s exact-model profile, provider default, then Models.dev. When none of them says whether a cloud model takes tools (a model released after the catalog, a custom id, or the catalog unreachable), Jazz assumes it does. A local server’s model gets tools only once the server reports them. Whenever a run sends no tools to an agent that has some, every surface warns: stderr for jazz run, the chat transcript, toolsDisabled: true in the JSON envelope, and a notice under the bot’s answer. A llama.cpp budget control is never assumed from a model family: declare it only when the active template accepts it.

Transports name the request field Jazz sends, not a vendor. OpenAI-compatible providers (llamacpp, vllm, sglang, nvidia, orcarouter) accept all three openai-compatible.chat.* transports:

When a profile lists efforts, a requested level the model does not list is lowered to the nearest listed level below it, or raised to the lowest listed level when none is below it; Jazz never raises it further. disable on a profile with "canDisableReasoning": false becomes the lowest level, or keeps reasoning on for a toggle. The /reasoning picker and jazz agent create/edit offer only the levels the model’s profile accepts; with no profile they offer every level. A typed /reasoning <level> or jazz run --reasoning <level> the model does not accept prints the level it runs at, and jazz agent edit shows it next to the saved level.

A bare llama-server serves the one model loaded at launch and ignores the requested model name, and that model can change between runs. Jazz therefore treats the model chosen at agent creation as a hint: at the start of each run it reads the actually-served model from /v1/models and the real context window from /props, so the displayed model and context accounting match what the server is running. A pinned llm.numCtx still overrides the server-reported window.

Slow first tokens from local models

Jazz abandons a provider stream that stays silent for llm.streamIdleTimeoutMs milliseconds once output is flowing, 120000 by default, and reports Provider stream produced no additional part for 120s and was abandoned. The timer restarts on every streamed part, so it never caps a long answer, and tools run between streams rather than inside one.

The wait for the first token has its own budget: five minutes, or llm.streamIdleTimeoutMs when that is higher. That covers NVIDIA NIM, which can queue a request for over three minutes, and large reasoning models that prefill for a long time. Ollama, llama.cpp, vLLM, or SGLang loading a large model from disk and then prefilling a long prompt can take longer still, so raise the budget for those hosts:

jazz config set llm.streamIdleTimeoutMs 600000

JAZZ_STREAM_IDLE_TIMEOUT_MS sets the same budget for one process; the saved value wins over it. llm.ollama.keep_alive avoids paying the cold start again.

Gemini naming

The Jazz provider ID is gemini; its SDK and environment variable retain Google’s upstream naming. Existing google agent and configuration identifiers are migrated to gemini when read.

Offline operation

JAZZ_OFFLINE=1 disables Jazz’s own update, hosted model-catalog, and library requests. It does not make a hosted provider work offline. Use Ollama, llama.cpp, vLLM, or SGLang, preinstall every required skill dependency, and enforce the network boundary outside Jazz. Follow Local and air-gapped models.

Diagnose provider failures

• Authentication errors: confirm the agent’s provider ID matches the key you supplied, and run jazz config show to see which provider settings Jazz resolved.
• Stream idle errors say whether the provider produced no first part or stopped between parts. The former points to queuing, model loading, or prompt prefill (allowed five minutes); the latter means generation had already started (allowed 120000 ms). Local servers that legitimately need longer can set llm.streamIdleTimeoutMs or JAZZ_STREAM_IDLE_TIMEOUT_MS.
• Retries: each model call gets maxRetries retries in total. After three streaming failures the remaining attempts use a plain request; a rate limit (429) keeps streaming. Backoff is jittered and follows the provider’s Retry-After; a provider asking for more than two minutes fails the call instead of waiting.
• “Prompt is too long”: Jazz compacts the conversation (or trims it when it cannot compact) and retries once. A second rejection fails the run; lower the agent’s llm.maxContextTokens to match what the server really holds.
• Unknown model: rerun agent editing after the provider catalog is reachable; do not copy a model name from an old documentation page.
• Local connection errors: start the server and verify its base URL from the Jazz host, not from your laptop when Jazz runs elsewhere.
• Tool-call failures on llama.cpp: confirm the model template supports tools and the server was started with --jinja.
• Tool-call failures on vLLM: confirm the model’s chat template, automatic tool choice, and matching parser are configured on the server.
• Tool-call failures on SGLang: confirm the model’s chat template and matching --tool-call-parser are configured on the server.
• Unexpected context truncation on Ollama: pin llm.numCtx on the agent and ensure the server can allocate it.

Read Creating agents, Local models, and Adding a provider next.