# Configure model providers An agent pins a provider and model. Changing either does not change its persona, tools, memory scopes, workflow, or deployment surface. Run `jazz agent create` for the normal setup path. The wizard discovers configured providers, fetches their current model catalogs, and stores the selected provider and model on the agent. ## Supported providers Where a provider is known by two names, Jazz accepts both. `GEMINI_API_KEY` is what Google's own documentation and CLI use, so it works as well as the `GOOGLE_GENERATIVE_AI_API_KEY` the AI SDK reads; set either. When both are set the canonical one wins. The provider identifiers below come from `AVAILABLE_PROVIDERS` in [`packages/core/src/constants/models.ts`](../../packages/core/src/constants/models.ts). Model names are not maintained in this page: hosted catalogs change frequently, so Jazz resolves them at runtime. | Provider ID | API-key environment variable | | ------------ | ----------------------------------------------------------------- | | `ai_gateway` | `AI_GATEWAY_API_KEY` | | `alibaba` | `ALIBABA_API_KEY` | | `anthropic` | `ANTHROPIC_API_KEY` | | `cerebras` | `CEREBRAS_API_KEY` | | `chatgpt` | None: sign in with a ChatGPT plan (see below) | | `deepseek` | `DEEPSEEK_API_KEY` | | `fireworks` | `FIREWORKS_API_KEY` | | `gemini` | `GOOGLE_GENERATIVE_AI_API_KEY`, or `GEMINI_API_KEY` | | `groq` | `GROQ_API_KEY` | | `llamacpp` | `LLAMACPP_API_KEY` when the server requires bearer authentication | | `minimax` | `MINIMAX_API_KEY` | | `mistral` | `MISTRAL_API_KEY` | | `moonshotai` | `MOONSHOT_API_KEY` | | `nvidia` | `NVIDIA_API_KEY`, or `NIM_API_KEY` | | `ollama` | `OLLAMA_API_KEY` for Ollama Cloud or protected servers | | `openai` | `OPENAI_API_KEY` | | `openrouter` | `OPENROUTER_API_KEY` | | `orcarouter` | `ORCAROUTER_API_KEY` | | `sglang` | `SGLANG_API_KEY` when the server requires bearer authentication | | `togetherai` | `TOGETHER_AI_API_KEY` | | `vllm` | `VLLM_API_KEY` when the server requires bearer authentication | | `xai` | `XAI_API_KEY` | | `zhipuai` | `ZHIPU_API_KEY` | ## Store credentials Jazz resolves a provider key in this order: 1. an agent-specific `config.llm.apiKeys` override; 2. the provider's environment variable; 3. the stored global provider key (keyring, then config-file value). The configuration wizard writes secrets to macOS Keychain or libsecret when available. On a host without a keyring it falls back to the protected Jazz secrets file. `jazz config show` and `jazz config get` redact resolved secrets, including those read from the keyring or the environment; add `--reveal` to print them. Each Jazz home has separate credentials. After changing `JAZZ_HOME` or `--data-dir`, configure keys and MCP OAuth again for that home. To set a key by hand, run `jazz config set ` (for example `jazz config set anthropic`) and paste the key when asked, so it never lands in your shell history. Jazz trims a pasted key and, for providers with a free endpoint to ask (OpenAI, Anthropic, Gemini, OpenRouter, xAI, Cerebras, DeepSeek, Fireworks, Groq, Mistral, Together AI), checks it before saving: a key rejected with `401` is asked for again. A `403` saves the key with a warning because a restricted key may allow inference without allowing model listing; the first model request determines whether it can run. A provider pointed at a custom `base_url` is not checked. In the interactive `/settings` → **Model providers** and **Web search** menus, a configured key stays masked. Press `e` to replace it or `c` to copy it to the system clipboard. Press `Esc` at the key prompt to keep the saved key and return to the provider list. Jazz never prints the key in the terminal. ChatGPT's OAuth credential can also be copied from its provider settings. It includes the refresh token. For hosted CI, set the copied JSON bundle as `JAZZ_CHATGPT_CREDENTIAL`; Jazz removes it from its child-process environment, stores it in its normal private credential store, and refreshes it automatically. A workflow must persist rotated bundles if its runners are ephemeral. For CI and containers, inject the environment variable from the platform's secret store. Do not commit provider keys in an agent JSON file merely because `llm.apiKeys` exists. ## ChatGPT subscription The `chatgpt` provider runs OpenAI models on a ChatGPT Plus or Pro plan instead of API credits, through the same sign-in the Codex CLI uses. Usage counts against the plan's limits, so Jazz shows no per-token cost for these models. Run `jazz config set chatgpt`, or run `jazz`, type `/settings`, then choose **Model providers**, then **ChatGPT**, and pick how to sign in: - **Open a browser on this machine** opens OpenAI's sign-in page and receives the result on `localhost:1455`. Finish or cancel any Codex CLI sign-in first, because it uses the same port. - **Enter a code on another device** shows a code to enter at `auth.openai.com/codex/device` from any phone or laptop. Use this over SSH and on servers without a browser. Choosing ChatGPT for an agent in `jazz agent create` starts the same sign-in when you are not signed in yet. The tokens are stored in the keyring (or the Jazz secrets file on hosts without one), and `config.json` records only the account ID and plan. Jazz refreshes the token on its own. The model list comes from your plan, so it only shows models the plan can use. Web search uses OpenAI's built-in search unless you have chosen an external search provider. For hosted CI, set `JAZZ_CHATGPT_CREDENTIAL` to the JSON bundle copied from `/settings` → **Model providers** → **ChatGPT** (`c`). Jazz reads it only when no stored ChatGPT credential exists and removes the variable before launching child processes. Refresh tokens rotate, so ephemeral workflows must persist a newly refreshed bundle before the next run; serialize all workflows that use the same credential. See [CI runs](../surfaces/ci.md) for unattended setup and the [pull-request reviewer](../guides/pr-review.md) for a working workflow. If a request fails with `Not signed in to ChatGPT`, OpenAI has rejected the stored refresh token. Choose **ChatGPT** under `/settings` → **Model providers** and press `r`. Jazz tries to refresh the token, and when OpenAI rejects it, Jazz starts a new sign-in. Press `e` there to sign in again or sign out. ## OpenRouter for model portability OpenRouter is useful when the workflow should stay stable while the underlying hosted model changes. `openrouter/free` routes to an available free model; it is useful for experiments but not a reliability guarantee. `openrouter/auto` is also a router rather than a fixed model, so exact capabilities and pricing depend on the selected upstream model. Jazz requests OpenRouter usage accounting and uses its per-call `usage.cost` when returned, falling back to the catalog estimate otherwise. See OpenRouter's [usage accounting guide](https://openrouter.ai/docs/use-cases/usage-accounting). Create separate agents when you need a pinned production model and an experimental router. That keeps evaluation and cost attribution honest. ```bash export OPENROUTER_API_KEY="..." jazz agent create ``` The [CI reviewer](../guides/pr-review.md) shows the same workflow running through OpenRouter or a self-hosted model. ## Vercel AI Gateway Vercel AI Gateway can select among upstream providers and reports the per-call amount in `providerMetadata.gateway.cost`. Jazz uses that amount when available. See Vercel's [AI Gateway pricing guide](https://vercel.com/academy/ai-gateway/ai-gateway-pricing). ## OrcaRouter OrcaRouter exposes an OpenAI-compatible catalog of provider-prefixed model IDs, plus routed models such as `orcarouter/auto`. Jazz requests OrcaRouter's per-call cost annotation and uses its reported charge in the footer when present; otherwise the footer falls back to the catalog-based estimate. The router's automatically selected upstream can differ between calls, so its reported charge is more accurate than estimating from the configured router model. See OrcaRouter's [per-request cost documentation](https://docs.orcarouter.ai/operations/per-request-cost). ```bash export ORCAROUTER_API_KEY="..." jazz agent create ``` ## NVIDIA NIM The `nvidia` provider talks to NVIDIA's hosted NIM API at `https://integrate.api.nvidia.com/v1`. Get a key from [NVIDIA Build](https://build.nvidia.com/); new accounts receive free inference credits. Jazz lists the models your key can call from NIM's `/v1/models`, leaving out embedding, reranking, guardrail and document-parsing models, and fills in context windows and tool support from the models.dev catalog. A model the catalog lists only under another host keeps that entry's context window and tool support but shows no price, since the price belongs to that host. A model the catalog does not list at all gets a 128,000-token window and is assumed to call tools, like any cloud model nothing describes. If it cannot, NIM rejects the request; set `"supportsTools": false` for it under [`llm.capabilityOverrides`](#model-capability-overrides). Setting `"supportsTools": true` there also makes `jazz agent create` offer tool selection for it. ```bash export NVIDIA_API_KEY="nvapi-..." jazz agent create ``` Jazz sends no reasoning control to NIM by default, so each model reasons the way its deployment is configured and an agent's reasoning setting has no effect. NIM rejects request fields a model's schema does not declare, and models differ in which reasoning field they accept, so a guessed control would fail the request. To control reasoning for a model you have tested, declare it under [`llm.capabilityOverrides`](#model-capability-overrides). Check which field the model's NIM page documents before adding an entry. This one declares DeepSeek V4.1 Flash's `reasoning_effort` ladder: NIM accepts `low`, `high`, `max` and `none` for it and rejects `medium`, so Jazz runs a `medium` agent at `low`: ```json { "llm": { "capabilityOverrides": { "nvidia": { "deepseek-ai/deepseek-v4.1-flash": { "reasoning": { "kind": "effort", "transport": "openai-compatible.chat.reasoning-effort", "efforts": ["low", "high", "max"], "canDisableReasoning": true } } } } } } ``` ## Ollama Ollama agents require no key when the server is local and unprotected. Jazz discovers models from the running server. ```bash ollama pull qwen3-coder ollama serve jazz agent create ``` The default API base URL is `http://127.0.0.1:11434/api`. The first time `jazz agent create` uses Ollama, it asks for the server URL and saves it; override it later with `llm.ollama.base_url`, `OLLAMA_BASE_URL`, or `/settings` → **Model providers** in the `jazz` home screen, which accepts a bare `host:port` and fills in the scheme and REST path; saved configuration wins over the environment. ```json { "llm": { "ollama": { "base_url": "http://ollama.internal:11434/api", "keep_alive": "30m" } } } ``` During agent creation Jazz asks for `llm.numCtx`, the context window sent to Ollama as `num_ctx`. Set it to a value the model and host can actually sustain. Without it, Jazz reads the window Ollama is running the model at from `/api/ps` once the model is loaded, which reflects a server-level `OLLAMA_CONTEXT_LENGTH`. Before the model is loaded, or when the server does not answer, it falls back to the estimate and may compact later than the server truncates. To point one agent at a different Ollama server than the global `llm.ollama.base_url`, set `llm.baseUrls` in its agent file. The same field works for `llamacpp`, `vllm` and `sglang`; the agent's URL wins over the global setting and the provider's environment variable. ```json { "llm": { "provider": "ollama", "model": "saluki-27b", "baseUrls": { "ollama": "http://ollama.internal:11434" } } } ``` `keep_alive` controls how long Ollama keeps the model loaded. `-1` keeps it resident; an omitted value uses Ollama's default. Models tagged `:cloud` or `-cloud` execute through Ollama Cloud and need `OLLAMA_API_KEY` or an authenticated local Ollama installation. They are not local merely because the provider ID is `ollama`. ## llama.cpp Jazz connects to `llama-server` through the `llamacpp` provider. The default base URL is `http://127.0.0.1:8080/v1`; the first time `jazz agent create` uses it, Jazz asks for the server URL and saves it. You can also set `llm.llamacpp.base_url`, `LLAMACPP_BASE_URL`, or use `/settings` → **Model providers** in the `jazz` home screen (a bare `host:port` is enough). llama.cpp needs no API key unless the server runs with `--api-key`; when it answers 401, `jazz agent create` asks for the key, and `/settings` → **Model providers** can set it. ```bash llama-server -m /models/model.gguf --jinja --port 8080 jazz agent create ``` Use `--jinja` when the model should call tools. Jazz reads `/props` for context and chat-template metadata when the server provides it. For reasoning models, Jazz maps the agent's reasoning effort string to llama.cpp's supported thinking controls; behavior depends on a recent server and a compatible template. ## vLLM Use the separate `vllm` provider for a vLLM server. Its default base URL is `http://127.0.0.1:8000/v1`. On first use, `jazz agent create` asks for the address and saves `llm.vllm.base_url`; you can also set `VLLM_BASE_URL` or use `/settings` → **Model providers** in the `jazz` home screen. Jazz reads `/v1/models`: it selects the sole served ID automatically, or asks you to choose when the server lists several. On each run, Jazz refreshes the model list and uses the saved ID while it is still listed; if it disappears, Jazz uses the first live ID and its context window. Configure `llm.vllm.api_key` or `VLLM_API_KEY` if the server uses `--api-key`; an authorization error during agent creation prompts for the key. ```bash vllm serve --port 8000 jazz agent create ``` Jazz uses vLLM's OpenAI-compatible chat API. Tool calls require a compatible model and server configuration, commonly `--enable-auto-tool-choice` with the matching `--tool-call-parser`; `/v1/models` alone cannot prove those flags are enabled. Follow [vLLM's tool-calling guide](https://docs.vllm.ai/en/latest/features/tool_calling/) for the model-specific setup. For a server reachable beyond the host, protect the deployment at the network boundary: [vLLM documents](https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/) that `--api-key` does not authenticate every endpoint. ## SGLang Use the `sglang` provider for an SGLang server. The default base URL is `http://127.0.0.1:30000/v1`. On first use, `jazz agent create` asks for the address and saves `llm.sglang.base_url`; `SGLANG_BASE_URL` and `/settings` → **Model providers** in the `jazz` home screen also work. Jazz discovers current IDs from `/v1/models`, selecting the sole model automatically or offering a picker for multiple IDs such as loaded LoRA adapters. At the start of each run it keeps the saved ID if still served, otherwise uses the first current ID. It also reads the live `max_model_len` for context accounting; a LoRA card without a length inherits its listed base model's length. ```bash python -m sglang.launch_server --model-path --port 30000 jazz agent create ``` Set `llm.sglang.api_key` or `SGLANG_API_KEY` if the server requires bearer authentication. Jazz prompts for a key when model listing returns 401 or 403. SGLang uses the OpenAI-compatible chat API. Tool calls need a compatible model and the matching `--tool-call-parser`; reasoning output needs an appropriate `--reasoning-parser` for the model. `/v1/models` does not report parser configuration, so test those features against your server. See [SGLang's reasoning guide](https://docs.sglang.ai/advanced_features/separate_reasoning.html) and [server argument reference](https://docs.sglang.ai/advanced_features/server_arguments.html). `llm.numCtx` limits Jazz's accounting; it does not change the server's context length. ## Model capability overrides Models.dev supplies broad metadata such as context length, tool support, and whether a model reasons. It does not describe every provider or local-template reasoning control. Jazz applies its own exact `provider + model` capability profiles after catalog metadata. For a private model or a self-hosted template, an operator can provide a strict local correction under `llm.capabilityOverrides`; overrides use only Jazz-defined transports and cannot inject arbitrary provider request fields. ```json { "llm": { "capabilityOverrides": { "llamacpp": { "Qwen3-32B-Instruct": { "reasoning": { "kind": "budget", "transport": "openai-compatible.chat.template-thinking-budget", "minimumBudgetTokens": 256, "maximumBudgetTokens": 32768, "canDisableReasoning": true }, "supportsTools": true } } } } } ``` Keys are exact server-facing model IDs. To set one field from the command line, quote a model ID that contains dots: `jazz config set 'llm.capabilityOverrides.nvidia."deepseek-ai/deepseek-v4.1-flash".supportsTools' true`. An override also corrects the model lists `jazz agent create` and `jazz agent edit` show, so their tool and reasoning steps match what requests do. Resolution is operator override, live local-server metadata, Jazz's exact-model profile, provider default, then Models.dev. When none of them says whether a cloud model takes tools (a model released after the catalog, a custom id, or the catalog unreachable), Jazz assumes it does. A local server's model gets tools only once the server reports them. Whenever a run sends no tools to an agent that has some, every surface warns: stderr for `jazz run`, the chat transcript, `toolsDisabled: true` in the JSON envelope, and a notice under the bot's answer. A llama.cpp budget control is never assumed from a model family: declare it only when the active template accepts it. Transports name the request field Jazz sends, not a vendor. OpenAI-compatible providers (`llamacpp`, `vllm`, `sglang`, `nvidia`, `orcarouter`) accept all three `openai-compatible.chat.*` transports: | Transport | Kind | Request field | | ------------------------------------------------- | -------- | ------------------------------------------------------------- | | `openai-compatible.chat.reasoning-effort` | `effort` | top-level `reasoning_effort`, `"none"` to disable | | `openai-compatible.chat.template-enable-thinking` | `toggle` | `chat_template_kwargs.enable_thinking` | | `openai-compatible.chat.template-thinking-budget` | `budget` | `chat_template_kwargs.thinking_budget`, omitted when disabled | When a profile lists `efforts`, a requested level the model does not list is lowered to the nearest listed level below it, or raised to the lowest listed level when none is below it; Jazz never raises it further. `disable` on a profile with `"canDisableReasoning": false` becomes the lowest level, or keeps reasoning on for a toggle. The `/reasoning` picker and `jazz agent create`/`edit` offer only the levels the model's profile accepts; with no profile they offer every level. A typed `/reasoning ` or `jazz run --reasoning ` the model does not accept prints the level it runs at, and `jazz agent edit` shows it next to the saved level. A bare `llama-server` serves the one model loaded at launch and ignores the requested model name, and that model can change between runs. Jazz therefore treats the model chosen at agent creation as a hint: at the start of each run it reads the actually-served model from `/v1/models` and the real context window from `/props`, so the displayed model and context accounting match what the server is running. A pinned `llm.numCtx` still overrides the server-reported window. ## Slow first tokens from local models Jazz abandons a provider stream that stays silent for `llm.streamIdleTimeoutMs` milliseconds once output is flowing, 120000 by default, and reports `Provider stream produced no additional part for 120s and was abandoned`. The timer restarts on every streamed part, so it never caps a long answer, and tools run between streams rather than inside one. The wait for the first token has its own budget: five minutes, or `llm.streamIdleTimeoutMs` when that is higher. That covers NVIDIA NIM, which can queue a request for over three minutes, and large reasoning models that prefill for a long time. Ollama, llama.cpp, vLLM, or SGLang loading a large model from disk and then prefilling a long prompt can take longer still, so raise the budget for those hosts: ```bash jazz config set llm.streamIdleTimeoutMs 600000 ``` `JAZZ_STREAM_IDLE_TIMEOUT_MS` sets the same budget for one process; the saved value wins over it. `llm.ollama.keep_alive` avoids paying the cold start again. ## Gemini naming The Jazz provider ID is `gemini`; its SDK and environment variable retain Google's upstream naming. Existing `google` agent and configuration identifiers are migrated to `gemini` when read. ## Offline operation `JAZZ_OFFLINE=1` disables Jazz's own update, hosted model-catalog, and library requests. It does not make a hosted provider work offline. Use Ollama, llama.cpp, vLLM, or SGLang, preinstall every required skill dependency, and enforce the network boundary outside Jazz. Follow [Local and air-gapped models](../getting-started/local-models.md). ## Diagnose provider failures - Authentication errors: confirm the agent's provider ID matches the key you supplied, and run `jazz config show` to see which provider settings Jazz resolved. - Stream idle errors say whether the provider produced no first part or stopped between parts. The former points to queuing, model loading, or prompt prefill (allowed five minutes); the latter means generation had already started (allowed 120000 ms). Local servers that legitimately need longer can set `llm.streamIdleTimeoutMs` or `JAZZ_STREAM_IDLE_TIMEOUT_MS`. - Retries: each model call gets `maxRetries` retries in total. After three streaming failures the remaining attempts use a plain request; a rate limit (429) keeps streaming. Backoff is jittered and follows the provider's `Retry-After`; a provider asking for more than two minutes fails the call instead of waiting. - "Prompt is too long": Jazz compacts the conversation (or trims it when it cannot compact) and retries once. A second rejection fails the run; lower the agent's `llm.maxContextTokens` to match what the server really holds. - Unknown model: rerun agent editing after the provider catalog is reachable; do not copy a model name from an old documentation page. - Local connection errors: start the server and verify its base URL from the Jazz host, not from your laptop when Jazz runs elsewhere. - Tool-call failures on llama.cpp: confirm the model template supports tools and the server was started with `--jinja`. - Tool-call failures on vLLM: confirm the model's chat template, automatic tool choice, and matching parser are configured on the server. - Tool-call failures on SGLang: confirm the model's chat template and matching `--tool-call-parser` are configured on the server. - Unexpected context truncation on Ollama: pin `llm.numCtx` on the agent and ensure the server can allocate it. Read [Creating agents](../getting-started/create-an-agent.md), [Local models](../getting-started/local-models.md), and [Adding a provider](../maintainers/add-a-provider.md) next.