Providers & models

This page explains how Jazz stays provider-agnostic, and what it does about the places providers genuinely differ.

Source: adapters/src/llm/ai-sdk-service.ts · adapters/src/llm/reasoning/ · core/utils/models-dev.ts

One port, 23 providers

core/ defines an LLMService interface. adapters/src/llm/ai-sdk-service.ts is the only implementation, and it delegates to the Vercel AI SDK.

flowchart TB
    CORE["<b>core/interfaces/llm.ts</b><br/>LLMService port<br/><i>the agent loop only knows this</i>"]
    IMPL["<b>adapters/src/llm/ai-sdk-service.ts</b><br/>the single adapter"]
    SDK["Vercel AI SDK"]

    subgraph cloud["Cloud"]
        direction TB
        C1["OpenAI · Anthropic · Google · xAI"]
        C2["Mistral · DeepSeek · Groq · Cerebras"]
        C3["Fireworks · TogetherAI · Alibaba"]
        C4["Moonshot · MiniMax · Zhipu · NVIDIA NIM"]
    end

    subgraph aggregators["Aggregators"]
        A1["OpenRouter"]
        A2["Vercel AI Gateway"]
        A3["OrcaRouter"]
    end

    subgraph local["Local: no API key"]
        L1["Ollama"]
        L2["llama.cpp"]
        L3["vLLM"]
        L4["SGLang"]
    end

    CORE --> IMPL --> SDK
    SDK --> cloud
    SDK --> aggregators
    SDK --> local

    classDef core fill:#4f9d9d,stroke:#2f6d6d,color:#ffffff
    classDef free fill:#f9a03f,stroke:#b3541e,color:#1a1a1a
    class CORE,IMPL core
    class local free

Adding a provider means adding an SDK package and a catalog entry, not a new streaming implementation, not a new tool-calling translation. The cost is that Jazz is bounded by what the SDK normalizes, and inherits its bugs.

Configuration and API keys: Integrations.

The model catalog

Jazz needs two things per model that providers don’t reliably report: context window and pricing. Both come from models.dev, with an on-disk snapshot so this never becomes a hard dependency.

flowchart TD
    NEED(["Need metadata for<br/>provider/model"]) --> LOCAL{"Local provider?<br/>ollama / llamacpp / vllm / sglang"}

    LOCAL -->|yes| ASK["Ask the local server<br/>Ollama /api/tags + /api/show<br/>llama.cpp /props<br/>vLLM + SGLang /v1/models<br/><i>no catalog needed at all</i>"]
    LOCAL -->|no| OFF{"JAZZ_OFFLINE?"}

    OFF -->|no| FRESH{"Snapshot under<br/>6 hours old?"}
    FRESH -->|yes| READ
    FRESH -->|no| FETCH["Fetch models.dev<br/>(or JAZZ_MODELS_DEV_URL mirror)"]
    FETCH --> SNAP["Refresh the snapshot<br/>~/.jazz/cache/models-dev.json"]
    OFF -->|yes| READ["Read the snapshot if present"]

    SNAP --> USE
    READ --> USE
    ASK --> USE
    READ -->|"no snapshot"| FALL["Provider-reported metadata,<br/>else 128k default"]
    FALL --> USE
    USE(["contextWindow · pricing<br/>· reasoning support"])

    classDef local fill:#f9a03f,stroke:#b3541e,color:#1a1a1a
    class ASK,READ local

Consequences worth knowing:

• Local models need no catalog. Model lists, context windows, and tool-calling support are read straight from the local server. An airgapped install works with an empty cache.
• The 80% compaction threshold tracks the real window. A 1M-context model gets a 1M-based threshold, not a guess.
• A brand-new model may be missing. You get the provider’s own metadata or a 128k default, and pricing display may be blank. Nothing breaks.
• The snapshot is reused for six hours, per catalog URL. Switching JAZZ_MODELS_DEV_URL selects that source’s own memory cache and disk snapshot, including offline. The default catalog uses models-dev.json; mirrors use models-dev.<sha256-of-url>.json under the same cache directory. In-flight loads cannot populate another source’s cache.
• JAZZ_MODELS_DEV_URL points at an internal mirror for air-gapped networks that still want metadata. See Local models.

Ollama /api/show and /api/ps probes share requests started within the last ten seconds. Each caller can cancel its own wait without cancelling another caller’s request. The shared request still has a five-second timeout; cancelling every waiter can leave that bounded request running to warm the cache.

Reasoning: the messiest difference

Providers expose “thinking” in incompatible ways. Some return it as a structured field. Some interleave <think>…</think> tags in the text stream. Some reject a reasoning-effort parameter outright. Local models frequently emit reasoning tags that their own advertised capabilities never mention.

flowchart TB
    STREAM["Model stream"] --> SELECT{"selectParser(provider,<br/>modelId, chatTemplate,<br/>capabilities)"}

    SELECT -->|"a factory claims it"| P1["That parser"]
    SELECT -->|"Harmony format detected<br/>&lt;|channel|&gt;analysis"| NONE["No parser,<br/>passthrough would mangle<br/>the delimiters"]
    SELECT -->|"nothing claims it"| P2["<b>Defensive TagPairParser</b><br/>passthrough on plain text;<br/>acts only if a real<br/>&lt;think&gt; tag appears"]

    P1 --> SPLIT
    P2 --> SPLIT
    SPLIT["Split each delta into<br/>visibleText + thinkingText"]
    SPLIT --> EV["thinking_start / thinking_chunk /<br/>thinking_complete events"]
    SPLIT --> TXT["text_chunk events"]

    classDef guard fill:#f9a03f,stroke:#b3541e,color:#1a1a1a
    class P2,NONE guard

The default is defensive rather than strict, and that’s a deliberate call. A strict factory gate (only parse when metadata says the model reasons) silently leaks <think> tags into the user-visible answer for the many local models that don’t declare it. The fallback parser is a passthrough until it actually sees an opening tag, so the cost of being wrong is zero. The one format explicitly refused is Harmony, where naive tag-pair parsing would visibly corrupt the channel delimiters.

Parsers are stateful per request and buffer across chunk boundaries, so a <think> tag split across two network packets is stitched correctly rather than half-rendered.

Reasoning effort (low / medium / high / disable) is normalized per provider. Models without reasoning support error if you ask for it, which is why --reasoning disable exists and why the Discord/Telegram bridges’ /model command sets it automatically from whichever source knows that model’s capabilities (a local Ollama’s own reporting, the models.dev catalog, or the provider’s own model-listing endpoint), for any provider, not just local models.

Retries and timeouts

Jazz owns retry policy. The AI SDK’s internal retries are turned off (AI_SDK_MAX_RETRIES = 0) so there aren’t two retry loops with different opinions about backoff.

A 15-minute ceiling exists because slow reasoning models on a long prompt genuinely take minutes, and a tighter timeout would fail runs that were about to succeed. Rate-limit errors are typed (LLMRateLimitError) so they’re distinguishable from real failures.

Cost accounting

Pricing usually comes from the catalog, per million input and output tokens. Providers that report an authoritative per-call charge can supply that instead:

ownCost = promptTokens/1e6 × inputPrice + completionTokens/1e6 × outputPrice
total   = ownCost + Σ(sub-agent cost)

A figure is emitted whenever either side is known: a free local parent that delegated to a paid cloud sub-agent still reports real spend. A known zero is included; when neither side is priced (an uncatalogued local model), cost stays omitted rather than being reported as $0.00.

Per-run records land in ~/.jazz/telemetry/ as local JSON. Nothing is transmitted anywhere. Command-risk classifier tokens are stored separately (classifierUsage on the run, purpose: "classifier" on each llm_usage) because that call uses the cheap harness model, not the agent’s, and mixing the two would hide both numbers. See Observability.

Switching models

Running a cheap model for compaction while the main agent runs an expensive one is the highest-value version of this, and it’s one field.

Related

• Integrations: providers: API keys and setup
• Local and air-gapped models: local-only operation
• Context management: what the context window is used for
• Architecture: provider boundaries and dependency direction