Computer use

Experimental. Computer use may change or break between releases. It runs on macOS with Apple Silicon only, and only in a terminal conversation while you watch. Screenshots and on-screen text are sent to your model provider, and nothing here is an operating-system sandbox: Jazz limits what the agent does, not what an app can do.

Computer use lets an agent read the windows of desktop apps you choose and click, scroll, and type in them, without taking over your cursor. Use it for work that only a desktop app can do. For web pages, the browser tools is the better tool.

Set it up

Computer use needs a desktop driver, cua-driver, installed on your Mac. Jazz finds it on your PATH, or at the path in JAZZ_COMPUTER_DRIVER. The driver needs two macOS permissions, Accessibility and Screen Recording, granted to it in System Settings under Privacy & Security.

1. Turn the tools on for an agent. Run jazz agent edit <agent> and enable Computer use (experimental), or add the tool names to the agent’s config.tools:

   {
     "tools": [
       "computer_apps",
       "computer_observe",
       "computer_pointer",
       "computer_input",
       "computer_foreground",
       "computer_handoff",
       "computer_end"
     ]
   }

2. Check the machine and the driver with jazz computer doctor. It exits 0 when ready, 1 when something needs attention, and 2 when computer use cannot run here. If the driver is missing, the agent can offer computer_install_driver; you approve its installation in the conversation. Grant Accessibility and Screen Recording to the driver in System Settings under Privacy & Security.

3. Ask the agent to use an app. It can request computer_grant_app for that app in chat; approve the grant there. To see bundle ids yourself, run jazz computer apps. After the first successful session has pinned the driver, you can also grant an app in the terminal:

   jazz computer grant com.apple.Notes --expires 2h

Enabling the tools authorizes computer use. The first successful session pins the driver’s SHA-256 digest. If that binary later changes, Jazz stops before starting it and offers an in-chat approval for the new build. The tools load on demand when the agent uses the desktop.

How the agent uses it

In Notes, open the note called Shopping and add "oat milk" to the list.

The agent calls computer_apps to see what it may control, then computer_observe to read a window. An observation is a text outline of the window’s controls. Each control it may act on ends with a ref such as [ref=c3.12], and the outline names its observation, such as c3:

observation: c3
app: Notes (com.apple.Notes)
window: "Shopping"
access: full control
elements: 14

- AXToolbar
  - AXButton "New Note" [ref=c3.4]
- AXTextArea "Note body" [ref=c3.9]

computer_pointer clicks a ref, clicks a pixel of the observation’s screenshot, or scrolls. computer_input types into a ref or presses a named key or shortcut. Any of the action tools can ask for the after-state with captureAfter: the result then carries a fresh observation of the window, so the effect and the new refs arrive in the same reply. computer_wait watches the window the agent last looked at or acted on until it settles (two identical reads) or changes, and returns a fresh observation. It replaces the observe-again round trip after an action that makes the window busy, such as opening a document or loading a page.

After the window changes, the agent observes again or waits for it to settle. A new observation retires the refs of the earlier one, and acting through a retired ref is an error, not a click on whatever moved into its place.

computer_observe with screenshot saves a PNG of the window and returns its path. The agent passes the path to analyze_media, so a model that cannot see images can still use it. Jazz keeps the latest five screenshots and deletes them all when the run ends.

See the tool inventory for disclosure and approval pairs.

What each app allows

Jazz decides how much an agent may do in an app from its bundle id. You cannot raise it.

Terminals are refused because controlling one would bypass the shell tool’s approvals, denylist, and secret redaction. An app whose bundle id Jazz cannot read is refused. A view-only window is observed without refs, so there is nothing in it to click.

Signing in

The agent never types a password it made up. Two things keep credentials out of its hands:

• A password field accepts only a secret you collected with ask_user_secret. The agent passes the placeholder as the text, Jazz asks you before entering it, and the model only ever sees the placeholder.
• For anything else, the agent calls computer_handoff with what you should do, such as signing in. Computer use pauses until you approve, nothing is read or acted on while you work, and every earlier observation is cleared afterwards.

Approvals

Reading a window is automatic once you grant the app. Everything the agent reads from a window is another party’s text: it arrives inside the untrusted-content envelope and marks the run as having read external content. After that, other outbound tools stop auto-approving below the high-risk policy, so in the default safe mode expect a prompt for each action the agent takes after its first look. computer_input and computer_foreground always need approval unless the policy is high-risk.

Some actions are never taken, whatever the approval: the shortcuts that empty the Trash, lock the screen, or log out; text that downloads and runs code or deletes data; and key presses that would type a character, which go through computer_input so its checks apply.

Stopping

• jazz computer stop ends the run that is using the computer. It takes effect at the agent’s next action and stops the driver at once, and the run cannot use the computer again.
• jazz computer revoke <bundle-id> takes an app away, effective on the very next action.
• A grant expires after the time you gave it, at most seven days. A run that sits idle past the shortest idle timeout among its grants, 30 minutes by default, loses access for good.
• Only one run uses the computer at a time. A second is told who holds it, and a run that crashed leaves nothing locked.
• Jazz shows a desktop notification when a run starts using the computer and when it finishes, where your terminal or system can show one.

What Jazz records

jazz computer log shows what computer use did: the app, the action, the element or window, and how it turned out. The record never holds a screenshot or typed text. An element’s or window’s label is shortened and has secrets redacted. It is stored in $JAZZ_HOME/computer/ledger.jsonl.

Limits

• Computer use is withheld from every run nobody is watching: scripts, schedules, goals, loops, webhooks, peers, bots, and detached jobs. The tools are simply absent. A sub-agent never gets them either.
• It runs on macOS with Apple Silicon only.
• Acting in the background works through the driver and differs between apps. Apps that draw their own content, such as design tools and games, may need the foreground tool, which brings the app to the front and needs --foreground on its grant.
• Screenshots and on-screen text reach your model provider. Keep apps you do not want it to see out of the grant, and close their windows.
• The security model lists what this does not protect against.

Related

• Computer use safety: what the desktop tools can and cannot reach
• jazz computer: the operator commands
• Browser tools: reading and acting on web pages
• Model companions: analyze_media for reading screenshots