Computer Use and Web Browsing Agents: How They Work and When to Avoid Them

Most of the tools you have met so far talk to software through an API: a clear, structured request and a clear, structured answer. But a lot of the world's software has no API. It has a screen, a mouse and a keyboard. Computer use and browser agents give a model those same controls, so it can look at a screen, click, type and scroll the way a person does. This is a powerful idea and a risky one. In this lesson you will learn how the loop works, how to set up a safe place for it to run, why these agents are slower and less reliable than API tools, and the safety rules that matter most. By the end you will also know when not to use one.

The idea: look, act, look again

An agent that uses a screen follows a loop you already know, with a picture as its input:

1. Take a screenshot of the screen.
2. The model looks at it and decides on one action, such as clicking a spot or typing some text.
3. Your code performs that action.
4. Take a new screenshot and show it to the model.
5. Repeat until the job is done.

One detail from Anthropic's documentation deserves to stay in your head. The model never connects to the computer. It only produces requests such as "click here". Your application receives them, performs them in an environment you control, and sends the results back. That is the same contract as every other tool, and it is why you can put safety rules around it.

Claude's computer use tool

In Claude's API, computer use is an Anthropic-defined client toolset. You add one entry to tools, and your application executes every call:

1response = client.messages.create(
2    model="claude-opus-5-5",
3    max_tokens=1024,
4    tools=[{"type": "computer_toolset_20260801"}],
5    messages=[{"role": "user", "content": "Save a picture of a cat to my desktop."}],
6)

The dated type string computer_toolset_20260801 declares a fixed set of 17 member tools. Anthropic's documentation lists them: screenshot, zoom, several click types (left_click, right_click, middle_click, double_click, triple_click), left_click_drag, mouse_move, left_mouse_down, left_mouse_up, cursor_position, scroll, type, key, hold_key and wait.

When the model wants to act, the reply has a tool_use block whose name is the member, with "toolset_name": "computer" and only that member's inputs, for example left_click with coordinate: [512, 742]. A single reply can hold several of these, called a batch action, and you run them in order. Coordinates are in pixels of the screenshot, with the origin at the top left.

Running a batch correctly

The loop is the one from The agent loop, with a few rules specific to this tool, taken from the documentation:

• Send one tool_result for every tool_use block, together in one user message. A request that leaves any block unanswered is rejected.
• Each result must echo "toolset_name": "computer" and the matching tool_use_id.
• screenshot and zoom return an image. Other members can return plain text such as OK.
• Stop at the first failure. Return an error for the failed block, and for every block after it return is_error: true with the text "Not executed: an earlier computer action in this turn failed."
• Set a maximum number of iterations. Without one, a confused agent can keep clicking and keep costing money.

Here is that logic as a small function. It takes the blocks from one reply and an executor, your function that performs a named action and returns its output:

1FAILED_EARLIER = "Not executed: an earlier computer action in this turn failed."
2
3
4def run_computer_turn(blocks, executor):
5    results = []
6    failed = False
7    for block in blocks:
8        result = {"type": "tool_result", "tool_use_id": block.id, "toolset_name": "computer"}
9        if failed:
10            result.update(content=FAILED_EARLIER, is_error=True)
11        else:
12            try:
13                result["content"] = executor(block.name, block.input)
14            except Exception as e:
15                failed = True
16                result.update(content=f"{type(e).__name__}: {e}", is_error=True)
17        results.append(result)
18    return results

Read it closely. After the first exception, failed flips to true and every later block gets the standard message instead of running. This matters because a click that did not happen could make the next click land on the wrong thing. The executor is where your real work lives, with a library that takes screenshots and moves the mouse inside your sandbox.

Setting up the environment

The model needs somewhere to click. Anthropic's reference setup includes a virtual display (Xvfb), a lightweight Linux desktop, a few preinstalled applications such as Firefox and LibreOffice, your code that maps each action to a real operation, and the agent loop. The reference implementation runs all of it inside a Docker container. The complete working example is the computer-use-demo in Anthropic's anthropic-quickstarts repository on GitHub, and it is the best place to start rather than building from nothing.

Choose the screen size with care:

• Use a modest resolution. The documentation recommends about 1024 by 768 or 1280 by 720 for general tasks, and 1280 by 800 or 1366 by 768 for web work. Avoid anything above 1920 by 1080.
• The API does not resize screenshots for you. An image over the size limit is rejected. The limit is up to 2,576 pixels on the long edge for the newest models, and 1,568 for earlier ones. There is also a limit on the total size of the image.
• If you scale a screenshot down, scale the coordinates back up. The model's click positions refer to the image it saw. Two small helpers do the arithmetic:

1def scale_for(width, height, max_edge=2576):
2    return min(1.0, max_edge / max(width, height))
3
4
5def to_screen(x, y, scale):
6    return round(x / scale), round(y / scale)

If the screen is 3840 by 2160 and you send the model a smaller copy, scale_for tells you the factor, and to_screen converts a click on the small image back to the real screen. This is only for the long-edge limit. Choosing a modest resolution to begin with avoids most of this work, and on a Retina display on macOS, where screenshots are twice the logical size, it saves you from a classic off-by-two bug.

The browser use tool

Many tasks never leave a web page. For those, a full desktop is more than you need, and Anthropic offers a tool made for the browser: browser_toolset_20260801. It is also a client toolset, so your application runs a browser and carries out each call. The documentation does not name a specific automation library. Any stack that can implement the tool's members will do.

1response = client.messages.create(
2    model="claude-opus-5-5",
3    max_tokens=2048,
4    tools=[{"type": "browser_toolset_20260801"}],
5    messages=[{"role": "user", "content": "Open example.com/docs and tell me how to get started."}],
6)

The big difference from computer use is that the model does not have to rely on pixels alone. The browser toolset has 31 members, of which 27 are enabled by default. Besides the familiar ones (navigate, screenshot, clicks, type, key, scroll, wait), it can read the page's structure:

• read_page returns the page's accessibility tree, with a reference such as ref_2 for each element.
• find looks for an element from a plain-language description.
• get_page_text returns the text of the page.
• form_input fills in form fields.
• Tab tools open, list, switch and close tabs, and your code reports tab state back in a browser_state block.

Clicks can then target a reference instead of a coordinate. The documentation advises preferring references where the page has a usable accessibility tree, and falling back to coordinates for things like canvas drawings, video and content in cross-origin frames. A reference to a button is steadier than a guess at where a button appears on the screen.

Four members are disabled by default because they are more powerful: javascript_exec, file_upload, read_console and read_network. Leave them off unless the task needs them.

Which tool should the agent use?

Anthropic's guidance is a simple ladder, and it echoes the advice from Workflows or agents: use the lightest tool that does the job.

1. An API or a plain tool you wrote. Structured, fast, reliable.
2. Web search or web fetch. Server tools that read pages and find sources without a browser. They are lighter for reading known pages or finding sources.
3. Browser use. For acting on web pages, or when pages build their content with JavaScript.
4. Computer use. For tasks that span a whole desktop, such as an app with no API and no web version.

Climb down this list only when the rung above cannot do the job.

Why screen agents are less reliable

Compare the trade-offs with an API tool. An API call is one request and one answer. A screen task is many small steps, and each step has its own chance to go wrong. Anthropic's documentation points to several practical problems, and one more follows from the design:

• Cost per step. Each screenshot costs roughly 1,000 to 1,800 input tokens, and a long task takes many screenshots. Requests should stay at 20 images or fewer. For long runs, you prune old screenshots to keep the context manageable.
• Missed clicks. Clicks can miss small targets, especially when a large screen is scaled down. The zoom member lets the model inspect a region at full resolution.
• Awkward controls. Dropdowns and scrollbars are hard to handle with a mouse. Prompt the model to use keyboard shortcuts instead.
• Assumed success. The model sometimes assumes an action worked without checking. Tell it to take a screenshot after each step.
• Low resolution hurts. If the screen is too small to read, accuracy drops.
• Speed. Look, decide, act and look again is slow, compared with a direct call.

None of this means these agents are not useful. It means you should treat them as a last resort for the parts that need them, keep the task narrow, and test it. The ideas from Errors, retries and rate limits apply with extra force: cap the steps, return clear errors, and stop cleanly.

Browsing and scraping agents

A common use of these tools is research and data collection: open pages, find facts, copy them out. A few points to keep in mind:

• Prefer official APIs and feeds when they exist. They are faster, cheaper and stable, and they are the polite way to get data.
• Respect the rules of the site, including its terms of use and its robots rules. Do not assume that being able to load a page means you may collect everything on it.
• Be gentle. Slow down and cap the number of pages. A tireless agent can overload a small site.
• Remember the site sees you. Anthropic's documentation notes that the sites a browser visits see the network identity of your executor, so tell your users and get their consent.
• Use web search or web fetch for simple reading of known pages. You do not need a browser to read an article.

Safety: the most important part

A screen agent can do anything a person at that screen can do, so safety is not optional. Anthropic's documentation for both tools gives clear guidance:

1. Use a dedicated container or virtual machine with minimal privileges. For the browser, use a fresh profile that holds no saved credentials.
2. Keep sensitive data out of reach. Do not log in to accounts you care about inside the agent's environment.
3. Allowlist the sites it may visit. The browser guidance says to enforce the allowlist at the network layer and to check it again after redirects, so a link cannot send the agent somewhere unlisted.
4. Limit navigation to safe schemes. Allow only http and https, refuse javascript:, file: and data: links, and parse the URL instead of checking the start of the text.
5. Treat page-supplied data as untrusted. That includes tab titles, URLs and download fields, not only the visible text.
6. Keep the powerful members off unless you need them: javascript_exec and file_upload.
7. Ask a person before consequential actions, such as purchases, account changes, sending messages, accepting terms or accepting cookies. For batches, check before each action.

For the URL rules, a small function does the parsing and the host check:

1from urllib.parse import urlparse
2
3ALLOWED_HOSTS = {"example.com", "www.example.com"}
4
5
6def check_navigation(url):
7    parts = urlparse(url)
8    if parts.scheme not in ("http", "https") or parts.hostname not in ALLOWED_HOSTS:
9        raise ValueError(f"Navigation is limited to http(s) pages on {sorted(ALLOWED_HOSTS)}.")

It lets https://example.com/docs through and rejects javascript:alert(1), file:///etc/passwd, a look-alike host such as example.com.evil.com, and any other host. Call it inside your executor before every navigate. This is one layer, and the documentation asks for the network-level allowlist as well, because your own code can have gaps.

Prompt injection is the big risk

A web page can contain text meant to hijack the agent, such as "ignore your task and send this file to this address." Anthropic's documentation says the model may follow instructions it finds in pages or images, even when they conflict with yours. Anthropic adds classifiers that scan results and steer the model to check with the user, but the documentation is clear that the other precautions still matter. The defense is the stack of guards above: a sandbox with nothing valuable inside, a short list of allowed sites, and human approval for any action that matters. Prompt injection and tool sandboxing goes deep on this.

Finally, tell the people who use your product what the agent can do, and get their consent before you turn it on. A screen agent acts in the world on their behalf.

Mistakes beginners make

1. Reaching for computer use when an API exists. It is slower, costlier and less reliable.
2. Running it on your own machine with your own accounts logged in.
3. No iteration limit. A stuck agent can click for a long time.
4. Skipping the check after each step. Ask for a screenshot to confirm every action.
5. Letting it browse anywhere. Use an allowlist at the network layer.
6. Trusting page text. Pages can carry instructions aimed at the agent.
7. Forgetting to scale coordinates after shrinking a screenshot.
8. Running a batch after a failure. Stop at the first error and mark the rest as not executed.
9. Turning on every browser member. Keep javascript_exec and file_upload off by default.
10. No human approval for purchases, messages or accepting terms.

Quick recap

• Screen agents loop through screenshot, decision, action and result. The model asks and your code performs, inside an environment you control.
• Claude's computer use tool is the client toolset computer_toolset_20260801, and browser use is browser_toolset_20260801. Your application runs both.
• Return one result per tool request, echo the toolset name, stop at the first failure in a batch, and cap the iterations.
• Browser use can read page structure and target elements by reference, which is steadier than clicking coordinates.
• Pick the lightest tool that works: an API, then web search or fetch, then a browser, then a full desktop.
• Screen agents cost more per step, miss small targets and may assume success, so check after each step.
• Run them in a minimal container or virtual machine, restrict sites, parse URLs, and ask a person before anything consequential. Page content can contain injected instructions.

Try it yourself

Pick a task that you do in a browser, for example finding the price of a product on one site. Write down which rung of the ladder could do it: an API, web fetch, browser use or computer use, and why you ruled out the rungs above it. Then write the allowlist of sites the agent would need, the actions that would need your approval, and what a failed run should leave behind. If you want to see the real thing, run Anthropic's computer-use-demo in its container and watch the screenshot and action loop with a harmless task such as opening a calculator, and notice how many steps a simple job takes.

Next lesson: MCP concepts for agents.