Deploying AI Agents: Hosting, Queues, State, Secrets and Safe Releases

An agent that works on your laptop is a demo. An agent that works for real users, at 3 a.m., while a deploy happens and a model provider has a bad minute, is a service. The gap between the two is mostly not about the model. It is about what happens when a run takes five minutes instead of five seconds, when a worker dies halfway through a refund, when the same request arrives twice, and when you need to ship a change without breaking runs that are already in flight. This lesson covers those problems and builds the heart of the solution, a small durable job runner, in tested Python.

What changes in production

Agent runs differ from ordinary web requests in four ways:

• They are slow. A run makes many model calls, each taking seconds, plus tool calls. Minutes are normal, and hours are possible.
• They are stateful. The agent's progress, its messages and its tool results are the work. Losing them midway loses the work.
• They have side effects. A run may send an email or issue a refund. Doing that twice is worse than doing it zero times.
• They are unpredictable. Cost, duration and the number of steps vary from run to run.

An ordinary web server handles a request in milliseconds and forgets it. An agent needs the opposite: it should remember where it was, and it should be safe to retry.

Where to run it

There are four common choices.

1. A serverless function

The simplest option: run the agent inside a request handler. It works for short agents, and it is easy to deploy. The limit is time. Platforms cap how long a function can run. As an example, Vercel's documentation says that with its default compute settings a function has a default maximum duration of 300 seconds on all plans, that Pro and Enterprise teams can raise it to 800 seconds, and that an extended limit of 30 minutes is in beta for them. These numbers change, so check your platform's current page. The principle holds anywhere: a run that might exceed the limit must not live inside one request.

Two related limits: Anthropic's SDK validates that non-streaming requests are not expected to exceed a 10-minute timeout, and its documentation recommends streaming for long requests. For any model call that may run long, stream it.

2. Workers behind a queue

The classic answer for long work. The request handler does not run the agent. It creates a job and returns a job id at once. A pool of workers pulls jobs from a queue and runs them, and the client checks the job's status or receives updates. You control the machines, the timeouts and the concurrency. This is the layout in the diagram, and it is what the code below builds.

3. A durable workflow engine

A workflow engine stores each step of a long-running process, so it survives crashes and deploys. Temporal's documentation, for example, describes a workflow that can run for seconds or months. If a worker crashes, the service gives the work to another worker, which replays the event history and resumes where execution stopped. It also offers retries with backoff for calls to outside services, durable timers, and ways for a running workflow to receive input from people. Some hosting platforms offer a similar service. Vercel's documentation points to its Workflows for work that needs to pause, resume and keep state without duration limits.

You get durability without writing it yourself, at the price of learning the engine's model and running or renting it.

4. A provider-hosted agent

Some model providers run the agent loop for you. Anthropic's Managed Agents, for example, is described in its documentation as a pre-built agent harness running in managed infrastructure, suited to long-running and asynchronous work. You define the agent and an environment (a managed cloud sandbox, or a self-hosted one), start a session, and exchange events with it. Responses stream back, and event history is stored on the server. At the time of writing it was in beta, and its documentation says it is not eligible for Zero Data Retention or HIPAA coverage, because sessions store transcripts until you delete them. Hosted options save you the infrastructure and tie you to the vendor, as in Agent frameworks compared.

Choosing

A reasonable path: start with a function for short tasks, move to a queue and workers when runs outgrow the time limit, and adopt a workflow engine when you need strong guarantees.

The life of a job

The states are simple:

1. Queued. The job exists and waits for a worker.
2. Running. A worker has claimed it and holds a lease, a time-limited right to work on it. While it works, it keeps renewing the lease. This is a heartbeat.
3. Waiting, optionally, for a person to approve something, as in Human-in-the-loop agents. The state is saved, so the job can wait for hours.
4. Succeeded, failed or cancelled. The end states.

Two rules make this robust:

• Save a checkpoint after every step. If the worker dies, the next one starts from the last checkpoint and does not redo finished work.
• Use leases. If a worker dies without telling anyone, its lease runs out, and another worker can claim the job. Without a lease, a dead worker's job would sit as "running" forever.

The code

This is the core of a durable job runner. A dictionary stands in for a database, so the sketch runs anywhere, and the logic carries over to a real queue and database.

1import itertools
2
3
4class PermanentError(Exception):
5    """A failure that retrying cannot fix, such as bad input."""
6
7
8class JobStore:
9    """A dict standing in for a database table. In production this is Postgres, Redis or a queue service."""
10
11    def __init__(self):
12        self.jobs, self.keys, self.effects, self._ids = {}, {}, {}, itertools.count(1)
13
14    def submit(self, task, key=None, deadline=None):
15        if key in self.keys:                      # the same request twice returns the same job
16            return self.keys[key]
17        job_id = f"job-{next(self._ids)}"
18        self.jobs[job_id] = {"id": job_id, "task": task, "status": "queued", "state": None, "attempts": 0,
19                             "lease_until": 0, "deadline": deadline, "result": None, "error": None}
20        if key:
21            self.keys[key] = job_id
22        return job_id
23
24    def claim(self, now, lease=60):
25        """Take a queued job, or one whose worker stopped sending heartbeats."""
26        for job in self.jobs.values():
27            if job["status"] == "queued" or (job["status"] == "running" and job["lease_until"] <= now):
28                job.update(status="running", lease_until=now + lease, attempts=job["attempts"] + 1)
29                return job
30        return None
31
32    def heartbeat(self, job_id, state, now, lease=60):
33        self.jobs[job_id].update(state=state, lease_until=now + lease)
34
35    def once(self, job_id, name, fn):
36        """Run a side effect (send an email, issue a refund) once per job, even across retries.
37        shortcut: a crash between the effect and this record repeats it, so fn also gets a key to send as the service's idempotency key."""
38        if (job_id, name) not in self.effects:
39            self.effects[(job_id, name)] = fn(f"{job_id}:{name}")
40        return self.effects[(job_id, name)]
41
42
43def run_job(store, job, step, now, max_steps=20, max_attempts=3, lease=60):
44    """step(state, once) -> (state, done, result). State is saved after every step, so a restart resumes there."""
45    jid = job["id"]
46    if job["attempts"] > max_attempts:
47        job.update(status="failed", error="too many attempts")
48        return job
49    try:
50        for _ in range(max_steps):
51            if job["status"] == "cancelled":
52                return job
53            if job["deadline"] is not None and now() > job["deadline"]:
54                job.update(status="failed", error="deadline passed")
55                return job
56            state, done, result = step(job["state"], lambda name, fn: store.once(jid, name, fn))
57            store.heartbeat(jid, state, now(), lease)
58            if done:
59                job.update(status="succeeded", result=result)
60                return job
61        job.update(status="failed", error="too many steps")
62    except PermanentError as exc:
63        job.update(status="failed", error=str(exc))
64    except Exception as exc:                      # a transient error goes back in the queue for another try
65        job.update(status="queued" if job["attempts"] < max_attempts else "failed", error=f"{type(exc).__name__}: {exc}")
66    return job

How it works

• submit creates a job. If you pass a key that was used before, it returns the existing job and does not create a second one. That protects you when a client retries a request.
• claim gives a worker a queued job, or a running job whose lease has expired, which is how a crashed worker's job is recovered. It counts the attempt.
• heartbeat saves the checkpoint and renews the lease.
• once runs a side effect, such as sending an email, one time per job, even if the step runs again after a retry. It also hands the effect a stable key, such as job-1:email, to send to the outside service as its own idempotency key. That matters because of a gap: if a worker crashes after sending but before recording, the effect would repeat, and only the service's own idempotency check can catch that. The comment in the code says so.
• run_job loops over steps. Before each step it checks for cancellation and for the deadline. After each step it saves the checkpoint. A PermanentError fails the job at once. Any other error puts the job back in the queue for another try, until the attempt limit. A job that has been claimed too many times, for example one that keeps killing its workers, is given up on.

The tests

1import pytest
2from jobs import *
3
4class Crash(BaseException):
5    """Stands in for the process being killed: nothing in run_job catches it."""
6
7clock = {"t": 0}
8now = lambda: clock["t"]
9
10def counting_step(calls):
11    def step(state, once):
12        n = (state or 0) + 1
13        calls.append(n)
14        return n, n == 3, "finished"
15    return step
16
17def test_submit_is_idempotent_and_claim_orders_jobs():
18    s = JobStore()
19    a, b = s.submit("one", key="k1"), s.submit("two")
20    assert s.submit("one again", key="k1") == a and a != b and len(s.jobs) == 2
21    assert s.claim(0)["id"] == a and s.claim(0)["id"] == b and s.claim(0) is None
22
23def test_runs_to_completion_with_checkpoints():
24    s, calls = JobStore(), []
25    s.submit("t"); job = s.claim(0)
26    run_job(s, job, counting_step(calls), now)
27    assert job["status"] == "succeeded" and job["result"] == "finished" and job["state"] == 3 and calls == [1, 2, 3]
28
29def test_a_crashed_worker_is_resumed_from_its_checkpoint():
30    s, calls, clock["t"] = JobStore(), [], 0
31    s.submit("t"); job = s.claim(0, lease=60)
32    def dying(state, once):
33        if (state or 0) == 2: raise Crash
34        return counting_step(calls)(state, once)
35    with pytest.raises(Crash): run_job(s, job, dying, now)
36    assert job["status"] == "running" and job["state"] == 2 and s.claim(30) is None     # lease not expired yet
37    clock["t"] = 60                                                                     # exactly when the lease runs out
38    again = s.claim(now())
39    assert again["id"] == job["id"] and again["attempts"] == 2
40    run_job(s, again, counting_step(calls), now)
41    assert again["status"] == "succeeded" and calls == [1, 2, 3]                         # steps 1 and 2 were not repeated
42
43def test_side_effects_happen_once_across_retries():
44    s, sent, clock["t"] = JobStore(), [], 0
45    s.submit("t"); job = s.claim(0)
46    fail_once = {"left": 1}
47    def step(state, once):
48        once("email", lambda key: sent.append(key) or "sent")
49        if fail_once["left"]:
50            fail_once["left"] -= 1
51            raise TimeoutError("slow")
52        return 1, True, "ok"
53    run_job(s, job, step, now)
54    assert job["status"] == "queued" and "TimeoutError" in job["error"]
55    job = s.claim(0); run_job(s, job, step, now)
56    assert job["status"] == "succeeded" and sent == ["job-1:email"]   # one send, with a stable idempotency key
57
58def test_transient_errors_stop_after_max_attempts():
59    s = JobStore(); s.submit("t")
60    def always(state, once): raise ConnectionError("down")
61    for _ in range(3):
62        job = s.claim(0); run_job(s, job, always, now)
63    assert job["status"] == "failed" and job["attempts"] == 3 and "ConnectionError" in job["error"]
64
65def test_permanent_error_fails_at_once():
66    s = JobStore(); s.submit("t"); job = s.claim(0)
67    def bad(state, once): raise PermanentError("empty task")
68    assert run_job(s, job, bad, now)["status"] == "failed" and job["error"] == "empty task" and job["attempts"] == 1
69
70def test_cancel_deadline_and_step_limit():
71    s = JobStore(); s.submit("t"); job = s.claim(0); job["status"] = "cancelled"
72    assert run_job(s, job, counting_step([]), now)["status"] == "cancelled"
73    clock["t"] = 100; s.submit("late", deadline=50); job = s.claim(0)
74    assert run_job(s, job, counting_step([]), now)["error"] == "deadline passed"
75    clock["t"] = 0; s.submit("loop"); job = s.claim(0)
76    assert run_job(s, job, lambda st, once: (st, False, None), now, max_steps=4)["error"] == "too many steps"
77
78def test_a_job_that_keeps_killing_its_worker_is_given_up_on():
79    s, clock["t"] = JobStore(), 0
80    s.submit("poison")
81    def dies(state, once): raise Crash
82    for _ in range(3):
83        job = s.claim(now())
84        with pytest.raises(Crash): run_job(s, job, dies, now)
85        clock["t"] += 61
86    job = s.claim(now())
87    assert job["attempts"] == 4 and run_job(s, job, dies, now)["error"] == "too many attempts"

They simulate a worker being killed with an exception that nothing catches, so the job stays "running" until its lease expires. They check that the next worker resumes from the checkpoint and does not repeat finished steps, that an email is sent once across a retry, and that deadlines, cancellation, step limits and permanent errors behave.

I broke the code ten ways to check that the tests notice. Eight were caught at first. Two survived. One was the lease boundary, whether a lease that ends at exactly time 60 is expired at time 60, and the other was the guard that gives up on a job claimed too many times. I added a test for each, and both changes now fail. Boundaries and give-up rules are where durable systems quietly go wrong.

What this sketch leaves out

A real version needs a database or queue service with atomic claiming, so two workers never take the same job. It needs separate worker processes, graceful shutdown, a retry delay with backoff, a dead-letter queue for jobs that keep failing, and a way for users to see progress. The ideas stay the same.

Idempotency

Idempotent means doing something twice has the same effect as doing it once. It is the main safety tool for anything retried, and agents retry a lot.

Stripe's documentation describes the standard pattern. The client generates a unique key and sends it in an Idempotency-Key header. The server saves the result of the first request made with that key, and a retry with the same key returns the saved result and does not repeat the work. In its API version 1, keys can be removed after at least 24 hours, and the server compares the parameters of a retry with the original and returns an error if they differ. Stripe suggests random identifiers such as V4 UUIDs, and advises against sensitive data such as email addresses as keys.

Apply it in two places:

• At the front door. The client sends a key with each request, and submit returns the same job for the same key, so a double click or a network retry does not start two runs.
• At each side effect. Pass a key derived from the job and the step to the outside service, as once does, so a retried step does not charge a card twice. Not every service supports idempotency keys, so for those, record the effect yourself and check before repeating it.

State

• Keep state outside the process. In a database, not in a worker's memory. Workers should be disposable.
• Checkpoint what you need to resume: the messages so far, the step number, tool results, and anything the agent has decided. A resumed run should be able to rebuild exactly where it was.
• Version the state. When you change its shape, jobs saved in the old shape may still be running. Handle both, or let old jobs finish on old code.
• Delete it on a schedule. State holds user data. See Privacy, bias and red teaming.

Secrets

• Never put secrets in prompts, context, logs or traces. See Prompt injection and tool sandboxing.
• Store them in your platform's secret manager or environment settings, not in code or the repository.
• Give each tool its own narrow credential, and the user's rights where it acts for a user.
• Use different keys for development, staging and production, and rotate them.
• Set spend limits on provider keys, so a bug or an attack has a ceiling.

Concurrency, limits and cost

• Size the worker pool to your model provider's rate limits, not only to your hardware. Add retries with backoff, as in Errors, retries and rate limits.
• Limit per user or tenant, so one customer cannot use the whole queue.
• Put a budget on every job, in steps and dollars, as in Agent cost and latency.
• Run code and risky tools in a sandbox, as in Prompt injection and tool sandboxing.

Monitoring

Add to the traces from Observability and tracing a few facts about the queue:

• Queue depth and the age of the oldest job. They show whether workers keep up.
• Success, failure and retry rates.
• Stuck jobs, running longer than they should.
• Jobs given up on, which need a person.
• Cost per job.
• Latency from submit to finish, not only model time.

Alert on the few that mean users are hurt, such as the oldest job's age passing a limit.

Releases and rollbacks

Releasing an agent has a problem web apps do not: runs are still in flight when you deploy. Anthropic's write-up of its multi-agent research system notes that deployments can break agents that are running, and that it avoids this with rainbow deployments, which shift traffic gradually from the old version to the new so that running agents are not cut off. It also says the system resumes from checkpoints when failures happen, and does not restart from the beginning.

Practical habits:

• Pin the model. Use a specific model id, and check whether your provider offers aliases that can move. A silent model change can change behavior.
• Version your prompts and tools, and record the version in each trace.
• Gate releases on evals, as in Evaluating AI agents. Run the suite before and after.
• Roll out gradually. Send a small share of new jobs to the new version first, and watch the metrics.
• Keep rollback easy: the previous version should still be deployable, and its state format should still be readable.
• Let in-flight jobs finish on the version they started with, or make sure new code can resume old checkpoints.
• Use feature flags to switch off a risky new tool or behavior without a deploy.

Telling users what is happening

A job that runs for minutes needs feedback. Stream progress such as "searching" or "drafting", or let the client poll the job's status. Show a job id, and allow cancelling. For actions that need a person, send a notification and wait on a saved state, as in Human-in-the-loop agents.

A launch checklist

1. Does the request return quickly, with a job id?
2. Does every step checkpoint, and can a new worker resume?
3. Are side effects idempotent?
4. Is every job bounded in steps, time and cost?
5. Are secrets out of code, prompts and logs?
6. Are risky tools sandboxed?
7. Do I have traces, queue metrics and alerts?
8. Can I roll back, and do in-flight jobs survive a deploy?
9. Do my evals gate the release?
10. What happens when the model provider is down?

Mistakes beginners make

1. Running a long agent inside one web request.
2. Keeping state only in memory.
3. No checkpoints, so a crash restarts everything.
4. Sending the email twice because a retry repeated the step.
5. No lease, leaving dead jobs "running" forever.
6. Unlimited retries.
7. No budget or step limit per job.
8. Secrets in prompts or logs.
9. Deploying while jobs run, without a plan for them.
10. Changing the prompt and model together, with no evals.

Quick recap

• Production agents are slow, stateful, side-effecting and unpredictable, so they need to be resumable and safe to retry.
• Run them in a function for short tasks, in workers behind a queue for long ones, in a durable workflow engine for strong guarantees, or in a provider-hosted harness for the least infrastructure.
• A job moves through queued, running, waiting and finished states. Workers hold leases and heartbeat, and a checkpoint is saved after every step.
• Idempotency keys make retries safe, at the front door and at every side effect.
• Keep state in a database, secrets in a secret manager, and every job bounded in steps, time and cost.
• Monitor queue depth, job age, failures and cost.
• Release gradually, pin versions, gate on evals, and let in-flight jobs finish or resume.
• Platform limits and provider terms change, so check current documentation.

Try it yourself

Run the tests, then break the runner on purpose. Make a step raise an error on its second call and watch the retry. Make one raise the uncatchable crash in the middle of a four-step job, advance the clock past the lease, and watch the next worker resume. Add a side effect with once and confirm it happens once. Then sketch how you would put the same logic on a real database and queue, and write down which of your agent's side effects are not idempotent yet.

Next lesson: Walkthrough: a support agent.