Self-Critique and Evaluator-Optimizer: Improve Agent Output with a Review Loop

The first answer an agent gives is often not its best one. A person would read a draft back, spot the weak parts and fix them. This lesson teaches the same habit for agents: a generate and evaluate loop, where one step writes, another step judges, and the writer revises using the feedback. Anthropic calls this the evaluator-optimizer workflow. You will learn when it helps, what can go wrong, how to build it, and how to stop it before it wastes money.

The idea in one picture

Two roles do the work:

• The generator (the optimizer) writes a draft, and later revises it.
• The evaluator reads the draft and returns a score and a list of problems.

The loop repeats until the evaluator is satisfied or a stop rule fires. Both roles can be the same model with different prompts, or different models, or code, or a person.

When a review loop pays off

Anthropic's guide to building effective agents describes two signs that this pattern fits. First, the output gets clearly better when someone gives feedback on it. Second, the model is able to give that feedback. Its examples are literary translation, where a reviewer notices nuance the first pass missed, and complex research, where a reviewer decides whether more searching is needed.

Research supports caution. A few findings, all from the papers' own abstracts and tests:

• Self-Refine (Madaan et al., 2023) has a model write, critique and revise its own output with no extra training. Its abstract reports about a 20 percent absolute improvement on average over single-step generation, across seven tasks.
• Reflexion (Shinn et al., 2023) has an agent write short notes about what went wrong and keep them for later attempts. Its abstract reports 91 percent pass@1 on the HumanEval coding benchmark, against 80 percent for GPT-4 in the comparison it cites.
• "Large Language Models Cannot Self-Correct Reasoning Yet" (Huang et al., ICLR 2024) found that without external feedback, models struggled to correct their reasoning, and accuracy sometimes got worse after self-correction. The authors note that self-correction does help with goals like tone and safety, where the criteria are clear. The paper also argues that some of the earlier gains came from feedback that used the correct answers.

Read these together. A loop helps when the feedback is real information: a failing test, a rule that was broken, a clear rubric, a fact that was checked. It helps much less when the model is only asked "are you sure?" about a hard reasoning answer it cannot verify. The lesson from the research is to give the evaluator something solid to judge against.

Three kinds of evaluator

Use the strongest, cheapest kind that can judge the thing you care about. Run code checks first, because they cost almost nothing. Add a model judge for what code cannot see. Add a person where the stakes demand it, as in Human-in-the-loop agents.

Self-critique or a separate evaluator

You can ask the same model to review its own draft. This is self-critique, and it is the simplest to build. It has one known weakness: a model that wrote a mistake may not notice it, because the same blind spot produced it.

A separate evaluator reduces this. It can be a fresh call with a different prompt and no memory of how the draft was written. It can be a different model. It can be code. The more independent the evaluator is, the more useful its feedback is. Give it only the task, the rubric and the draft, so it judges the result and not the writer's intentions.

Write a rubric the evaluator can use

A model judge is only as good as its instructions. Compare:

• Weak: "Is this good?"
• Strong: "Score from 0 to 1. Subtract for each problem: over 60 words, no mention of the refund, not ending in a full sentence. For each problem, say what to change."

A good rubric:

1. Lists a few concrete criteria, not a general feeling.
2. Asks for a score and a list of problems, so your code can decide whether to continue.
3. Asks for problems that say what to change, because "unclear" gives the generator nothing to act on.
4. Is asked to judge only against the rubric, so it does not invent new demands every round.

The code

Here is the whole loop. It has a rule-based evaluator, a model evaluator, a reviser and the loop itself. The evaluator and reviser are passed in as functions, so the loop works with any of them.

1import json
2
3MODEL = "claude-haiku-5-5"
4
5
6def check_summary(text, max_words=60):
7    """A rule-based evaluator: cheap, exact, and it cannot be talked into a pass."""
8    problems = []
9    words = len(text.split())
10    if words > max_words:
11        problems.append(f"Too long: {words} words, the limit is {max_words}.")
12    if "refund" not in text.lower():
13        problems.append("Does not mention the refund.")
14    if not text.rstrip().endswith((".", "!", "?")):
15        problems.append("Does not end with a full sentence.")
16    return {"score": 1 - len(problems) / 3, "problems": problems}
17
18
19def llm_evaluate(client, task, draft, rubric):
20    """A model evaluator: use it for criteria that code cannot check."""
21    reply = client.messages.create(
22        model=MODEL, max_tokens=512,
23        system="You are a strict reviewer. Judge only against the rubric. "
24               'Reply with JSON only: {"score": 0 to 1, "problems": ["..."]}. '
25               "Each problem must say what to change.",
26        messages=[{"role": "user", "content": f"Task:\n{task}\n\nRubric:\n{rubric}\n\nDraft:\n{draft}"}],
27    )
28    try:
29        data = json.loads(reply.content[0].text)
30        return {"score": float(data["score"]), "problems": [str(p) for p in data["problems"]]}
31    except (ValueError, KeyError, TypeError):
32        return {"score": 0.0, "problems": ["The evaluator reply was not valid JSON."]}
33
34
35def revise(client, task, draft, problems):
36    prompt = f"Task:\n{task}\n\nDraft:\n{draft}\n\nFix only these problems and keep everything else:\n"
37    reply = client.messages.create(
38        model=MODEL, max_tokens=1024,
39        messages=[{"role": "user", "content": prompt + "\n".join(f"- {p}" for p in problems)}],
40    )
41    return reply.content[0].text
42
43
44def refine(draft, evaluate, revise_fn, max_rounds=3, pass_score=1.0):
45    best, best_review = draft, evaluate(draft)
46    history = [(draft, best_review)]
47    if best_review["score"] >= pass_score:
48        return best, history, "passed"
49    current, review = draft, best_review
50    for _ in range(max_rounds):
51        new = revise_fn(current, review["problems"])
52        if new == current:
53            return best, history, "no change"
54        new_review = evaluate(new)
55        history.append((new, new_review))
56        if new_review["score"] > best_review["score"]:
57            best, best_review = new, new_review
58        if new_review["score"] >= pass_score:
59            return best, history, "passed"
60        if new_review["score"] <= review["score"]:
61            return best, history, "no progress"
62        current, review = new, new_review
63    return best, history, "max rounds"

The tests for this code use a stand-in client that returns scripted replies, so they run without the API.

The evaluators

check_summary is the cheap kind. It counts words, looks for a required word, and checks the ending, then returns a score and a list of problems. The reply format, {"score", "problems"}, is the same one the model evaluator returns, so the loop does not care which one it has.

llm_evaluate asks a model for the same format. Notice two things. The system prompt says to judge only against the rubric. And the reply is parsed defensively: model output is data from outside your code, so a reply that is not valid JSON becomes a score of zero with a clear message, not a crash. For stricter guarantees, use structured outputs as in Tool definitions and structured output.

The reviser

revise sends the task, the current draft and the list of problems, and asks the model to fix only those problems and keep everything else. The words "keep everything else" matter. Without them, a reviser may rewrite parts that were fine and introduce new faults.

The loop

refine does the following:

1. Scores the first draft. If it already passes, it returns at once with no revision calls.
2. Otherwise it revises, scores the new draft, and records it in the history.
3. It remembers the best draft seen, not just the latest one.
4. It stops for a reason and says which reason it was.

Stopping rules

A loop with no end condition can run forever and cost a lot. Anthropic's guide also recommends stopping conditions such as a maximum number of iterations. The code has four:

• Passed. The score reached the bar. This is the happy ending.
• Max rounds. A hard limit, such as three. Pick it from your own data: measure how much each extra round adds, and stop where it stops paying.
• No progress. The new score is no better than the last. More rounds are unlikely to help, so stop.
• No change. The reviser returned the same text, so another round would be identical.

You can add a fifth: a budget, in tokens or dollars, for the whole loop. See Agent cost and latency.

Return the best draft, not the last

Revisions can make a draft worse. The reviser fixes one problem and breaks another. This is the reason refine keeps best separately. In the tests, a revision that scores lower stops the loop, and the function returns the earlier, better draft. Without that rule, a loop can finish with something worse than the first attempt.

A worked example

This trace is illustrative: the replies are scripted, as in the tests, not taken from a live model.

Task: write a customer reply of at most 60 words that mentions the refund.

1. Draft 1: "Sorry, it is late." The evaluator scores it 0.33. Problems: does not mention the refund, does not end with a full sentence.
2. Draft 2: "Sorry for the delay. We will refund you." Score 1.0. Passed after one revision, two evaluator calls and one reviser call.

If draft 2 had scored lower than draft 1, the loop would stop with reason "no progress" and hand back draft 1. That is the behavior to want.

What a review loop costs

Every round adds at least two model calls, one to revise and one to evaluate, and the revise call re-sends the draft. Three rounds is roughly seven calls instead of one, plus the added delay. So:

• Run the cheap checks first, and call a model judge only when the cheap ones pass.
• Use a smaller model as the evaluator when the rubric is simple, and test that it agrees with a stronger one.
• Stop early when the score passes. Do not loop for the sake of looping.
• Measure. Compare the quality of first drafts and final drafts on a sample of real tasks. If the final is not clearly better, remove the loop. Evaluating AI agents shows how.

How a review loop goes wrong

1. A vague evaluator. "Needs improvement" gives the generator nothing to work with.
2. Rubric gaming. The generator learns to satisfy the letter of the rubric and not its purpose, such as stuffing a required word in an odd place. Keep checks tied to what you actually care about, and spot-check results yourself.
3. A lenient judge. A model that writes and judges the same text can be too ready to approve it. Use a separate prompt, or a different model, or code.
4. A harsh judge. An evaluator that always finds something wrong makes the loop run to the limit every time.
5. Moving goalposts. Different demands each round make the draft wander. Fix the rubric and send the same one every time.
6. Over-editing. The reviser rewrites good parts. Tell it to change only what the problems name.
7. Using it where it cannot know. Asking a model to re-check a fact or a calculation it cannot verify adds cost but no information. Give it a tool, such as a calculator, a search or a test runner, so the check is real. Common agent tools covers some.

Where this fits with other patterns

A review loop is a workflow, because you decide the shape in advance: write, judge, revise, stop. That makes it easy to test and predict, which Anthropic's guide favors when the task allows. It pairs well with others in this module. The ReAct loop reasons while acting, and planning designs decide steps ahead of time. A review loop can wrap either one, to check the final answer. The next lesson, Chaining, routing and parallelization, covers more fixed workflows. If you are unsure which family to start from, read Workflows or agents.

Mistakes beginners make

1. Asking "are you sure?" and calling it a review. Give the evaluator a rubric and real evidence.
2. No stop rule. Always set a round limit.
3. Returning the last draft instead of the best one.
4. Evaluating with the writer's own context, which carries its blind spots.
5. Skipping cheap code checks and paying a model for what a few lines can verify.
6. A changing rubric from round to round.
7. Never measuring whether the loop beat the first draft.
8. Trusting the score. A model's score of 0.9 is a judgment, not a measurement.

Quick recap

• A review loop has a generator, an evaluator and a stop rule. Anthropic calls it the evaluator-optimizer workflow.
• It works best when the output improves with feedback and the evaluator can give useful feedback.
• Research shows gains when the feedback is real, and little or negative effect when a model just second-guesses its own hard reasoning.
• Prefer code or tests as evaluators, add a model judge for fuzzy qualities, and use a person for high stakes.
• Give the evaluator a concrete rubric, a score and a list of problems that say what to change.
• Stop on pass, on a round limit, on no progress, or on no change, and return the best draft seen.
• Each round costs at least two calls. Measure whether the loop is worth it.

Try it yourself

Run refine with check_summary and a reviser function that you script by hand, giving it a good draft, a bad draft that improves, and a draft that gets worse. Confirm each stop reason. Then add one more rule to check_summary, such as "must not contain the word unfortunately", and watch how the score and problems change. Finally, write a rubric for a task of your own, such as a product description, and ask yourself which of its criteria code could check and which need a model.

Next lesson: Chaining, routing and parallelization.