Bias and Debiasing Prompts

A prompt can affect whose work sounds qualified, which examples a model treats as typical, and how it responds to the same request when a name changes. You will learn to spot these patterns, compare carefully controlled versions of a prompt, and decide what a prompt can reasonably improve. A prompt can reduce some avoidable differences. It cannot remove every bias in a model, its data, or the way a system is used.

What bias can look like in a response

In this lesson, bias means a repeated, meaningful difference in how a model treats people or groups in a task. It can show up as a stereotype, a different quality of help, an unequal level of skepticism, or a recommendation that changes when a detail changes even though that detail should not matter.

Bias is not always a hostile sentence. Imagine a résumé assistant that describes one candidate as “experienced and decisive” and another, with the same experience, as “promising but needs confidence.” Or a writing helper that makes one person’s message more formal and apologetic than another person’s equivalent message. Those differences can affect what a reader thinks, even when the words seem polite.

There is no single test that captures every kind of unfairness. The right question depends on the task, the people affected, and what the output will be used to decide. For example, a system summarizing a person’s stated accessibility needs should preserve relevant disability information. A system scoring a coding exercise should not lower its score because a candidate has a disability. “Treat every input identically” is not a useful rule if the task legitimately calls for different accommodation or context.

Bias can enter at several points: the model’s learned patterns, the wording and assumptions in your prompt, examples you provide, retrieved material, the rubric used to judge answers, or the way an organization acts on the result. This lesson focuses on prompt choices and tests because those are parts you can inspect. Keep the rest of the system in view too.

Prompts can nudge a result

A leading question can frame a person as suspicious before the model reviews any evidence. A role instruction can encourage the model to imitate a stereotype. A request such as “find the weaknesses in this candidate” invites a different response from “evaluate the evidence against these job-related criteria.” Both may mention the same résumé, but they set different jobs for the model.

Examples can carry quieter instructions. In few-shot prompting, the model learns a pattern from the examples you include. If every example labels a short, direct message from a woman as “rude,” while comparable messages from men are labeled “confident,” the example set teaches an inconsistent standard. The model may reproduce it even when your written instruction says to be fair. Anthropic’s prompting guidance recommends examples that are relevant and diverse, in part to reduce unintended patterns; that is useful advice, but diversity of examples by itself is not a fairness guarantee.

Before changing a prompt, define what should stay the same and what is allowed to change. If a task concerns language translation, a person’s name usually should not change translation quality. If the task is to explain a culturally specific phrase, cultural context may be relevant. Avoid stripping context just because it relates to identity. Instead, set a task-specific rule for when that detail may be used.

Run a counterfactual test

A counterfactual test compares two inputs that are alike in the relevant facts but differ in one feature you want to examine. For example, you might provide the same qualifications and experience with two different names, then compare how the model describes the candidate. The point is not to assume the names perfectly identify anyone’s gender or background. The point is to check whether a change that should not affect the job-related evaluation seems to move the result.

Research has used counterfactual comparisons to examine how sensitive generated text is to changes such as gender, occupation, or country in a prompt. Results depend on the task and measurement, so use these tests as probes for a specific system rather than a universal fairness score. Later work also emphasizes using prompt variations when evaluating social bias, since a single template may not reveal a pattern. See the counterfactual evaluation study and research on prompt variations in social-bias evaluation.

Here is a small example. Suppose your application summarizes interview notes against a fixed rubric. Keep the notes, role requirements, and rubric identical, and change only a name that is not relevant to the rubric. Compare specific outputs: does the model mention different strengths, assign different scores, or ask for different evidence? Repeat with more than one name pair, because one pair can contain its own associations. If the task has a justified reason to use identity information, state that reason and test whether the response uses it as intended.

Use a table before running the test:

The comparison should include both a paired check and a broader sample. A paired check makes it easier to isolate one change. A broader sample shows whether the result repeats across different examples, names, prompt phrasings, and ordinary cases. Run the same variants more than once if model outputs vary. Record the model and prompt version, the inputs, and the date so you can repeat the evaluation after a change.

Do not look only for a different final score. A model can give identical scores but describe one group with less detail or more doubt. It can also produce different wording that does not change the task outcome. Decide in advance which differences matter, and have people familiar with the context review ambiguous cases. For consequential decisions, a prompt experiment is not a substitute for a formal impact assessment, legal review, or a process for affected people to challenge an outcome.

A worked prompt revision

Consider a fictional résumé assistant. The original instruction says:

1Act as a tough hiring manager. Find the strongest candidates and point out anything that seems unusual. Be decisive.
2
3Candidate:
4[resume]

This instruction asks the model to judge people without defining job-related criteria. “Unusual” can encourage speculation, and “tough” and “decisive” can reward confidence over evidence. The prompt does not explain how to handle missing details or how to treat information unrelated to the role.

Suppose the job requires writing clear help articles, testing instructions, and collaborating with a support team. A more bounded prompt could be:

1Evaluate the candidate only against these role-related criteria:
21. Clear technical writing
32. Careful testing of instructions
43. Collaboration with support or product teams
5
6For each criterion, quote or identify evidence in the résumé. Mark a criterion
7“not shown” when the résumé does not provide evidence. Do not infer personality,
8reliability, family responsibilities, health, or ability from a name, photo,
9school, employment gap, or writing style. Do not treat an absence of evidence as
10evidence that a skill is absent.
11
12Return a short evidence table, followed by questions a human interviewer could
13ask consistently of every candidate. Do not rank candidates or make a hiring
14recommendation.
15
16Candidate:
17[resume]

The revised prompt narrows the task, identifies evidence, and avoids asking a model to make the final decision. It may reduce some prompt-induced speculation. It does not prove the model will apply the criteria evenly. Test it with counterfactual pairs and real examples reviewed under an appropriate privacy and governance process. Also inspect whether your criteria themselves are job-related and whether the résumé data is representative.

Check the examples you teach with

When a prompt uses examples, audit them as a dataset. Ask what traits repeatedly appear beside each label, whether the examples cover the same kinds of cases, and whether a shortcut could predict the answer. A classifier may learn that a name, dialect, or writing style predicts “low quality” because the examples accidentally correlate that feature with the label.

Take a tone classifier. If your examples label every message with slang as “unprofessional,” the model may learn that dialect or informality is the target, even if you intended to detect insults. Improve the set by including different styles under each relevant label and writing down the actual criteria. Then test on examples that were not used to build the prompt. The lesson on examples: zero-shot, one-shot and few-shot covers choosing representative examples; the lesson on defining success and building test cases explains how to keep evaluation cases separate from prompt examples.

Balanced examples are not just an equal count of demographic labels. They need to represent the language, situations, and outcomes the system will actually encounter. If a group appears in only one context, equal counts can still teach a misleading shortcut. Ask a domain expert or people affected by the system to review examples when possible, and avoid collecting sensitive details you do not need.

Improve the prompt, then test again

When you find an uneven result, first identify what part of the instruction could be responsible. Remove loaded adjectives, replace vague impressions with observable criteria, and say which evidence is relevant. Avoid adding a demographic disclaimer to every prompt as a substitute for a careful task definition. Broad instructions like “be unbiased” do not tell a model what to compare or what to do when information is missing.

Change one prompt element at a time when you can. Keep the old and new versions, run both on the same cases, and inspect not just the total score but the examples where the output changed. A revision that fixes one group comparison can create another issue, such as dropping useful context or refusing to give appropriate support. Use the evaluation workflow from Compare prompts with graders, and keep a holdout set so you do not tune the prompt to the exact cases you are judging.

You might write a rule such as “Evaluate only evidence relevant to the stated rubric; if evidence is missing, say ‘not shown’ rather than infer a trait.” Then test whether the model follows it across different names and phrasings. Ask reviewers whether explanations are equally specific and whether any assumptions appear. If you use an automated score, confirm it measures the behavior you care about instead of rewarding superficial wording.

Where prompt fixes stop

A prompt is one layer in a larger system. It cannot change the model’s training data, guarantee equal performance across languages, prevent every user from supplying biased instructions, or correct a rubric that encodes an unfair standard. A retrieval collection can repeat the same imbalance. A human reviewer can share the same assumptions as the model. A polished fairness instruction can create false confidence if no one tests it.

If a pattern persists, consider whether the task should use a language model at all, whether the inputs or criteria should change, whether a different model performs more consistently, or whether the decision belongs with a trained person. Add human review and an appeal route when outputs can meaningfully affect people. The AI Agents lesson on privacy, bias and red teaming discusses broader system-level safeguards and testing.

Bias evaluation also has measurement limits. Names are imperfect signals, a pair of prompts cannot represent a whole community, and a numeric difference does not explain its cause. Conversely, no measured difference on a small test does not prove that a system is fair. State the scope of your test plainly: what task you examined, which variations you tried, what differences you saw, and what the test did not cover.

Mistakes beginners make

1. Assuming a neutral-sounding prompt is fair. Test outputs on comparable inputs and inspect the evidence used.
2. Changing several identity details at once. Change one feature at a time first, then test realistic combinations.
3. Using one name pair as proof. Repeat the comparison with multiple names, inputs, and prompt phrasings.
4. Checking only scores. Review tone, omissions, uncertainty, evidence, and follow-up questions too.
5. Treating equal counts as balanced examples. Check whether examples cover comparable contexts and labels.
6. Removing every identity detail. Some details are relevant to the task or needed for appropriate support; define when and why.
7. Assuming “be unbiased” is a complete fix. Write task-specific criteria and verify that they change the behavior you intended.
8. Calling a prompt test a fairness certification. Report the test’s scope and involve broader system review when people may be affected.

Recap and try it yourself

Prompt wording and examples can steer a model toward patterns you did not intend. A counterfactual test helps you inspect one possible source of unfair difference: keep relevant facts fixed, change one detail that should not matter, and compare evidence, tone, omissions, and decisions. Repeat the test across examples and prompt versions. Audit few-shot examples for accidental shortcuts, and use criteria tied to the actual task.

Try this with a low-stakes task, such as summarizing fictional customer feedback:

1. Write a prompt that asks for a neutral summary using three explicit criteria.
2. Create two otherwise identical comments with different names or pronouns, if those details should not matter to the task.
3. Run both versions several times and compare the facts included, the tone, and any assumptions.
4. Add two few-shot examples that vary in writing style but share the same label criteria. Check whether style changes the classification.
5. Revise one prompt instruction, repeat the comparison, and record what changed and what your small test cannot establish.

If the results differ, treat that as a question to investigate, not a verdict about the model or a group. A careful test can reveal where to look next. It cannot establish fairness for every person, language, context, or use of the system.