Automatic Prompt Engineering

A model can help write the instructions you give to another model. You can ask it to make a prompt clearer, generate several alternatives, or suggest a fix after seeing a failure. That is useful when treated as a way to explore options. It does not remove the need to test the prompts or decide what good means.

In this lesson, you will learn a practical way to use prompt generators, what automatic prompt search means, and how both differ from soft prompt tuning. The examples are illustrative. I have not run them against a live model.

Ask for a candidate, not a final answer

A prompt generator is a model or tool that drafts instructions for another model. A metaprompt is a prompt whose task is to create or improve another prompt. For example, you might give a generator your task, audience, success criteria, a few examples, and a failed answer. It can return a revised instruction for you to inspect.

The generator is proposing text, not certifying quality. A prompt can sound precise while being too long, contradictory, or poorly suited to the actual input. Keep the useful parts, edit anything vague or unsupported, and evaluate the candidate against your existing cases.

The Automatic Prompt Engineer (APE) paper describes generating candidate instructions with a language model and selecting among them using a chosen score function. Its experiments used specific tasks and evaluation setups. They show how generation plus measurement can be automated for a defined problem; they do not mean any prompt a model writes will outperform a human-written prompt. The Promptbreeder paper explores an iterative variant that evolves task prompts and the prompts used to mutate them. Both approaches depend on an evaluation signal that rewards the behavior you actually want.

Give the generator useful material

A generator cannot infer your product’s definition of success from “make this prompt better.” Give it enough detail to propose a testable change:

1. Task: what the target model should do.
2. Audience and use: who will use the result and where.
3. Criteria: what must be included, avoided, or checked.
4. Examples: a few representative inputs, including a failure if available.
5. Boundaries: what the target model should do when information is missing.
6. Output request: ask for a prompt candidate and a short explanation of the changes.

For example, suppose a delivery-note prompt sometimes invents a prediction when a note does not contain one. Instead of asking “improve this,” provide the generator with the failed input, the bad answer, the expected behavior, and the current instruction. Ask it to propose two focused alternatives, each addressing the missing-time error without changing unrelated requirements.

The generator might return an instruction that says, “Use only times present in the note. If a prediction is missing, say it was not provided.” You still need to ask whether this is clear, whether it conflicts with another rule, and whether it preserves other required details. A model can invent improvements that do not address the failure, or remove a useful constraint while making the wording sound cleaner.

The previous lesson on debugging a failing prompt describes how to preserve the failure and identify its cause. Give that diagnosis to the generator. The more specific the observed issue, the more useful the candidate is likely to be.

Use a small, controlled search

You can try more than one candidate instead of accepting the first suggestion. Keep the search manageable:

1. Save the current prompt as the baseline.
2. Ask a generator for a small number of candidates, each based on a clear hypothesis.
3. Read the candidates and remove ones that contradict your requirements.
4. Run the remaining candidates on the same development cases and graders.
5. Inspect failures, compare results with the baseline, and keep a candidate only if it improves the criteria that matter.
6. Use holdout cases after you finish tuning, not as a repeated source of ideas.

This is a small form of prompt search: explore candidate instructions and select based on measured results. You can also generate variations from a successful prompt, such as shorter wording, clearer priorities, or a format example. Change one meaningful feature per candidate where possible. If one candidate changes the task, audience, examples, and output shape all at once, it becomes difficult to tell what caused a difference.

Prompt search can be manual or automated. A person can ask a model for three variants, then test them in the prompt lab. A script can generate candidates, run them across a dataset, and sort them by a score. The automated version can explore more candidates, but it also spends more calls and can optimize for a flawed score. More search does not make a weak grader trustworthy.

Keep the test honest

The generator must not grade itself using only the examples it saw while proposing candidates. Keep a stable development set for finding and fixing failures, then check the selected candidate on holdout cases. If you expose all cases and scores to the search process, it can learn to exploit those particular examples rather than improve the task more broadly.

Use the same criteria for every candidate and the baseline. If you change the grading rules midway, record that and rerun all candidates under the revised rules. Read the individual outputs. A higher average score can hide a serious regression on a small but important category, such as missing information or a valid no-answer case.

The lesson on comparing prompts with graders covers code checks, model graders, human review, repeated trials, and reading failure reports. Automatic prompt generation belongs inside that comparison process. It does not replace it.

Avoid optimizing for the wrong score

Search methods optimize whatever score you give them. If you reward answers for containing a particular phrase, candidates may repeat that phrase even when it does not fit. If a model grader prefers long answers, the search may produce bloated prompts and outputs. If the score rewards exact wording, useful paraphrases may be rejected.

Use separate checks for important properties. For delivery summaries, a code check can confirm that a required time appears, a factual comparison can verify it against the input, and a reviewer can judge tone. Keep safety and “do not guess” behavior as explicit criteria, not as optional preferences that can be traded away for a higher style score.

Before launching a large search, manually test the score on a few known-good and known-bad outputs. If the score ranks a known bad answer above a good one, fix the evaluation before generating more candidates. The search process can only be as reliable as its target.

Example: ask for alternatives after a failure

Here is a metaprompt for the missing-prediction problem. The bad output and suggested candidates are illustrative, not generated from a live model.

1You are helping revise an instruction for a delivery-note summarizer.
2
3Task:
4Write one neutral sentence of no more than 25 words. Preserve predicted and actual arrival times when they are present. Do not invent a cause.
5
6Failure:
7Input: “The bus arrived at 8:32. The app did not show a prediction.”
8Output: “The app predicted 8:10, but the bus arrived at 8:32 because of traffic.”
9
10Success criteria:
11- Do not invent a predicted time or cause.
12- State that the prediction was unavailable.
13- Preserve the actual arrival time.
14- Keep the reply within 25 words.
15
16Suggest two concise prompt revisions. Change only instructions that could
17address this failure. Keep the audience and task the same. For each option,
18explain which criterion it targets. Do not claim either option is tested.

Review the candidates before running them. A useful candidate might add “If a time is absent, say it was not provided.” Another might explicitly say “Use only facts stated in the note.” These are related but not identical. Test them separately if you want to learn which instruction helps. Then run each with the baseline on the same examples, including an ordinary note that has both times.

If the first model-generated prompt passes the missing-time example but fails to include an available prediction, it is not a finished fix. It may have overcorrected by omitting predictions altogether. The ordinary case catches this regression. The holdout set checks whether the change generalizes to different wording.

What prompt tuning means

Prompt tuning often refers to a training technique that learns a set of continuous values, sometimes called soft prompts, which are added to a model’s internal input representation. A person does not normally read or edit these values as ordinary sentences. The technique is trained with examples and requires access to a suitable model-training process.

This differs from prompt engineering in a chat or API call. A written prompt is discrete text you can inspect and revise directly. A soft prompt is learned numerical information associated with model input. The prompt-tuning research paper describes learning soft prompts through backpropagation while keeping the underlying model weights frozen in its experiments. The phrase “prompt tuning” is sometimes used more loosely for iterative editing of normal text prompts, so check how a tool or article defines it.

For most people writing prompts for an existing chat product, the relevant workflow is still to write text, test it on representative cases, and compare results. Soft prompt tuning is a distinct technical option for teams that can train and evaluate a model in an appropriate setup.

Keep human judgment in the loop

Automatic generation is most useful when it expands the options you can test. It can suggest wording you did not consider, summarize patterns from failures, or make a set of alternatives for a narrow task. You remain responsible for deciding whether a candidate expresses the intended behavior, respects boundaries, and passes checks that reflect real use.

Do not paste sensitive customer records or confidential instructions into an external prompt generator unless your organization allows it. You can often make a useful example by replacing personal details with fictional values while keeping the failure pattern intact. Prompt generation is still a model call, and the same privacy rules apply as for any other use.

Record the winning prompt, the cases and graders used, and why you selected it. That makes it possible to revisit the decision when inputs, task requirements, or models change. The next lesson covers keeping prompt versions and documenting those choices.

Mistakes beginners make

1. Accepting the first generated prompt. Treat it as a candidate and compare it with the baseline.
2. Asking only “make it better.” Give the generator the task, criteria, input, failure, and boundaries.
3. Generating too many variants at once. A large search costs more and makes it harder to inspect why a winner scored well.
4. Changing several prompt features in every candidate. Keep candidates tied to clear hypotheses.
5. Letting search see and tune to the holdout set. Use holdout examples only after choosing a candidate, then refresh them if they guide further edits.
6. Trusting the score without inspecting outputs. A broken grader can reward the wrong behavior very consistently.
7. Optimizing one criterion while breaking another. Keep must-pass requirements visible and inspect category-level failures.
8. Confusing soft prompt tuning with writing a better sentence. One edits readable text; the other learns continuous values through a training process.

Recap and try it yourself

A prompt generator can draft or revise instructions, while prompt search tests several candidates against a score or set of graders. Give the generator the failure and success criteria, then inspect its suggestions. Run every candidate and the baseline on the same development cases. Preserve holdout examples for a later check, and keep human judgment for issues the graders cannot decide. Soft prompt tuning learns numerical inputs through training and is different from editing readable prompt text.

Try this with one prompt you have already tested:

1. Choose one recurring failure from your development set.
2. Write a metaprompt that includes the current task, failure, expected behavior, and success checks.
3. Ask a model to propose two focused alternatives. If you do not have an approved model tool, draft two alternatives yourself from the same failure; the comparison steps are identical.
4. Review and run both candidates and the baseline against the same cases and graders.
5. Inspect the failures, then keep a candidate only if it improves the important checks without breaking other cases.

The value is not in having a model write the prompt. The value is in using its suggestions to find a change that survives a fair test.