Prompt engineering & evals

This is the 4D Description from Session 1, held to an engineering standard: every vague word in your prompt is a decision you delegated to the model. Name the audience, the format, the length, the edge-case behavior. The test: could a new hire execute your prompt without asking a follow-up question?

Notice what "after" pins down: audience, sections, limits, evidence rule, empty-case behavior, dialect, length. None of that is clever - it is just complete. Most prompt "engineering" is finishing your own specification.

Once a prompt contains more than one KIND of thing - instructions plus a document plus examples - prose runs together and Claude can confuse your data with your rules. XML tags draw hard boundaries. Claude is specifically trained to respect them, and you can reference tags by name ("the document in <context> tags").

Common tags: <context>, <instructions>, <examples>, <data>, <output_format>. The names are yours to choose - consistency matters more than the vocabulary. Tags also defuse a classic failure: a document that itself contains instruction-like sentences no longer hijacks your prompt, because your instructions live outside the <context> block.

The highest-leverage technique per minute spent: 1-3 gold examples of input → ideal output routinely beat paragraphs of format description. An example carries tone, structure, level of detail, and edge-case handling all at once - things prose describes badly.

Rules of thumb: pick examples that span your real variety (one easy, one messy, one edge case); keep them SHORT - the model imitates length too; and make them genuinely gold, because Claude will faithfully reproduce any sloppiness you include. If you cannot produce a gold example yourself, that is a sign the task is underspecified, not a prompting problem.

Prompts fail quietly and unevenly: a change that fixes your favorite test case often breaks three you never look at. The eval workflow is the same loop as ML model development:

• 1 · Define the task - inputs, expected outputs, and what "correct" means, in writing.
• 2 · Build a test dataset - 20-100 realistic cases with expected answers, including the ugly ones.
• 3 · Run - execute the prompt against every case.
• 4 · Grade - code-based checks for structured tasks, model-based rubrics for subjective ones.
• 5 · Iterate - change ONE thing in the prompt, rerun, compare pass rates. Keep what wins.

Writing 50 test cases by hand is why people skip evals. So don't: have Claude generate them, then human-review before trusting. You are curating, not authoring.

Then the non-negotiable step: read every generated case and fix or delete the bad ones. Claude's label IS your answer key - an unreviewed answer key silently corrupts every eval you run against it. Add real production examples (anonymized) as they accumulate; synthetic data starts the flywheel, real data keeps it honest.

If the output is structured - a category, a JSON object, a SQL string - grade it with plain Python. Exact match, regex, or schema checks: free, instant, deterministic, and immune to flattery.

Design your task to be code-gradable where you can - forcing a category from a fixed list, or JSON with known keys, is often a better prompt AND a free grader. Save the expensive grading for outputs that genuinely need judgment.

Summaries, emails, explanations - no regex can score "faithful and clear". Instead, a second Claude call grades the first against a written rubric. The rubric is the whole game: grade specific, checkable qualities, never "is this good?".

Note the 6.1 moves reused: temperature 0, JSON prefill, validation. Spot-check the grader against your own judgment on 10 cases before trusting it at scale - a grader is a prompt, and prompts need evals too. Which grader when:

Everything above, assembled. A JSONL dataset, a loop, a grader, a pass rate. This is genuinely all an eval needs to be useful - frameworks add convenience later, not correctness.

Workflow: change the prompt, bump PROMPT_VERSION, rerun, read the failures - they tell you exactly what to fix next. Keep a log of version → pass rate and prompt tuning becomes boring, measurable engineering. Which was the point.