Why vocabulary is a leadership tool
There is a specific moment this session retires: the status update rolls past - "we added HITL gates after the evals caught a tool-calling regression" - and you nod, and you make a mental note to look it up later, and later never comes. The cost of that nod is not embarrassment; it is every question you did not ask. Each term below is presented three ways: the word, the plain-English meaning, and the question it lets you ask. The third column is the one that changes your meetings.
The machine, in six words 8 min live
Every agent system - whatever the vendor, whatever the framework - is built from the same handful of parts. Six words describe the whole machine. Learn these six and every architecture diagram becomes readable.
LiveLLM · agent · workflow - the identity words4 min▶
- LLM - the large language model: the text engine that predicts the next word astonishingly well. The engine, not the vehicle. The question it lets you ask: "which model is this running on, and what does each use cost us?"
- Agent - a system where the model chooses its own steps and tools (session a1's definition, now one word). "What exactly is it allowed to decide on its own?"
- Workflow - a fixed flowchart with a model filling in text at some steps. "Did we fix the steps in advance, or does the model pick them?"
Notice the pattern already: the plain meaning is nice to have, but the question is what you take into the room. That pattern holds for all 22 terms.
LiveOrchestration · tool calling · state - the moving parts4 min▶
- Orchestration - the code that sequences models, tools, and checks into one system; the conductor. "Who is conducting - our code, or the model itself?" (That is the workflow/agent question wearing engineering clothes.)
- Tool calling - the model asking to run a real function: search the wiki, query the database, send the email. Tool calling is where AI stops writing about the world and starts acting on it. "What can it actually DO, beyond producing text?"
- State - everything the system remembers mid-task: the conversation so far, tool results, progress. "What happens if it fails halfway through - do we resume, restart, or lose the work?" This single question routinely reveals whether a system was built for demos or for production.
The question that found the gap. A operations director (anonymized org) asked exactly the state question - "what happens if it fails halfway?" - in a project review. Silence, then honesty: a failure at step four of five meant starting over, including re-sending one customer email. The fix took a sprint. The question took ten seconds.
The operations words 9 min live
Six words describe the machine; the rest describe running it responsibly. Four are governance words - your words, as a leader. Four are plumbing words that explain most of the sentences engineers say. The remaining eight live in the full table below.
LiveHITL · evals · traces · observability - the governance four4 min▶
- HITL (human-in-the-loop) - mandatory human approval at defined points; the machine pauses, a person signs off. "Where exactly does a human sign off - and who is that human?"
- Evals - automated tests that score AI output quality, run repeatedly. The AI equivalent of financial controls. "How do we know it is still good next month, not just in the demo?"
- Traces - the step-by-step record of one run: every decision, every tool call, in order. "Show me exactly what it did last Tuesday." If your team can answer that in minutes, you have real observability. If not, you have a black box with a logo.
- Observability - dashboards and alerts over ALL runs: costs, errors, quality drift. "Would we notice if it degraded quietly - or would customers tell us first?"
These four are not technical trivia. They are the same four instincts you already apply to finance: approval limits, controls, audit trails, and monitoring. The vocabulary is new; the governance is not.
LiveRAG · context window · token · MCP - the plumbing four4 min▶
- RAG (retrieval-augmented generation) - fetch relevant passages from YOUR documents first, then answer from those. "Is it answering from our data, or from its general memory?" The difference decides whether the answer is grounded or merely plausible.
- Context window - how much text the model can consider at once. Finite, like working memory. "On long tasks, what falls out of its memory - and does anything important?"
- Token - the billing unit; roughly three-quarters of a word. "What actually drives our bill - and which use case is the expensive one?"
- MCP (Model Context Protocol) - an open standard for plugging tools and data sources into AI systems; USB for AI tools. "Are our integrations standard and reusable, or is every one a custom build we will pay to maintain?"
Self-studyThe full 22-term table - keep it beside your calendar6 min read▶
The remaining eight terms - guardrails, prompt injection, the lethal trifecta (session a4 is built around it), hallucination, LLM-as-judge, checkpointing, agent washing, multi-agent - complete the set. The whole vocabulary, one line each:
| Term | Plain English | The question it lets you ask |
|---|---|---|
| LLM | The text engine that predicts the next word, astonishingly well | "Which model, and what does each use cost?" |
| Agent | A system where the model chooses its own steps and tools | "What is it allowed to decide alone?" |
| Workflow | A fixed flowchart; the model fills in text at some steps | "Did we fix the steps, or does the model?" |
| Orchestration | The code that sequences models, tools, and checks | "Who is conducting - our code or the model?" |
| Tool calling | The model asking to run a real function (search, SQL, email) | "What can it actually DO beyond text?" |
| State | Everything the system remembers mid-task | "What happens if it fails halfway?" |
| HITL | Mandatory human approval at defined points | "Where does a human sign off, and who?" |
| Evals | Automated tests that score AI output quality | "How do we know it is still good next month?" |
| Traces | The step-by-step record of one run | "Show me what it did last Tuesday." |
| Observability | Dashboards and alerts across all runs | "Would we notice quiet degradation?" |
| RAG | Fetch our documents first, then answer from them | "Our data, or its general memory?" |
| Context window | How much text the model can consider at once | "What falls out of memory on long tasks?" |
| Token | The billing unit; roughly three-quarters of a word | "What actually drives our bill?" |
| MCP | Open standard for plugging tools into AI systems | "Standard integrations, or bespoke builds?" |
| Guardrails | Hard limits on what the system may do or say | "What can it NEVER do, by construction?" |
| Prompt injection | Malicious text that hijacks the model's instructions | "Can outside text give our agent orders?" |
| Lethal trifecta | Private data + untrusted content + external comms in one agent | "Do any of ours have all three?" (session a4) |
| Hallucination | Confident, fluent, wrong | "How do we catch it before customers do?" |
| LLM-as-judge | Using one model to grade another's output | "Who grades the grader?" |
| Checkpointing | Saving state so a run can pause, resume, or undo | "Can we stop it mid-flight and rewind?" |
| Agent washing | Relabeling chatbots or RPA as "agents" | "Does the model actually choose the steps?" |
| Multi-agent | Several agents handing work to each other | "What happens when one feeds errors to the next?" |
Decode the status update ★ 12 min · glossary in hand
Here is a realistic, anonymized engineering status update - the kind that produces the nod. Your job is to translate it, line by line, until there is nothing left to nod at.
"The agent kept blowing past the context window on long threads, so we added a summarization step in the orchestration layer and moved document lookup to RAG over the policy wiki. Evals caught a tool-calling regression after the model swap - traces show it hallucinating a refund tool that doesn't exist - so we've put a HITL gate on anything touching payments until the new guardrails land."
Read it once at normal speed and notice, honestly, which phrases blur past. Those are your terms for today.
Underline every glossary term: context window, orchestration, RAG, evals, tool calling, traces, hallucinating, HITL, guardrails. Nine terms in three sentences - a normal density.
Translate line by line with the table. Line 1: it forgot things on long conversations, so they compress the history and now fetch policy answers from our own wiki. Continue for lines 2 and 3 yourself.
Write the one-sentence executive version. Something like: "the system had a memory limit and briefly invented a payment action; tests caught it, and payments now require human sign-off until the fix ships."
Now the leadership move: find the question hiding in it. (Strong candidate: "the evals caught it - AFTER how many production runs?")
Your first three questions ★ 8 min · your real project
Vocabulary you have not used within a week evaporates. Pick three terms and spend them on your actual project before Friday.
Pick the ONE live (or proposed) AI initiative in your organization you care most about. Write its name at the top of a page.
Scan the 22-term table's third column and pick the three questions that make your stomach tighten slightly when you imagine the answer. That feeling is good judgment - trust it.
Rewrite each question in your project's language. Not "show me a trace" but "show me the trace of what the onboarding assistant did for the customer who complained last week."
Put them in the agenda of your next review meeting - as curiosity, not audit. "I've been learning the vocabulary and I'd love to see..." opens doors that "I require..." closes.
Before session a3 ◐ 30 min total
- Read the full 22-term table once, slowly, and mark the terms you could not yet explain to a colleague over coffee. Re-read only those.
- Run the decoding-partner prompt on one REAL status update, email, or slide from your own inbox. This is the exercise that makes it stick.
- Ask your three questions from Exercise 2 in an actual meeting this week, and note which answer surprised you.
- Preview for a3: skim the "lethal trifecta" row in the table. Next session's failure stories will make it vivid; session a4 makes it actionable.
- Can I see a trace of one real run of our most important agent or workflow - every step, every tool call?
- Which of our AI systems have evals today, when did they last fail, and what happened next?
- Where are our human-in-the-loop points, and who exactly is the human at each one?
- On our longest-running tasks, what falls out of the context window - and does anything important?
- Are our tool integrations built on a standard like MCP, or is each one a custom build we maintain alone?
Official sources covered
Every definition on this page is grounded in the official docs and published engineering guidance - simplified in wording, never in meaning. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The term "state" equips you to ask which leadership question?
State is everything the system remembers mid-task. The halfway-failure question routinely separates demo-grade systems from production-grade ones - in ten seconds.
2 · A trace is best described as...
One run, every step, every tool call, replayable. It is the audit trail your auditors will ask for - and the single most useful thing to request in a review.
3 · Your team says the assistant "uses RAG over the policy wiki". That means...
RAG = retrieve first, then generate from what was retrieved. No retraining, no editing rights - and the reason its answers can cite your actual policies.