Where DataDesk stands
DataDesk is a team now: analyst, checker and writer under a supervisor, on a persistent graph with approval gates and cited retrieval. What it does not have is proof. You cannot see inside a run without print statements, and "it seems better" is your entire quality methodology. Tonight fixes both - and closes the loop this track opened in b1.
See everything 5 min live
Debugging an agent with print statements is archaeology. Tracing is the live feed: every model call, every tool call, every token and millisecond, nested exactly as they happened - and for LangChain/LangGraph apps it costs two environment variables.
LiveTracing in two env vars - zero code changes3 min▶
Because DataDesk is built on LangChain and LangGraph, the framework already emits structured run data. LangSmith just needs to know where to send it:
Open the project at smith.langchain.com and every run is a collapsible tree:
- Every model call - full prompt in, full response out, per-call token counts and cost.
- Every tool call - arguments and returns, so "why did it retrieve THAT chunk?" is a click, not a guess.
- Latency per step - the waterfall shows exactly where a slow run spent its time.
- The multi-agent tree - supervisor → analyst → checker → writer as nested spans; b9's cascade experiment becomes literally visible.
LiveFree-tier reality check2 min▶
The Developer tier is genuinely free: $0, 5,000 traces per month, 14-day retention, one seat. For a team pilot of DataDesk - tens of runs a day - that is comfortable headroom. Paid tiers add seats, longer retention, higher volumes and the deployment features in Part 3.
| What | Developer (free) | When you outgrow it |
|---|---|---|
| Price | $0 | Per-seat + usage pricing above |
| Traces | 5k / month | A busy production app in days |
| Retention | 14 days | Audit or compliance needs longer |
| Seats | 1 | The moment a teammate asks "can I see?" |
Numbers as of 2026-07 - re-check the pricing page before you present this to anyone with a budget.
Self-study@traceable + the business model, seen clearly2 min read▶
Non-framework code: your plain Python functions (a pandas cleanup step, a custom API call) do not auto-trace. Decorate them with @traceable from the langsmith package and they appear in the same tree, nested where they ran. Your whole pipeline, not just the LLM parts, becomes one waterfall.
The business model, now visible: b1 told you "frameworks free, trust layer paid" - tonight you see the mechanics. LangChain and LangGraph are MIT and always will be; the moment you NEED to see inside a production agent - and you will - the tool that shows you is the product. That is not a criticism, it is the deal, and you should walk into it with open eyes: the free tier is real, and so is the gravity.
From vibes to numbers 6 min live
"It seems better after my change" is how agent projects die. An eval suite is 15 real questions with known-good answers, scored automatically on every change. It is unit testing for behavior - and it is the single highest-leverage habit this track can leave you with.
LiveDatasets + evaluate() - unit tests for behavior3 min▶
Three pieces: a dataset (inputs + reference outputs, curated from questions people actually asked DataDesk), a target function (your agent, wrapped), and evaluators (functions that score each output). client.evaluate(...) runs the whole grid and posts an experiment you can compare against every previous one.
- Curate from reality: your best test questions are the ones DataDesk already got wrong - b8's "not in the docs" probes, b9's poisoned CSV. Every production miss becomes a dataset row.
- 15 is enough to start. A small suite you run on every change beats a grand one you run never.
- Experiments diff: change the model, a prompt, a chunk size - the experiment view shows exactly which questions improved and which regressed. That is the "regression gate" your team can actually enforce.
LiveLLM-as-judge via openevals - scoring at scale3 min▶
Exact match works for "how many rows?" but not for the writer's COO summary - correct answers phrased a thousand ways. An LLM-as-judge evaluator hands the question, the reference and the agent's answer to a model with a grading rubric. The openevals package ships prebuilt ones - CORRECTNESS_PROMPT is the workhorse.
- Judge choice matters: use a strong model as judge even if the agent runs on the cheap one - a weak judge grades noise.
- Spot-check the judge: read 5 of its verdicts yourself before trusting 500. Judges have biases (verbosity, agreeableness); calibrate once, then scale.
- Two evaluators beat one: exact match for the factual rows, judge for the prose rows - Demo 2 wires both.
Self-studyOnline evals, pairwise, and the regression-gate habit2 min read▶
Online evals run evaluators on sampled LIVE traffic - catching drift your golden dataset has not learned yet (new question types, a changed upstream doc). Pairwise experiments show a judge two versions' answers side by side and ask which is better - often more reliable than absolute scores for prose.
The habit that outlives this course: treat the eval suite like CI. No prompt change, model swap, or chunk-size tweak merges without an eval run attached. Teams that adopt this bore their stakeholders with steady numbers; teams that skip it entertain everyone with incidents.
Shipping options 4 min live
DataDesk runs on your laptop. Shipping it to the team means choosing a lane: run the open-source stack on your own infra for free, or pay for LangSmith Deployment to host it. Know both lanes AND both names - "LangGraph Platform" was renamed "LangSmith Deployment" in late 2025, and the old name still dominates search results.
LiveChoosing a lane without religion2 min▶
The honest framing: everything you built in b1-b9 runs anywhere Python runs - the OSS lane is real, not a demo tier. What LangSmith Deployment sells is the platform work you would otherwise build around it: an agent server exposing your graph as an API, assistants (versioned configurations of one graph - prompts and models tuned per use case without redeploying), and Studio, the visual debugger for stepping through live graph runs.
- Pilot for a five-person data team? OSS lane: your graph + Postgres checkpointer + FastAPI is an afternoon (you own patching it).
- Org-wide with SLAs and no platform team? The managed lane earns its bill fast.
- Regulated industry? Hybrid and fully self-hosted deployment exist at the Enterprise tier - data plane on your metal, control plane theirs, or everything yours.
- Search-result decoder: anything titled "LangGraph Platform" or "LangGraph Studio" is the same product family pre-rename - read on, mentally substituting the new names.
Self-studyWhat production needs beyond deploy2 min read▶
Deployment is where this course stops, not where production ends. Still ahead of a real rollout: monitoring dashboards and alerting on live traffic, online evals as drift detection, cost budgets and rate limits per user, access control, and an incident playbook for when the agent is confidently wrong in public. The free Academy courses on observability and monitoring production agents are the natural next step - you now have every prerequisite they assume, and DataDesk is a better lab than their examples.
Turn the lights on ★ 10 min · everyone builds
Two env vars, three queries, and nine sessions of guessing what DataDesk does become a waterfall you can read.
Create the free account at smith.langchain.com, generate an API key, and export the two env vars from Part 1. Nothing else changes - pip install langsmith if it is not already present as a dependency.
Run three DataDesk queries you know well: a pure CSV question, a wiki question from b8, and the full b9 team request ("analyze march.csv and draft a COO summary").
Open the project and read the multi-agent trace as a tree: supervisor → analyst (watch its tool calls) → checker → writer. Every prompt, every return value, expandable.
Find the slowest step in the waterfall and the total token cost of the b9 run. Say the number out loud - this is the multi-agent price tag from the honest gate, now in dollars.
Find the checker catching the bad number (re-run the b9 cascade experiment if needed). The fix you built blind is now a visible span with the wrong value in its input and the bounce in its output.
ChatOllama reports no token-usage metadata, so cost columns stay empty on the local path; latency and structure still show in full.
DataDesk graduates ★ 12 min · build your own
Build the golden dataset, wire two evaluators, run the first experiment - and settle the b1 question with evidence.
Curate 15 questions from your real DataDesk usage this course: 8 factual (CSV stats, wiki definitions - exact expected answers), 4 grounding probes ("not in the docs" cases), 3 prose (summary requests with reference summaries).
Create the dataset and run the script below: one exact-match evaluator for the factual rows, one LLM-as-judge for correctness.
Read the experiment in the UI: score per question, judge reasoning attached, total cost. Find your worst question and read its full trace - score and diagnosis, one click apart.
Revisit your b1 sentence: "for DataDesk, the framework is/is not worth it because...". Rewrite it with tonight's numbers - accuracy on 15 real questions, cost per run, the checker's saves. That sentence is now a design-review answer.
Graduation checklist, out loud: persistence ✓ (b6) · approval gates ✓ (b7) · grounded RAG ✓ (b8) · team + checker ✓ (b9) · traces + evals ✓ (tonight). DataDesk is no longer a course exercise - it is a system you can defend.
The experiment that ended a meeting. A data team argued for three weeks about whether the cheap model was "good enough" for their assistant. One golden dataset and two experiments later: 93% vs 89% on their real questions, at a quarter of the cost. The argument died in an afternoon - not because anyone won, but because the question finally had a unit.
Try it yourself - this week ◐ 45-60 min total
- Run the full eval suite on BOTH engines -
init_chat_model("claude-haiku-4-5-20251001")and("ollama:llama3.1")- and compare honestly: score, latency, cost (local = $0 but slower and weaker on tool routing?). Write the three-line verdict you would give your team. - Add 5 new dataset rows from this week's real questions, including at least one that failed. The dataset grows forever; start the habit now.
- Adopt the regression gate: before your next prompt or model change, run the suite, change, re-run, and diff the experiments. Once, deliberately.
- Read your rewritten b1 sentence to a colleague and defend it for two minutes. If it survives, you have graduated.
- Next steps when ready: the free Academy courses on LangSmith observability and monitoring production agents - DataDesk is your lab.
Official sources covered
This track teaches from the official docs and the free LangChain Academy curricula (login required for lesson content; certificates stay with the Academy - all free). This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Turning on full tracing for a LangChain/LangGraph app requires...
The framework already emits structured run data; the env vars just route it. @traceable exists for NON-framework code you want in the same tree. Deployment is a separate, optional product.
2 · The regression-gate habit means...
Evals as CI: a small suite of real questions, run on every change, compared against the last experiment. Online evals on live traffic exist too - but the gate is the habit that prevents regressions from shipping.
3 · A 2025 blog post raves about "LangGraph Platform". You know that...
Renamed late 2025, and the old name still owns the search results. Same product family: managed hosting for your graphs, alongside the always-available OSS self-run lane.