API Pagination Extractor
Paginated API pulls are hand-coded each time - one generic extractor speaks offset, page, cursor, and link-follow, with 429 retry/backoff built in.
Each one started with something breaking in a real pipeline: a metric that disagreed with itself, a parser that read the same bytes two ways, a model that looked fine until it shipped. Source code and a runnable notebook on every one. Browse by what you are trying to do.
It lives somewhere else and it has to land here, on a schedule, without losing rows.
Paginated API pulls are hand-coded each time - one generic extractor speaks offset, page, cursor, and link-follow, with 429 retry/backoff built in.
Every Monday, someone on the team downloads CSVs from five different SaaS tools. This replaces that.
Backfills are error-prone manual scripts - chunk the date range, persist every status, retry the flaky, quarantine the broken, and resume after a crash without repeating a success.
Loading data into our database is a manual nightmare - copy-paste into SQL, fight with column types, pray nothing breaks.
Our data lake is a pile of CSVs - convert flat files into hive-partitioned, compressed parquet and prove the query speedup with a stopwatch.
Full reloads take hours every night - load only what changed with watermark extraction, key upsert, and soft-delete tombstones.
We can't compute rolling metrics live - event-time windows with watermarks in ~100 lines, with the same semantics Flink and Spark charge a cluster for.
The bytes arrived. What they MEAN is a decision, and every layer decides differently.
headers["X-Request-Id"] is the wrong shape, and not for style reasons. RFC 9110 makes field names case-insensitive; RFC 9113 makes them lowercase on an HTTP/2 wire or the message is…
A file has no lines in it. It has bytes. Lines are produced by a splitter, and every runtime ships a different one. wc -l counts LF bytes.
A numeric string does not contain a number. It contains characters. A reader assigns the number - and a reader is four things multiplied together: a symbol table, a grouping rule, a…
Round three equal rows to one decimal place and the column reads 99.9%. Every row is correct.
A string does not contain a boolean. A reader assigns one - and a reader is three separate decisions: an accept table (which spellings mean true), a normalisation (case, whitespace…
A rounding function returns a number. It cannot return the fact that the rows no longer add up.
Reading a fixed-width file is one line: pd.read_fwf(path, colspecs=...). That line decodes the record to a string and then slices it by character position.
Turn a messy CSV export into a tidy, analysis-ready file - with an auditable report of every change.
Duplicate records everywhere - normalize the variants, cluster by match keys, and merge each cluster into a golden record with field-level survivorship and full provenance.
Find byte-identical files hiding under different names - efficiently - and reclaim the space. So a shared drive full of duplicates stops costing you storage.
csv.Sniffer().sniff() returns a Dialect or raises csv.Error. It has no third answer.
Every API response is nested. Every warehouse table is flat.
Normalising a local timestamp to UTC is one line: ts.replace(tzinfo=ZoneInfo(zone)).astimezone(utc).
Everything imported as string. Typing the columns is easy - typing them without silently corrupting data is the actual job.
ORDER BY name reads like a total order on a column. It is not. It is a collation applied to bytes, and a collation is three decisions the SQL never states: which sequence the characters are…
Turning a DataFrame into a markdown table is one line. That line is also a lossy encoder, and the GitHub Flavored Markdown table extension has no error state to say so.
parse(text) -> timedelta cannot answer the question people actually have about a duration string, which is not "how long is this" but "does the parser at the other end of this pipe agree with me".
sanitise(name) -> str is the wrong shape, and not because of which characters are on the deny-list.
slugify("Node.js at scale") and slugify("NodeJS at scale") both return nodejs-at-scale. Neither call is wrong. Neither raises.
"Truncate to 20" does not name an operation. A truncator is three decisions - a unit of length, a boundary rule, and a policy for the piece it removes - and the integer 20 carries none of…
Something changed upstream. Find out before a dashboard does.
Bad values slip into columns and nobody notices until a dashboard breaks - scan every column and get triaged, explainable alerts before they do damage.
Producers break schemas without warning - put a YAML contract in CI so breaking changes block the merge and violating data never lands.
Stop guessing how bad your data is - get a 0-100 score, a letter grade, and a ranked list of what to fix first.
Between yesterday's snapshot and today's, what actually changed? Nobody can answer without eyeballing two files - this diffs the same dataset over time on a key column and tells you exactly…
No tests on our models - this profiles a table and auto-generates a paste-ready dbt schema.yml.
Our data-quality rules live in a wiki nobody enforces - by the time a bad batch is caught, it's already in a report.
Setting up data validation is tedious - so most teams skip it until something breaks in prod.
A FastAPI microservice that validates JSON payloads against a schema and returns every error at once - so broken API contracts surface before they corrupt data.
df.isna().sum() answers the wrong question.
Database migrations break things silently - catch schema changes before they hit production.
A producer changes a column type and three downstream jobs break silently - we find out from the dashboard, not before.
We copied a table from the source system into the warehouse and can't prove it matched - row counts drift and values silently differ.
Who owns this column, who may read it, and can you show an auditor?
"Active user" means five different things to five teams, so every metric argument starts from zero - this gives each business term one owned, versioned definition and flags the governance gaps that let ambiguity creep back in.
Table-level lineage tells you model B reads model A. It can't tell you that renaming orders.amount will silently break total_revenue three hops away.
Run a document against a policy ruleset before the auditor does.
We process personal data for purposes the user never agreed to - and can't prove otherwise when asked.
"Who can see the PII table?" takes a week of digging, so over-privileged and stale access piles up unnoticed - ingest your access grants and get triaged, explainable findings for a least-privilege review in seconds.
Point it at a table, get a documented data dictionary - types, PII flags, descriptions.
Paste SQL, see what breaks when you change a table.
We keep everything forever - storage bloats and we are one audit away from a finding for holding personal data past its purpose.
"We don't have database diagrams" - paste your CREATE TABLE statements and get an instant ER diagram.
Nobody knows what tables and columns exist, what they mean, or who owns them - so every analysis starts by pinging three people on Slack.
"No model documentation." - a model with no card is a black box: nobody knows what it's for, what it trained on, or where it breaks.
"We accidentally share sensitive data in staging" - scan CSV data for PII and mask it before it leaves production.
"We masked the PII, so the extract is safe to share." Two things are usually wrong with that.
Legal templates cost money and are still generic - answer a short questionnaire and get a GDPR/CCPA-aware policy draft plus a checklist of the compliance work the document can't do for you.
A subject access request arrives with one email address. Finding the rows is easy.
The pipeline fails at 3am. The question is whether anyone finds out before the meeting.
A dead-man's-switch for scheduled jobs - heartbeat in, email out when a job goes silent. So you find out batch jobs failed before your users do.
Dashboards show stale data and nobody notices - this tool catches it first.
"We discover pipeline failures from angry users" - an agent that catches failures, slow runs, row-count drops, and silent staleness before anyone downstream notices.
One screen of database visibility - cache hits, latency, locks, replication lag - scored red/amber/green so problems surface before users notice.
A pre-flight check for environment variables - declare what your app needs, validate it before you ship, and fail the deploy if anything's missing or malformed.
Incident playbooks exist but nobody follows them - this agent executes the runbook step-by-step and escalates when a remediation step fails.
The dashboard said the index was 89% fresh, and 27% of the answers it served quoted a fact the source had already changed.
Every hop of the RAG chain has its own latency dashboard, and adding them up said we were 792 ms over the SLO.
The dashboard said the upgrade cut refusals by two thirds. The model was refusing 39% more requests than before, twice as many of them legitimate - it had started saying "I'm not able to"…
Stop grepping logs by hand. Parse mixed-format logs into structured records, see the severity breakdown and top errors, and fire alerts when errors spike.
We notice metric drops late - this watches a metric and fires the day it moves, with three independent detectors.
Models degrade silently in production - this watches a live window against the training reference via PSI and alerts before accuracy quietly rots.
A pipeline that lands late, runs slow, goes stale, or arrives thin usually breaks a consumer before anyone on the data team notices.
A new dataset landed. Two hours of profiling before you can say anything about it.
Drop in a CSV, get the whole first-look in seconds - shape, missingness, dtypes, distributions, correlations, and the quality problems worth a human's attention.
"Which features relate?" - a correlation heatmap plus VIF to catch the redundant, collinear features that quietly wreck a model.
Find your natural customer groups with KMeans - auto-picks the number of segments, names each one, and shows how distinct they really are.
"Which rows are weird, and why?" - Isolation Forest finds the anomalies; per-feature z-scores explain each one.
"Too many features to see" - project high-dimensional data down to 2D and actually look at its structure.
Turn a raw survey export into a type-aware summary in seconds - NPS, Likert scores, choice breakdowns, and open-text sentiment.
"What are these documents about?" - discover the themes in a text pile automatically, no labels required.
We have 10K reviews and no insights - this turns raw feedback into a sentiment split, ranked complaints, ranked praises, and the one thing to fix first.
One metric, three dashboards, three answers. Define it once and serve it.
"d = 0.5, a medium effect" is read as a fact about how big something is - so this holds d exactly fixed, changes nothing but the shape of the distribution, and asks the other effect-size metrics whether they agree.
"Retention analysis takes our analyst 2 days" - turn a raw event log into a cohort retention heatmap in seconds.
Dashboards built from vague asks - this reads the data's shape and recommends the right charts, with the reasoning.
See exactly where users drop off - biggest leak called out, step-by-step conversion, and a segment comparison.
Adding a sparkline column to a table is one line of pandas. That line autoscales every row to its own min and max, which is what makes a row that grew 3.5% and a row that grew 3.5% off a…
Execs ask for the same metrics every Monday - this turns a metric time series into the answer automatically.
"Why did revenue move?" - this splits a KPI's change across its drivers exactly, with no residual.
Which metrics exist, and who owns them? A registry with tiers, dependencies, and automated governance that flags the metrics nobody is minding.
Every dashboard computes revenue differently - a metrics layer makes one governed definition the single source of truth.
Non-analysts can't query the warehouse - NL→SQL lets them ask in English, behind guardrails that keep it read-only and in-schema.
"Conversion is up 5% this week!" - but is it a real shift or just noise? Metric Diff pairs a period-over-period delta with a significance test so you only act on changes that are real.
Nobody reads the pivot table.
Get sales forecasts out of someone's spreadsheet - a reproducible forecast with a confidence band and an honest accuracy check.
Weekly reports sent by hand - this defines a report once on a cron and previews every scheduled send (a dry run that never emails).
Analysts get pinged for every number - a self-serve explorer lets anyone pivot, aggregate, and filter without SQL or a ticket.
Slow queries, no idea why? This lints SQL for the anti-patterns that quietly force full table scans.
The line went up. Whether that means anything is a separate question.
The test said +10%, the launch delivered +2%, and the chart everyone blames it on cannot tell you whether the effect faded or whether you measured different users each week.
Shapiro-Wilk says your data is not normal, so you abandon the t-test. Across 63 measured cells that gate fires 29 times on a t-test that was already correct, and the substitute it sends you…
"p = 0.049" gets written up as a finding, but nobody has ever watched the same study run twice - so this replays one true effect forty thousand times and reports what the p-value actually does.
The test won overall, then somebody sliced it by country, device, plan and tenure and found the segment it hurt - and on a launch that helped every single user equally, that slicing invents a harmed segment 40% of the time, with a number the…
The experiment dashboard says the split came out 49.3 / 50.7. Somebody decides that is close enough to even and reads the result.
The A/B test said +26%. It shipped to everybody and nothing moved. Nobody made a mistake: a split test measures the difference between the arms, and the decision needed the total.
Somebody opens the experiment dashboard every morning and ships the variant the day it goes green.
Stop eyeballing significance. Enter your A/B numbers, get a proper two-proportion z-test - p-value, confidence interval, and a clear verdict - plus a sample-size planner.
Fitting ten distributions and ranking them by AIC is four lines of scipy. The ranking will always produce a winner - including when the data came from something that is not on the list.
The ANOVA came back significant and now someone has to say which groups differ - and the five usual ways of finding out name different pairs on the same data.
We measured the same stores before and after, then tested them as two separate groups - and a change the data clearly shows came out as "no significant difference".
"Compare groups in survey data" - contingency table, chi-square test, effect size, and the exact cells driving the difference.
"We ran a t-test" does not name a procedure. On 50 users against 10 with unequal spread, the textbook t-test calls a coin-flip significant 29% of the time while claiming 5%.
0 of 20 converted in control, 4 of 20 in the variant: two standard tests say it moved, two say it did not - and which one you ran decides the launch.
Two defensible tests, one dataset, both significant, opposite conclusions - because "just use Mann-Whitney" swaps the question, not just the method.
The test found no significant difference and everyone read it as "the two are the same" - but a small study says that about a difference that matters two times in three.
"The error bars overlap, so it is not significant" throws away up to half of the real results on the chart - and on paired data it fails the other way.
We could not randomise this one, so we compared the change in the group that got the policy against the change in the group that did not, and checked that the lines looked parallel beforehand.
"How many users do we need?" gets asked at the start of every experiment and answered by vibes.
"Which statistical test do I use?" - describe the data, get the right test (parametric or not), and run it.
"We checked the variances first." Run Levene, then pick Student's or Welch's. Across 14 measured designs that two-stage move never once beat simply using Welch, and on four of them it…
One market got the intervention and there was never going to be a control group, so a weighted average of the other markets is fitted to stand in for one - and the pre-period plot everybody publishes as proof was chosen to look good, which is why…
One analyst reported "significant, p = 0.047", the other ran a Bayesian test on the same numbers and reported "strong evidence of no effect" - and nobody could say which of them was wrong.
The test needs six weeks, and somebody says CUPED will halve it. The first claim is arithmetic.
Fit something, then find out honestly whether it is better than doing nothing.
"The forecast was excellent in-sample" - a single train/test split reports one draw of an origin, and the model that won it was chosen by it.
"The point forecast is useless" - so the chart gets a 95% band, and the band's claim is never checked, because the check almost nobody runs has almost no power.
"We got 0.87 AUC." Compared to what?
Scoring new data shouldn't be a manual copy-paste ritual.
Spot at-risk customers before they leave - ranked risk scores, churn drivers, and an honest model-quality check.
"Our fraud model ignores the minority class." - a classifier that predicts "not fraud" every time scores 98% accuracy and catches zero fraud.
A model outputs a probability. A decision needs a cutoff. Almost everyone ships 0.5 - a number nobody chose, which quietly assumes a false positive and a false negative hurt exactly the…
The model scores 0.99 in cross-validation and falls apart in production. That gap is usually leakage.
Every new ML project you rewrite the same impute / scale / one-hot boilerplate, get the column lists slightly wrong, and leak test statistics into training.
"Stakeholders don't trust the model." One importance number is easy to doubt - three that agree is evidence.
"Forecast any metric" - Holt-Winters (trend + seasonality) in pure numpy, with honest backtest accuracy.
We grid-search by hand - nudge n_estimators, re-run a cell, keep whatever looked better.
"Which model is actually in production?" should have a one-word answer.
Binning a feature and printing its Information Value takes twenty lines. Knowing whether the number you just printed means anything is the actual job - and the conventional 0.02 / 0.1 / 0.3…
A "0.9" from your model should mean it happens 90% of the time. Often it doesn't.
"We have no 'you may also like'" - item-item collaborative filtering, with an explanation for every recommendation.
"Auto-tag our support tickets" - train a TF-IDF + logistic-regression classifier and route new text by category.
Model selection is ad-hoc - train a few classifiers, eyeball one accuracy number, ship whichever looked best on one lucky split.
The answer is in a 200-page PDF nobody will read.
Bad chunks = bad answers. Prove which chunking strategy retrieves best - on evidence, not vibes.
Turn scattered competitor notes into one comparison matrix + strategic takeaways.
Turn a 3-day legal read into a 30-second clause + risk triage.
Stop answering the same question daily - turn your docs into a ready FAQ.
Your LLM pipeline returns JSON. Nothing checks it. json.loads is not a contract.
Turn scattered docs into a searchable, cited knowledge base - before the expert leaves.
"Meetings produce no actionable output" - paste the transcript, get TL;DR + decisions + action items + open questions.
"Pull the names, orgs, money, and dates out of this text" - structured entities, no heavy model download.
A duplicate in a vector index does not cost you storage. It costs you the answer.
"Nobody reads the 200-page policy manual" - upload it, ask questions, get cited answers.
Rank a stack of resumes by skills fit - a triage aid, not a hiring decision.
Search by meaning, not just exact words - what Ctrl+F can't do.
Stop your pipeline from crashing when the LLM returns almost-JSON.
It sounds right. Sounding right is not a measurement.
We labelled 240 real conversations once, called it our eval set, and a year later it still said 90% while customers were getting 84%.
Our prompt CI said eight cases broke on a change that fixed a typo, so we stopped reading it - and then a real regression shipped.
"We don't know if our agents are actually good." Run a fixed eval suite against two agent versions and get pass rate, quality, latency, and per-category regressions - numbers instead of vibes.
We upgraded the model, the eval score didn't move, so we shipped it - and the refunds team said it had stopped working.
Stop shipping the same static few-shot examples for every input - pick the ones that actually resemble the query.
Flag the claims your RAG answer makes that the retrieved context never supported.
No more surprise API bills - log every call, roll spend up, watch the budget.
Model outputs unsafe / off-topic text? Put cheap deterministic checks around it.
Stop paying frontier-model prices for "what's the capital of France?"
Prompts are the only part of an LLM system that ships with no review gate.
Prompts get pasted into random .py files and Slack threads - no history, no diff, no rollback. This gives them the same discipline as code and models.
You changed the chunking or swapped the embedding model. Did RAG get better or worse? "It feels better" is not an answer - this harness gives you numbers.
Stop paying for repeated similar queries - reuse an answer when the meaning matches.
Our agent eval said the refund bot got the answer right, and it had sent the refund to the wrong order.
We told the model to cite every sentence, citation coverage went from 74% to 98%, and the passages it cited supported 8 points fewer of its claims.
Context recall said 83% of the facts the answers needed were in the window, and 34% of the queries lost a fact that was there - more than were lost to retrieval, and the dashboard counts every one of them as a success.
Price an LLM feature before you build it - not after the invoice lands.
The task is small, correct, and done by hand every single week.
We don't test for accessibility until the end - paste HTML and get WCAG-referenced findings, severity levels, and a 0-100 score in seconds.
Writing DAG boilerplate is repetitive - describe the pipeline in ~30 lines of YAML and get a validated, ready-to-deploy Airflow DAG.
Scan any repo into a structured profile and generate a README - by template offline, or with Claude for richer prose. Wire it into GitHub Actions to auto-open a docs PR on every push.
"PRs sit unreviewed for days" - an agent that gives every PR an instant first pass so reviewers focus on logic, not nitpicks.
"Tickets go to the wrong team 40% of the time" - an agent that routes tickets by weighted keyword signal, and escalates to human triage instead of guessing when confidence is low.
One place for the team's standup. Everyone submits yesterday / today / blockers; the bot rolls it into a single digest with blockers surfaced first. No more scrolling Slack threads.
Writing dbt staging models from scratch is repetitive boilerplate - paste your SQL DDL and get production-ready dbt artifacts instantly.
"I spend 2 hours daily on email" - an agent that drafts replies, checks its own work against the original ask, and revises before handing it back.
Write the weekly report in Markdown, get a branded, print-ready PDF - cover header, styled tables, page numbers. No more hand-formatting in Word.
"New hires miss steps, ramp-up takes 3 months" - a multi-step agent that resolves onboarding step dependencies and flags what's overdue or blocked before it cascades.
0 0 13 5 is the line somebody writes for Friday the 13th. It fires on 61 days a year instead of 3, because when both day fields are restricted cron takes their union.
Monthly reports take 2 days of copy-paste - a team of role-specialized agents turns a table of metrics into a ready-to-send report in seconds.
backoff(attempt) returns a number of seconds. That is the whole interface, and it is the wrong shape for the problem.
People ask the same questions in #general every day - this agent answers them from a knowledge base via RAG, and escalates anything it isn't confident about to a human instead of guessing.
A FastAPI microservice that turns unreadable SQL into a clean house style - and lints it for danger (DELETE with no WHERE) and style (SELECT , implicit joins).
The number is on the screen. Someone still has to decide, and later justify it.
A decision log without a prediction attached is a diary: it records what was chosen and can never say whether choosing it was any good.
Every experiment review has a checklist of counter-metrics. Somebody confirms none of them moved, and the change ships.
Revenue lands a month after anyone could have done something about it, so somebody builds a table of candidate upstream metrics, correlates each against revenue at a few lags, sorts by the biggest number and puts the top three on a dashboard.
A proxy metric gets picked because it correlates with the outcome somebody cares about. It becomes a target. It goes up.
Somebody asks what next quarter's number should be, and six people in the room have six defensible answers.
Finance forwards the warehouse bill and asks which team spent it. The question sounds like accounting and it is not.
Somebody asks what to do first, so you score the backlog - RICE, or WSJF, or cost-of-delay-over-duration - and the meeting ends with a ranked list. But a list is not what you pay.
A catalog is only useful if somebody arriving with a problem can find the thing that solves it.
Somebody asks which option to take, so you build the spreadsheet: probability times payoff, one row per option, pick the biggest.
We argue about priorities without data - RICE turns "I feel strongly about X" into a rankable, defensible number.
OKRs get set and forgotten - this tracks progress against pace and flags which key results are behind while there's still time to act.
Roadmaps are PowerPoints that go stale - generate the roadmap from data instead: a Gantt-style timeline, grouped by lane, colored by status, with a "today" line.
Retros are unstructured and repetitive - feed in the sprint numbers and get grounded observations and concrete action items in your team's format.
We build before we validate - this scores an idea across five dimensions, builds a Lean Canvas, and hands you the cheapest experiment to test your riskiest assumption first.
A pre-mortem is cheap and it works: assume the project already failed, then write down why. What happens next is the problem.
Writing good user stories takes practice - paste raw feature ideas and get INVEST-scored stories with Given/When/Then acceptance criteria.
We told the AI agent to hand anything under 90% confidence to a person. It sent 3,655 tickets a day to a team that can review 700, and the day cost more than letting the agent decide…
The control plane the tools plug into: identity, permissions, audit, connectors, orchestration.