Where we are
Session 2 named the decision. Session 3 wrote the contract the data has to honour: units, lineage, ownership, classification. Tonight is the layer under the contract - the platform that runs the pipeline, holds the state, and decides what happens when a node dies at 2am on a Sunday. This session does not add a new Tideline defect of its own. It could not: none of the twenty planted defects live at the platform layer, because that layer is invisible on a rendered screen. What it adds is the argument for why the platform lens still belongs in the instrument - the defects it would catch are the ones that do not exist yet, because right now Tideline has been lucky.
★ What you walk out with tonight. A working map of six architecture principle sets that between them cover reliability, cost, consistency and deployability. A four-part tradeoff record you can attach to any platform decision so the next incident review does not start from zero. And the one distinction, CAP versus PACELC, that most data leads have heard of but cannot actually apply under pressure.
LiveA platform defect is a data defect with a delay3 min▶
Tideline's dashboard has no freshness stamp, that is session 5's D1. But ask why the freshness could ever slip in the first place, and the answer is not on the screen at all: it is a retry policy, a partition tolerance choice, a queue that fills up quietly. The interface layer shows you the symptom. The platform layer is where the actual decision got made, usually months earlier, usually by someone who is no longer in the room when the symptom shows up.
That is the honest case for this session. It will not hand you a 21st defect to add to the count, because you cannot see a platform choice by staring at a rendered dashboard. What you can do is learn to ask the platform question before the data question, because by the time the data question is worth asking, the platform already answered it.
Tideline's ingestion job retries a failed batch exactly once, silently, with no idempotency key. Nobody wrote that down as a decision. It was a default in a library, six years and two data leads ago. It has not caused a visible problem yet. When it does, the on-call engineer will spend the first two hours believing the dashboard is lying, because nobody told them the platform makes this exact kind of lie possible.
Self-studyWhy this layer resists a defect count3 min▶
Sessions 3, 6 and 7 catch things you can point at: a missing unit, a truncated axis, thirteen fill colours. A platform review catches things you can only infer: a single point of failure that has not failed yet, a consistency model nobody chose on purpose, a cost tradeoff that was reasonable at last year's volume and is not reasonable now. None of that renders as a pixel, so none of it can be planted as a countable defect the way D1 through D20 were. The honest framing is that this lens changes what an incident review looks like, not what a screenshot review counts.
Six pillars, six questions for an analytics platform
7 min liveAWS's Well-Architected Framework was written to review cloud workloads generally, and it holds up almost unchanged when the workload is an analytics pipeline rather than a web service. Each pillar translates into one question worth asking about the platform under any dashboard.
LiveThe six pillars, primary-verified5 min▶
AWS's own framework page names six pillars, unchanged since the sustainability pillar was added. Nothing here is analytics-specific in its wording, which is exactly why it transfers cleanly: a data platform is still a system that runs, gets attacked, fails, costs money, and draws power.
| Pillar | What it asks of an analytics platform |
|---|---|
| Operational excellence | Can the team run and monitor the pipeline as code, and learn from every failure without a hero? |
| Security | Is access to raw and derived data controlled at the layer where the breach would actually happen? |
| Reliability | Does the pipeline recover from a component failure without a human noticing at 3am? |
| Performance efficiency | Is compute matched to the actual shape of the workload, not over-provisioned out of habit? |
| Cost optimization | Is spend visible per pipeline, so a runaway job gets caught before the monthly bill does? |
| Sustainability | Is anything being recomputed, stored or refreshed that nobody actually reads? |
Tideline's nightly job reprocesses the full twelve-month retention window every run, because the incremental version was never finished. That is a performance-efficiency and a sustainability question at once, and it has been quietly answered the same way, by default, for two years.
Self-studyThe Data Analytics Lens, and why its wording is secondary4 min▶
AWS also publishes a Data Analytics Lens that specialises the six pillars for analytics workloads specifically. The count of general design principles is high confidence - six - but the exact wording below is read through a secondary AWS-domain source rather than the primary lens document itself, so treat the phrasing as close but not verbatim.
Verification note: the lens document is dated 2023-12-22. The principle count is high confidence; the exact wording of each principle is secondary-sourced. Quote the theme, not the sentence, until you have read the primary PDF yourself.
- Automate data ingestion end to end, so a new source does not mean a new manual step.
- Design ingestion for failures and duplicates - the technical name for this is idempotency, and it is the single principle worth memorising from this list.
- Design scalable, reliable analytics pipelines rather than ones sized for today's volume.
- Describe data with metadata, so a dataset is self-explaining rather than tribal knowledge.
- Manage the data-asset lifecycle and its governance from creation to retirement.
- Enforce data hygiene and model standards before data is allowed to reach a consumer.
Principle 2, idempotency, is the one worth carrying forward. An ingestion step that can safely run twice on the same input is a step that can be retried without a human deciding whether it is safe. Tideline's single silent retry, mentioned above, fails this principle specifically - a second attempt at the same batch is not guaranteed to produce the same result.
Twelve factors, written for an app, aimed at a pipeline
6 min liveThe 12-Factor App was written for web services, but every factor has a clean analytics translation, and two of them are the ones that quietly break the most pipelines: config, and dev/prod parity.
LiveAll twelve, applied to a pipeline rather than a web app5 min▶
The factors are primary-verified at 12factor.net, unchanged since publication. Read as a checklist for a pipeline rather than a web service, most translate directly.
| Factor | Analytics translation |
|---|---|
| I Codebase | One tracked codebase per pipeline, many deploys, not a folder of ad hoc notebooks. |
| II Dependencies | Every library version pinned; nothing assumed to already be on the runner. |
| III Config | Credentials and connection strings come from the environment, never from a script. |
| IV Backing services | The warehouse, queue and object store are attachable resources, swappable without a code change. |
| V Build, release, run | The three stages stay separate; nobody edits code on the box that is running. |
| VI Processes | A transform job keeps no state of its own; the warehouse is the source of truth. |
| VII Port binding | A pipeline exposes its own trigger or API rather than depending on a host-level scheduler hack. |
| VIII Concurrency | Scale by running more workers on more partitions, not by making one worker bigger forever. |
| IX Disposability | A killed job can restart from a clean state with no half-written table left behind. |
| X Dev/prod parity | Dev runs against a real slice of production data and the same warehouse engine, not a sample CSV. |
| XI Logs as event streams | Row counts, timings and failures stream out as structured events, not a log file nobody tails. |
| XII Admin processes | A manual backfill runs as a tracked, versioned job, not a query pasted into a console. |
LiveConfig and dev/prod parity: the two that bite hardest4 min▶
Config breaks analytics pipelines more than any other factor because a data pipeline typically touches more external systems than a web app: a source database, a queue, a warehouse, sometimes a third-party API. Each one hard-codes a connection string somewhere, usually in whichever script a departed contractor wrote first, and each one is a separate outage waiting for someone to change a password.
Dev/prod parity fails almost as often, for a specific analytics reason: production data has skew, nulls, duplicates and volume that a hand-built sample never reproduces. A transform that passes every test against a thousand clean sample rows can still fall over on the one production row where a customer typed an emoji into a name field.
Tideline's transform layer was developed against a curated sample export from eighteen months ago. The retention calculation that produces the truncated bar chart in session 6 was never tested against a month where a customer churned and returned within the same billing cycle, because that customer does not exist in the sample.
CAP tells you about a partition. PACELC asks about the rest of the time
7 min liveCAP is the most misquoted result in distributed systems, usually stated as "pick two of three" when the actual claim is narrower and more useful: during a network partition, a system must choose between availability and consistency. Most of the time, there is no partition. PACELC is the extension that asks the question that actually governs day-to-day analytics: else - when the network is fine - do you optimise for latency or for consistency?
LiveCAP, proven, and usually overstated4 min▶
Eric Brewer stated the conjecture in a PODC keynote in 2000; Gilbert and Lynch proved it formally in 2002. The actual claim: a distributed data store can provide at most two of Consistency, Availability and Partition tolerance at the moment a partition occurs. Because a network partition on modern infrastructure is a real possibility, not a hypothetical, partition tolerance is usually treated as non-negotiable, which leaves the real choice as consistency versus availability during that partition.
The overstatement to correct out loud: CAP says nothing about behaviour when there is no partition, which is most of the time. Treating CAP as a complete description of a system's tradeoffs is the mistake PACELC exists to fix.
LivePACELC adds the question that actually governs a normal Tuesday4 min▶
Daniel Abadi's 2012 paper in IEEE Computer, "Consistency Tradeoffs in Modern Distributed Database System Design", extends CAP with else: if there is a Partition, trade Availability against Consistency (PAC); Else, trade Latency against Consistency (LC). Every analytics platform makes the else choice constantly, whether or not anyone ever wrote it down - it is the difference between a dashboard that answers instantly from a slightly stale replica, and one that waits for the freshest possible read.
The question worth asking in a platform review is never just "is this system CP or AP". It is "what does this system do on a Tuesday with no partition at all", because that is the branch it lives in essentially all the time.
Self-studyThe Reactive Manifesto's four traits4 min▶
The Reactive Manifesto, version 2.0, dated 2014-09-16, names four traits a system should have, in a dependency order: message driven underlies elastic and resilient, which together enable responsive.
- Responsive - the system replies in a timely way, which is the trait users and downstream dashboards actually experience.
- Resilient - the system stays responsive in the face of failure, by isolating components so one crash does not cascade.
- Elastic - the system stays responsive under varying load, scaling resources up and down to match.
- Message driven - components communicate through asynchronous message passing, which is what makes isolation and elasticity possible in the first place.
A batch pipeline is not automatically reactive just because it runs on cloud infrastructure. Tideline's nightly job is a single long-running process with no isolation between its stages - a failure in the transform step takes down the whole run rather than isolating to that stage alone, which is precisely the resilience trait this manifesto names.
Self-studyThe Lakehouse, honestly3 min▶
The Lakehouse is frequently summarised as "the N principles of the lakehouse". It is not. The CIDR 2021 paper by Zaharia, Ghodsi, Xin, Armbrust et al. describes an architecture, not a numbered principle set, and it should never be cited as one. The themes it actually enumerates: open direct-access file formats, ACID transactions on object storage, schema enforcement and governance, decoupled compute and storage, and unified BI plus ML workloads on one copy of the data.
Treat the lakehouse as a pattern to describe, the same way you would describe Kimball versus Inmon: correctly, by its themes, and never as a countable list that somebody else can go verify and find missing.
Write Tideline's tradeoff record
★ 16 min · everyone buildsA tradeoff nobody wrote down is a tradeoff nobody can defend, revisit, or blame correctly when it goes wrong. Pick one real Tideline platform decision - the silent single retry on ingestion is a good default choice if you want a shared example - and fill in all four parts.
1. The choice. State the actual decision in one sentence: what does the platform do, concretely? For the retry example: "A failed ingestion batch is retried exactly once, automatically, with no idempotency key."
2. The pillar it trades against. Name which Well-Architected pillar took the hit for this choice to be cheap or simple. The single retry is a reliability tradeoff made in the name of operational simplicity - nobody had to build a dedup step.
3. The PACELC position. Write both halves. During a partition: does this choice favour availability or consistency? In the normal case (else): does it favour latency or consistency? For the retry example: during a partition it favours availability, the batch gets marked done either way; in the normal case it favours latency over consistency, because nobody pays the cost of verifying the batch actually landed correctly.
4. Who signed it off. Name a person or a role, not "the team". If the honest answer is "nobody decided this, it was a library default", write that down exactly. That sentence is often the most useful line in the whole record.
A tradeoff record does not need to argue the choice was wrong. Its job is to make the choice visible enough that the next person to hit its consequences does not have to reverse-engineer it from a stack trace.
Before session 5
20 min- Pick the pipeline behind the dashboard you use most. Find one config value that is hard-coded rather than injected from the environment. Just find it - do not fix it tonight.
- Ask your platform or data-engineering owner, in one message: "during a network partition, does this pipeline favour availability or consistency, and what does it do the rest of the time?" Write down whether they can answer in one sentence.
- Write one more tradeoff record, for a different platform decision on the same pipeline, using all four parts from tonight's build-along.
- Bring both records. Session 5 asks what promise the platform underneath is actually able to keep, and a tradeoff record you cannot defend is the fastest way to find a promise you cannot keep either.
Three questions
What does the CAP theorem NOT tell you about a distributed data store?
CAP describes behaviour only at the moment of a partition. It is silent on the vastly more common case where the network is fine, which is exactly the gap PACELC's else clause fills.
Which 12-Factor App factor is most directly violated by an ingestion step that produces a different result if the same batch is retried?
A non-idempotent retry is precisely what the Analytics Lens principle warns against: ingestion must be designed so a failure or a duplicate attempt does not corrupt the result. Idempotency is the property a safe retry depends on.
Why does this session add no new Tideline defect to the count of twenty?
The twenty planted defects are all properties of a rendered screen that a program can detect. A platform choice like an un-idempotent retry or an unnamed PACELC position is not visible that way - it becomes visible only after it causes a future data problem.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 4 cheat sheet · pin this
The platform decides what is possible. Session 5 asks what the team actually promised on top of it - a freshness promise with a target and an error budget - and owns the three defects an interface lens can never see on its own: no freshness stamp anywhere, no promise to be late against, and no evidence the data was tested before it landed.