Storage engineering

Analytics questions almost always touch a few columns across many rows - "total revenue by month," not "give me everything about order 4471." Columnar storage is built for exactly that shape.

• Compression: a column holds similar values (all dates, all amounts), which compresses far better than mixed rows. Smaller files mean a smaller storage bill.
• Column pruning: a query that needs 3 of 40 columns reads only those 3. On a warehouse priced per byte scanned, that is money saved on every single query.
• Parquet is the default: it has become the standard analytics format across the modern stack for these reasons - most tools read and write it natively.

Columnar is not always right. CSV and JSON win when a human needs to read the file, when you are handing data to a partner who expects a universal format, or when you are writing one record at a time rather than analyzing millions. Row formats are also fine as the landing format straight out of a source - you convert to Parquet once, at the storage step, before analytics. The rule of thumb: raw and human-facing can be row-based; anything queried repeatedly for analytics should be columnar. Your team makes this call at the storage stage of the pipeline.

You may hear engineers say "Iceberg" or "Delta" or "Hudi." These are table formats - a thin layer on top of Parquet files that adds database-like features: transactions, safe concurrent writes, and time-travel to older versions. You do not need the internals. The leader-level takeaway: they let a cheap pile of files in object storage behave more like a real database, which is the idea behind the "lakehouse." When your team proposes one, the right question is "what does it let us do that plain Parquet does not, and is that worth the added complexity right now?"

Object storage - Amazon S3 and its equivalents - is where the modern stack keeps its raw data. It is cheap, effectively infinite, and extraordinarily durable, and it separates storage from compute so you pay for each independently.

• Cheap and durable: designed to almost never lose data, at a fraction of the cost of database storage. This is why "land everything in S3 first" is a common pattern.
• Storage separate from compute: you store data once and point different engines at it, rather than paying a database to hold data it is not currently querying.
• The base of the lakehouse: Parquet files on object storage, with a table format on top, is the storage foundation most modern platforms are built on.

Two words you will hear a lot. A data lake is raw files (often Parquet) sitting cheaply in object storage - flexible, low-cost, less structured. A data warehouse is a managed system optimized for fast, governed queries on modeled data - more structure, higher cost, more speed. From a storage-cost angle, the lake is where you keep the bulk cheaply and the warehouse is where you put the curated data people query all day. How the warehouse models that curated data - star schemas, dimensions, marts - is the depth owned by the learn-data-warehouse course. Here we only care about the storage split and its cost.

None of these need a technical answer from you - they need an honest one from your team.

• Are we storing raw data forever at full price? Raw data is worth keeping, but not on the hot tier once it is old. If nobody has a retention-and-tiering answer, that is the finding.
• Do we tier cold data down automatically? Cloud providers can move data to cheaper tiers on a rule (for example, after 90 days). If it is all still hot, you are overpaying by default.
• Is our format choice costing us on every query? If analytics runs on CSV or JSON instead of Parquet, every query scans more bytes than it needs - and on usage-priced warehouses, that is real money, repeatedly.

The trap with storage is that it is never urgent. No dashboard breaks because logs sat on the hot tier for three years - the bill just quietly grows. That is exactly why it needs a standing rhythm rather than a one-off cleanup. Put storage on a monthly or quarterly review, look at what is growing fastest and whether it is on the right tier and format, and set lifecycle rules so cooling happens automatically instead of waiting for a human to remember. Treat it as a habit and it stays a rounding error. Treat it as a project you do once and it silently becomes one of your largest line items.