Where analysis starts answering questions
Sessions 1 to 4 got you a clean, trustworthy frame. That was setup. This is the payoff. Almost every real question you'll ever ask of data comes down to two operations: split the data into groups and compute something per group, and join separate tables into one. That's it. Median salary by experience level? Group. Top skills across postings? Group and count. Do bigger companies pay more? Join, then group. Master these and you can interrogate any dataset - today you interrogate the job market.
salary_year by role and experience level, a ranked list of the most-demanded skills across all postings, and a pivot grid of pay by experience level crossed with remote status. Three answers you could put on a slide - and the summary frames you'll chart in Session 6.
groupby: split-apply-combine 5 min live
The single most useful mental model in data analysis. When you write df.groupby("col")["val"].mean(), pandas does three things - and once you see them, groupby stops being magic.
Livegroupby, agg, and sorting the answer3 min▶
The basic form is df.groupby("key")["value"].func(). Where it gets powerful:
- Multiple functions at once with
.agg():df.groupby("level")["salary_year"].agg(["median", "count"])gives you the typical pay and how many postings back that number - always report both. - Group by several keys:
groupby(["level", "remote"])nests the split. - Sort the result with
.sort_values("median", ascending=False)or grab the extremes with.nlargest(10). A groupby result is just another frame - you keep working on it.
On the LinkedIn data, jobs.groupby("experience_level")["salary_year"].median() answers a question people actually care about in one line: does the pay ladder climb the way you'd expect from entry to director? The count alongside it keeps you honest - a "median" backed by 12 postings is a rumor, not a finding.
Self-studyWhy count belongs next to every average2 min read▶
An aggregate hides its sample size. A group of 5,000 postings and a group of 8 both collapse to a single median, and on a chart they look equally solid. Always carry the count: .agg(["median", "count"]). In your Session 8 report, a footnote of group sizes is the difference between "credible" and "made up." Small groups are where confident wrong conclusions live.
Pivot tables and reshaping 4 min live
groupby gives you a list of answers; a pivot table lays them out as a readable 2D grid. Same computation, better shape for the eye - and the reverse move, melt, turns wide back to long when a tool needs it.
Livepivot_table, crosstab, and when to use which3 min▶
pivot_table(index=, columns=, values=, aggfunc=)- the workhorse.indexbecomes the rows,columnsthe columns, and each cell isaggfuncapplied tovaluesfor that combination. It's groupby-on-two-keys, laid out as a grid.pd.crosstab(a, b)- a shortcut for counting combinations (a frequency table). Reach for it when you just want "how many postings are remote × senior."meltvspivot-meltgoes wide -> long (many columns collapse into key/value rows),pivotgoes long -> wide. Charting libraries usually want long; humans usually want wide.
pivot errors if any index/column pair repeats. pivot_table takes an aggfunc so duplicates get summarized instead of crashing - on messy real data, reach for pivot_table by default.
Joining the tables 5 min live
The dataset isn't one table - it's several that share key columns: postings, companies, skills, salaries. The full picture only appears when you stitch them together on those keys. That's what merge does.
LiveThe four join types and the keys3 min▶
pd.merge(left, right, on="key", how=...) combines two frames on a shared column. The how decides which rows survive:
- inner - only rows with a match on both sides. The safe default when you need complete pairs.
- left - keep all left rows, attach right where it matches,
NaNwhere it doesn't. This is the everyday analyst join: keep every posting, enrich it. - right / outer - keep all right rows / keep everything from both.
- keys -
on="company_id"when both sides name it the same;left_on=/right_on=when they differ. Watch one-to-many: one company matched to many postings multiplies rows - expected here, but always check the shape after.
pd.concat([a, b]) is different - it stacks frames with the same columns (more rows) rather than matching on a key. Use it to append, not to enrich.
Join company_size onto postings with a left join on company_id, then group by size and take the median salary_year - and you've answered "do bigger companies pay more?" with your own data. The left join guarantees you don't quietly drop postings whose company is missing from the lookup table; they just get a NaN size you can filter later.
Highest-paying roles by group ★ 8 min · everyone groups
Load your cleaned frame from Session 4 and keep only rows that disclosed pay.
Group by experience level and a title bucket, take median pay and count, then sort.
summary[summary["count"] >= 50]) before you quote a number.
The most-demanded skills ★ 7 min · everyone merges
The skills live in their own table keyed to job_id. Merge it onto postings, then count.
If a posting stores several skills in one cell instead, split and explode first: df.assign(skill=df["skills"].str.split(",")).explode("skill"), then value_counts.
This is the finding people screenshot: the top 15 skills across 124k postings, ranked. SQL and Python usually fight for the top; the surprises further down (a cloud platform, a BI tool) are what make the post worth reading. You built it with one merge and one value_counts.
Pay pivot: experience × remote ★ 6 min · follow along
Lay median pay out as a grid so the story reads at a glance.
Read the story out of the grid: does remote pay more at every level, or only for senior roles? Note what you see.
Save the summary frames - Session 6 turns these exact tables into charts.
This week ◐ 40 min total
- Kaggle Pandas Lessons 4 & 6 - Grouping & Sorting, then Renaming & Combining. These are exactly groupby and merge, with good reps.
- Answer three project questions on your own joined data, in a notebook: (1) the top-paying roles, (2) the most-demanded skills, (3) the remote-vs-onsite pay gap. Save each as a summary frame.
- Write one sentence per finding - the plain-English takeaway. These sentences become your Session 8 report bullets.
- Read McKinney ch8 (Join, Combine, Reshape) and ch10 (Aggregation & Group Operations) - the canonical references for today.
- Optional: try grouping by a third key (location or company size) and see which cut of the data surprises you most.
Three questions before you go 🎯 ◐ 90 seconds
1 · What does "split-apply-combine" actually mean?
That's the engine under every groupby: split by the key, apply the aggregate per group, combine the per-group answers into one summary frame.
2 · Which merge keeps every row from the left table and matches the right where it can?
how="left" keeps all left rows and fills NaN where the right side has no match - the everyday "keep every posting, enrich it" join.
3 · How does pivot_table differ from a plain groupby?
Same computation, different shape: pivot_table spreads two grouping keys across rows and columns for the eye; groupby hands back a result indexed by the group.
Official sources covered
This session teaches the working content of the pandas groupby, merge/join/concat, and pivot guides, the free Kaggle Pandas micro-course, and Wes McKinney's open-access Python for Data Analysis (3e). Certificates and graded exercises stay on those platforms - links provided.