Real data is dirty, and that's the job
People imagine analysis is clever models and pretty charts. In reality, cleaning is roughly 80% of the work - and it's where analyses go wrong most quietly. A dropped-vs-filled decision made without looking, a salary column mixing hourly and yearly pay, a dtype that's secretly text: none of these throw an error. They just hand you a confident, wrong answer. Today we do it deliberately. We look first with missingno, decide with reasons, and only then touch the data. By the end you have a cleaned frame you can trust - and Session 5 explores it.
salary_year column (min/max combined, pay period normalized to annual), sane dtypes across the board, tidy lowercased titles, no duplicate postings, and - just as important - a written record of what you dropped and why. Saved to a new file, raw untouched, ready to explore.
See the missingness first 5 min live
Before you drop a single row or fill a single gap, look at the shape of what's missing. The rule this course keeps hammering: profile before you fix. missingno turns a wall of NaNs into three pictures that tell you what to do.
LiveThe three missingno plots3 min▶
missingno gives you three complementary views. You run all three in about ten seconds and they answer different questions:
msno.matrix(df)- the row-by-row view above. White streaks show where gaps fall. If missingness clusters in blocks, something structural is going on; if it's random static, that's a different story.msno.bar(df)- one bar per column showing the count of present values. The fastest way to rank columns from cleanest to worst.msno.heatmap(df)- the nullity correlation: which columns tend to go missing together. A value near +1 means "when this one is missing, that one usually is too."
Run these on the LinkedIn postings frame and the ~70% missing salary is visible instantly - the bar for min_salary and max_salary is a stub next to the full-height title bar. The heatmap then shows min_salary and max_salary go missing together at nearly +1: when a poster omits pay, they omit all of it. That single fact shapes every salary decision you make today.
Self-studyQuantify what you saw2 min read▶
Pictures point you at the problem; a number lets you decide. Pair the plots with one line:
fillna or dropna before seeing the pattern. A column that's 2% missing at random and a column that's 70% missing structurally need opposite treatments. The plot tells you which one you're holding.
Handling missing values 5 min live
There is no default answer to a missing value. There's a decision, and it depends on how much is missing and why. Here's the framework analysts actually use.
LiveThe four tools and when to reach for each3 min▶
isna()/notna()- boolean masks.df["col"].isna().sum()counts gaps;df[df["col"].notna()]keeps only the rows that have a value (the "salaried subset" move).dropna()- remove gaps.how="any"vshow="all",thresh=Nto require N non-nulls, andsubset=[...]to only judge specific columns. Dropping on the whole frame is usually too aggressive - scope it withsubset.fillna()- fill gaps with a value: a median, a mode, or a sentinel like"Unknown". Powerful and dangerous.- Mean vs median - salary is right-skewed (a few huge numbers pull the mean up). For skewed data, median is the honest center. Reach for mean only on roughly symmetric data.
df.fillna(0) across a whole frame is a classic disaster - it silently turns "we don't know this salary" into "this job pays $0," and every average you compute afterward is wrong. Fill a column at a time, with a reason.
Self-studyOur salary strategy, decided out loud2 min read▶
Applying the framework to the LinkedIn salary columns:
Salary is ~70% missing and it's the most interesting variable in the dataset - so it lands in the bottom-right box. We do not impute it: filling 70% of rows with one median would fabricate the majority of the data and bias every finding toward that single number. Instead we keep every row, build a clean salary_year where it exists, and analyze the ~30% salaried subset for pay questions - while saying clearly in the report that pay figures describe postings that disclosed salary. Honest, and still useful.
Fixing types, text, and dupes 4 min live
With missingness handled, three mechanical clean-ups remain: get the dtypes right, tidy the free-text fields, and drop duplicate postings. These are the moves that turn a frame from "loads without error" into "safe to compute on."
LiveCoerce the dtypes1.5 min▶
A number stored as text won't add up; a date stored as text won't sort. Three workhorses fix this:
astype()- direct conversion when you know the data is clean:df["views"].astype("int64").pd.to_numeric(s, errors="coerce")- convert to number, and turn anything unparseable intoNaNinstead of crashing. Essential for messy salary text.pd.to_datetime(s)- parse thepostedcolumn into real timestamps so you can sort and resample later.
"competitive" in a salary column throws and stops your whole notebook. With it, that value becomes NaN, the conversion finishes, and you decide what to do with the gaps deliberately.
LiveTame the free-text with .str1.5 min▶
Job titles are gloriously inconsistent: "Data Analyst", " data analyst ", "DATA ANALYST II". The .str accessor cleans a whole column at once:
.str.strip()- kill leading/trailing whitespace..str.lower()- normalize case so grouping works..str.contains("analyst")- boolean filter for title bucketing..str.replace(...)and.str.extract(r"...")- swap noise out, pull structured bits (like a level number) out with a regex.
LiveParse salary to a yearly number1 min▶
The core project transform. Salary lives across min_salary, max_salary, and pay_period (HOURLY / MONTHLY / YEARLY). We coerce the numbers, take the midpoint, then scale by period to a single annual figure:
Rows with no salary stay NaN - exactly what we want. We're building the column, not filling the gaps.
Self-studyFind and drop duplicates2 min read▶
Scrapes double-post. df.duplicated().sum() counts exact duplicate rows; df.duplicated(subset=["job_id"]).sum() counts repeats of a business key. df.drop_duplicates(subset=["job_id"]) keeps the first of each. Always check the shape before and after so you know how many you removed - a silent 10% drop is worth noticing.
See the mess with missingno ★ 8 min · everyone profiles
Reload the postings frame from Session 1 into jobs. We profile before we touch anything.
Run all three missingno plots and read them out loud together.
Write down the three worst columns by missing %. Confirm with jobs.isna().mean().sort_values(ascending=False).head(5).
Your three worst columns will almost certainly be the salary trio. The heatmap makes the story concrete: min_salary and max_salary light up together near +1. You now know that "handle missing salary" is one decision, not two - and Demo 2 acts on it.
Build the clean salary column ★ 9 min · everyone builds
Coerce both salary bounds to numeric so stray text becomes NaN instead of crashing.
Combine min and max into a midpoint, then normalize the pay period to an annual figure.
Decide the missing strategy out loud: keep every row, do not impute the ~70% gap, plan to analyze the salaried subset with jobs[jobs["salary_year"].notna()] in Session 5.
.describe(). If the max salary reads $5,000,000 you probably have an hourly rate that wasn't scaled, or a typo row. Catch it now, before it becomes a headline.
Tidy titles, dedupe, save clean ★ 5 min · follow along
Normalize the title text so grouping works next week.
Save to a new file - never overwrite the raw download.
postings.csv is your one source of truth; keep it read-only in your head. Every save this course does uses a new name.
This week ◐ 40 min total
- Kaggle Pandas Lesson 5 - Data Types & Missing Values. Short, and the exercises drill exactly today's moves.
- Fully clean your frame - build
salary_year, tidy titles, drop duplicates onjob_id, and save it aspostings_clean.csv. Session 5 loads this file, so this is not optional. - Write your cleaning log - one markdown cell listing what you dropped, what you left missing, and why. This becomes a paragraph in your Session 8 report.
- Read McKinney ch7 - Data Cleaning and Preparation. It's the canonical reference for everything today.
- Optional: run a sweetviz report comparing the raw frame to your cleaned frame - a satisfying before/after that shows what your work changed.
Three questions before you go 🎯 ◐ 90 seconds
1 · Why run missingno before you start dropping or filling?
A 2%-random gap and a 70%-structural gap need opposite treatments. The plots tell you which one you're holding - profile before you fix.
2 · What does pd.to_numeric(s, errors="coerce") do?
It's your seatbelt: one stray "competitive" in a salary column becomes NaN and the conversion finishes, instead of throwing and stopping the notebook.
3 · Why NOT impute the ~70% missing salary with the mean?
Filling 70% of rows with one number invents the majority of your data. Keep the rows, analyze the ~30% that disclosed pay, and state that clearly in the report.
Official sources covered
This session teaches the working content of the pandas missing-data and text guides, the free Kaggle Pandas micro-course, Wes McKinney's open-access Python for Data Analysis (3e), and the missingno and ydata-profiling docs. Certificates and graded exercises stay on those platforms - links provided.