Pandas - lesson 5 | HackIndia Skills

Pandas - lesson 5 | HackIndia Skills | HackIndia

Common mistake

Avoid chained assignment

Older tutorials often show chained assignment. In pandas 3.0, Copy-on-Write means that pattern no longer works as people expect.

Use .loc[row_condition, column_name] = value when you update part of a DataFrame.

# Avoid this pattern
# df[df["fee"] < 0]["fee"] = pd.NA

# Use this instead
df.loc[df["fee"] < 0, "fee"] = pd.NA

Example

Clean a fees file

This example cleans common college fest registration data. It finds missing names, fixes bad fee values, standardises city text, and removes rows with no student_id.

Notice the order. Clean symbols first, convert to number, then handle missing and bad values.

import pandas as pd
import numpy as np

raw = pd.DataFrame({
    "student_id": [101, 102, None, 104, 105],
    "name": ["Asha", "", "Ravi", None, "Meena"],
    "city": [" Pune ", "MUMBAI", "mumbai", "Delhi", None],
    "fee": ["₹500", "", "-100", "Rs 300", None]
})

# Check missing values
print(raw.isna().sum())

clean = raw.copy()

# Treat empty names as missing
clean["name"] = clean["name"].replace("", pd.NA)

# Standardise city text
clean["city"] = clean["city"].str.strip().str.title()
clean["city"] = clean["city"].fillna("Unknown")

# Clean fee text and convert
clean["fee"] = (
    clean["fee"]
    .astype("str")
    .str.replace("₹", "", regex=False)
    .str.replace("Rs", "", regex=False)
    .str.strip()
)
clean["fee"] = pd.to_numeric(clean["fee"], errors="coerce")

# Mark impossible fees as missing
clean.loc[clean["fee"] < 0, "fee"] = pd.NA

# Fill safe default for missing fee
clean["fee"] = clean["fee"].fillna(0)

# Drop rows without key ID
clean = clean.dropna(subset=["student_id"])

print(clean)

Clean Missing and Bad Values

One rule

What to clean

Pick a fix

Avoid chained assignment

Clean a fees file

Write your reason

Missing affects charts

Do this now

Quick check