Model Evaluation - lesson 4 | HackIndia Skills

Model Evaluation - lesson 4 | HackIndia Skills | HackIndia

Example

Simple sklearn split

This works for a normal classification dataset where each row is independent. For example, predicting whether a support ticket is urgent or not urgent.

Check the class counts after splitting. If your validation set has almost no positive cases, your precision or recall will not mean much.

from sklearn.model_selection import train_test_split

X = df.drop(columns=["label"])
y = df["label"]

X_train, X_val, y_train, y_val = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

print("Train size:", len(X_train))
print("Validation size:", len(X_val))
print("Train class ratio:")
print(y_train.value_counts(normalize=True))
print("Validation class ratio:")
print(y_val.value_counts(normalize=True))

Example

Group split example

Use this when rows are not independent. The validation set will contain groups not seen during training.

This is useful for projects with repeated users, sellers, patients, classrooms, or devices.

from sklearn.model_selection import GroupShuffleSplit

X = df.drop(columns=["label", "student_id"])
y = df["label"]
groups = df["student_id"]

splitter = GroupShuffleSplit(
    n_splits=1,
    test_size=0.2,
    random_state=42
)

train_idx, val_idx = next(splitter.split(X, y, groups=groups))

X_train = X.iloc[train_idx]
X_val = X.iloc[val_idx]
y_train = y.iloc[train_idx]
y_val = y.iloc[val_idx]

print("Train students:", groups.iloc[train_idx].nunique())
print("Validation students:", groups.iloc[val_idx].nunique())

Make a clean validation split

One clean rule

What validation means

Make the split

Simple sklearn split

Common leakage

Split by group

Group split example

Do this now

Quick check