How to Clean Messy Data for Machine Learning in Python: A Beginner's Guide (Food Delivery ML Series, Part 3)

The short answer

Real data is messy: typos, missing values, wrong units, raw timestamps and duplicates. In this guide you clean 3,040 messy delivery orders and build a scikit-learn pipeline that goes from a raw order straight to a prediction. Cleaning alone cuts the average error from 13.0 minutes to 4.5.

Quick answer: To clean data for machine learning in Python: (1) inspect it with info(), isna().sum() and value_counts(), (2) remove duplicates, (3) standardise text and fix wrong units, (4) engineer features such as hour and weekday from timestamps, (5) fill missing values with SimpleImputer, (6) turn categories into numbers with OneHotEncoder, and (7) wrap every step and the model in a Pipeline so new data is always cleaned the same way.

What you will build

In Parts 1 and 2 the data was perfect: every order had every value, spelled the same way, in the right units. Real company data never looks like that. In this part the delivery company hands you an export from its systems, and it's a mess:

• the same city zone spelled 14 different ways (Old Town, old town, OLD TOWN, TechPark …);
• weather stored as text in 8 different versions (Rain, RAIN, Rainy, Sunny …);
• 299 orders with no kitchen prep time and 152 riders with no experience record;
• one partner app that sends distance in metres, so some "6 km" orders say 6,500 km;
• raw timestamps instead of a neat "is it peak hour?" column;
• 40 orders logged twice after a network glitch.

You will find each problem, fix it, and package all the fixes together with the model into a single pipeline. You then feed it a deliberately messy new order (zone typed as " old TOWN", distance in metres, prep time missing) and still get a sensible prediction.

How this part connects to the series

This is Part 3 of the Delivery ML series. Part 2 ended with a warning: the saved model didn't include the add_interactions step, so anyone using it had to remember to run that step by hand. The pipeline you build here fixes that bug for good: cleaning, feature engineering and the model travel together as one object. Part 4 turns that pipeline into an API.

Prerequisites

You need Parts 1 and 2 (or the same knowledge), basic pandas, and the same Python setup. Nothing new to install.

From earlier in the series

Knowledge

Software

Same as Part 2; nothing new to install.

Files you'll create

This part uses three small files in one folder, the first step towards a real project structure:

In Colab, create the first two files with the %%writefile delivery_data.py command at the top of a cell.

Hardware, time and cost

• Computer: any laptop or Google Colab; everything trains in a few seconds.
• Time: 60–75 minutes reading every explanation; about 20 minutes if you only run the code.
• Cost: free. No downloads, GPU or paid API.

Step 1: Set up your project folder

Reuse your environment from Parts 1–2 and create a folder for this part's three files.

cd first-ml-model
source venv/bin/activate          # Windows: venv\Scripts\activate
mkdir part3-clean-data
cd part3-clean-data

Starting fresh? Create a virtual environment as in Part 1, Step 1, then:

pip install scikit-learn pandas numpy joblib jupyter

Using Colab? Open a new notebook. You'll create the helper files with %%writefile in Steps 2 and 3.

Why three files instead of one?

In Parts 1 and 2 everything lived in one script. That's fine for learning, but it caused Part 2's bug: the feature function lived in the script, so the saved model couldn't be used without it. From now on we separate:

• data (delivery_data.py): where orders come from;
• features (delivery_features.py): how a raw order becomes model inputs;
• training (clean_messy_data.py): what we do with them.

The features file is the one that matters most: in Step 7 you'll see that the saved pipeline needs to import it, and in Part 4 the API will import it too.

Step 2: Load the messy orders and find the problems

Five quick commands reveal six kinds of problem: 14 spellings of 4 zones, 8 versions of the weather, 451 missing values, distances in metres, raw timestamps and 40 duplicates. Finding problems comes before fixing them.

Create the data file

Save this as delivery_data.py. It plays the role of the company's database export: it creates 3,000 realistic orders using a richer version of our hidden recipe, then deliberately adds real-world mess. You don't need to understand every line; treat it as the data you were handed.

What's new in the recipe since Part 2: each order now has a city zone (Old Town's narrow streets are 35% slower; the Suburbs' wide roads 15% faster), a cuisine (biryani takes about 25 minutes to prepare, burgers about 12), and weekends add about 3 minutes.

"""Delivery ML Series: generate realistic, messy food delivery orders."""
import numpy as np
import pandas as pd


def make_messy_orders(n=3000, seed=7):
    rng = np.random.default_rng(seed)

    # When was each order placed? 4 weeks of September 2026, busiest at lunch and dinner
    day = rng.integers(0, 28, n)
    hours = np.arange(10, 24)
    weights = np.array([2, 3, 6, 6, 3, 2, 2, 3, 4, 7, 7, 6, 3, 2], dtype=float)
    hour = rng.choice(hours, n, p=weights / weights.sum())
    minute = rng.integers(0, 60, n)
    order_time = (pd.Timestamp("2026-09-01")
                  + pd.to_timedelta(day, unit="D")
                  + pd.to_timedelta(hour, unit="h")
                  + pd.to_timedelta(minute, unit="min"))

    cuisines = np.array(["Pizza", "Burger", "Biryani", "Chinese", "Sushi", "South Indian"])
    cuisine_prep = {"Pizza": 15, "Burger": 12, "Biryani": 25, "Chinese": 18, "Sushi": 20, "South Indian": 14}
    zones = np.array(["Central", "Suburbs", "Old Town", "Tech Park"])
    zone_speed = {"Central": 1.0, "Suburbs": 0.85, "Old Town": 1.35, "Tech Park": 1.0}

    cuisine = rng.choice(cuisines, n)
    zone = rng.choice(zones, n, p=[0.35, 0.3, 0.15, 0.2])
    distance = rng.uniform(0.5, 12, n).round(1)
    prep = np.clip(np.array([cuisine_prep[c] for c in cuisine]) + rng.normal(0, 5, n), 5, 45).round()
    raining = rng.random(n) < 0.2
    experience = rng.integers(0, 49, n)

    is_peak = np.isin(hour, [12, 13, 19, 20, 21])
    is_weekend = order_time.dayofweek >= 5
    speed = np.array([zone_speed[z] for z in zone])
    travel = distance * 3 * speed * (1 + 0.5 * raining) * (1 + 0.3 * is_peak)
    total = prep + travel - 0.1 * experience + 3 * is_weekend + rng.normal(0, 5, n)

    orders = pd.DataFrame({
        "order_time": order_time.strftime("%Y-%m-%d %H:%M"),
        "city_zone": zone,
        "cuisine": cuisine,
        "distance_km": distance,
        "prep_time_min": prep,
        "weather": np.where(raining, "Rain", "Clear"),
        "rider_experience_months": experience.astype(float),
        "delivery_min": np.clip(total, 5, None).round(1),
    })

    # ---- Real-world mess ----
    # 1. Inconsistent text: typed by hand in different apps
    messy_zone = rng.random(n) < 0.15
    zone_variants = {"Central": ["central", "CENTRAL", " Central"], "Suburbs": ["suburbs", "Suburbs "],
                     "Old Town": ["old town", "Old town ", "OLD TOWN"], "Tech Park": ["tech park", "TechPark"]}
    orders.loc[messy_zone, "city_zone"] = [rng.choice(zone_variants[z]) for z in orders.loc[messy_zone, "city_zone"]]
    messy_weather = rng.random(n) < 0.15
    weather_variants = {"Rain": ["rain", "RAIN", "Rainy"], "Clear": ["clear", "CLEAR", "Sunny"]}
    orders.loc[messy_weather, "weather"] = [rng.choice(weather_variants[w]) for w in orders.loc[messy_weather, "weather"]]
    # 2. Missing values: restaurants that don't report prep time, riders with no HR record
    orders.loc[rng.random(n) < 0.10, "prep_time_min"] = np.nan
    orders.loc[rng.random(n) < 0.05, "rider_experience_months"] = np.nan
    # 3. Wrong units: one partner app sends distance in metres
    metres = rng.random(n) < 0.02
    orders.loc[metres, "distance_km"] = (orders.loc[metres, "distance_km"] * 1000).round()
    # 4. Duplicates: the same order logged twice after a network retry
    dupes = orders.sample(40, random_state=seed)
    orders = pd.concat([orders, dupes]).sample(frac=1, random_state=seed).reset_index(drop=True)
    return orders

First look

from delivery_data import make_messy_orders

orders = make_messy_orders()
orders.head(8)

Output:

         order_time  city_zone       cuisine  distance_km  prep_time_min weather  rider_experience_months  delivery_min
0  2026-09-12 20:23    Central        Burger          5.7           11.0   Clear                     24.0          40.5
1  2026-09-17 21:58    Suburbs         Pizza         11.7           17.0   Rainy                      5.0          76.2
2  2026-09-23 21:20  Tech Park         Pizza          0.8            5.0   clear                     46.0           5.0
3  2026-09-06 18:00    Central       Chinese          2.6           15.0   Clear                     43.0          18.5
4  2026-09-10 11:29    Suburbs        Burger         10.1           13.0    Rain                     32.0          41.9
5  2026-09-17 19:32   Old Town         Pizza          6.6           16.0   Clear                      4.0          45.6
6  2026-09-20 20:37    Suburbs  South Indian          9.6           16.0   Clear                      8.0          51.3
7  2026-09-19 14:56    suburbs  South Indian       6500.0           13.0   Clear                     10.0          21.2

Even in eight rows you can spot trouble: Rainy vs Rain vs clear, suburbs in lower case, and row 7, a 6,500 km delivery that took 21 minutes. That's obviously 6.5 km logged in metres.

Detective tool 1: missing values

print(orders.shape)
print(orders.isna().sum())
(3040, 8)
order_time                   0
city_zone                    0
cuisine                      0
distance_km                  0
prep_time_min              299
weather                      0
rider_experience_months    152
delivery_min                 0

299 orders have no prep time and 152 have no rider experience. Models in scikit-learn refuse to train on missing values, so we must deal with them.

Detective tool 2: category spellings

print(orders["city_zone"].value_counts())
Central      914
Suburbs      766
Tech Park    538
Old Town     402
suburbs       64
TechPark      59
Suburbs       59
 Central      55
central       44
tech park     44
CENTRAL       38
Old town      22
old town      21
OLD TOWN      14

Four real zones, 14 spellings. Look closely: there are two Suburbs rows, because one has an invisible trailing space ("Suburbs "). The weather column has the same disease: Clear, CLEAR, clear, Sunny, Rain, RAIN, rain and Rainy.

Detective tool 3: impossible values

print(orders["distance_km"].describe().round(1))
count     3040.0
mean       132.6
std        988.6
min          0.5
25%          3.5
50%          6.4
75%          9.3
max      11700.0

The median is 6.4 km but the mean is 132.6 km and the max is 11,700 km. When the mean is far above the median, a few huge values are dragging it up. Always ask: is this value physically possible? A food delivery of 11,700 km is not.

Detective tool 4: duplicates and data types

print("Duplicates:", orders.duplicated().sum())
orders.info()
Duplicates: 40
...
 0   order_time               3040 non-null   str

40 exact duplicate rows, and order_time is stored as plain text, not a date. The model can't use "2026-09-17 21:58" directly; we'll extract hour and day of the week from it.

The problem list

Step 3: Clean the data and engineer features

One function fixes spelling, units and weather text, turns timestamps into peak-hour and weekend features, and adds Part 2's interactions. After it runs, 14 zone spellings become 4 and the longest delivery is 12 km instead of 11,700.

Remove duplicates first

orders = orders.drop_duplicates()
print(orders.shape)
(3000, 8)

Duplicates go before anything else, and before the train/test split. Otherwise the same order could land in both sets, and the model would be tested on an order it has already seen: a quiet form of cheating.

Write the cleaning function

Save this as delivery_features.py:

"""Delivery ML Series: turn raw, messy order rows into clean model features."""
import pandas as pd

ZONES = {"central": "Central", "suburbs": "Suburbs", "oldtown": "Old Town", "techpark": "Tech Park"}
WEATHER = {"rain": 1, "rainy": 1, "clear": 0, "sunny": 0}
PEAK_HOURS = [12, 13, 19, 20, 21]


def clean_orders(df):
    df = df.copy()

    # 1. Fix wrong units: no delivery is over 50 km, so those values are metres
    df["distance_km"] = df["distance_km"].where(df["distance_km"] <= 50, df["distance_km"] / 1000)

    # 2. Standardise text: " OLD TOWN", "old town", "Old town " -> "Old Town"
    zone_key = df["city_zone"].str.strip().str.lower().str.replace(" ", "")
    df["city_zone"] = zone_key.map(ZONES)

    # 3. Turn weather text into a 1/0 feature
    df["is_raining"] = df["weather"].str.strip().str.lower().map(WEATHER)

    # 4. Engineer features from the timestamp
    order_time = pd.to_datetime(df["order_time"])
    df["is_peak_hour"] = order_time.dt.hour.isin(PEAK_HOURS).astype(int)
    df["is_weekend"] = (order_time.dt.dayofweek >= 5).astype(int)

    # 5. Interaction features from Part 2
    df["distance_x_rain"] = df["distance_km"] * df["is_raining"]
    df["distance_x_peak"] = df["distance_km"] * df["is_peak_hour"]
    df["distance_x_rain_x_peak"] = df["distance_km"] * df["is_raining"] * df["is_peak_hour"]

    return df.drop(columns=["order_time", "weather"])

Each fix, explained

The cleaning code, line by line

The table above explains why each fix exists. This one explains the pandas syntax, so you can write similar rules for your own data.

Notice the order of the steps: the units fix comes first, because the interaction features at the end multiply distance_km. Fix the distance after building distance_x_rain and the interaction column would still contain 6,500.

Check the cleaning worked

Never assume a cleaning step worked. Check it:

from delivery_features import clean_orders

cleaned = clean_orders(orders)
print(cleaned["city_zone"].value_counts())
print(cleaned["is_raining"].value_counts())
print(cleaned["distance_km"].max())

Output:

city_zone
Central      1038
Suburbs       873
Tech Park     636
Old Town      453

is_raining
0    2401
1     599

12.0

Four zones, two weather values, and a maximum distance of 12 km. Every row is accounted for: 1,038 + 873 + 636 + 453 = 3,000, so no spelling slipped through as missing.

What's deliberately *not* in this function

Notice what clean_orders doesn't do: it doesn't fill in missing values or turn zones into numbers. That's on purpose.

Everything in clean_orders is a fixed rule that treats each order on its own ("divide by 1,000 if over 50"). Filling a missing prep time with "the typical prep time" is different: the typical value has to be learned from data, and it must be learned from the training orders only. If you calculated it from all 3,000 orders, information from the test orders would leak into training. Learned steps belong inside the pipeline (Step 4), which fits them on training data automatically.

Why not just delete rows with missing values?

orders.dropna() is tempting, but here it would throw away 14.9% of the training orders. Worse, the model would have no way to handle a new order with a missing value, so in production it would simply fail on roughly one order in seven. Filling in sensible values keeps both the data and the service working.

Theory break: missing values, categories and scaling

Models only understand complete tables of numbers. Three standard techniques get you there: imputation fills gaps, one-hot encoding turns categories into numbers, and scaling puts numbers on a similar range.

1. Missing values: delete, fill, or flag

We use the median for numbers rather than the mean because the median ignores extreme values. If a few orders had a 45-minute prep time, the mean would be pulled up; the median wouldn't.

2. Categories: one-hot encoding

A model can't multiply "Old Town" by a weight. The obvious idea, numbering the zones (Central = 1, Suburbs = 2, Old Town = 3, Tech Park = 4), is a trap: the model would think Tech Park is "four times" Central and that Old Town sits halfway between Suburbs and Tech Park. Zones have no order.

One-hot encoding gives each category its own yes/no column instead:

Exactly one column is "hot" (1) per row, hence the name. Now the model can learn a separate effect for each zone, such as "Old Town adds several minutes".

Why spelling matters here: without Step 3's cleaning, one-hot encoding would create 14 zone columns, one per spelling. The model would learn about "OLD TOWN" from just 14 orders instead of all 453, and a brand-new spelling it has never seen would match no column at all.

OneHotEncoder(handle_unknown="ignore") tells scikit-learn what to do if a category appears later that it never saw in training: set all that feature's columns to 0 instead of crashing.

3. Scaling: putting numbers on the same footing

Our numbers live on very different ranges: distance 0.5–12, prep time 5–45, rider experience 0–48. StandardScaler rescales each column so its average is 0 and its typical spread is 1.

• Plain linear regression gives the same predictions with or without scaling. It's harmless, and it makes the coefficients comparable with each other.
• Many other models (k-nearest neighbours, support vector machines, neural networks, regularised regression) work much better with scaled inputs.
• Tree models (random forests, gradient boosting) don't care about scale at all.

We include scaling so the pipeline works well whichever model you plug in later in the series.

The golden rule: learn from training data only

The median for imputation, the list of categories for one-hot encoding, and the averages for scaling are all learned from data. They must be learned from the training set only and then applied unchanged to the test set and new orders. Doing that by hand is fiddly and easy to get wrong. A pipeline does it automatically, which is exactly what we build next.

Step 4: Build a pipeline

A pipeline chains every step, from cleaning to imputing, encoding, scaling and the model, into one object. You call **.fit()** once on raw training orders and **.predict()** on raw new orders; it does everything else in the right order.

Split the raw data

Notice we split the raw, uncleaned orders. The pipeline will clean them itself:

from sklearn.model_selection import train_test_split

X = orders.drop(columns="delivery_min")   # raw order columns, mess and all
y = orders["delivery_min"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Preprocessing: different treatment for numbers and categories

Numbers and categories need different handling. A ColumnTransformer sends each group of columns down its own mini-pipeline, then glues the results back together side by side:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_features = ["distance_km", "prep_time_min", "rider_experience_months", "is_raining",
                    "is_peak_hour", "is_weekend", "distance_x_rain", "distance_x_peak",
                    "distance_x_rain_x_peak"]
categorical_features = ["city_zone", "cuisine"]

preprocess = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric_features),
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                          OneHotEncoder(handle_unknown="ignore")), categorical_features),
])

The full pipeline

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import FunctionTransformer

from delivery_features import clean_orders

pipeline = make_pipeline(
    FunctionTransformer(clean_orders),   # Step 3's fixed rules
    preprocess,                          # learned: medians, categories, scaling
    LinearRegression(),                  # the model
)

FunctionTransformer wraps your own clean_orders function so it can sit inside a pipeline like any scikit-learn step.

Read it top to bottom: fixed rules first, then the steps that learn from training data (inside the dashed box), then the model. .fit() and .predict() both run this exact path.

What happens when you call `.fit()` and `.predict()`

The left column only ever sees training data, and the right column never re-learns anything. That's the golden rule from the theory break, enforced automatically. It also means the Part 2 bug, forgetting to run a feature step on new data, can no longer happen.

Theory break: fit, transform, and what preprocessing learns

In Part 2 you saw that a model learns parameters (weights and a bias) while you choose hyperparameters. Preprocessing steps work exactly the same way: the imputer learns a median, the encoder learns a list of zones, the scaler learns averages. **fit** learns them; **transform** applies them.

Every step has a `fit` and a `transform`

When you call pipeline.fit(X_train, y_train), the pipeline runs fit_transform on each preprocessing step in order, then fits the model. When you call pipeline.predict(new_orders), it runs only transform on each step, then predict. Nothing is re-learned from new data.

Hyperparameters vs learned values, step by step

The same distinction from Part 2 applies to every step in the pipeline:

That last row says 19 weights: 9 number columns plus 4 zone columns plus 6 cuisine columns after one-hot encoding. Cleaning and encoding change how many features the model actually sees.

See what your pipeline learned

Every learned value is stored inside the fitted pipeline, and scikit-learn marks them with a trailing underscore, just like coef_ in Part 2:

preprocess_fitted = pipeline.named_steps["columntransformer"]
num_steps = preprocess_fitted.named_transformers_["num"]
cat_steps = preprocess_fitted.named_transformers_["cat"]

print("Median prep time:", num_steps.named_steps["simpleimputer"].statistics_[1])
print("Average distance:", num_steps.named_steps["standardscaler"].mean_[0].round(2))
print("Most common zone, cuisine:", cat_steps.named_steps["simpleimputer"].statistics_)
print("Zones learned:", list(cat_steps.named_steps["onehotencoder"].categories_[0]))
print("Columns reaching the model:", len(pipeline.named_steps["linearregression"].coef_))

Output:

Median prep time: 17.0
Average distance: 6.25
Most common zone, cuisine: ['Central' 'Chinese']
Zones learned: ['Central', 'Old Town', 'Suburbs', 'Tech Park']
Columns reaching the model: 19

make_pipeline names each step automatically after its class in lower case ("simpleimputer", "columntransformer" and so on), which is how named_steps[...] finds them.

Why "learn from training data only" really matters: data leakage

Data leakage means information the model shouldn't have at prediction time sneaks into training. It makes test scores look better than real life will be. Three forms you've now met in this series:

In our data, the median prep time happens to be 17.0 whether it's calculated from the training orders or from all 3,000, so preprocessing leakage wouldn't change the score much here. The habit still matters: with time-based data (learning from January to predict March) or small datasets, leaked values can be very different, and the damage is invisible until the model fails in production.

Step 5: Train, evaluate, and measure what each cleaning step is worth

The full pipeline predicts delivery time within 4.5 minutes on average (R² 0.884). Switching the cleaning steps on one at a time shows that fixing the metres alone nearly halves the error.

Train and evaluate

from sklearn.metrics import mean_absolute_error, r2_score

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
print(f"Linear regression  MAE: {mean_absolute_error(y_test, y_pred):.2f} min  R2: {r2_score(y_test, y_pred):.3f}")

Output:

Linear regression  MAE: 4.49 min  R2: 0.884

One .fit() call cleaned 2,400 raw training orders, learned the medians, categories and scaling, and trained the model. One .predict() call pushed 600 raw test orders through the same steps.

Why isn't this as low as Part 2's 3.82? This is a harder problem with more going on (zones, cuisines, weekends, missing values), and one effect isn't fully captured yet: Old Town's slower streets matter more on longer trips, but our zone columns add a fixed amount. In Step 6 you'll try a model that can discover effects like this on its own, and you could add distance × zone interactions yourself as an exercise.

How much is each cleaning step worth?

The most convincing argument for data cleaning is to measure it. We built six versions of the pipeline, each adding one more cleaning step, and scored each on the same test orders:

Three lessons stand out:

1. One bad column can wreck everything. Just 2% of distances in metres made the model nearly useless: 13.03 minutes MAE is barely better than always guessing the average (13.72 minutes). Values like 11,700 km dominate a linear model's sums. Fixing them nearly halved the error in one line of code.
2. Hidden information pays. The weather text and the timestamp were sitting in the data all along, but unusable until cleaned and engineered. Together they were worth almost 1.5 minutes.
3. Every step helps, none is enough alone. Data cleaning is many small, boring fixes that add up to the difference between a broken model and a useful one.

Optional: reproduce the table yourself

This code rebuilds the pipeline six times, switching steps on progressively. It's longer than the rest of the guide, so skip it on a first read.

basic = ["distance_km", "prep_time_min", "rider_experience_months"]
variants = [
    ("Raw numbers only",              False, basic, []),
    ("+ fix metres to km",            True,  basic, []),
    ("+ rain from weather text",      True,  basic + ["is_raining"], []),
    ("+ peak/weekend from timestamp", True,  basic + ["is_raining", "is_peak_hour", "is_weekend"], []),
    ("+ zone and cuisine (one-hot)",  True,  basic + ["is_raining", "is_peak_hour", "is_weekend"],
                                             ["city_zone", "cuisine"]),
    ("+ interactions (full pipeline)", True, basic + ["is_raining", "is_peak_hour", "is_weekend",
                                             "distance_x_rain", "distance_x_peak", "distance_x_rain_x_peak"],
                                             ["city_zone", "cuisine"]),
]

for name, fix_units, num_cols, cat_cols in variants:
    def prepare(df, fix_units=fix_units):
        out = clean_orders(df)
        if not fix_units:
            out["distance_km"] = df["distance_km"]   # undo the units fix
        return out

    steps = [("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num_cols)]
    if cat_cols:
        steps.append(("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                                           OneHotEncoder(handle_unknown="ignore")), cat_cols))
    variant = make_pipeline(FunctionTransformer(prepare), ColumnTransformer(steps), LinearRegression())
    variant.fit(X_train, y_train)
    print(f"{name:32s} MAE = {mean_absolute_error(y_test, variant.predict(X_test)):5.2f} min")

Output:

Raw numbers only                 MAE = 13.03 min
+ fix metres to km               MAE =  7.14 min
+ rain from weather text         MAE =  6.18 min
+ peak/weekend from timestamp    MAE =  5.66 min
+ zone and cuisine (one-hot)     MAE =  5.07 min
+ interactions (full pipeline)   MAE =  4.49 min

This technique, removing or adding one component at a time to measure its value, is called an ablation study. Researchers use it in almost every machine learning paper, and it's just as useful for justifying data-cleaning work to your team.

Theory break: how to judge a model trained on messy data

One overall score hides the most important question for messy data: does the model work as well on the messy orders as on the clean ones? Split the errors by data quality and by category. Here, orders with a missing prep time are off by 6.55 minutes on average, against 4.30 for complete orders.

Parts 1 and 2 taught the core metrics (precision and recall, MAE, RMSE and R²) and the habit of looking at error patterns. With messy data, there are new places for problems to hide.

1. Errors by data quality: did imputation work?

Imputation fills a gap with a guess. Check how good the guesses are by comparing orders that had a missing value with those that didn't:

check = clean_orders(X_test)
check["error"] = y_test.values - y_pred
check["abs_error"] = check["error"].abs()
check["prep_was_missing"] = X_test["prep_time_min"].isna().values

print(check.groupby("prep_was_missing")["abs_error"].agg(["count", "mean"]).round(2))
                  count  mean
prep_was_missing
False               549  4.30
True                 51  6.55

The overall MAE of 4.49 hides a 52% bigger error on the 1 in 12 orders with a missing prep time. That's the cost of guessing "17 minutes" for a biryani that usually takes 25 (Step 7 shows a single example of exactly this). It tells you where to spend effort: getting restaurants to report prep time would help more than any algorithm change. Or try filling the gap with the median for that cuisine, then rerun this check.

Rule: whenever you impute, report the error separately for imputed and complete rows. If the imputed rows are much worse, your users with incomplete data are getting a worse service, and an overall score won't show it.

2. Errors by category: is any group left behind?

print(check.groupby("city_zone")[["abs_error", "error"]].mean().round(2))
           abs_error  error
city_zone
Central         4.15   0.62
Old Town        5.75   0.80
Suburbs         4.53  -0.03
Tech Park       4.21  -1.28

Old Town is the problem area. Grouping its errors by distance shows why, exactly as rain did in Part 2:

old_town = check[check["city_zone"] == "Old Town"]
print("Old Town, long trips (> 8 km):  ", round(old_town[old_town["distance_km"] > 8]["error"].mean(), 2))
print("Old Town, short trips (<= 3 km):", round(old_town[old_town["distance_km"] <= 3]["error"].mean(), 2))
Old Town, long trips (> 8 km):   2.81
Old Town, short trips (<= 3 km): -2.55

Long Old Town trips arrive about 2.8 minutes later than predicted; short ones about 2.6 minutes earlier. The one-hot zone column adds a fixed amount for Old Town, but its narrow streets slow every kilometre by 35%, so the effect should grow with distance. It's the same blind spot as Part 2's rain, and the same fix applies: an interaction feature distance × is_old_town. Gradient boosting (Step 6) learns it on its own: its average error on long Old Town trips is just −0.09 minutes. (There are only 9 short Old Town trips in the test set, too few to judge reliably.)

Rule: check every important category separately. In real products those categories are often cities, customer segments or people, and a model that's worse for one group is a fairness problem as well as an accuracy one.

3. Check the cleaning itself, automatically

A cleaning step that silently fails is worse than no cleaning. Turn the checks from Step 3 into assertions: lines that stop the program with an error if something is wrong.

cleaned = clean_orders(orders)
assert not orders.duplicated().any(), "Duplicates remain"
assert cleaned["city_zone"].notna().all(), "A zone spelling wasn't recognised"
assert cleaned["is_raining"].notna().all(), "A weather value wasn't recognised"
assert cleaned["distance_km"].max() <= 50, "A distance is still in metres"

If next month's export contains a new spelling such as "Old-Town", the second assertion fails loudly instead of quietly turning it into a missing value. In production, run checks like these every time new data arrives.

4. A checklist for judging a model on messy data

Quick self-check

1. A model has an MAE of 4 minutes overall, but 12 minutes on the 5% of orders with a missing value. Is that a problem if the business only looks at the overall MAE?
2. Why does the one-hot "Old Town" column under-predict long Old Town trips?
3. What's the difference between an assertion failing and a value becoming NaN after .map()?

Answers: (1) Yes. One in 20 customers gets a far worse estimate, and the overall score hides it. Report both numbers. (2) It adds the same fixed number of minutes to every Old Town order, but the real slowdown grows with distance, so long trips get too little extra time. (3) An assertion stops the program and tells you something's wrong. A NaN from .map() silently passes the problem on to the imputer, which hides it.

Step 6: Swap in a different model, keep the cleaning

With a pipeline, trying a new model is a one-line change. Gradient boosting scores 4.6 minutes without any hand-made interaction features, almost matching our carefully engineered linear model.

One line to swap the model

from sklearn.ensemble import GradientBoostingRegressor

boosted = make_pipeline(
    FunctionTransformer(clean_orders),
    preprocess,
    GradientBoostingRegressor(random_state=42),   # the only change
)
boosted.fit(X_train, y_train)
y_pred_gb = boosted.predict(X_test)
print(f"Gradient boosting  MAE: {mean_absolute_error(y_test, y_pred_gb):.2f} min  R2: {r2_score(y_test, y_pred_gb):.3f}")

Output:

Gradient boosting  MAE: 4.61 min  R2: 0.881

All the cleaning, imputing and encoding came along for free. Without a pipeline you'd have to repeat every step by hand for each model you try, and keep them perfectly in sync.

Does gradient boosting need our interaction features?

Tree-based models can discover interactions such as "rain matters more on long trips" by themselves. To test that, give it only the basic features:

basic_numeric = ["distance_km", "prep_time_min", "rider_experience_months",
                 "is_raining", "is_peak_hour", "is_weekend"]

preprocess_basic = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), basic_numeric),
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                          OneHotEncoder(handle_unknown="ignore")), categorical_features),
])

for name, model in [("Linear regression", LinearRegression()),
                    ("Gradient boosting", GradientBoostingRegressor(random_state=42))]:
    p = make_pipeline(FunctionTransformer(clean_orders), preprocess_basic, model).fit(X_train, y_train)
    print(f"{name:18s} without interactions  MAE: {mean_absolute_error(y_test, p.predict(X_test)):.2f} min")
Linear regression  without interactions  MAE: 5.07 min
Gradient boosting  without interactions  MAE: 4.58 min

The pattern is clear:

• Linear regression depends on you. Without the interaction features it loses more than half a minute, because it can only add fixed amounts.
• Gradient boosting finds the interactions itself. It doesn't need them (it even does marginally better without them), because its trees naturally learn rules like "if far and raining, add more".
• The engineered linear model still wins narrowly. Domain knowledge, turned into features, is hard to beat. But gradient boosting got close with no domain knowledge at all, and it's using default settings.

That last point is a cliffhanger: gradient boosting has many settings (how many trees, how deep, how fast it learns) that we haven't touched. Part 5 tunes them, and Part 7 explores boosting in depth.

How does gradient boosting work, briefly?

It builds a sequence of small decision trees. The first tree makes rough predictions; the second tree learns to predict the first tree's errors; the third corrects what's left, and so on for 100 trees by default. Each tree is weak on its own, but together they're strong. Its training MAE here is 3.78 minutes against 4.61 on test, a small gap, so it isn't badly overfitting.

Step 7: Predict a raw, messy order and save the whole pipeline

Feed the pipeline an order with a misspelled zone, distance in metres and no prep time, and it still predicts 63.8 minutes. Save the pipeline and that same messy order works in a brand-new Python session.

The ultimate test: a messy new order

import pandas as pd

new_order = pd.DataFrame([{
    "order_time": "2026-10-03 20:15",   # a Saturday, at dinner time
    "city_zone": "  old TOWN",          # messy spelling
    "cuisine": "Biryani",
    "distance_km": 6000,                # sent in metres by a partner app
    "prep_time_min": None,              # restaurant didn't report it
    "weather": "RAINY",
    "rider_experience_months": 6,
}])

print("Prediction:", pipeline.predict(new_order).round(1))

Output:

Prediction: [63.8]

To see what the model actually received, run just the cleaning step:

print(clean_orders(new_order).T)
city_zone                Old Town
cuisine                   Biryani
distance_km                     6
prep_time_min                None
rider_experience_months         6
is_raining                      1
is_peak_hour                    1
is_weekend                      1
distance_x_rain                 6
distance_x_peak                 6
distance_x_rain_x_peak          6

Every problem was handled: " old TOWN" became Old Town, 6,000 metres became 6 km, "RAINY" became 1, and the timestamp revealed a Saturday at peak hour. The missing prep time was then filled in by the pipeline's imputer.

A limitation worth seeing: imputation is a guess

The imputer filled the missing prep time with the training median: 17 minutes. But this is a biryani, and biryanis in our data take about 25 minutes on average. If the restaurant had reported 25, the prediction would be quite different:

reported = new_order.copy()
reported["prep_time_min"] = 25
print(pipeline.predict(reported).round(1))
[71.5]

That's almost 8 minutes more. A single overall median ignores what we know about the order. Smarter options include filling with the median for that cuisine, or adding a "was missing" flag with SimpleImputer(add_indicator=True). Try one as an exercise; the lesson is that filling a gap is never as good as having the real value, so it's worth asking the restaurants for their data.

Save the whole pipeline

import joblib

joblib.dump(pipeline, "delivery_pipeline.joblib")

This single file now contains every learned step: the medians, the category lists, the scaling and the model coefficients.

Load it in a brand-new session

In a new Python file or notebook, in the same folder as **delivery_features.py**:

import joblib
import pandas as pd

pipeline = joblib.load("delivery_pipeline.joblib")

order = pd.DataFrame([{
    "order_time": "2026-10-03 20:15", "city_zone": "  old TOWN", "cuisine": "Biryani",
    "distance_km": 6000, "prep_time_min": None, "weather": "RAINY", "rider_experience_months": 6,
}])
print(pipeline.predict(order).round(1))
[63.8]

Same answer, no manual feature steps. This fixes Part 2's bug.

Why must **delivery_features.py** be in the same folder? joblib saves a reference to your clean_orders function ("it lives in delivery_features"), not the function's code. When loading, Python must be able to import that module. This is why we put the function in its own file in Step 1. In Part 4, the API server will import this exact file and load this exact pipeline.

Security note: as before, only load .joblib files you created or fully trust, and use the same scikit-learn version for saving and loading.

The complete script

With **delivery_data.py** (Step 2) and **delivery_features.py** (Step 3) in the same folder, save this as **clean_messy_data.py** and run **python clean_messy_data.py**.

"""Delivery ML Series, Part 3: clean messy data and build a pipeline."""
import joblib
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

from delivery_data import make_messy_orders
from delivery_features import clean_orders

# 1-2. Load the messy orders and inspect them
orders = make_messy_orders()
print(orders.shape)
print(orders.isna().sum())
print(orders["city_zone"].value_counts())
print("Duplicates:", orders.duplicated().sum())

# 3. Remove duplicates, then preview the cleaning
orders = orders.drop_duplicates()
cleaned = clean_orders(orders)
print(cleaned["city_zone"].value_counts())
print(cleaned["is_raining"].value_counts())
print(cleaned["distance_km"].max())

# Split the RAW data: the pipeline cleans it
X = orders.drop(columns="delivery_min")
y = orders["delivery_min"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 4. Build the pipeline
numeric_features = ["distance_km", "prep_time_min", "rider_experience_months", "is_raining",
                    "is_peak_hour", "is_weekend", "distance_x_rain", "distance_x_peak",
                    "distance_x_rain_x_peak"]
categorical_features = ["city_zone", "cuisine"]

preprocess = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric_features),
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                          OneHotEncoder(handle_unknown="ignore")), categorical_features),
])

pipeline = make_pipeline(
    FunctionTransformer(clean_orders),
    preprocess,
    LinearRegression(),
)

# 5. Train and evaluate
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
print(f"Linear regression  MAE: {mean_absolute_error(y_test, y_pred):.2f} min  R2: {r2_score(y_test, y_pred):.3f}")

# 6. Swap the model, keep the cleaning
boosted = make_pipeline(
    FunctionTransformer(clean_orders),
    preprocess,
    GradientBoostingRegressor(random_state=42),
)
boosted.fit(X_train, y_train)
y_pred_gb = boosted.predict(X_test)
print(f"Gradient boosting  MAE: {mean_absolute_error(y_test, y_pred_gb):.2f} min  R2: {r2_score(y_test, y_pred_gb):.3f}")

# 7. Predict a raw, messy new order, then save the whole pipeline
new_order = pd.DataFrame([{
    "order_time": "2026-10-03 20:15",   # a Saturday, at dinner time
    "city_zone": "  old TOWN",          # messy spelling
    "cuisine": "Biryani",
    "distance_km": 6000,                # sent in metres by a partner app
    "prep_time_min": None,              # restaurant didn't report it
    "weather": "RAINY",
    "rider_experience_months": 6,
}])
print("Prediction:", pipeline.predict(new_order).round(1))
print(clean_orders(new_order).T)

joblib.dump(pipeline, "delivery_pipeline.joblib")

Expected output (key lines):

(3040, 8)
Duplicates: 40
12.0
Linear regression  MAE: 4.49 min  R2: 0.884
Gradient boosting  MAE: 4.61 min  R2: 0.881
Prediction: [63.8]

Common mistakes and how to fix them

Data cleaning habits worth building now: inspect before you fix, check after you fix, fix with rules you can explain, keep learned steps inside the pipeline, and save the whole pipeline, never just the model.

Practice project: clean a messy smart-meter export

The energy company from the Part 1 and 2 practice projects now sends you its real export: 2,050 meter readings from four cities, with old city names, "Y/TRUE/yes" for solar, missing values, duplicates, and one meter brand that reports in watt-hours, including in the column you're trying to predict.

Get the data

Save this as energy_data_messy.py. Don't read it closely before doing Task 1: discovering the problems yourself is the point.

"""Practice project (Part 3): messy smart-meter export for home electricity use."""
import numpy as np
import pandas as pd


def make_messy_homes(n=2000, seed=21):
    rng = np.random.default_rng(seed)
    month = rng.integers(1, 13, n)
    day = rng.integers(1, 29, n)
    reading_date = pd.to_datetime({"year": 2026, "month": month, "day": day})
    is_summer = np.isin(month, [4, 5, 6]).astype(int)          # April to June

    cities = np.array(["Mumbai", "Bengaluru", "Chennai", "Pune"])
    city = rng.choice(cities, n, p=[0.3, 0.3, 0.2, 0.2])
    city_ac = {"Mumbai": 1.0, "Bengaluru": 0.6, "Chennai": 1.3, "Pune": 0.9}   # how hot the city runs

    size = rng.uniform(30, 200, n).round()
    residents = rng.integers(1, 7, n)
    has_solar = (rng.random(n) < 0.25).astype(int)
    wfh = (rng.random(n) < 0.3).astype(int)
    age = rng.integers(0, 16, n)
    base_ac = np.where(is_summer == 1, rng.normal(6, 2.5, n), rng.normal(1.5, 1, n)).clip(0, 14)
    ac_hours = (base_ac * np.array([city_ac[c] for c in city])).clip(0, 16).round(1)

    kwh = (50 + 0.9 * size + 35 * residents + 45 * ac_hours * (size / 100) + 4 * age + 60 * wfh)
    kwh = (kwh * (1 - 0.35 * has_solar) + rng.normal(0, 30, n)).clip(20).round()

    homes = pd.DataFrame({
        "reading_date": reading_date.dt.strftime("%Y-%m-%d"),
        "city": city,
        "home_size_sqm": size,
        "residents": residents.astype(float),
        "solar_panels": np.where(has_solar == 1, "Yes", "No"),
        "works_from_home": wfh,
        "appliance_age_years": age.astype(float),
        "ac_hours_per_day": ac_hours,
        "monthly_kwh": kwh,
    })

    # ---- Real-world mess ----
    # 1. City names: different spellings AND old names
    variants = {"Mumbai": ["mumbai", "MUMBAI", "Bombay"], "Bengaluru": ["Bangalore", "bengaluru", "BLR"],
                "Chennai": ["chennai", "Madras"], "Pune": ["pune", "PUNE "]}
    messy = rng.random(n) < 0.2
    homes.loc[messy, "city"] = [str(rng.choice(variants[c])) for c in homes.loc[messy, "city"]]
    # 2. Yes/No typed inconsistently
    solar_variants = {"Yes": ["yes", "Y", "TRUE"], "No": ["no", "N", "FALSE"]}
    messy = rng.random(n) < 0.15
    homes.loc[messy, "solar_panels"] = [str(rng.choice(solar_variants[s])) for s in homes.loc[messy, "solar_panels"]]
    # 3. Missing values
    homes.loc[rng.random(n) < 0.12, "appliance_age_years"] = np.nan
    homes.loc[rng.random(n) < 0.05, "residents"] = np.nan
    # 4. Wrong units: one meter brand reports watt-hours (Wh), not kWh, including the target!
    wh = rng.random(n) < 0.03
    homes.loc[wh, "monthly_kwh"] = homes.loc[wh, "monthly_kwh"] * 1000
    # 5. Duplicates from a re-sent meter upload
    dupes = homes.sample(50, random_state=seed)
    homes = pd.concat([homes, dupes]).sample(frac=1, random_state=seed).reset_index(drop=True)
    return homes

Your tasks

• 1. Detective work (Step 2). Load homes = make_messy_homes(). Use shape, isna().sum(), value_counts(), describe() and duplicated().sum() to list every problem you can find, in a table like the one at the end of Step 2.
• 2. Remove duplicates (Step 3). How many rows are left?
• 3. Fix the target (new!). Some monthly_kwh values are in watt-hours. Write a fixed rule to convert them back to kWh. (Hint: no home uses 20,000 kWh in a month.) Why must this happen before the train/test split, and outside the pipeline? (Hint: what does a pipeline transform, X or y?)
• 4. Write **clean_homes(df)** (Step 3) in its own file, energy_features.py. It should:map all city spellings and old names to Mumbai, Bengaluru, Chennai and Pune. Lower-casing alone won't turn "Bombay" into "Mumbai"; you need a lookup table;
• turn the solar column into a 1/0 has_solar;
• create is_summer from the reading date (April to June);
• add Part 2's ac_x_size interaction feature.

Check your results

Think about it: the city clearly affects electricity use (Chennai runs hotter than Bengaluru), so why did removing the city column change nothing? *(Hint: look at how **ac_hours_per_day* is created in the generator. Which column already carries the city's effect?) And if the city doesn't help the model, was cleaning the city names wasted effort? (Hint: think about who else uses this data: reports, dashboards, billing.)

Share your energy_features.py and your answers in the comments. Comparing different people's cleaning rules is one of the best ways to learn what you missed.

Frequently asked questions

How do I clean data for machine learning in Python?

Inspect with info(), isna().sum(), value_counts() and describe(); remove duplicates; standardise text; fix impossible values; engineer features from timestamps; then fill missing values and encode categories inside a scikit-learn pipeline so the same steps apply to new data.

How do I handle missing values in scikit-learn?

Use SimpleImputer inside a pipeline: strategy="median" for numbers and strategy="most_frequent" for categories. Add add_indicator=True if the fact that a value is missing is itself informative. Avoid deleting rows unless very few are affected.

What is one-hot encoding?

A way to turn a category into numbers by giving each possible value its own yes/no column. A row for "Old Town" gets a 1 in the Old Town column and 0 in the others. It avoids implying a false order between categories.

What is a scikit-learn pipeline and why use one?

A pipeline chains preprocessing steps and a model into one object with a single .fit() and .predict(). It learns every step from training data only, applies identical steps to new data, prevents data leakage, and saves as one file.

What is ColumnTransformer used for?

It applies different preprocessing to different columns, such as imputing and scaling the numbers while imputing and one-hot encoding the categories, then combines the results into one table for the model.

Should I clean data before or after the train/test split?

Remove duplicates and apply fixed rules (spelling, units) before splitting. Anything learned from data, such as medians, category lists or scaling, must be fitted on training data only. A pipeline handles that automatically.

How much does data cleaning improve a model?

It depends on the data, but often more than changing algorithms. In this guide, cleaning took the average error from 13.03 minutes to 4.49; fixing a single unit error accounted for almost six of those minutes.

Glossary

What to learn next: the Delivery ML series

This guide is Part 3 of an 8-part series. Every part uses the same food delivery problem and starts where the previous part ended.

Next up: Part 4, Deploy Your Model as an API. Keep this folder exactly as it is: delivery_features.py and delivery_pipeline.joblib are what the API will load.

Conclusion

You've turned a messy company export into a reliable pipeline that takes raw orders, typos and all, to a prediction within about 4.5 minutes, and you've measured that cleaning was worth more than any change of algorithm.

Most real machine learning work looks like this guide, not like Parts 1 and 2: finding problems, fixing them with rules you can explain, and packaging every step so it runs the same way in training and in production. Try adding a new mess to delivery_data.py, such as rider experience recorded in years for some riders, and see whether you can detect and fix it.