How to Tune a Machine Learning Model in Python: A Beginner's Guide to Hyperparameters and GridSearchCV (Food Delivery ML Series, Part 5)

The short answer

Tuning a model means choosing its settings, its hyperparameters, by testing many options on validation data and keeping the best. In this tutorial, tuning turns a gradient boosting model that lost to simple linear regression (4.61 vs 4.49 minutes of error) into one that ties it overall (4.50) and does better on the hardest orders. Along the way you'll learn why an honest "tie" is a real result.

Quick answer: To tune a machine learning model in Python: (1) keep the test set aside; (2) pick the hyperparameters to try, such as max_depth, learning_rate and n_estimators; (3) score each combination with 5-fold cross-validation on the training data, using scikit-learn's GridSearchCV (every combination) or RandomizedSearchCV (a random sample); (4) take the best settings and check them once on the test set; (5) compare with your old model using a range, not a single number, because small differences can be luck.

What you will build

By the end of this tutorial you will have:

The rainy Saturday biryani from Parts 3 and 4 now gets:

{
  "predicted_minutes": 68.8,
  "promised_minutes": 76,
  "model_version": "part5-gb-tuned-v1"
}

How this part fits the series

This is Part 5 of the Delivery ML series. It takes the gradient boosting model from Part 3, Step 6, which scored worse than linear regression, tunes it properly, and swaps it into the API you built in Part 4. Part 6 (explaining predictions) works with either model, and Part 7 goes deeper into boosting with XGBoost and LightGBM, where tuning matters even more.

Prerequisites

You need Parts 2 and 3 of this series, Part 4 for the redeploy step, and about 90 minutes. No new libraries: everything in this part comes with scikit-learn.

From earlier parts

Knowledge

Software

Hardware, time and cost

• Any laptop from the last five years. Searches use all CPU cores (n_jobs=-1); the biggest one takes about a minute.
• Google Colab works too: run the same code in cells.
• Free: no cloud services or API keys.

Theory break: parameters, hyperparameters, and the three-way split

Parameters are what the model learns from the data; hyperparameters are the settings you choose before training. Tuning means trying different settings and keeping the best, but "best" has to be judged on data the model never trained on, and never on the test set.

A quick refresher from Part 2

Gradient boosting's three main dials

Gradient boosting (Part 3, Step 6) builds a team of small decision trees, one after another. Each new tree tries to fix the mistakes of the trees before it.

n_estimators and learning_rate work as a pair: smaller steps need more trees.

Why you can't tune on the test set

Imagine a student who sees the final exam, practises those exact questions, and scores 95%. The score says nothing about what the student knows.

If you try 50 settings and keep the one with the best test MAE, you've done the same thing. The test score is now optimistic, because you picked it partly for luck. The test set stops being a fair final exam.

The fix is a third pot of data for choosing settings:

In Step 2 we'll carve validation data out of the training set by hand. In the cross-validation theory break, we'll find a way to use all the training data for both learning and validating.

Step 1: Set up and measure the baseline

Rebuild Part 3's data, split and both models. Gradient boosting fits the training orders much more closely than linear regression (3.78 vs 4.50 minutes of error) but does worse on the test set (4.61 vs 4.49). That gap is the problem tuning will try to fix.

Create a new file, tune_delivery.py, in the same folder as Part 3's delivery_data.py and delivery_features.py:

import time

import joblib
import numpy as np
import pandas as pd
from scipy.stats import loguniform, randint, uniform
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import (GridSearchCV, KFold, RandomizedSearchCV, cross_val_predict,
                                     cross_validate, train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

from delivery_data import make_messy_orders
from delivery_features import clean_orders

orders = make_messy_orders().drop_duplicates()
X = orders.drop(columns="delivery_min")
y = orders["delivery_min"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

numeric_features = ["distance_km", "prep_time_min", "rider_experience_months", "is_raining",
                    "is_peak_hour", "is_weekend", "distance_x_rain", "distance_x_peak",
                    "distance_x_rain_x_peak"]
categorical_features = ["city_zone", "cuisine"]


def make_preprocess():
    return ColumnTransformer([
        ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric_features),
        ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                              OneHotEncoder(handle_unknown="ignore")), categorical_features),
    ])


def make_boosted(**settings):
    """A Part 3 gradient boosting pipeline with the hyperparameters you choose."""
    return make_pipeline(FunctionTransformer(clean_orders), make_preprocess(),
                         GradientBoostingRegressor(random_state=42, **settings))


linear = make_pipeline(FunctionTransformer(clean_orders), make_preprocess(), LinearRegression())
linear.fit(X_train, y_train)
default_boosted = make_boosted().fit(X_train, y_train)

for name, model in [("Linear", linear), ("Boosted (default)", default_boosted)]:
    train_mae = mean_absolute_error(y_train, model.predict(X_train))
    print(f"{name:18s} train MAE {train_mae:.2f}")

Output:

Linear             train MAE 4.50
Boosted (default)  train MAE 3.78

Most of this is Part 3's code. The new parts:

Reading the baseline

We already know the test scores from Part 3, so let's put everything side by side:

The linear model makes almost exactly the same size of error on orders it has seen and orders it hasn't. Gradient boosting is 0.8 minutes better on the orders it trained on, and 0.1 minutes worse on new ones. It has learned some patterns that are really just noise in the training orders.

Could different settings fix that? That's the question for the rest of this tutorial.

From here on, the test set is locked. We've looked at it once for these two models (in Part 3). Every decision from now until Step 5 is made without it.

Step 2: Tune one hyperparameter by hand

Split off 20% of the training orders as validation data and try six tree depths. Depth 2 wins with 4.51 minutes. But with a different random split, depth 4 wins instead, so one validation split isn't enough to trust.

We start with max_depth, the most important dial: how many questions each tree can ask.

X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=0)
for depth in [1, 2, 3, 4, 5, 6]:
    model = make_boosted(max_depth=depth).fit(X_fit, y_fit)
    print(f"max_depth={depth}  validation MAE {mean_absolute_error(y_val, model.predict(X_val)):.2f}")

Output:

max_depth=1  validation MAE 5.36
max_depth=2  validation MAE 4.51
max_depth=3  validation MAE 4.56
max_depth=4  validation MAE 4.59
max_depth=5  validation MAE 4.69
max_depth=6  validation MAE 4.81

What the numbers say

• Depth 1 is too simple (5.36). A tree that can ask only one question can't learn that rain matters more at rush hour. This is underfitting.
• Deeper than 2 gets steadily worse. Each extra level lets the trees memorise more quirks of these 1,920 orders. This is overfitting.
• Depth 2 is the sweet spot, beating the default depth 3.

The catch

Change one number, random_state=0 to random_state=1, so that a different 480 orders become the validation set:

max_depth=1  validation MAE 5.44
max_depth=2  validation MAE 4.36
max_depth=3  validation MAE 4.30
max_depth=4  validation MAE 4.27
max_depth=5  validation MAE 4.41
max_depth=6  validation MAE 4.58

Now depth 4 wins. Nothing about the model changed; only which orders were used for checking. With 480 validation orders, the luck of the draw moves the scores by about as much as the differences we're trying to measure.

We need a way to validate on more orders without touching the test set. That's cross-validation.

Theory break: cross-validation

Cross-validation splits the training data into 5 parts and takes turns: each part is the validation set once while the other 4 train the model. Every order gets used for validation, so the score is far less a matter of luck.

How 5-fold cross-validation works

Cut the 2,400 training orders into 5 equal folds of 480. Then run five rounds:

Average the five validation scores. Like a student who sits five different practice exams instead of one, the average tells you far more than any single result.

The cost: five times as much training. For our small dataset, that's seconds.

The same depth question, cross-validated

cv = KFold(n_splits=5, shuffle=True, random_state=42)
for depth in [1, 2, 3, 4, 5, 6, 8]:
    scores = cross_validate(make_boosted(max_depth=depth), X_train, y_train, cv=cv,
                            scoring="neg_mean_absolute_error", return_train_score=True, n_jobs=-1)
    print(f"max_depth={depth}  train {-scores['train_score'].mean():.2f}  "
          f"validation {-scores['test_score'].mean():.2f}")

Output:

max_depth=1  train 5.13  validation 5.33
max_depth=2  train 4.11  validation 4.52
max_depth=3  train 3.71  validation 4.40
max_depth=4  train 3.33  validation 4.42
max_depth=5  train 2.83  validation 4.50
max_depth=6  train 2.16  validation 4.63
max_depth=8  train 0.88  validation 4.85

Because the pipeline (cleaning, imputing, scaling, model) is refitted inside every round, the imputer's median is learned only from that round's 4 training folds. That's Part 3's leakage lesson, handled automatically.

Reading the validation curve

This picture is called a validation curve, and it's one of the most useful plots in machine learning:

• Training error falls forever. At depth 8 the model is within 0.9 minutes on orders it has seen: it is memorising them.
• Validation error falls, then rises. The bottom of the U is the best setting. Left of it the model is too simple (underfitting); right of it, it learns noise (overfitting).
• The gap between the lines is overfitting you can see. At depth 3 it's 0.7 minutes; at depth 8 it's 4 minutes.

Cross-validation picks depth 3 (4.40), with depth 4 a hair behind (4.42). The single splits in Step 2 picked depth 2 and depth 4; averaging five rounds settles it.

Notice something else: at depth 3, cross-validation says boosting beats linear regression (4.40 vs 4.53), even though linear won on the test set. Hold that thought; it comes back in Step 5.

Step 3: Search a grid of settings with GridSearchCV

**GridSearchCV** tries every combination of the settings you list, cross-validates each one, and keeps the best. 4 depths × 4 learning rates × 3 tree counts = 48 settings and 240 fits, in under a minute. The winner: depth 2, learning rate 0.1, 400 trees, with a validation MAE of 4.36.

Depth isn't the only dial, and the dials affect each other: shallower trees may need more of them. So we search all three together.

param_grid = {
    "gradientboostingregressor__max_depth": [1, 2, 3, 4],
    "gradientboostingregressor__learning_rate": [0.03, 0.05, 0.1, 0.2],
    "gradientboostingregressor__n_estimators": [100, 200, 400],
}
start = time.perf_counter()
grid = GridSearchCV(make_boosted(), param_grid, cv=cv, scoring="neg_mean_absolute_error", n_jobs=-1)
grid.fit(X_train, y_train)
print(f"Tried {len(grid.cv_results_['params'])} settings in {time.perf_counter() - start:.0f} s")
print("Best settings:", grid.best_params_)
print(f"Best validation MAE: {-grid.best_score_:.3f}")

Output (your timing will differ; ours is a 2-core machine):

Tried 48 settings in 57 s
Best settings: {'gradientboostingregressor__learning_rate': 0.1, 'gradientboostingregressor__max_depth': 2, 'gradientboostingregressor__n_estimators': 400}
Best validation MAE: 4.358

Look beyond the winner

The top five tell an important story. grid.cv_results_ holds every score; turn it into a table:

results = pd.DataFrame(grid.cv_results_)
top = results.sort_values("rank_test_score").head(5)
print(pd.DataFrame({
    "max_depth": top["param_gradientboostingregressor__max_depth"],
    "learning_rate": top["param_gradientboostingregressor__learning_rate"],
    "n_estimators": top["param_gradientboostingregressor__n_estimators"],
    "val_MAE": -top["mean_test_score"].round(3),
    "spread": top["std_test_score"].round(3),
}).to_string(index=False))

Output:

 max_depth  learning_rate  n_estimators  val_MAE  spread
         2           0.10           400    4.358   0.093
         3           0.03           400    4.373   0.074
         2           0.05           400    4.374   0.087
         3           0.05           400    4.378   0.071
         3           0.10           200    4.381   0.073

• **rank_test_score** is 1 for the best setting, 2 for the next, and so on.
• **spread** (std_test_score) is how much the score varied across the 5 folds.

Two things to notice:

1. The top five are within 0.023 minutes of each other (about 1.4 seconds), while each one's score wobbles by about 0.08 between folds. They are effectively tied. The "winner" won by luck as much as by merit.
2. The same pattern repeats: shallow trees + many small steps. Depth 2 with 400 trees and depth 3 with a slow learning rate both work, because both learn gradually. That's the learning_rate and n_estimators partnership from the first theory break.

For comparison, linear regression on the same 5 folds:

linear_cv = cross_validate(linear, X_train, y_train, cv=cv, scoring="neg_mean_absolute_error")
print(f"Linear validation MAE: {-linear_cv['test_score'].mean():.3f}")
Linear validation MAE: 4.534

So cross-validation says the tuned model is about 0.18 minutes better than linear regression. Keep that number in mind.

Step 4: Search bigger spaces with RandomizedSearchCV

When there are too many combinations to try them all, **RandomizedSearchCV** tries a random sample. 40 random settings across 6 hyperparameters found a quite different model (Huber loss, depth 3, 180 trees) that scored 4.34, only 0.02 better than the grid. Many different settings are equally good.

Gradient boosting has more dials than the three we searched. Adding three more, with a handful of values each, would make thousands of combinations. Trying them all would take hours.

The trick: instead of a list of values, give each hyperparameter a range, and let the search pick random combinations.

param_distributions = {
    "gradientboostingregressor__max_depth": randint(1, 6),
    "gradientboostingregressor__learning_rate": loguniform(0.01, 0.3),
    "gradientboostingregressor__n_estimators": randint(100, 800),
    "gradientboostingregressor__min_samples_leaf": randint(1, 50),
    "gradientboostingregressor__subsample": uniform(0.5, 0.5),
    "gradientboostingregressor__loss": ["squared_error", "absolute_error", "huber"],
}
start = time.perf_counter()
search = RandomizedSearchCV(make_boosted(), param_distributions, n_iter=40, cv=cv,
                            scoring="neg_mean_absolute_error", n_jobs=-1, random_state=42)
search.fit(X_train, y_train)
print(f"Tried 40 settings in {time.perf_counter() - start:.0f} s")
for name, value in search.best_params_.items():
    value = round(value, 3) if isinstance(value, float) else value
    print(f"  {name.split('__')[1]}: {value}")
print(f"Best validation MAE: {-search.best_score_:.3f}")

Output:

Tried 40 settings in 125 s
  learning_rate: 0.08
  loss: huber
  max_depth: 3
  min_samples_leaf: 14
  n_estimators: 180
  subsample: 0.629
Best validation MAE: 4.336

The ranges, explained

What is `loss="huber"`?

The loss is the error the model tries to shrink while it trains (Part 2 introduced squared error, the default). Squared error punishes big misses very hard, so a few freak orders, like a rider who got lost, can pull the trees around. Huber loss behaves like squared error for small misses and like absolute error for big ones, so it's calmer about freak orders. Since we judge the model by MAE, that's a sensible pick, and the search found it on its own.

Grid or random?

The two winners look completely different (depth 2 with 400 trees vs depth 3 with 180 trees and Huber loss) but score within 0.02 minutes of each other. That's typical: there's usually a wide plateau of good settings, not one perfect peak. Once you're on the plateau, more searching mostly finds luck.

We'll carry the grid winner forward. Its settings are simpler to explain and reproduce, and the 0.02-minute difference is far smaller than the fold-to-fold spread (about 0.09).

Step 5: Unlock the test set, once, and compare honestly

On the test set, tuning cuts gradient boosting's error from 4.61 to 4.50 minutes, but linear regression scores 4.49. The tuned model is genuinely better than the default one; against linear regression it's a tie. It does better on three of four hard groups of orders, but each group is too small to prove it.

The searches are finished, so now, and only now, we score the candidates on the 600 locked test orders:

tuned = grid.best_estimator_
candidates = {"Linear (Part 3)": linear, "Boosted, default": default_boosted,
              "Boosted, grid-tuned": tuned, "Boosted, random-tuned": search.best_estimator_}
for name, model in candidates.items():
    print(f"{name:22s} test MAE {mean_absolute_error(y_test, model.predict(X_test)):.2f}")

Output:

Linear (Part 3)        test MAE 4.49
Boosted, default       test MAE 4.61
Boosted, grid-tuned    test MAE 4.50
Boosted, random-tuned  test MAE 4.51

Cross-validation promised the tuned model would beat linear regression by about 0.18 minutes. On the test set it's 0.01 minutes behind. Which do we believe?

Is a difference real, or luck?

A test set is a sample of 600 orders. A different 600 would give slightly different scores. To see how much, we can bootstrap: draw 600 orders at random with replacement from the test set (some orders twice, some not at all), compare the two models on that sample, and repeat 2,000 times.

def compare(model_a, model_b, rounds=2000):
    """How much better is model_b than model_a on the test set? Returns a 95% range."""
    err_a = np.abs(model_a.predict(X_test) - y_test.values)
    err_b = np.abs(model_b.predict(X_test) - y_test.values)
    rng = np.random.default_rng(0)
    diffs = []
    for _ in range(rounds):
        idx = rng.integers(0, len(err_a), len(err_a))
        diffs.append(err_a[idx].mean() - err_b[idx].mean())
    return np.percentile(diffs, [2.5, 97.5]).round(2)


print("Tuned vs default, minutes saved:", compare(default_boosted, tuned))
print("Tuned vs linear, minutes saved: ", compare(linear, tuned))

Output:

Tuned vs default, minutes saved: [0.01 0.21]
Tuned vs linear, minutes saved:  [-0.19  0.17]

How to read the ranges:

• Tuned vs default: 0.01 to 0.21 minutes saved. The whole range is above zero, so tuning really did help, if only a little.
• Tuned vs linear: −0.19 to +0.17. The range comfortably includes zero. Neither model is reliably better. The 0.18 that cross-validation suggested sits at the very edge of what's plausible.

This is the most important lesson in this tutorial: a single score difference smaller than the noise is not a result. Many real projects "improve" a model by 1% on one test set and celebrate luck.

Where the models differ

An overall tie can hide real differences. Part 3 and Part 4 flagged the hardest orders; let's compare the models there:

cleaned_test = clean_orders(X_test)
hard_groups = {
    "Old Town": cleaned_test["city_zone"] == "Old Town",
    "Rain at peak hour": (cleaned_test["is_raining"] == 1) & (cleaned_test["is_peak_hour"] == 1),
    "Prep time missing": cleaned_test["prep_time_min"].isna(),
    "Longer than 10 km": cleaned_test["distance_km"] > 10,
}
rows = []
for group, mask in hard_groups.items():
    row = {"group": group, "orders": int(mask.sum())}
    for name, model in [("linear", linear), ("default", default_boosted), ("tuned", tuned)]:
        errors = np.abs(model.predict(X_test[mask.values]) - y_test[mask.values])
        row[name] = round(errors.mean(), 2)
    rows.append(row)
print(pd.DataFrame(rows).to_string(index=False))

Output:

            group  orders  linear  default  tuned
         Old Town      82    5.75     5.59   5.23
Rain at peak hour      71    4.30     4.73   4.39
Prep time missing      51    6.55     6.50   6.27
Longer than 10 km      94    4.63     4.73   4.42

• mask is a column of True/False, one per test order; X_test[mask.values] keeps only that group's orders.
• The tuned model is best in three groups, most clearly Old Town (0.5 minutes better than linear), the zone Part 3 found hardest.
• But these groups have only 51–94 orders. Bootstrapping Old Town alone gives a range of −0.24 to +1.34 minutes: promising, not proven.

The decision

We'll deploy the tuned model in Step 7, because it ties overall and does better on the orders customers complain about most. We'll keep the linear model file so we can switch back in seconds, and Part 4's monitoring-by-group will show within weeks whether the Old Town improvement is real. That's a sensible engineering call, not a proof, and your reports should say so.

Step 6: Choose a decision threshold: the promise buffer

A prediction is an average guess, so about half of orders arrive later than predicted. To promise a time customers can trust, add a buffer. Chosen on validation predictions, a 7-minute buffer gets 89.9% of orders there on time; on the locked test set it scores 88.5%.

Why a prediction isn't a promise

Our model predicts the typical delivery time. Real deliveries scatter around it, so if the app promised exactly the prediction, about half the customers would be disappointed. Part 1 met the same idea for classification: the model gives a probability, and you choose the threshold that turns it into a decision. Here the decision is the promise, and the threshold is how many minutes to add.

The buffer is a setting we choose, so it's tuned like a hyperparameter: on validation data, never on the test set.

Get honest validation predictions

val_pred = cross_val_predict(make_boosted(**{k.split("__")[1]: v for k, v in grid.best_params_.items()}),
                             X_train, y_train, cv=cv, n_jobs=-1)
for buffer in [0, 2, 4, 6, 7, 8, 10]:
    on_time = (y_train.values <= val_pred + buffer).mean()
    print(f"buffer {buffer:2d} min  on time {on_time:.1%}  average promise {np.mean(val_pred + buffer):.1f} min")

Output:

buffer  0 min  on time 49.7%  average promise 40.1 min
buffer  2 min  on time 64.2%  average promise 42.1 min
buffer  4 min  on time 76.8%  average promise 44.1 min
buffer  6 min  on time 85.9%  average promise 46.1 min
buffer  7 min  on time 89.9%  average promise 47.1 min
buffer  8 min  on time 93.2%  average promise 48.1 min
buffer 10 min  on time 96.9%  average promise 50.1 min

Why not use grid.best_estimator_.predict(X_train)? That model trained on these very orders, so its predictions are too good. The buffer would come out too small, and customers would find out.

Choosing

There's no formula for the right buffer. It's a business decision the curve makes visible:

• +0 minutes: honest-looking promises, but half of customers wait longer than promised.
• +10 minutes: 97% on time, but every customer is told 10 minutes more than typical, and some will order elsewhere.
• +7 minutes reaches the 90% target that many delivery teams aim for. Each extra minute after that buys less: going from +7 to +10 adds 7 percentage points for 3 more minutes, while +4 to +7 bought 13.

We choose 7 minutes. Now, and only now, check it on the locked test set:

BUFFER = 7
test_on_time = (y_test.values <= tuned.predict(X_test) + BUFFER).mean()
print(f"Test set, buffer {BUFFER}: on time {test_on_time:.1%}")
Test set, buffer 7: on time 88.5%

Very close to the 89.9% the validation data promised: the buffer generalises. (A difference of 1.4 points is well within luck for 600 orders.)

A better buffer later: one buffer for every order is crude. Old Town orders are harder to predict, so they could use a bigger buffer. Gradient boosting can also predict the 90th percentile directly with loss="quantile", alpha=0.9. Both are good practice-project extensions.

Step 7: Redeploy the tuned model

Save the tuned pipeline, point the Part 4 API at it, bump the model version and add the promise time. The old test for 63.8 minutes now fails, which is exactly what tests are for: you update it on purpose, and 7 tests pass.

Save the tuned pipeline

At the end of tune_delivery.py:

joblib.dump(tuned, "delivery_pipeline_tuned.joblib")

tuned is grid.best_estimator_: the whole pipeline (cleaning, preprocessing and the tuned gradient boosting model), refitted on all 2,400 training orders. Copy the file into your Part 4 project folder. Keep the old **delivery_pipeline.joblib** there too: it's your way back.

Update `app.py`

Only a few lines change. Everything else, including all the input checks, stays exactly as in Part 4:

"""Delivery ML Series, Part 5: the Part 4 API, now serving the tuned model."""
import math  # NEW

import joblib
# ... the other imports are unchanged ...

MODEL_VERSION = "part5-gb-tuned-v1"  # CHANGED
PROMISE_BUFFER_MIN = 7  # NEW: chosen in Part 5, Step 6: about 90% of orders arrive on time

pipeline = joblib.load("delivery_pipeline_tuned.joblib")  # CHANGED

app = FastAPI(title="Delivery Time API", version="2.0")  # CHANGED

# ... class Order is unchanged ...


class Prediction(BaseModel):
    predicted_minutes: float
    promised_minutes: int  # NEW
    model_version: str


# ... /health and the input checks in /predict are unchanged ...

    minutes = float(pipeline.predict(row)[0])
    promised = math.ceil(minutes + PROMISE_BUFFER_MIN)  # NEW
    return Prediction(predicted_minutes=round(minutes, 1), promised_minutes=promised,  # CHANGED
                      model_version=MODEL_VERSION)

Run the tests and watch one fail

Run Part 4's tests unchanged:

.F....
FAILED test_app.py::test_predict_matches_part3 - assert 68.8 == 63.8
1 failed, 5 passed

Good! The test was written to catch any change in the answer for the rainy biryani, and the model really has changed its answer: the tuned model predicts 68.8 minutes instead of 63.8. All five input-checking tests still pass, so the safety rules survived the swap.

A failing test after a deliberate change is a prompt to ask, "Is this change what I wanted?" Here it is (Step 5 explains why we accept it), so we update the test on purpose, and add one for the new field:

def test_predict_matches_part5():
    response = client.post("/predict", json=MESSY_ORDER)
    assert response.status_code == 200
    body = response.json()
    assert body["predicted_minutes"] == 68.8
    assert body["promised_minutes"] == 76
    assert body["model_version"] == "part5-gb-tuned-v1"


def test_promise_is_always_later_than_prediction():
    for distance in [1, 5, 15, 30]:
        body = client.post("/predict", json={**MESSY_ORDER, "distance_km": distance}).json()
        assert body["promised_minutes"] >= body["predicted_minutes"] + 7

Replace the old test_predict_matches_part3 with these two and run pytest: 7 passed.

The second test checks a rule rather than one number: whatever the distance, the promise is at least 7 minutes after the prediction. Rule tests keep working when you retrain the model next month; number tests have to be updated each time.

Try it

Start the server (uvicorn app:app --reload) and send the rainy biryani from /docs:

{
  "predicted_minutes": 68.8,
  "promised_minutes": 76,
  "model_version": "part5-gb-tuned-v1"
}

The app can now say "Arriving by 21:31" (20:15 + 76 minutes) and keep that promise about 9 times in 10.

Rolling back

If monitoring shows the tuned model doing worse, go back by changing two lines: the file name in joblib.load and MODEL_VERSION. That's why we kept the old file. (Docker users: rebuild the image as in Part 4.)

Using Part 6? Part 6 explains whichever pipeline the API loads. Its numbers were written for the Part 3 linear model; with the tuned model you'll see different, but equally additive, explanations.

Theory break: how to judge a tuned model

A tuned model's best validation score is always a little too optimistic, because you picked it as the best of many. Judge tuning with five questions: did the test set stay locked, is the gain bigger than the noise, how optimistic was the search, is there room left to improve at all, and is the simpler model just as good?

1. Did the test set stay locked?

Count how many times you looked at the test set while making choices. In this tutorial: zero between Step 1 and Step 5, then once for the models and once for the buffer, both after the choices were made. If you catch yourself re-running a search because the test score disappointed you, the test set has become a validation set, and you need a fresh one.

2. Is the gain bigger than the noise?

Two kinds of noise matter:

If an improvement is smaller than these, report it as a tie.

3. How optimistic was the search? (the winner's curse)

Imagine 48 students of equal ability taking the same practice exam. Someone gets the top mark, partly by luck, and on the real exam they'll probably score a bit lower. Hyperparameter search works the same way: the best of 48 validation scores is the best and the luckiest.

Compare validation and test scores:

The untuned model did about as well on the test set as validation predicted. The tuned one did worse. Part of that is luck in our 600 test orders; part is the winner's curse. The more settings you try, the bigger the curse, which is one reason not to search thousands of settings on a small dataset.

The thorough fix is nested cross-validation: run the whole search inside each fold of an outer cross-validation, so every score comes from data the search never saw. It costs 5× the time, and it's worth knowing the name: you'll see it in research papers.

4. Is there room left to improve?

Part 2 introduced irreducible error: randomness no model can predict. Part 3's delivery_data.py adds random noise with a typical size of 5 minutes to every delivery. For that kind of noise, even a perfect model would have an MAE of about 4.0 minutes (roughly 0.8 × 5).

Our models score 4.36–4.50. So at most about half a minute is left to win, and some of that is lost for good to the missing prep times Part 3 imputed. When a model is this close to the floor, no amount of tuning will produce a big jump. Better data (real prep times, rider GPS) would help more than better settings.

5. Is the simpler model just as good?

When two models tie, the simpler one is usually the better choice: easier to explain, faster to retrain, fewer things to break. Here the choice was close, and we chose the tuned model for its edge on the hard orders (Step 5). Choosing linear regression would also have been defensible. What matters is that the reasoning is written down.

The complete code

One script, **tune_delivery.py**, runs every step of this tutorial: baseline, hand tuning, cross-validation, grid search, random search, the honest test, the promise buffer, and saving the tuned pipeline. It takes about 3–4 minutes on a 2-core machine; the random search is the slowest part.

Put it next to Part 3's delivery_data.py and delivery_features.py. Each section prints a STEP heading so you can match its output to this article.

"""Delivery ML Series, Part 5: tune the gradient boosting model."""
import time

import joblib
import numpy as np
import pandas as pd
from scipy.stats import loguniform, randint, uniform
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import (GridSearchCV, KFold, RandomizedSearchCV, cross_val_predict,
                                     cross_validate, train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

from delivery_data import make_messy_orders
from delivery_features import clean_orders

# ---- Step 1: same data, same split as Part 3 ----
orders = make_messy_orders().drop_duplicates()
X = orders.drop(columns="delivery_min")
y = orders["delivery_min"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

numeric_features = ["distance_km", "prep_time_min", "rider_experience_months", "is_raining",
                    "is_peak_hour", "is_weekend", "distance_x_rain", "distance_x_peak",
                    "distance_x_rain_x_peak"]
categorical_features = ["city_zone", "cuisine"]


def make_preprocess():
    return ColumnTransformer([
        ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric_features),
        ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                              OneHotEncoder(handle_unknown="ignore")), categorical_features),
    ])


def make_boosted(**settings):
    """A Part 3 gradient boosting pipeline with the hyperparameters you choose."""
    return make_pipeline(FunctionTransformer(clean_orders), make_preprocess(),
                         GradientBoostingRegressor(random_state=42, **settings))


linear = make_pipeline(FunctionTransformer(clean_orders), make_preprocess(), LinearRegression())
linear.fit(X_train, y_train)
default_boosted = make_boosted().fit(X_train, y_train)

print("STEP 1")
for name, model in [("Linear", linear), ("Boosted (default)", default_boosted)]:
    train_mae = mean_absolute_error(y_train, model.predict(X_train))
    print(f"{name:18s} train MAE {train_mae:.2f}")

# ---- Step 2: one validation split, by hand ----
print("STEP 2")
X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=0)
for depth in [1, 2, 3, 4, 5, 6]:
    model = make_boosted(max_depth=depth).fit(X_fit, y_fit)
    print(f"max_depth={depth}  validation MAE {mean_absolute_error(y_val, model.predict(X_val)):.2f}")

# ---- Theory break: cross-validation ----
print("CV")
cv = KFold(n_splits=5, shuffle=True, random_state=42)
for depth in [1, 2, 3, 4, 5, 6, 8]:
    scores = cross_validate(make_boosted(max_depth=depth), X_train, y_train, cv=cv,
                            scoring="neg_mean_absolute_error", return_train_score=True, n_jobs=-1)
    print(f"max_depth={depth}  train {-scores['train_score'].mean():.2f}  "
          f"validation {-scores['test_score'].mean():.2f}")

# ---- Step 3: grid search ----
print("STEP 3")
param_grid = {
    "gradientboostingregressor__max_depth": [1, 2, 3, 4],
    "gradientboostingregressor__learning_rate": [0.03, 0.05, 0.1, 0.2],
    "gradientboostingregressor__n_estimators": [100, 200, 400],
}
start = time.perf_counter()
grid = GridSearchCV(make_boosted(), param_grid, cv=cv, scoring="neg_mean_absolute_error", n_jobs=-1)
grid.fit(X_train, y_train)
print(f"Tried {len(grid.cv_results_['params'])} settings in {time.perf_counter() - start:.0f} s")
print("Best settings:", grid.best_params_)
print(f"Best validation MAE: {-grid.best_score_:.3f}")
linear_cv = cross_validate(linear, X_train, y_train, cv=cv, scoring="neg_mean_absolute_error")
print(f"Linear validation MAE: {-linear_cv['test_score'].mean():.3f}")
results = pd.DataFrame(grid.cv_results_)
top = results.sort_values("rank_test_score").head(5)
print(pd.DataFrame({
    "max_depth": top["param_gradientboostingregressor__max_depth"],
    "learning_rate": top["param_gradientboostingregressor__learning_rate"],
    "n_estimators": top["param_gradientboostingregressor__n_estimators"],
    "val_MAE": -top["mean_test_score"].round(3),
    "spread": top["std_test_score"].round(3),
}).to_string(index=False))

# ---- Step 4: random search ----
print("STEP 4")
param_distributions = {
    "gradientboostingregressor__max_depth": randint(1, 6),
    "gradientboostingregressor__learning_rate": loguniform(0.01, 0.3),
    "gradientboostingregressor__n_estimators": randint(100, 800),
    "gradientboostingregressor__min_samples_leaf": randint(1, 50),
    "gradientboostingregressor__subsample": uniform(0.5, 0.5),
    "gradientboostingregressor__loss": ["squared_error", "absolute_error", "huber"],
}
start = time.perf_counter()
search = RandomizedSearchCV(make_boosted(), param_distributions, n_iter=40, cv=cv,
                            scoring="neg_mean_absolute_error", n_jobs=-1, random_state=42)
search.fit(X_train, y_train)
print(f"Tried 40 settings in {time.perf_counter() - start:.0f} s")
for name, value in search.best_params_.items():
    value = round(value, 3) if isinstance(value, float) else value
    print(f"  {name.split('__')[1]}: {value}")
print(f"Best validation MAE: {-search.best_score_:.3f}")

# ---- Step 5: the test set, once ----
print("STEP 5")
tuned = grid.best_estimator_
candidates = {"Linear (Part 3)": linear, "Boosted, default": default_boosted,
              "Boosted, grid-tuned": tuned, "Boosted, random-tuned": search.best_estimator_}
for name, model in candidates.items():
    print(f"{name:22s} test MAE {mean_absolute_error(y_test, model.predict(X_test)):.2f}")


def compare(model_a, model_b, rounds=2000):
    """How much better is model_b than model_a on the test set? Returns a 95% range."""
    err_a = np.abs(model_a.predict(X_test) - y_test.values)
    err_b = np.abs(model_b.predict(X_test) - y_test.values)
    rng = np.random.default_rng(0)
    diffs = []
    for _ in range(rounds):
        idx = rng.integers(0, len(err_a), len(err_a))
        diffs.append(err_a[idx].mean() - err_b[idx].mean())
    return np.percentile(diffs, [2.5, 97.5]).round(2)


print("Tuned vs default, minutes saved:", compare(default_boosted, tuned))
print("Tuned vs linear, minutes saved: ", compare(linear, tuned))

cleaned_test = clean_orders(X_test)
hard_groups = {
    "Old Town": cleaned_test["city_zone"] == "Old Town",
    "Rain at peak hour": (cleaned_test["is_raining"] == 1) & (cleaned_test["is_peak_hour"] == 1),
    "Prep time missing": cleaned_test["prep_time_min"].isna(),
    "Longer than 10 km": cleaned_test["distance_km"] > 10,
}
rows = []
for group, mask in hard_groups.items():
    row = {"group": group, "orders": int(mask.sum())}
    for name, model in [("linear", linear), ("default", default_boosted), ("tuned", tuned)]:
        errors = np.abs(model.predict(X_test[mask.values]) - y_test[mask.values])
        row[name] = round(errors.mean(), 2)
    rows.append(row)
print(pd.DataFrame(rows).to_string(index=False))

# ---- Step 6: choose the promise buffer on validation predictions ----
print("STEP 6")
val_pred = cross_val_predict(make_boosted(**{k.split("__")[1]: v for k, v in grid.best_params_.items()}),
                             X_train, y_train, cv=cv, n_jobs=-1)
for buffer in [0, 2, 4, 6, 7, 8, 10]:
    on_time = (y_train.values <= val_pred + buffer).mean()
    print(f"buffer {buffer:2d} min  on time {on_time:.1%}  average promise {np.mean(val_pred + buffer):.1f} min")
BUFFER = 7
test_on_time = (y_test.values <= tuned.predict(X_test) + BUFFER).mean()
print(f"Test set, buffer {BUFFER}: on time {test_on_time:.1%}")

# ---- Step 7: save for the API ----
joblib.dump(tuned, "delivery_pipeline_tuned.joblib")
print("Saved delivery_pipeline_tuned.joblib")

Run python tune_delivery.py. Every number should match this article except the timings.

Common mistakes

Practice project: tune the electricity model

Tune a gradient boosting model for the home-electricity data from the earlier practice projects, compare it honestly with linear regression, and choose a bill buffer so that about 90% of homes use less than the estimate. Here, unlike the delivery data, tuning uncovers a real weakness in the linear model.

In Parts 1–4 you predicted monthly electricity use (kWh) for homes in Mumbai, Bengaluru, Chennai and Pune, cleaned the messy smart-meter export in Part 3, and served a linear model as an API in Part 4.

What you start with

• energy_data_messy.py (Part 3 practice) and energy_features.py with clean_homes (Part 4 practice).
• Rebuild the data exactly as in the Part 4 practice: drop duplicates, divide Wh readings (≥ 20,000) by 1,000, then train_test_split(test_size=0.2, random_state=42). You should have 1,600 training homes and 400 test homes.
• The numeric columns: home_size_sqm, residents, has_solar, works_from_home, appliance_age_years, ac_hours_per_day, is_summer, ac_x_size; the category column: city.

Your task

• 1. Baseline (Step 1). Write a make_boosted(**settings) factory for the energy pipeline. Compare the training MAE of linear regression and default gradient boosting, and their 5-fold validation MAE (KFold(5, shuffle=True, random_state=42)).
• 2. Validation curve (theory break). Cross-validate max_depth from 1 to 6. Which depth is best? Where does the gap between training and validation error get large?
• 3. Grid search (Step 3). Use the same 48-setting grid as this tutorial. What wins, and with what validation MAE?
• 4. Test once (Step 5). Score linear, default and tuned on the test set. Bootstrap tuned vs linear. Is it a clear win?
• 5. Look at groups (Step 5). Compare test MAE for homes with and without solar panels, for linear and tuned.
• 6. Bill buffer (Step 6). With cross_val_predict, find the smallest buffer (in kWh) that keeps about 90% of training homes at or below the estimate. Check it once on the test set.
• Stretch: add estimate_with_buffer_kwh to your Part 4 energy_app.py, bump the model version, and update the tests on purpose

Expected results

Your numbers should match these (timings will differ):

Check your thinking

• Task 4: here, switching to gradient boosting helps more than tuning it does. Default and tuned boosting are tied; both beat linear regression by about 1–1.5 kWh. Don't credit the tuning for what the algorithm did.
• Task 5: this is the real story. Solar panels cut a bill by a percentage, but a linear model can only subtract a fixed amount, so it gets solar homes badly wrong. Trees can learn "solar × size" on their own, and the error for solar homes drops from 43.0 to 29.5 kWh. The cost: homes without solar get a little worse (29.5 → 32.4). An overall tie can hide a big shift between groups, which is why you always check them.
• Your Part 4 home (the May home in "Bombay" with solar): linear said 636.0 kWh; the tuned model says 520.4. Which do you trust more for a solar home, given Task 5?

Frequently asked questions

What is hyperparameter tuning?

Choosing the settings a model can't learn by itself, such as tree depth or learning rate, by trying several options and keeping the one with the best validation score. The model's own parameters are still learned from the data as usual.

What is the difference between GridSearchCV and RandomizedSearchCV?

GridSearchCV tries every combination of the values you list. RandomizedSearchCV tries a fixed number of random combinations from ranges you give. Use grid search for 2–3 hyperparameters, random search for more.

Why use cross-validation instead of one validation set?

One validation set is a small sample, so its winner can be luck: in this tutorial, two different splits picked depth 2 and depth 4. Cross-validation uses every training row for validation once and averages the results, which is much more stable.

How many folds should I use?

5 is the usual default and a good balance. Use 10 for small datasets if you can afford the time; 3 if training is very slow.

Can I tune on the test set if I'm careful?

No. Any choice made by looking at the test score makes that score optimistic. Tune with cross-validation on the training data and use the test set once, at the end.

Why didn't tuning make a big difference here?

Because the models are already close to the irreducible error: random noise in delivery times that no model can predict. When that's the case, better data helps more than better settings.

Glossary

Series roadmap

Next up, Part 6: the API now promises "76 minutes", and customers will ask why so long? Part 6 opens up the model with permutation importance and SHAP, splitting a prediction into reasons like "rain +9 minutes, Old Town +8", and adds them to the API.

Conclusion

You can now tune a model the way professionals do, and, just as important, judge whether the tuning worked:

• Hyperparameters are tuned on validation data, never on the test set.
• Cross-validation gives stable scores; a single validation split can pick the wrong winner.
• The validation curve shows underfitting on one side and overfitting on the other; the best setting is the bottom of the U.
• Grid search tries everything; random search covers more ground for the same time. Both found a plateau of equally good settings.
• Test once, and compare with a range. Tuning made gradient boosting better than its default version, and tied it with linear regression overall.
• Turn predictions into decisions deliberately. A 7-minute buffer, chosen on out-of-fold predictions, keeps about 90% of promises.

The most valuable result in this part isn't a lower number. It's knowing when a number is real.