How to Use XGBoost and LightGBM in Python: A Beginner's Guide to Gradient Boosting (Food Delivery ML Series, Part 7)

The short answer

Gradient boosting builds hundreds of small decision trees, each one fixing the mistakes of the trees before it. XGBoost and LightGBM are fast, battle-tested versions of that idea. On our delivery data, tuned XGBoost reaches a test error of 4.43 minutes, the best in this series, and is the most accurate model on every hard group of orders. It searches settings about four times faster than scikit-learn's version. Overall it still only edges linear regression, and this tutorial shows you why.

Quick answer: To use XGBoost or LightGBM in Python: (1) pip install xgboost lightgbm; (2) create XGBRegressor() or LGBMRegressor() and use it like any scikit-learn model, inside a pipeline; (3) don't trust the defaults: they overfit small datasets; (4) use early stopping to choose the number of trees; (5) let the library handle missing values and categories itself; (6) tune with RandomizedSearchCV and compare against a simple baseline on the test set, once.

What you will build

By the end of this tutorial you will have:

The rainy Saturday biryani now gets:

{
  "predicted_minutes": 69.5,
  "promised_minutes": 77,
  "model_version": "part7-xgb-v1"
}

How this part fits the series

This is Part 7 of the Delivery ML series. Part 3 introduced gradient boosting; Part 5 tuned scikit-learn's version and found it tied with linear regression. Part 7 opens up how boosting works, then moves to XGBoost and LightGBM, the libraries most used in industry and in data-science competitions on tables of data. Part 6's SHAP tools work on these models too. Part 8 turns the API's predictions into friendly customer messages with generative AI.

Prerequisites

You need Parts 3 and 5 of this series (Part 4 for the redeploy step), two new libraries, and about 90 minutes.

From earlier parts

Knowledge

Software

On a Mac, LightGBM may also need OpenMP: brew install libomp. In Google Colab, both libraries are usually installed already.

Hardware, time and cost

• Any laptop from the last five years. The whole script runs in about a minute on 2 CPU cores.
• Free: no cloud services, GPUs or API keys.

Checklist before you start

• python -c "import xgboost, lightgbm; print(xgboost.__version__, lightgbm.__version__)" prints two versions
• Part 3's delivery_data.py and delivery_features.py are in your project folder
• You can say what cross_validate and RandomizedSearchCV do
• (For Step 7) Part 5's API tests pass

Theory break: how gradient boosting works

Gradient boosting starts with a rough guess (the average) and then adds small decision trees one at a time. Each new tree is trained on the current mistakes, the residuals, and only a small fraction of its correction is added. After hundreds of trees, the small corrections add up to an accurate model.

A golf analogy

You're putting a golf ball towards a hole you can't quite see.

1. Your first putt, aimed at the middle of the green, stops 20 metres short. That's the average prediction.
2. A friend tells you "20 metres short, a bit left". You aim your next putt to fix that error. That's tree 1, trained on the residuals.
3. But you deliberately hit it gently, covering only part of the distance, so one misjudged putt can't send you into the lake. That's the learning rate.
4. You keep going: each putt fixes what's left. After many small putts, you're in the hole, or close.

The same thing for delivery times

(The tree 1 number, 33.3 minutes, is real: you'll see that exact tree in Step 1.)

The "gradient" in the name comes from calculus: for squared error, the residual is exactly the direction that reduces the error fastest, the same idea as gradient descent. You don't need the maths; "fit the next tree to what's still wrong" is the whole idea.

Why small trees and small steps?

• Small trees (depth 2–4) are weak learners: each one is only slightly better than nothing, so no single tree can memorise the training data.
• A small learning rate means each tree only nudges the prediction. Mistakes made by one tree get corrected by later ones.
• Together they make boosting powerful but easy to overfit if you add too many trees, which is why Step 4 teaches early stopping.

Boosting vs random forests

You may have heard of random forests, which also use many trees. The difference:

Where XGBoost and LightGBM come in

scikit-learn's GradientBoostingRegressor, which we used in Parts 3 and 5, follows the recipe above directly. XGBoost (2014) and LightGBM (2016, from Microsoft) follow the same recipe with clever engineering:

• Histograms: instead of trying every possible split point, they first sort each feature into about 256 bins. Much faster.
• Regularisation: extra penalties that discourage trees from making extreme corrections.
• Built-in handling of missing values and categories (Step 5).
• Early stopping built into training (Step 4).

They're the go-to tools for tabular data: rows and columns like our orders, as opposed to images or text.

Step 1: Set up, then build gradient boosting by hand

A loop of 10 lines, "fit a small tree to the residuals, add 10% of its prediction", repeated 400 times, cuts the training error from 13.56 to 3.77 minutes. Its predictions are identical to scikit-learn's **GradientBoostingRegressor** with the same settings.

Setup

Create boost_delivery.py next to Part 3's files. The start is familiar from Part 5, with two changes: the numeric features are split into base and interaction lists (for the experiment after Step 3), and there's a second preprocessing recipe without the scaler, because trees don't care about scale.

import time

import joblib
import numpy as np
import pandas as pd
from lightgbm import LGBMRegressor, early_stopping
from scipy.stats import loguniform, randint, uniform
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import (KFold, RandomizedSearchCV, cross_val_predict, cross_validate,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeRegressor, export_text
from xgboost import XGBRegressor

from boosting_features import prepare_for_boosting
from delivery_data import make_messy_orders
from delivery_features import clean_orders

orders = make_messy_orders().drop_duplicates()
X = orders.drop(columns="delivery_min")
y = orders["delivery_min"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

base_features = ["distance_km", "prep_time_min", "rider_experience_months", "is_raining",
                 "is_peak_hour", "is_weekend"]
interaction_features = ["distance_x_rain", "distance_x_peak", "distance_x_rain_x_peak"]
numeric_features = base_features + interaction_features
categorical_features = ["city_zone", "cuisine"]
cv = KFold(n_splits=5, shuffle=True, random_state=42)


def linear_preprocess(numeric):
    return ColumnTransformer([
        ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric),
        ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                              OneHotEncoder(handle_unknown="ignore")), categorical_features),
    ])


def tree_preprocess(numeric):
    """Like Part 3's, minus the scaler: trees don't care about scale."""
    return ColumnTransformer([
        ("num", SimpleImputer(strategy="median"), numeric),
        ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                              OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
         categorical_features),
    ], verbose_feature_names_out=False)


def score(model, label):
    """Training MAE and 5-fold validation MAE. The test set stays locked until Step 6."""
    result = cross_validate(model, X_train, y_train, cv=cv, scoring="neg_mean_absolute_error",
                            return_train_score=True)
    train, val = -result["train_score"].mean(), -result["test_score"].mean()
    print(f"{label:28s} train {train:.2f}  validation {val:.3f}")
    return val

Boosting in 10 lines

prep = tree_preprocess(numeric_features)
X_fit_matrix = prep.fit_transform(clean_orders(X_train))
names = list(prep.get_feature_names_out())
target = y_train.values

prediction = np.full(len(target), target.mean())
print(f"Start: predict the average, {target.mean():.2f} min. Training MAE {np.abs(target - prediction).mean():.2f}")
trees = []
for round_number in range(1, 401):
    residuals = target - prediction
    tree = DecisionTreeRegressor(max_depth=2, random_state=42).fit(X_fit_matrix, residuals)
    prediction += 0.1 * tree.predict(X_fit_matrix)
    trees.append(tree)
    if round_number in (1, 2, 3, 10, 50, 100, 200, 400):
        print(f"After {round_number:3d} trees: training MAE {np.abs(target - prediction).mean():.2f}")
print(export_text(trees[0], feature_names=names, decimals=1))

Output:

Start: predict the average, 40.12 min. Training MAE 13.56
After   1 trees: training MAE 12.74
After   2 trees: training MAE 12.05
After   3 trees: training MAE 11.45
After  10 trees: training MAE 8.73
After  50 trees: training MAE 4.83
After 100 trees: training MAE 4.13
After 200 trees: training MAE 3.93
After 400 trees: training MAE 3.77
|--- distance_km <= 6.5
|   |--- distance_km <= 3.5
|   |   |--- value: [-16.4]
|   |--- distance_km >  3.5
|   |   |--- value: [-5.0]
|--- distance_km >  6.5
|   |--- distance_x_rain <= 8.8
|   |   |--- value: [8.9]
|   |--- distance_x_rain >  8.8
|   |   |--- value: [33.3]

Read the first tree

The first tree is a perfectly sensible summary of delivery times:

• Short orders (≤ 3.5 km) are about 16 minutes faster than average.
• Medium orders (3.5–6.5 km) are about 5 minutes faster.
• Long orders in clear weather are about 9 minutes slower.
• Long orders in rain (distance_x_rain > 8.8 means at least 8.8 rainy kilometres) are about 33 minutes slower.

Only 10% of each correction is used, so after tree 1 a long rainy order moves from 40.1 to 43.4 minutes. Tree 2 then fixes what's left, and so on.

Is the loop really "gradient boosting"?

Yes. Fit scikit-learn's model with the same settings and compare:

sk_model = GradientBoostingRegressor(max_depth=2, n_estimators=400, random_state=42)
sk_model.fit(X_fit_matrix, target)
print(np.abs(sk_model.predict(X_fit_matrix) - prediction).max())
5.684341886080802e-14

That's 0.00000000000006 minutes: the largest difference between our hand-built predictions and scikit-learn's, across all 2,400 orders. It's just the tiny rounding every computer makes with decimals. The two models are the same. There's no magic inside; the libraries just do the same loop faster, with more options. (This check isn't in the complete script; it's a one-off.)

The training error keeps falling with every tree. But as Part 5 showed, training error always falls. Whether all 400 trees help on new orders is a question for Step 4.

Step 2: Meet XGBoost, and don't trust its defaults

**XGBRegressor** drops into a scikit-learn pipeline like any other model. With its default settings it fits the training orders almost perfectly (1.11 minutes of error) and validates worst of all (4.90). The defaults are built for big datasets; ours has 2,400 rows.

score(make_pipeline(FunctionTransformer(clean_orders), linear_preprocess(numeric_features), LinearRegression()),
      "Linear regression")
score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    GradientBoostingRegressor(max_depth=2, n_estimators=400, random_state=42)),
      "scikit-learn, Part 5 tuned")
score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    XGBRegressor(random_state=42)), "XGBoost, default")

Output:

Linear regression            train 4.50  validation 4.534
scikit-learn, Part 5 tuned   train 3.70  validation 4.358
XGBoost, default             train 1.11  validation 4.903

Why the defaults overfit here

With deep trees and big steps, XGBoost memorises the training orders: 1.11 minutes of training error, but 4.90 on validation. That's the overfitting gap from Part 5's validation curve, at its most extreme.

This isn't a flaw in XGBoost. Its defaults suit datasets with hundreds of thousands of rows, where a 64-group tree still has plenty of orders in each group. On small data, always tune. We'll fix it with early stopping (Step 4) and a search (Step 6).

Step 3: Meet LightGBM

LightGBM grows its trees differently, splitting whichever leaf helps most rather than one level at a time. Its defaults overfit less than XGBoost's (validation 4.56) but still trail Part 5's tuned model (4.36).

score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    LGBMRegressor(random_state=42, verbose=-1)), "LightGBM, default")

Output:

LightGBM, default            train 2.85  validation 4.564

• **LGBMRegressor** is LightGBM's scikit-learn-style regressor.
• **verbose=-1** silences LightGBM's training messages, which are useful for debugging but noisy in a tutorial.

Level-wise vs leaf-wise trees

The two libraries grow trees in different orders:

Leaf-wise growth is often more accurate for the same number of leaves, and faster on big datasets. But it can dig very deep into small corners of the data, so on small datasets keep num_leaves modest and min_child_samples (the fewest orders allowed in a leaf, default 20) reasonably high.

The scoreboard so far (validation MAE, minutes)

Out of the box, neither famous library beats a carefully tuned simple one. Before we tune them, let's see what boosting does that linear regression can't.

Theory break: automatic interactions

An interaction is when one feature's effect depends on another: rain costs more minutes on a long trip than a short one. Linear regression can only learn interactions you build by hand. Boosted trees find them on their own: remove our three hand-made interaction features and linear regression gets 0.52 minutes worse, while boosting gets just 0.01 minutes worse.

What linear regression can't do

Part 2 explained that linear regression adds up one fixed effect per feature: "+x minutes for rain", whatever the distance. Real deliveries don't work like that. In rain, every extra kilometre is slower. That's why Parts 2 and 3 created distance_x_rain, distance_x_peak and distance_x_rain_x_peak by hand: so a linear model could learn "rain matters more on long trips".

The catch: someone had to know that interaction existed and type it in. With 10 features there are 45 possible pairs, and 120 possible triples. Nobody types them all.

Why trees do it automatically

Look again at the first tree from Step 1. Its right-hand branch asks:

Is distance > 6.5 km? and then is distance_x_rain > 8.8?

Any tree deeper than one question is an interaction: "if this and that". A depth-2 tree can say "long and rainy = +33 minutes", which no single linear weight can express. That's also why Part 5's validation curve showed depth-1 trees, which ask a single question, doing worst by far (5.33 minutes).

The experiment

Take the interaction features away and see who suffers:

for label, numeric in [("with interactions", numeric_features), ("base features only", base_features)]:
    score(make_pipeline(FunctionTransformer(clean_orders), linear_preprocess(numeric), LinearRegression()),
          f"Linear, {label}")
    score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric),
                        GradientBoostingRegressor(max_depth=2, n_estimators=400, random_state=42)),
          f"Boosting, {label}")

Output:

Linear, with interactions    train 4.50  validation 4.534
Boosting, with interactions  train 3.70  validation 4.358
Linear, base features only   train 5.02  validation 5.053
Boosting, base features only train 3.81  validation 4.371

The loop builds each pipeline twice: once with all 9 numeric features, once with only the 6 base ones. clean_orders still creates the interaction columns, but the ColumnTransformer only passes on the columns it's given and drops the rest.

Without our help, linear regression loses half a minute, more than every difference between models in Part 5. Boosting barely notices: it rebuilt the interactions from distance, rain and peak hour by itself.

So should you stop making interaction features?

Not quite:

• For boosting, hand-made interactions rarely hurt and sometimes help a little, as here.
• For linear models, they're essential, and you may want a linear model as a fallback or for explainability.
• The real lesson: if linear regression is doing well, it may be because someone already did the thinking boosting would do for free. On a new problem where nobody knows the interactions yet, boosting usually wins by a lot more than it does here.

That last point explains this whole series' results: linear regression ties with boosting on our data because Part 2 already handed it the interactions that matter.

Step 4: Let the model choose its number of trees with early stopping

Instead of guessing how many trees to build, ask for far too many (2,000) and give the model a validation set to watch. It stops when 50 trees in a row fail to improve validation error. XGBoost stops at 354 trees, LightGBM at 427, without any search.

In Part 5, n_estimators was a hyperparameter we searched. Early stopping makes it cheaper: one training run, watching validation error as it goes.

X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=0)
prep = tree_preprocess(numeric_features)
fit_matrix = prep.fit_transform(clean_orders(X_fit))
val_matrix = prep.transform(clean_orders(X_val))
xgb = XGBRegressor(n_estimators=2000, learning_rate=0.05, max_depth=2, early_stopping_rounds=50,
                   eval_metric="mae", random_state=42)
xgb.fit(fit_matrix, y_fit, eval_set=[(val_matrix, y_val)], verbose=False)
history = xgb.evals_result()["validation_0"]["mae"]
for n in [50, 100, 200, 300]:
    print(f"After {n} trees: validation MAE {history[n - 1]:.3f}")
print(f"Best: {xgb.best_iteration + 1} trees, validation MAE {xgb.best_score:.3f}. Stopped after {len(history)} trees.")

lgbm = LGBMRegressor(n_estimators=2000, learning_rate=0.05, num_leaves=4, random_state=42, verbose=-1)
lgbm.fit(fit_matrix, y_fit, eval_X=(val_matrix,), eval_y=(y_val,), eval_metric="l1",
         callbacks=[early_stopping(stopping_rounds=50, verbose=False)])
print(f"LightGBM best: {lgbm.best_iteration_} trees, validation MAE {lgbm.best_score_['valid_0']['l1']:.3f}")

Output:

After 50 trees: validation MAE 6.511
After 100 trees: validation MAE 5.182
After 200 trees: validation MAE 4.502
After 300 trees: validation MAE 4.412
Best: 354 trees, validation MAE 4.396. Stopped after 404 trees.
LightGBM best: 427 trees, validation MAE 4.369

When it stops, the model keeps the best version, from tree 354, not tree 404. Predictions automatically use only the best trees.

Reading the curve

• Trees 1–150: both errors fall fast. Each tree fixes real patterns.
• Trees 150–354: validation improves slowly, about 0.3 minutes over 200 trees.
• After 354: training error keeps creeping down, but validation is flat. The extra trees are learning noise. After waiting 50 trees for an improvement that never comes, XGBoost stops.

Why patience?

Validation error wobbles a little from tree to tree. Stopping at the very first tree that doesn't improve would quit too early, on a wobble. A patience of 50 says "give it 50 more chances". Typical values: 20–100. With a smaller learning rate, use more patience, because each tree changes less.

One caution

Early stopping chose 354 by looking at those 480 validation orders, so its validation score (4.396) is slightly optimistic, the winner's curse again from Part 5. That's fine for choosing; for an honest score, cross-validate or use the locked test set later.

Step 5: Let the library handle missing values and categories

XGBoost and LightGBM can read missing values and text categories directly: no imputer, no one-hot encoding. On our data that's slightly better (validation 4.374 vs 4.397) and simpler. You need one small helper, and one line in it that is easy to forget.

Why skip the imputer?

Part 3 filled missing prep times with the median. That works, but it throws information away: Part 3 found that orders with a missing prep time behave differently (they had the biggest errors). Filling in 17 minutes hides that.

XGBoost and LightGBM do something smarter. At every split, they try sending the missing values left, then right, and keep whichever direction reduces the error more. If "prep time unknown" really means "probably a slow kitchen", the trees can learn that.

Why skip one-hot encoding?

One-hot turns city_zone into four 0/1 columns, so a tree needs several questions to say "Old Town or Suburbs". With native categories, a tree can split on a group of categories in one question: "is the zone one of {Old Town, Suburbs}?".

The helper

Create boosting_features.py:

"""Turns raw orders into what XGBoost needs: clean numbers, NaN for missing, categories for text."""
import pandas as pd

from delivery_features import clean_orders

NUMERIC_FEATURES = ["distance_km", "prep_time_min", "rider_experience_months", "is_raining",
                    "is_peak_hour", "is_weekend", "distance_x_rain", "distance_x_peak",
                    "distance_x_rain_x_peak"]
ZONE_NAMES = ["Central", "Old Town", "Suburbs", "Tech Park"]
CUISINE_NAMES = ["Biryani", "Burger", "Chinese", "Pizza", "South Indian", "Sushi"]


def prepare_for_boosting(df):
    """Clean, keep missing values as NaN, and mark text columns as pandas categories."""
    df = clean_orders(df)
    df[NUMERIC_FEATURES] = df[NUMERIC_FEATURES].astype(float)
    df["city_zone"] = pd.Categorical(df["city_zone"], categories=ZONE_NAMES)
    df["cuisine"] = pd.Categorical(df["cuisine"], categories=CUISINE_NAMES)
    return df[NUMERIC_FEATURES + ["city_zone", "cuisine"]]

Why a separate file? Because the saved pipeline stores a reference to this function, and the API must be able to import it, the same Can't get attribute lesson from Part 4. If you define it inside the training script, the API can't load the model.

Why the fixed category list matters

Without categories=, pandas builds the list from whatever values it sees. A batch of orders with no Tech Park order would get a three-category column, and "Suburbs" could end up with a different internal code than during training. The model would silently read it as the wrong zone. A fixed list means "Old Town" is always the same category.

Why `.astype(float)` matters

When the API receives "prep_time_min": null, pandas makes a one-row column containing None, whose type is object, not a number. scikit-learn's imputer doesn't mind. XGBoost does:

DataFrame.dtypes for data must be int, float, bool or category. ... Invalid columns:prep_time_min: object

LightGBM gives a similar error: pandas dtypes must be int, float or bool. Training would work fine, because the training data has real numbers alongside the gaps, and then the very first API request with a missing prep time would crash. .astype(float) turns None into NaN, which both libraries understand as missing.

Compare

score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    XGBRegressor(n_estimators=354, learning_rate=0.05, max_depth=2, random_state=42)),
      "XGBoost, imputed + one-hot")
score(make_pipeline(FunctionTransformer(prepare_for_boosting),
                    XGBRegressor(n_estimators=354, learning_rate=0.05, max_depth=2,
                                 enable_categorical=True, random_state=42)),
      "XGBoost, native")

Output:

XGBoost, imputed + one-hot   train 3.92  validation 4.397
XGBoost, native              train 3.89  validation 4.374

• **n_estimators=354** comes from early stopping in Step 4.
• **enable_categorical=True** tells XGBoost to use its category support. Without it, XGBoost refuses category columns. LightGBM reads pandas categories automatically.
• The native pipeline is two steps instead of three: no ColumnTransformer at all.

A small gain (0.02 minutes) and a simpler pipeline. We'll use the native version from here on.

Step 6: Tune both libraries, then compare everything honestly

A 30-setting random search takes about 20 seconds for each library, roughly four times faster per setting than scikit-learn's boosting in Part 5. On the test set, tuned XGBoost and LightGBM both score 4.43 minutes, the best in the series, and XGBoost wins in every hard group. But against linear regression the bootstrap range still includes zero.

Search

The search works exactly as in Part 5, now with the native pipeline from Step 5:

searches = {
    "XGBoost": (XGBRegressor(enable_categorical=True, random_state=42, n_jobs=1), {
        "xgbregressor__n_estimators": randint(100, 1000),
        "xgbregressor__learning_rate": loguniform(0.01, 0.2),
        "xgbregressor__max_depth": randint(1, 6),
        "xgbregressor__min_child_weight": randint(1, 30),
        "xgbregressor__subsample": uniform(0.5, 0.5),
        "xgbregressor__colsample_bytree": uniform(0.5, 0.5),
        "xgbregressor__reg_lambda": loguniform(0.01, 10),
    }),
    "LightGBM": (LGBMRegressor(random_state=42, verbose=-1, n_jobs=1), {
        "lgbmregressor__n_estimators": randint(100, 1000),
        "lgbmregressor__learning_rate": loguniform(0.01, 0.2),
        "lgbmregressor__num_leaves": randint(2, 32),
        "lgbmregressor__min_child_samples": randint(5, 60),
        "lgbmregressor__subsample": uniform(0.5, 0.5),
        "lgbmregressor__subsample_freq": [1],
        "lgbmregressor__colsample_bytree": uniform(0.5, 0.5),
        "lgbmregressor__reg_lambda": loguniform(0.01, 10),
    }),
}
best = {}
for name, (model, space) in searches.items():
    start = time.perf_counter()
    search = RandomizedSearchCV(make_pipeline(FunctionTransformer(prepare_for_boosting), model), space,
                                n_iter=30, cv=cv, scoring="neg_mean_absolute_error", n_jobs=-1,
                                random_state=42)
    search.fit(X_train, y_train)
    best[name] = search.best_estimator_
    print(f"{name}: 30 settings in {time.perf_counter() - start:.0f} s, validation MAE {-search.best_score_:.3f}")
    for key, value in search.best_params_.items():
        value = round(value, 3) if isinstance(value, float) else value
        print(f"    {key.split('__')[1]}: {value}")

Output (timings will differ):

XGBoost: 30 settings in 22 s, validation MAE 4.275
    colsample_bytree: 0.656
    learning_rate: 0.047
    max_depth: 2
    min_child_weight: 4
    n_estimators: 921
    reg_lambda: 0.012
    subsample: 0.921
LightGBM: 30 settings in 19 s, validation MAE 4.310
    colsample_bytree: 0.5
    learning_rate: 0.029
    min_child_samples: 7
    n_estimators: 458
    num_leaves: 7
    reg_lambda: 0.341
    subsample: 0.709
    subsample_freq: 1

The new hyperparameters:

Both winners agree with Part 5's lesson: small trees, small steps, many trees (XGBoost: depth 2, learning rate 0.047, 921 trees).

Speed: Part 5's scikit-learn random search needed about 125 seconds for 40 settings, about 3 seconds per setting. Here it's about 0.7 seconds per setting. On 2,400 orders that's a convenience; on 2 million it's the difference between minutes and hours.

Unlock the test set

All choices are made. Now, once, the test set:

linear = make_pipeline(FunctionTransformer(clean_orders), linear_preprocess(numeric_features),
                       LinearRegression()).fit(X_train, y_train)
sklearn_tuned = make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                              GradientBoostingRegressor(max_depth=2, n_estimators=400,
                                                        random_state=42)).fit(X_train, y_train)
candidates = {"Linear (Part 3)": linear, "scikit-learn (Part 5)": sklearn_tuned,
              "XGBoost, tuned": best["XGBoost"], "LightGBM, tuned": best["LightGBM"]}
errors = {name: np.abs(model.predict(X_test) - y_test.values) for name, model in candidates.items()}
for name, err in errors.items():
    print(f"{name:22s} test MAE {err.mean():.2f}")


def saved_vs_linear(name, rounds=2000):
    """Minutes saved per order compared with linear regression: estimate and 95% range."""
    rng = np.random.default_rng(0)
    diffs = []
    for _ in range(rounds):
        idx = rng.integers(0, len(y_test), len(y_test))
        diffs.append(errors["Linear (Part 3)"][idx].mean() - errors[name][idx].mean())
    gap = errors["Linear (Part 3)"].mean() - errors[name].mean()
    low, high = np.percentile(diffs, [2.5, 97.5])
    return gap, low, high


for name in ["scikit-learn (Part 5)", "XGBoost, tuned", "LightGBM, tuned"]:
    gap, low, high = saved_vs_linear(name)
    print(f"{name:22s} saves {gap:+.2f} min per order (95% range {low:+.2f} to {high:+.2f})")

Output:

Linear (Part 3)        test MAE 4.49
scikit-learn (Part 5)  test MAE 4.50
XGBoost, tuned         test MAE 4.43
LightGBM, tuned        test MAE 4.43
scikit-learn (Part 5)  saves -0.00 min per order (95% range -0.19 to +0.17)
XGBoost, tuned         saves +0.07 min per order (95% range -0.11 to +0.24)
LightGBM, tuned        saves +0.06 min per order (95% range -0.09 to +0.22)

This is Part 5's bootstrap, reorganised: errors stores each model's per-order errors once, and saved_vs_linear resamples them. ("−0.00" means a tiny loss that rounds to zero.)

XGBoost and LightGBM moved the estimate in the right direction, by about 4 seconds an order, but every range still crosses zero. Overall, it's still a tie.'

Where it matters

cleaned_test = clean_orders(X_test)
hard_groups = {
    "Old Town": cleaned_test["city_zone"] == "Old Town",
    "Rain at peak hour": (cleaned_test["is_raining"] == 1) & (cleaned_test["is_peak_hour"] == 1),
    "Prep time missing": cleaned_test["prep_time_min"].isna(),
    "Longer than 10 km": cleaned_test["distance_km"] > 10,
}
rows = [{"group": group, "orders": int(mask.sum()),
         **{name: round(err[mask.values].mean(), 2) for name, err in errors.items()}}
        for group, mask in hard_groups.items()]
print(pd.DataFrame(rows).to_string(index=False))

(The script prints the same numbers with the full model names as column headings. **{...} unpacks one column per model into each row.)

XGBoost has the lowest error in all four hard groups:

• Old Town: 0.73 minutes better than linear, the zone that has troubled the model since Part 3.
• Prep time missing: 0.65 better: Step 5's native missing-value handling at work. Instead of pretending the prep time was 17 minutes, the trees learned what "unknown" means.
• It even edges linear regression on rainy rush-hour orders, the one group where Part 5's model lost.

Each group is small, so no single row proves anything (Part 5's bootstrap of Old Town alone ranged from −0.24 to +1.34). But a model that is best in every hard group and best overall is a consistent pattern, not one lucky number.

Step 7: Decide, re-check the buffer, and redeploy

We switch the API from Part 5's scikit-learn model to tuned XGBoost: best overall, best in every hard group, just as fast to serve. A new model needs its promise buffer re-checked: 7 minutes still gives 90.5% on time in validation and 88.8% on the test set. Four small changes to the API, and all 7 tests pass.

The decision

That's a reasonable upgrade over Part 5's model. Against linear regression, it remains a judgement call, as Part 5 explained, and we keep the old model files for rollback.

Re-check the buffer

The 7-minute buffer was chosen for Part 5's model. A different model makes different mistakes, so check it again with out-of-fold predictions:

val_pred = cross_val_predict(best["XGBoost"], X_train, y_train, cv=cv)
for buffer in [6, 7, 8]:
    print(f"buffer {buffer} min: on time {(y_train.values <= val_pred + buffer).mean():.1%}")
BUFFER = 7
test_on_time = (y_test.values <= best["XGBoost"].predict(X_test) + BUFFER).mean()
print(f"Test set, buffer {BUFFER}: on time {test_on_time:.1%}")
joblib.dump(best["XGBoost"], "delivery_pipeline_xgb.joblib")

Output:

buffer 6 min: on time 86.2%
buffer 7 min: on time 90.5%
buffer 8 min: on time 93.6%
Test set, buffer 7: on time 88.8%

cross_val_predict refits a fresh copy of the tuned pipeline on each fold, so every prediction comes from a model that never saw that order. 7 minutes still hits the 90% target, so the buffer stays.

Update the API

Copy delivery_pipeline_xgb.joblib and boosting_features.py into your API folder. Then four changes:

1. **app.py**: three lines.

MODEL_VERSION = "part7-xgb-v1"  # CHANGED
PROMISE_BUFFER_MIN = 7  # re-checked in Part 7, Step 7: 90.5% on time with XGBoost

pipeline = joblib.load("delivery_pipeline_xgb.joblib")  # CHANGED

app = FastAPI(title="Delivery Time API", version="2.1")  # CHANGED

Everything else, including Part 4's input checks and Part 5's promise, is unchanged. app.py doesn't import boosting_features itself; joblib imports it when it loads the pipeline, which is why the file must be in the folder.

2. **requirements.txt**: add one line.

xgboost==3.4.1

Pinned to the exact version you trained with, as Part 4 taught. A model saved by one XGBoost version may not load in a much newer one. (Docker users: rebuild the image.)

3. **test_app.py**: the rainy biryani's expected answers change.

def test_predict_matches_part7():
    response = client.post("/predict", json=MESSY_ORDER)
    assert response.status_code == 200
    body = response.json()
    assert body["predicted_minutes"] == 69.5
    assert body["promised_minutes"] == 77
    assert body["model_version"] == "part7-xgb-v1"

4. Run the tests: pytest → 7 passed. The rule test from Part 5 ("the promise is always at least 7 minutes after the prediction") passed without any change, exactly as promised.

Try it

{
  "predicted_minutes": 69.5,
  "promised_minutes": 77,
  "model_version": "part7-xgb-v1"
}

The rainy biryani arrives "by 21:32". Notice the request still sends "prep_time_min": null: that's the case that would crash without .astype(float) in Step 5. The prediction test sends it on every run, so if someone ever deletes that line, the tests fail before customers notice.

Part 6 and SHAP: Part 6's make_explainer works on this pipeline unchanged, because it only calls pipeline.predict on raw orders. XGBoost also has a much faster built-in explainer (pred_contribs=True), which is worth exploring once the basics are clear.

Theory break: when is simple enough?

Boosting wins big when the patterns are complicated and nobody has spelled them out. Linear regression holds its own when the data is small, someone already built the right features, and the remaining error is mostly noise. Our delivery data is the second kind, which is exactly why every model here lands within seconds of the others.

The part is titled "When simple isn't enough", so here's the honest answer to the question it raises.

Three reasons our models tie

1. The interactions were handed over. The theory break after Step 3 showed linear regression only keeps up because of the three interaction features from Parts 2 and 3. Take them away and boosting leads by 0.7 minutes (4.371 vs 5.053 in validation).
2. There's little error left to remove. Part 5 worked out that random noise in our data sets a floor of about 4.0 minutes MAE. XGBoost is at 4.43. The best any model could still gain is around 0.4 minutes, and some of that is lost for good to missing prep times.
3. The data is small. With 2,400 training orders, the clever extras in XGBoost and LightGBM have little to work with. Their advantages grow with data.

Choosing, in general

That last row surprises people. A tree's answer is always an average of training orders in some leaf, so boosting can't extrapolate: for a delivery twice as long as anything in the training data, it predicts roughly the same as the longest ones it saw. Linear regression would continue the trend, for better or worse.

The professional habit

1. Always build the simple baseline first. Without Part 3's linear model, we'd have celebrated XGBoost's 4.43 without knowing it was only 0.07 better.
2. Measure with ranges, not single numbers (Part 5).
3. Look at the groups that matter: overall ties can hide real gains, like XGBoost's in Old Town.
4. Pay for complexity only when it buys something: a new library is one more thing to install, pin, update and explain.

In this series, boosting earns its place on the hard orders. On a richer dataset, with GPS traces, restaurant histories and live traffic, it would likely earn a lot more.

The complete code

Two files: **boosting_features.py** (the helper from Step 5, needed by the API too) and **boost_delivery.py**, which runs every step and saves the XGBoost pipeline. The script takes about a minute on a 2-core machine.

Your project folder needs Part 3's delivery_data.py and delivery_features.py, plus boosting_features.py exactly as shown in Step 5. Then boost_delivery.py:

"""Delivery ML Series, Part 7: gradient boosting by hand, then XGBoost and LightGBM."""
import time

import joblib
import numpy as np
import pandas as pd
from lightgbm import LGBMRegressor, early_stopping
from scipy.stats import loguniform, randint, uniform
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import (KFold, RandomizedSearchCV, cross_val_predict, cross_validate,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeRegressor, export_text
from xgboost import XGBRegressor

from boosting_features import prepare_for_boosting
from delivery_data import make_messy_orders
from delivery_features import clean_orders

# ---- Setup: same data, same split, same baselines as Parts 3 and 5 ----
orders = make_messy_orders().drop_duplicates()
X = orders.drop(columns="delivery_min")
y = orders["delivery_min"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

base_features = ["distance_km", "prep_time_min", "rider_experience_months", "is_raining",
                 "is_peak_hour", "is_weekend"]
interaction_features = ["distance_x_rain", "distance_x_peak", "distance_x_rain_x_peak"]
numeric_features = base_features + interaction_features
categorical_features = ["city_zone", "cuisine"]
cv = KFold(n_splits=5, shuffle=True, random_state=42)


def linear_preprocess(numeric):
    return ColumnTransformer([
        ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric),
        ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                              OneHotEncoder(handle_unknown="ignore")), categorical_features),
    ])


def tree_preprocess(numeric):
    """Like Part 3's, minus the scaler: trees don't care about scale."""
    return ColumnTransformer([
        ("num", SimpleImputer(strategy="median"), numeric),
        ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"),
                              OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
         categorical_features),
    ], verbose_feature_names_out=False)


def score(model, label):
    """Training MAE and 5-fold validation MAE. The test set stays locked until Step 6."""
    result = cross_validate(model, X_train, y_train, cv=cv, scoring="neg_mean_absolute_error",
                            return_train_score=True)
    train, val = -result["train_score"].mean(), -result["test_score"].mean()
    print(f"{label:28s} train {train:.2f}  validation {val:.3f}")
    return val


# ---- Step 1: gradient boosting by hand ----
print("STEP 1")
prep = tree_preprocess(numeric_features)
X_fit_matrix = prep.fit_transform(clean_orders(X_train))
names = list(prep.get_feature_names_out())
target = y_train.values

prediction = np.full(len(target), target.mean())
print(f"Start: predict the average, {target.mean():.2f} min. Training MAE {np.abs(target - prediction).mean():.2f}")
trees = []
for round_number in range(1, 401):
    residuals = target - prediction
    tree = DecisionTreeRegressor(max_depth=2, random_state=42).fit(X_fit_matrix, residuals)
    prediction += 0.1 * tree.predict(X_fit_matrix)
    trees.append(tree)
    if round_number in (1, 2, 3, 10, 50, 100, 200, 400):
        print(f"After {round_number:3d} trees: training MAE {np.abs(target - prediction).mean():.2f}")
print(export_text(trees[0], feature_names=names, decimals=1))

# ---- Step 2: XGBoost, default settings ----
print("STEP 2")
score(make_pipeline(FunctionTransformer(clean_orders), linear_preprocess(numeric_features), LinearRegression()),
      "Linear regression")
score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    GradientBoostingRegressor(max_depth=2, n_estimators=400, random_state=42)),
      "scikit-learn, Part 5 tuned")
score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    XGBRegressor(random_state=42)), "XGBoost, default")

# ---- Step 3: LightGBM, default settings ----
print("STEP 3")
score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    LGBMRegressor(random_state=42, verbose=-1)), "LightGBM, default")

# ---- Theory break: do boosted trees need our interaction features? ----
print("INTERACTIONS")
for label, numeric in [("with interactions", numeric_features), ("base features only", base_features)]:
    score(make_pipeline(FunctionTransformer(clean_orders), linear_preprocess(numeric), LinearRegression()),
          f"Linear, {label}")
    score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric),
                        GradientBoostingRegressor(max_depth=2, n_estimators=400, random_state=42)),
          f"Boosting, {label}")

# ---- Step 4: early stopping ----
print("STEP 4")
X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=0)
prep = tree_preprocess(numeric_features)
fit_matrix = prep.fit_transform(clean_orders(X_fit))
val_matrix = prep.transform(clean_orders(X_val))
xgb = XGBRegressor(n_estimators=2000, learning_rate=0.05, max_depth=2, early_stopping_rounds=50,
                   eval_metric="mae", random_state=42)
xgb.fit(fit_matrix, y_fit, eval_set=[(val_matrix, y_val)], verbose=False)
history = xgb.evals_result()["validation_0"]["mae"]
for n in [50, 100, 200, 300]:
    print(f"After {n} trees: validation MAE {history[n - 1]:.3f}")
print(f"Best: {xgb.best_iteration + 1} trees, validation MAE {xgb.best_score:.3f}. Stopped after {len(history)} trees.")

lgbm = LGBMRegressor(n_estimators=2000, learning_rate=0.05, num_leaves=4, random_state=42, verbose=-1)
lgbm.fit(fit_matrix, y_fit, eval_X=(val_matrix,), eval_y=(y_val,), eval_metric="l1",
         callbacks=[early_stopping(stopping_rounds=50, verbose=False)])
print(f"LightGBM best: {lgbm.best_iteration_} trees, validation MAE {lgbm.best_score_['valid_0']['l1']:.3f}")

# ---- Step 5: let the libraries handle categories and missing values ----
print("STEP 5")
score(make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                    XGBRegressor(n_estimators=354, learning_rate=0.05, max_depth=2, random_state=42)),
      "XGBoost, imputed + one-hot")
score(make_pipeline(FunctionTransformer(prepare_for_boosting),
                    XGBRegressor(n_estimators=354, learning_rate=0.05, max_depth=2,
                                 enable_categorical=True, random_state=42)),
      "XGBoost, native")

# ---- Step 6: tune both, then unlock the test set ----
print("STEP 6")
searches = {
    "XGBoost": (XGBRegressor(enable_categorical=True, random_state=42, n_jobs=1), {
        "xgbregressor__n_estimators": randint(100, 1000),
        "xgbregressor__learning_rate": loguniform(0.01, 0.2),
        "xgbregressor__max_depth": randint(1, 6),
        "xgbregressor__min_child_weight": randint(1, 30),
        "xgbregressor__subsample": uniform(0.5, 0.5),
        "xgbregressor__colsample_bytree": uniform(0.5, 0.5),
        "xgbregressor__reg_lambda": loguniform(0.01, 10),
    }),
    "LightGBM": (LGBMRegressor(random_state=42, verbose=-1, n_jobs=1), {
        "lgbmregressor__n_estimators": randint(100, 1000),
        "lgbmregressor__learning_rate": loguniform(0.01, 0.2),
        "lgbmregressor__num_leaves": randint(2, 32),
        "lgbmregressor__min_child_samples": randint(5, 60),
        "lgbmregressor__subsample": uniform(0.5, 0.5),
        "lgbmregressor__subsample_freq": [1],
        "lgbmregressor__colsample_bytree": uniform(0.5, 0.5),
        "lgbmregressor__reg_lambda": loguniform(0.01, 10),
    }),
}
best = {}
for name, (model, space) in searches.items():
    start = time.perf_counter()
    search = RandomizedSearchCV(make_pipeline(FunctionTransformer(prepare_for_boosting), model), space,
                                n_iter=30, cv=cv, scoring="neg_mean_absolute_error", n_jobs=-1,
                                random_state=42)
    search.fit(X_train, y_train)
    best[name] = search.best_estimator_
    print(f"{name}: 30 settings in {time.perf_counter() - start:.0f} s, validation MAE {-search.best_score_:.3f}")
    for key, value in search.best_params_.items():
        value = round(value, 3) if isinstance(value, float) else value
        print(f"    {key.split('__')[1]}: {value}")

linear = make_pipeline(FunctionTransformer(clean_orders), linear_preprocess(numeric_features),
                       LinearRegression()).fit(X_train, y_train)
sklearn_tuned = make_pipeline(FunctionTransformer(clean_orders), tree_preprocess(numeric_features),
                              GradientBoostingRegressor(max_depth=2, n_estimators=400,
                                                        random_state=42)).fit(X_train, y_train)
candidates = {"Linear (Part 3)": linear, "scikit-learn (Part 5)": sklearn_tuned,
              "XGBoost, tuned": best["XGBoost"], "LightGBM, tuned": best["LightGBM"]}
errors = {name: np.abs(model.predict(X_test) - y_test.values) for name, model in candidates.items()}
for name, err in errors.items():
    print(f"{name:22s} test MAE {err.mean():.2f}")


def saved_vs_linear(name, rounds=2000):
    """Minutes saved per order compared with linear regression: estimate and 95% range."""
    rng = np.random.default_rng(0)
    diffs = []
    for _ in range(rounds):
        idx = rng.integers(0, len(y_test), len(y_test))
        diffs.append(errors["Linear (Part 3)"][idx].mean() - errors[name][idx].mean())
    gap = errors["Linear (Part 3)"].mean() - errors[name].mean()
    low, high = np.percentile(diffs, [2.5, 97.5])
    return gap, low, high


for name in ["scikit-learn (Part 5)", "XGBoost, tuned", "LightGBM, tuned"]:
    gap, low, high = saved_vs_linear(name)
    print(f"{name:22s} saves {gap:+.2f} min per order (95% range {low:+.2f} to {high:+.2f})")

cleaned_test = clean_orders(X_test)
hard_groups = {
    "Old Town": cleaned_test["city_zone"] == "Old Town",
    "Rain at peak hour": (cleaned_test["is_raining"] == 1) & (cleaned_test["is_peak_hour"] == 1),
    "Prep time missing": cleaned_test["prep_time_min"].isna(),
    "Longer than 10 km": cleaned_test["distance_km"] > 10,
}
rows = [{"group": group, "orders": int(mask.sum()),
         **{name: round(err[mask.values].mean(), 2) for name, err in errors.items()}}
        for group, mask in hard_groups.items()]
print(pd.DataFrame(rows).to_string(index=False))

# ---- Step 7: re-choose the promise buffer for XGBoost, then save ----
print("STEP 7")
val_pred = cross_val_predict(best["XGBoost"], X_train, y_train, cv=cv)
for buffer in [6, 7, 8]:
    print(f"buffer {buffer} min: on time {(y_train.values <= val_pred + buffer).mean():.1%}")
BUFFER = 7
test_on_time = (y_test.values <= best["XGBoost"].predict(X_test) + BUFFER).mean()
print(f"Test set, buffer {BUFFER}: on time {test_on_time:.1%}")
joblib.dump(best["XGBoost"], "delivery_pipeline_xgb.joblib")
print("Saved delivery_pipeline_xgb.joblib")

Run python boost_delivery.py. Every number should match this article except the timings.

Common mistakes

Practice project: boosting the electricity model

Repeat this tutorial on the home-electricity data. This time the result is different: tuned XGBoost beats linear regression by a margin the bootstrap can confirm, because the electricity data has an interaction nobody built by hand.

You've predicted monthly electricity use (kWh) for homes in Mumbai, Bengaluru, Chennai and Pune since Part 1, cleaned the messy export in Part 3, served it in Part 4 and tuned scikit-learn boosting in Part 5.

What you start with

• energy_data_messy.py, and energy_features.py with clean_homes.
• The same data rebuild as before: drop duplicates, divide Wh readings (≥ 20,000) by 1,000, train_test_split(test_size=0.2, random_state=42), and KFold(5, shuffle=True, random_state=42).
• Numeric columns: home_size_sqm, residents, has_solar, works_from_home, appliance_age_years, ac_hours_per_day, is_summer, ac_x_size. Category column: city.

Your tasks

• 1. Defaults (Steps 2–3). Cross-validate linear regression, default XGBoost and default LightGBM (imputed + one-hot, no scaler). Which overfits worst?
• 2. Interactions (theory break). Remove ac_x_size, the one hand-made interaction (AC hours × home size). Compare linear regression and scikit-learn boosting (max_depth=2, n_estimators=400) with and without it.
• 3. Native categories (Step 5). Write energy_boosting.py with a prepare_homes(df) helper: clean_homes, .astype(float) on the numeric columns, and city as a category from the fixed list ["Bengaluru", "Chennai", "Mumbai", "Pune"].
• 4. Early stopping (Step 4). XGBoost with learning_rate=0.05, max_depth=2, enable_categorical=True, patience 50, on a validation split (test_size=0.2, random_state=0) of the training homes. How many trees?
• 5. Search and test (Step 6). Run this tutorial's 30-setting XGBoost search. Then score linear, Part 5's scikit-learn settings and tuned XGBoost on the test set, and bootstrap XGBoost against both.
• 6. Groups. Compare test MAE for homes with and without solar panels.
• Stretch: switch your Part 4 energy API to the tuned XGBoost pipeline, pin xgboost in requirements.txt, and update the tests on purpose.

Expected results

(scikit-learn scores 31.7 here, not Part 5's 31.6, because this pipeline skips the scaler. Trees ignore scale in theory, but tiny rounding differences can move a split point.)

Check your thinking

• Why the same winning settings as the delivery search? With random_state=42, the search draws the same 30 candidate settings from the same ranges. It's a coincidence that the same candidate wins on both datasets, but a telling one: small trees and many small steps suit both.
• Why does boosting clearly win here but only tie on deliveries? Solar panels cut a bill by a percentage, which is an interaction between solar and everything else, and nobody built a solar × size feature. Linear regression can't learn it; boosting finds it, and the error for solar homes drops from 43.0 to 28.7 kWh. On the delivery data, Parts 2 and 3 had already handed linear regression its key interactions.
• And **ac_x_size**? It matters enormously for linear regression (13.9 kWh) and barely for boosting (1.3 kWh): the theory break's lesson, even more strongly.

Frequently asked questions

What is gradient boosting, in one sentence?

A model built from many small decision trees, added one at a time, where each new tree learns to correct the mistakes the previous trees still make.

XGBoost or LightGBM: which should I use?

Both are excellent and usually score within a whisker of each other, as here (4.43 vs 4.43). LightGBM is often faster on large datasets; XGBoost has slightly more conservative defaults for tree shape. Try both, keep the one that validates better, and don't agonise.

Why did XGBoost's default settings do so badly?

Its defaults (trees 6 levels deep, learning rate 0.3) suit large datasets. On 2,400 rows, they memorise the training data. Use smaller trees, a smaller learning rate and early stopping.

What is early stopping?

Training with a validation set and stopping when the validation error hasn't improved for a set number of trees (the patience). The model keeps the best version, so you don't have to guess the number of trees.

Do I still need to impute missing values and one-hot encode?

Not for XGBoost or LightGBM: both handle missing values and pandas categories natively. Make sure numeric columns really are numbers (.astype(float)) and categories use a fixed list.

Is boosting always better than linear regression?

No. On our delivery data, with the key interactions already built by hand, they tie overall. On the electricity data, with an interaction nobody built, boosting clearly wins. Always compare against a simple baseline.

Glossary

Series roadmap

Next up, Part 8, the finale: our API now says "69.5 minutes, promised by 77", and Part 6 can say why: rain, Old Town, Saturday night. Part 8 hands those numbers and reasons to a large language model and gets back a friendly message for the customer, with guardrails so it never invents a reason or promises a time the model didn't give.

Conclusion

You've now seen inside the models that win most competitions on tabular data:

• Gradient boosting is a simple loop: fit a small tree to the residuals, add a fraction of it, repeat. Ten lines reproduce scikit-learn's model exactly.
• XGBoost and LightGBM are faster, more flexible versions, but their defaults can overfit small data badly.
• Early stopping chooses the number of trees for you.
• Native missing values and categories simplify the pipeline, and need a fixed category list and real numeric types.
• Boosting finds interactions automatically. That's its real advantage, and why it ties on our delivery data (where we built the interactions) but wins clearly on the electricity data (where we didn't).
• Keep the simple baseline. Tuned XGBoost scored 4.43 minutes, best in the series and best on every hard group, and we only know how much that's worth because linear regression was standing next to it.

Complex models are worth it when they buy something you can measure. Now you know how to measure it.