Build a Neural Network Without PyTorch. What Changes When It Learns?

You run a training script. A number called “loss” gets smaller. A few seconds later, your model makes better predictions. But what happened inside it? Where did the learning get stored? And why does running the same calculation again produce a different answer?

We’ll answer those questions by building a small neural network with Python and NumPy. Our example is a simulated plant-watering assistant. It receives soil moisture and temperature and estimates whether a plant needs water. We’ll follow one observation through the calculations, watch a weight change, and inspect the model’s predictions before and after training.

The short answer

A neural network is a machine-learning model made of connected calculation units called neurons. It transforms inputs, such as moisture and temperature, into an output, such as an estimated probability that a plant needs water.

It learns by changing its weights and biases. These adjustable numbers determine how inputs affect predictions. Training measures prediction error with a loss function, calculates gradients with backpropagation, and uses those gradients to update the parameters. We’ll unpack each term before implementing it.

For one plant observation in this tutorial, the estimated probability of needing water moves from 41.6% before training to 71.6% after training. You will be able to trace that change through the actual code.

The finished network reaches 95.7% accuracy on 300 held-out simulated observations. A baseline that always predicts “does not need water” reaches 71.7% on the same test set. We’ll calculate these results and explain the remaining mistakes.

What you will build

Imagine a greenhouse where sensors record two facts for each plant observation.

The target is needs_water. A value of 1 means the observation is labelled as needing water. A value of 0 means it is labelled as having enough moisture for the simulated conditions.

This is supervised learning, because training examples include the desired answer. It is binary classification, because there are two possible classes.

You will generate 1,800 observations, train a network with eight hidden neurons, evaluate it on held-out data, inspect one prediction before and after training, and save everything needed to reproduce that prediction.

Why this example works for beginners

The inputs are ordinary measurements. You can inspect individual rows, compare a dry plant with a wet one, and ask what happens when temperature rises while moisture stays constant.

The simulator contains a curved relationship. The moisture threshold rises more quickly at high temperatures, giving the hidden layer something meaningful to learn. Random variation introduces ambiguity near that threshold, creating examples that are hard to separate perfectly.

The formula is a teaching assumption that defines the labels for this experiment. A practical watering model would need observations covering plant species, soil, sensor behaviour, and actual watering requirements.

What is a neural network, in plain English?

Suppose you want a program to decide whether a plant needs water. You could start with a handwritten rule: “If soil moisture is below 30%, predict that it needs water.” That rule is easy to understand, but it gives the same answer at 20°C and 35°C. What if temperature changes how much moisture is enough?

You could write more rules. Or, if you have suitable examples, you could train a model to learn a relationship between the measurements and the answers.

A neural network takes the second approach. You choose its arrangement of calculations, give it examples, and use a training procedure to adjust the numbers inside those calculations. After training, it can apply the learned relationship to new inputs.

Think of it as a mathematical function with adjustable settings. The inputs enter the function, its current settings determine the calculations, and an output comes out. Learning changes those settings so the outputs better match the training targets.

In our example, the function takes two measurements and returns one probability estimate:

p̂ = f(soil moisture, temperature; weights and biases)

The symbol f means “the calculation performed by the network.” The hat in p̂ tells us this is an estimate. The weights and biases are the settings that training adjusts.

Why is it called “neural”?

The name comes from a loose inspiration from connected biological neurons. An artificial neuron in this tutorial is a small mathematical calculation: multiply inputs by weights, add them together with a bias, and apply an activation function.

That biological analogy explains the name, but you do not need neuroscience to understand the code. Our neurons operate on numbers. They have no built-in understanding of plants, heat, or thirst.

How is this different from writing an ordinary rule?

Learning does not mean that the program rewrites its Python functions. In this tutorial, forward keeps performing the same sequence of operations. The values in W1, b1, W2, and b2 change.

There is also a deliberate teaching shortcut here: we already know the formula that generates our labels. We could use that formula directly to describe the simulator’s threshold. We train a network so we can observe how learning works when the training procedure receives examples instead of the formula. The network cannot recover the individual random variation that we omit from its inputs.

What is inside our neural network?

Our model is a feedforward neural network: information moves from inputs through a hidden layer to an output, without a feedback loop inside a prediction. It is also called a multilayer perceptron, or MLP.

It has two input values, eight hidden neurons, and one output neuron. This arrangement is often written as 2 → 8 → 1.

The input layer holds the supplied values; it does not learn a separate set of weights. “Hidden” means the intermediate responses are neither the input measurements nor the target answer. We can still print and inspect every one of them.

What does one neuron do?

Imagine a neuron receiving the two scaled measurements. It first calculates a score:

Score = moisture input × moisture weight + temperature input × temperature weight + bias.

Each weight controls how its input contributes to that neuron’s score. Holding everything else fixed, a positive weight makes the score increase when that input increases; a negative weight makes it decrease. A weight’s magnitude describes this contribution at the neuron, but does not by itself tell us the feature’s importance to the whole network.

The bias adds an offset. Without it, this weighted score would always be zero when both scaled inputs are zero. With a bias, the neuron can shift its response independently of those current input values.

An activation function then transforms the score. Our hidden neurons use tanh, which smoothly maps scores into values between −1 and 1. Step 4 works through the arithmetic for one neuron.

Why have several neurons and an activation function?

All eight hidden neurons receive the same two inputs, but each has its own weights and bias. They can therefore produce different responses to the same observation. The output neuron learns how to combine those responses.

The nonlinear activation lets these combined responses represent more complicated relationships. Without a nonlinear transformation between layers, repeatedly multiplying by weights and adding biases would still collapse into one weighted sum plus an offset. Adding layers alone would not give this model a curved classification boundary.

In our simulator, the moisture threshold changes with the square of a temperature difference. The hidden layer gives the network a way to approximate that curved relationship from examples.

We do not designate one neuron as a “dryness expert” and another as a “heat expert.” Useful behaviour is learned across the parameters, and any interpretation of an individual neuron needs to be checked against its actual responses.

How does a neural network make a prediction?

Let’s follow an observation with 32% moisture and a temperature of 35°C. This is the same observation we will inspect throughout the code.

Stages 2–5 are the forward pass through this network. Using the trained model to make a prediction is called inference.

After training, this observation produces approximately 0.7155. This is the model’s estimated probability for class 1. It is neither a request for 71.55 millilitres of water nor a measurement of the soil. It answers the classification question we chose when defining the target.

The 0.5 cutoff is a decision rule we choose outside the learned weights. The network produces the probability; the cutoff converts that probability into a class. The numerical estimate also needs evaluation before we can assume that predictions near 0.72 correspond to positive outcomes about 72% of the time.

How does a neural network learn?

Before training, random weights already let the network produce a number. Producing a number is easy; producing a useful prediction requires fitting the parameters to data.

For supervised learning, each training example supplies both the measurements and the target answer. The training loop repeatedly performs four jobs:

1. Predict: run the examples through the current network.
2. Measure the error: use a loss function to compare predictions with their targets.
3. Calculate gradients: use backpropagation to work out how small parameter changes would affect the loss.
4. Update the parameters: use gradient descent to adjust weights and biases in a direction intended to reduce the loss.

The loop uses the targets to assess predictions, but the targets are not input features. For a new observation, we provide moisture and temperature without supplying the answer we are trying to predict.

For intuition, consider a training observation labelled 1 that receives probability 0.4. The training objective favours increasing its predicted probability. An observation labelled 0 pushes towards a lower probability instead. Because all observations share the same parameters, the update combines these competing contributions across the training batch.

Backpropagation calculates the gradients; gradient descent uses them to change the parameters. They are two distinct parts of learning. Step 5 shows both in code and follows one real weight update numerically.

What changes during training, and what stays fixed?

For the same [32, 35] observation, the prediction changes from 0.4162 before training to 0.7155 after training because the parameters changed. During ordinary inference in our code, the parameters remain fixed. Repeating a prediction for the same input does not teach the model anything new.

Learning should help with examples the model has not seen

A flexible model can fit training examples closely and still make poor predictions on new ones. This is overfitting. Our goal is generalization: useful performance on new observations drawn from the conditions we intend to support.

That is why we separate training, validation, and test data. Training adjusts the parameters. Validation helps select a saved parameter state. The test set checks the selected model on observations that were used for neither purpose. We will implement this separation in Step 3.

What are neural networks used for?

Neural networks are useful when examples contain patterns that a suitable model can learn, especially when writing all the relevant rules by hand is difficult. The inputs, architecture, training objective, and output interpretation depend on the task.

These systems do not all use the tiny architecture in this article. Image, speech, recommendation, and language tasks may require different network structures and much more data and computation. The shared idea is that trainable parameters shape how inputs become outputs.

If we changed our question from “Does this plant need water?” to “How many millilitres should it receive?”, we would need numerical training targets, a suitable output formulation, and a regression loss. Renaming the existing probability would not turn this classifier into a water-quantity predictor.

How do neural networks relate to AI, machine learning, and deep learning?

Artificial intelligence is the broad field. Machine learning includes methods that fit models from data. Neural networks are one family of those models; decision trees and linear models are other families. Deep learning uses neural networks with multiple successive learned layers, typically to learn increasingly transformed representations.

Our network has one hidden layer, so it is a small, shallow example. Its purpose is to make the mechanics visible. Understanding it gives you a foundation for studying larger networks, while leaving many architectural details still to learn.

When is a neural network unnecessary?

If a reliable formula already solves the problem, implementing that formula can be simpler and easier to inspect. If the data is a small table, a linear model or decision tree may already meet the goal. The choice should depend on evaluation and practical requirements.

For our watering task, a neural network would need to justify its complexity against simpler alternatives. The majority-class baseline later is a basic sanity check; outperforming it does not establish that the network is better than a tuned threshold rule or another trained model.

A network also depends on the information it receives. This one cannot distinguish two plants with identical moisture and temperature but different species, because species is absent from its inputs. More neurons cannot supply a missing measurement. Its strong score on simulated data tells us how well it learned this synthetic task, not how well it would manage a real greenhouse.

With that foundation, we can now build the calculations ourselves and see exactly where the learning is stored.

Prerequisites

The reference run used NumPy 2.3.5. A normal laptop CPU is sufficient. Allow about 90–120 minutes to work through the foundations, code, and experiments. All data is generated locally, and the example requires no paid service.

Matplotlib is optional for the plots. You can run the model before you understand every derivative, then revisit backpropagation with the printed results beside you.

The machine learning workflow at a glance

Run the Python blocks in Steps 2–7 in order in one notebook. Each block builds on earlier variables. The complete script later combines them into one file.

Step 1 — Set up your environment

Create a directory and virtual environment from your terminal.

mkdir numpy-neural-network
cd numpy-neural-network
python -m venv .venv

In Windows PowerShell, activate it with the following command.

.venv\Scripts\Activate.ps1

On macOS or Linux, use the following command.

source .venv/bin/activate

Install NumPy into that environment.

python -m pip install numpy

If your system uses python3, use that command consistently. In a notebook, select the kernel associated with the environment containing NumPy.

NumPy supplies arrays, matrix multiplication, and mathematical functions. We will implement the network, loss, gradients, and training loop ourselves.

Step 2 — Create and explore the data

Understand the hidden recipe

We sample moisture between 10% and 70% and temperature between 15°C and 40°C. Then we calculate a moisture threshold.

threshold = 25 + 0.04(temperature - 20)²

The label is 1 when moisture falls below this threshold plus a small random variation.

The number 25 sets the threshold at 20°C. Subtracting 20 measures distance from that reference temperature. Squaring that distance makes its contribution grow more quickly as temperature moves away from 20°C. The multiplier 0.04 controls the size of the contribution.

The square also produces a small rise below 20°C. That behaviour is part of this deliberately simple simulator.

Consider moisture of 32%. At 20°C, it sits above the 25% threshold. At 35°C, it sits below the 34% threshold. The same moisture reading can therefore receive a different label depending on temperature.

We add normally distributed variation with standard deviation 3 percentage points to the threshold. Most variations are relatively small, although larger ones are possible. This creates overlapping labels near the boundary, representing omitted influences in the synthetic problem.

Generate the observations

import numpy as np

def make_data(n=1800, seed=42):
    rng = np.random.default_rng(seed)
    moisture = rng.uniform(10, 70, n)
    temperature = rng.uniform(15, 40, n)
    threshold = 25 + 0.04 * (temperature - 20) ** 2
    variation = rng.normal(0, 3, n)
    needs_water = moisture < threshold + variation
    X = np.column_stack([moisture, temperature])
    y = needs_water.astype(float).reshape(-1, 1)
    return X, y

X_raw, y = make_data()
print(np.column_stack([X_raw[:5], y[:5]]))
print("Positive share:", y.mean())

np.column_stack places the feature columns side by side. reshape(-1, 1) makes the targets a column with one answer per observation. The -1 asks NumPy to infer the number of rows.

The fixed seed makes this data generation reproducible in our environment. The training functions receive feature arrays and target arrays. The threshold formula is used during data generation.

The first five observations, rounded for readability, look like this.

Read the last row as a short story. The soil moisture is low, the temperature is fairly high, and the observation is labelled as needing water. About 31.9% of the full dataset has label 1.

Theory break — Features, labels, and predictions

A feature is an input measurement available when making a prediction. An observation here has two features.

A label, also called a target, is the answer supplied for training or evaluation. Here, it records whether the observation needs water according to the simulator.

A prediction is the model’s estimate. A model might predict 0.72 for an observation whose label is 1. The loss measures how well that probability agrees with the label.

The word “target” also applies to regression. If we predicted millilitres of water, that quantity would be the regression target.

Step 3 — Split the data and scale the inputs

Give each dataset a specific job

We shuffle the observation indices and create three groups.

The generated observations are independent, making a shuffled split appropriate. Repeated measurements from the same plants would need a split that accounts for plant identity or time, depending on the intended use.

Why scale the inputs?

Hidden neurons multiply inputs by weights. Large resulting scores can push tanh close to −1 or 1, where its slope is small. Small slopes can weaken the gradient reaching earlier weights.

Standardization expresses a feature as its distance from the training mean, measured in training standard deviations.

scaled value = (value - training mean) / training standard deviation

Zero means “at the training average.” One means “one training standard deviation above average.”

Calculate these statistics from the training rows and use them for validation, test, and future observations. That keeps the scaling independent of the evaluation data and keeps each input’s meaning consistent.

indices = np.random.default_rng(19).permutation(len(X_raw))
train_ids = indices[:1200]
validation_ids = indices[1200:1500]
test_ids = indices[1500:]
mean = X_raw[train_ids].mean(axis=0, keepdims=True)
scale = X_raw[train_ids].std(axis=0, keepdims=True)
scale = np.where(scale == 0, 1.0, scale)
X_train = (X_raw[train_ids] - mean) / scale
X_validation = (X_raw[validation_ids] - mean) / scale
X_test = (X_raw[test_ids] - mean) / scale
y_train, y_validation, y_test = y[train_ids], y[validation_ids], y[test_ids]

permutation creates shuffled row indices. Slices select the three groups. The scale == 0 check handles a constant feature by using a divisor of 1.

The training moisture mean is approximately 39.499, with standard deviation 17.346. The temperature mean is 27.434, with standard deviation 7.321.

Our example, moisture 32% and temperature 35°C, becomes approximately [-0.4323, 1.0335]. Its moisture is below the training average and its temperature is above it.

Step 4 — Build the network and follow one prediction

Start with one neuron

A neuron calculates a weighted sum and adds a bias.

z = w₁x₁ + w₂x₂ + b

Each weight controls how strongly an input contributes to this neuron’s score. The bias shifts the score independently of the current input values.

For a made-up arithmetic example, suppose the scaled inputs are −0.4 and 1.0, the weights are −0.5 and 0.8, and the bias is 0.1. The score is (-0.4 × -0.5) + (1.0 × 0.8) + 0.1 = 1.1.

The neuron applies an activation function to that score. We use tanh, which maps a score into the interval between −1 and 1. A score of 1.1 produces an activation of about 0.8005.

Why use a hidden layer?

We create eight neurons, each with its own two weights and bias. They can respond differently to combinations of moisture and temperature. The output neuron combines their activations into a prediction.

“Hidden” means these intermediate values sit between observed inputs and the output. Training discovers useful intermediate responses through their effect on the final loss.

The nonlinear activation matters. If both layers only multiplied inputs and added biases, their composition would reduce to one affine transformation. With a 0.5 sigmoid threshold, it would produce a straight decision boundary in this two-feature space. Our simulator’s boundary curves with temperature. Applying tanh between the layers gives the model the ability to represent curved boundaries.

Count the trainable numbers

Initialize the weights

def initialize(seed=7, hidden_units=8):
    rng = np.random.default_rng(seed)
    return {
        "W1": rng.normal(0, 1 / np.sqrt(2), (2, hidden_units)),
        "b1": np.zeros((1, hidden_units)),
        "W2": rng.normal(0, 1 / np.sqrt(hidden_units), (hidden_units, 1)),
        "b2": np.zeros((1, 1)),
    }

Random weights give hidden neurons different starting responses. The standard deviations scale the initial values according to each layer’s input count. Biases begin at zero.

The seed controls the initial random draw. Different seeds may produce different parameter values even when the networks make similar predictions.

Implement the forward pass

def forward(X, parameters):
    z1 = X @ parameters["W1"] + parameters["b1"]
    hidden = np.tanh(z1)
    logits = hidden @ parameters["W2"] + parameters["b2"]
    probability = np.exp(-np.logaddexp(0, -logits))
    return hidden, logits, probability

A forward pass calculates outputs using the current parameters.

The @ operator performs matrix multiplication. A row of inputs combines with a column of weights to calculate one weighted sum. NumPy repeats this for every row and column.

Biases have one row. NumPy broadcasts it across observations, so each observation uses the same learned bias for a given neuron.

The output score is a logit. Sigmoid converts it into a probability.

p = 1 / (1 + e⁻ᶻ)

A zero logit gives probability 0.5. Positive logits give probabilities above 0.5, and negative logits give probabilities below it. We predict class 1 at probability 0.5 or higher.

The code writes sigmoid as np.exp(-np.logaddexp(0, -logits)). This equivalent expression avoids overflow from directly exponentiating a large positive number. logaddexp(a, b) calculates the logarithm of exp(a) + exp(b) stably.

Follow our plant through the initial network

The scaled input is approximately [-0.4323, 1.0335]. Hidden neuron 0 starts with weights 0.000870 and -0.348043, and bias zero.

Its score is approximately (-0.4323 × 0.000870) + (1.0335 × -0.348043) = -0.3601. Applying tanh gives −0.3453.

The other neurons perform their own calculations. Together, the eight activations are approximately [-0.3453, -0.4966, 0.4151, 0.4877, 0.2127, -0.3600, -0.0397, 0.0981].

The output neuron multiplies these by its eight weights and adds its bias, producing logit −0.3383. Sigmoid gives 0.4162, and the initial class prediction is 0.

These parameters came from initialization. Training will give them information from labelled observations.

Step 5 — Measure errors and teach the network

Why accuracy gives a weak training signal

Imagine a positive observation whose probability improves from 0.10 to 0.40. At threshold 0.5, both produce class 0. Accuracy counts the same incorrect answer even though the probability moved towards the target.

A loss function measures that progress continuously.

Binary cross-entropy, one observation at a time

For label 1, loss is -log(p). For label 0, it is -log(1 - p). Here log is the natural logarithm.

Across a batch, average the individual losses.

L = -(1/N) ∑ [yᵢ log(pᵢ) + (1 - yᵢ) log(1 - pᵢ)]

N is the observation count. y_i is row i’s label, and p_i is its predicted probability. Since the label is 0 or 1, one of the two terms vanishes for each row.

def loss(logits, targets):
    return np.mean(np.logaddexp(0, (1 - 2 * targets) * logits))

This implementation calculates the equivalent loss directly from logits. For label 1, it evaluates log(1 + exp(-logit)). For label 0, it evaluates log(1 + exp(logit)). logaddexp keeps these expressions stable for large scores.

What is a gradient?

A gradient describes how loss changes locally when a parameter changes. Imagine nudging one weight upward by a tiny amount, holding all others fixed. Does loss rise or fall, and how quickly?

A positive derivative means a small upward change tends to raise loss. A negative derivative means it tends to lower loss. Gradient descent moves the weight in the opposite direction to its derivative.

w_new = w_old - η (∂L / ∂w)

The learning rate η controls the movement’s size. This is a local estimate. A very large step can move beyond the region where that estimate was useful.

What backpropagation calculates

Loss depends on the probability, which depends on the output score, which depends on hidden activations, which depend on earlier weights. Backpropagation follows these dependencies backwards using the chain rule.

The chain rule combines rates of change along a calculation. If a weight changes a hidden activation, and that activation changes loss, multiplying the appropriate local derivatives tells us how the weight changes loss through that path. Contributions from multiple paths are added.

def backward(X, targets, hidden, probability, parameters):
    d_logits = (probability - targets) / len(X)
    dW2 = hidden.T @ d_logits
    db2 = d_logits.sum(axis=0, keepdims=True)
    d_hidden = d_logits @ parameters["W2"].T
    d_z1 = d_hidden * (1 - hidden ** 2)
    dW1 = X.T @ d_z1
    db1 = d_z1.sum(axis=0, keepdims=True)
    return {"W1": dW1, "b1": db1, "W2": dW2, "b2": db2}

First, calculate the output-score gradient. Combining sigmoid with binary cross-entropy gives (probability - targets) / len(X). Division by batch size accounts for averaging the loss.

For an illustrative row with label 1 and probability 0.4, the derivative is (0.4 - 1) / 1200 = -0.0005. Increasing that row’s logit would locally reduce its loss. A row with label 0 and probability 0.8 has a positive derivative, pushing in the other direction.

Second, calculate output-weight and bias gradients. hidden.T @ d_logits combines each activation with the output-score derivative and sums over rows. .T transposes an array, exchanging its row and column axes. Here, (8, 1200) @ (1200, 1) produces eight weight gradients.

The output bias is added to every row, so its derivative is the sum of the row contributions. keepdims=True preserves the two-dimensional shape matching the bias.

Third, propagate to the hidden activations. Multiplication by W2.T accounts for the output connections through which the activations affect the final score.

Fourth, pass through tanh. Its derivative is 1 - tanh(z)². Since hidden stores tanh(z1), we reuse it. The * here means element-by-element multiplication. An activation near zero has a derivative near one. An activation near −1 or 1 has a derivative near zero.

Finally, calculate first-layer gradients. X.T @ d_z1 combines inputs with hidden-score gradients. Summing d_z1 over observations gives hidden-bias gradients.

Each gradient array matches its parameter’s shape. Calculate every gradient before changing any parameters.

See a real weight update

The moisture-to-first-hidden-neuron weight starts at 0.00086985. On the first batch, its gradient is approximately −0.21537959. With learning rate 0.1, the update becomes

w_new = 0.00086985 - 0.1(-0.21537959) ≈ 0.02240781

The weight increases because its gradient is negative. This gradient combines contributions from all 1,200 training observations. Individual observations can push shared weights in different directions.

Run the training loop

parameters = initialize()
initial = {name: value.copy() for name, value in parameters.items()}
learning_rate = 0.1
epochs = 4000
best_validation_loss = float("inf")
best_parameters = None
best_epoch = 0
history = []

for epoch in range(epochs + 1):
    hidden, logits, probability = forward(X_train, parameters)
    training_loss = float(loss(logits, y_train))
    _, validation_logits, _ = forward(X_validation, parameters)
    validation_loss = float(loss(validation_logits, y_validation))
    history.append([epoch, training_loss, validation_loss])
    if validation_loss < best_validation_loss:
        best_validation_loss = validation_loss
        best_parameters = {name: value.copy() for name, value in parameters.items()}
        best_epoch = epoch
    if epoch in (0, 1, 100, 500, 1000, 4000):
        print(f"{epoch:4d}  train={training_loss:.6f}  validation={validation_loss:.6f}")
    if epoch == epochs:
        break
    gradients = backward(X_train, y_train, hidden, probability, parameters)
    for name in parameters:
        parameters[name] -= learning_rate * gradients[name]

parameters = best_parameters
print("Selected epoch:", best_epoch)

An epoch is a pass over the training set. We use the entire set for each update, so one update corresponds to one epoch. Larger projects often divide an epoch into smaller batches, each with its own update.

Epoch 0 records initialization. Epoch 4,000 records the network after 4,000 updates. The final break prevents an extra update after reporting that state.

The loop measures validation loss and copies parameters whenever that loss improves. NumPy arrays are mutable, so .copy() preserves the checkpoint as later updates occur.

The selected parameters come from the lowest-validation-loss checkpoint. Test observations take no part in gradient calculations or checkpoint selection.

Read the output

Epoch 4,000 is selected in this run. Most initial improvement arrives quickly. Later updates refine the boundary more gradually.

Validation loss is lower here because randomly sampled subsets can differ in difficulty. Watch the trends across training. Falling training loss accompanied by a sustained rise in validation loss is a reason to examine overfitting.

Step 6 — Evaluate the model and inspect what changed

Compare with a simple baseline

The training majority class is 0, so the baseline predicts 0 for every test observation. Choosing its class from training data keeps that choice independent of test labels.

def metrics(X, targets, parameters):
    _, logits, probability = forward(X, parameters)
    predicted = probability >= 0.5
    actual = targets == 1
    tp = int(np.sum(predicted & actual))
    tn = int(np.sum(~predicted & ~actual))
    fp = int(np.sum(predicted & ~actual))
    fn = int(np.sum(~predicted & actual))
    return {
        "loss": float(loss(logits, targets)),
        "accuracy": float(np.mean(predicted == actual)),
        "precision": tp / max(tp + fp, 1),
        "recall": tp / max(tp + fn, 1),
        "confusion": [[tn, fp], [fn, tp]],
    }
majority_class = int(y_train.mean() >= 0.5)
print(f"Baseline test accuracy: {np.mean(y_test == majority_class):.3f}")
print("Training metrics:", metrics(X_train, y_train, parameters))
print("Test metrics:", metrics(X_test, y_test, parameters))

Test accuracy is 287 / 300, about 95.7%. The network correctly classifies 72 more test observations than the baseline.

Understand the mistakes

Rows in the confusion matrix are actual labels. Columns are predicted labels.

A false positive is a watering prediction for an observation labelled 0. A false negative is a missed observation labelled 1.

Precision asks how often positive predictions were correct. Here, 79 / (79 + 7) is about 91.9%.

Recall asks how many actual positives the network found. Here, 79 / (79 + 6) is about 92.9%.

The metrics function uses a denominator of at least 1 to return zero when precision or recall would otherwise divide by zero. This is an explicit reporting convention for those cases.

These scores describe the synthetic distribution. The inputs omit the random variation used to generate labels, leaving some boundary cases ambiguous even with a well-learned relationship.

Follow the same observation before and after learning

example_raw = np.array([[32.0, 35.0]])
example = (example_raw - mean) / scale
for label, current in (("Before", initial), ("After", parameters)):
    hidden, logits, probability = forward(example, current)
    print(label, "hidden:", np.round(hidden, 4))
    print(label, "logit:", np.round(logits, 4), "probability:", np.round(probability, 4))
print("W1[0, 0]:", initial["W1"][0, 0], "->", parameters["W1"][0, 0])

For moisture 32% and temperature 35°C, the trace gives the following values.

Training updates the parameter arrays. The forward pass recomputes activations and output scores from those updated values.

The first weight moves substantially while the first activation changes only slightly. The neuron combines two weighted inputs and a bias, and those parameters change together. Its behaviour depends on the combined calculation.

A hidden weight influences the final prediction through an activation and an output weight. Reading its sign as the overall effect of moisture would miss those dependencies. Controlled input comparisons help us inspect the model’s behaviour directly.

Parameters and hyperparameters

Design and training choices are hyperparameters. Compare alternative settings using validation data. Repeated choices based on test performance make the test score less independent.

Step 7 — Predict new observations and save the model

Try a controlled “what if” comparison

scenarios = np.array([[20., 25.], [50., 25.], [32., 20.], [32., 35.]])
_, _, probabilities = forward((scenarios - mean) / scale, parameters)
print("Scenario probabilities:", np.round(probabilities.ravel(), 4))

The first pair holds temperature constant and changes moisture. The second holds moisture constant and changes temperature. Both agree with the broad structure of the simulator.

These probabilities are model estimates. Assessing how closely they match observed frequencies would require a calibration analysis with suitable held-out data.

Save the learned state and preprocessing

np.savez(
    "plant_water_model.npz",
    **parameters,
    mean=mean,
    scale=scale,
    feature_names=np.array(["soil_moisture_pct", "temperature_c"]),
    decision_threshold=np.array(0.5),
)
with np.load("plant_water_model.npz", allow_pickle=False) as saved:
    restored = {name: saved[name] for name in ("W1", "b1", "W2", "b2")}
    restored_input = (example_raw - saved["mean"]) / saved["scale"]
    _, _, restored_probability = forward(restored_input, restored)
    restored_class = restored_probability >= saved["decision_threshold"]
print("Reloaded probability:", float(restored_probability[0, 0]))
print("Reloaded needs_water:", int(restored_class[0, 0]))

The archive contains weights, biases, scaling statistics, feature names, and the classification threshold. The inference code supplies the network operations, including tanh and sigmoid.

Feature order is [soil_moisture_pct, temperature_c]. Putting temperature in the moisture column changes the computation’s meaning even though the array retains the expected shape.

Reloading reproduces probability 0.715550 and class 1, confirming that the saved parameters and preprocessing reproduce the prediction.

Visualize the learning

These optional blocks run after the tutorial. Install Matplotlib with python -m pip install matplotlib.

Plot training and validation loss

import matplotlib.pyplot as plt

history_array = np.array(history)
plt.plot(history_array[:, 0], history_array[:, 1], label="Training")
plt.plot(history_array[:, 0], history_array[:, 2], label="Validation")
plt.xlabel("Completed updates")
plt.ylabel("Mean binary cross-entropy")
plt.legend()
plt.tight_layout()
plt.show()

Each point describes a dataset’s average loss. That average can improve even when an individual prediction temporarily gets worse, because the parameters are shared across observations.

Plot the decision boundary

moisture_grid, temperature_grid = np.meshgrid(
    np.linspace(10, 70, 160), np.linspace(15, 40, 160)
)
grid_raw = np.column_stack([moisture_grid.ravel(), temperature_grid.ravel()])
_, _, grid_probability = forward((grid_raw - mean) / scale, parameters)
probability_surface = grid_probability.reshape(moisture_grid.shape)

fig, ax = plt.subplots()
filled = ax.contourf(
    moisture_grid, temperature_grid, probability_surface,
    levels=np.linspace(0, 1, 11), cmap="viridis"
)
ax.contour(moisture_grid, temperature_grid, probability_surface,
           levels=[0.5], colors="white")
temperatures = np.linspace(15, 40, 160)
ax.plot(25 + 0.04 * (temperatures - 20) ** 2, temperatures,
        color="red", linestyle="--", label="Simulator threshold before variation")
ax.set_xlabel("Soil moisture (%)")
ax.set_ylabel("Temperature (°C)")
fig.colorbar(filled, ax=ax, label="Probability of needing water")
ax.legend()
fig.tight_layout()
plt.show()

The white contour marks probability 0.5 and forms the model’s decision boundary. The red dashed curve is the simulator’s threshold before variation. Compare their shapes to see how closely the network approximates the relationship that generated the labels.

The complete script

Save this as neural_network.py and run python neural_network.py. It trains the model, prints the results and parameter trace, and writes plant_water_model.npz in the working directory. Small numerical differences can occur across environments.

import numpy as np


def make_data(n=1800, seed=42):
    rng = np.random.default_rng(seed)
    moisture = rng.uniform(10, 70, n)
    temperature = rng.uniform(15, 40, n)
    threshold = 25 + 0.04 * (temperature - 20) ** 2
    variation = rng.normal(0, 3, n)
    needs_water = moisture < threshold + variation
    X = np.column_stack([moisture, temperature])
    y = needs_water.astype(float).reshape(-1, 1)
    return X, y


def initialize(seed=7, hidden_units=8):
    rng = np.random.default_rng(seed)
    return {
        "W1": rng.normal(0, 1 / np.sqrt(2), (2, hidden_units)),
        "b1": np.zeros((1, hidden_units)),
        "W2": rng.normal(0, 1 / np.sqrt(hidden_units), (hidden_units, 1)),
        "b2": np.zeros((1, 1)),
    }


def forward(X, parameters):
    z1 = X @ parameters["W1"] + parameters["b1"]
    hidden = np.tanh(z1)
    logits = hidden @ parameters["W2"] + parameters["b2"]
    probability = np.exp(-np.logaddexp(0, -logits))
    return hidden, logits, probability


def loss(logits, targets):
    return np.mean(np.logaddexp(0, (1 - 2 * targets) * logits))


def backward(X, targets, hidden, probability, parameters):
    d_logits = (probability - targets) / len(X)
    dW2 = hidden.T @ d_logits
    db2 = d_logits.sum(axis=0, keepdims=True)
    d_hidden = d_logits @ parameters["W2"].T
    d_z1 = d_hidden * (1 - hidden ** 2)
    dW1 = X.T @ d_z1
    db1 = d_z1.sum(axis=0, keepdims=True)
    return {"W1": dW1, "b1": db1, "W2": dW2, "b2": db2}


def metrics(X, targets, parameters):
    _, logits, probability = forward(X, parameters)
    predicted = probability >= 0.5
    actual = targets == 1
    tp = int(np.sum(predicted & actual))
    tn = int(np.sum(~predicted & ~actual))
    fp = int(np.sum(predicted & ~actual))
    fn = int(np.sum(~predicted & actual))
    return {
        "loss": float(loss(logits, targets)),
        "accuracy": float(np.mean(predicted == actual)),
        "precision": tp / max(tp + fp, 1),
        "recall": tp / max(tp + fn, 1),
        "confusion": [[tn, fp], [fn, tp]],
    }


X_raw, y = make_data()
indices = np.random.default_rng(19).permutation(len(X_raw))
train_ids = indices[:1200]
validation_ids = indices[1200:1500]
test_ids = indices[1500:]
mean = X_raw[train_ids].mean(axis=0, keepdims=True)
scale = X_raw[train_ids].std(axis=0, keepdims=True)
scale = np.where(scale == 0, 1.0, scale)
X_train = (X_raw[train_ids] - mean) / scale
X_validation = (X_raw[validation_ids] - mean) / scale
X_test = (X_raw[test_ids] - mean) / scale
y_train, y_validation, y_test = y[train_ids], y[validation_ids], y[test_ids]

parameters = initialize()
initial = {name: value.copy() for name, value in parameters.items()}
learning_rate = 0.1
epochs = 4000
best_validation_loss = float("inf")
best_parameters = None
best_epoch = 0
history = []

for epoch in range(epochs + 1):
    hidden, logits, probability = forward(X_train, parameters)
    training_loss = float(loss(logits, y_train))
    _, validation_logits, _ = forward(X_validation, parameters)
    validation_loss = float(loss(validation_logits, y_validation))
    history.append([epoch, training_loss, validation_loss])
    if validation_loss < best_validation_loss:
        best_validation_loss = validation_loss
        best_parameters = {name: value.copy() for name, value in parameters.items()}
        best_epoch = epoch
    if epoch in (0, 1, 100, 500, 1000, 4000):
        print(f"{epoch:4d}  train={training_loss:.6f}  validation={validation_loss:.6f}")
    if epoch == epochs:
        break
    gradients = backward(X_train, y_train, hidden, probability, parameters)
    for name in parameters:
        parameters[name] -= learning_rate * gradients[name]

parameters = best_parameters
print("Selected epoch:", best_epoch)
majority_class = int(y_train.mean() >= 0.5)
print(f"Baseline test accuracy: {np.mean(y_test == majority_class):.3f}")
print("Training metrics:", metrics(X_train, y_train, parameters))
print("Test metrics:", metrics(X_test, y_test, parameters))

example_raw = np.array([[32.0, 35.0]])
example = (example_raw - mean) / scale
for label, current in (("Before", initial), ("After", parameters)):
    hidden, logits, probability = forward(example, current)
    print(label, "hidden:", np.round(hidden, 4))
    print(label, "logit:", np.round(logits, 4), "probability:", np.round(probability, 4))
print("W1[0, 0]:", initial["W1"][0, 0], "->", parameters["W1"][0, 0])

scenarios = np.array([[20., 25.], [50., 25.], [32., 20.], [32., 35.]])
_, _, probabilities = forward((scenarios - mean) / scale, parameters)
print("Scenario probabilities:", np.round(probabilities.ravel(), 4))

np.savez(
    "plant_water_model.npz",
    **parameters,
    mean=mean,
    scale=scale,
    feature_names=np.array(["soil_moisture_pct", "temperature_c"]),
    decision_threshold=np.array(0.5),
)
with np.load("plant_water_model.npz", allow_pickle=False) as saved:
    restored = {name: saved[name] for name in ("W1", "b1", "W2", "b2")}
    restored_input = (example_raw - saved["mean"]) / saved["scale"]
    _, _, restored_probability = forward(restored_input, restored)
    restored_class = restored_probability >= saved["decision_threshold"]
print("Reloaded probability:", float(restored_probability[0, 0]))
print("Reloaded needs_water:", int(restored_class[0, 0]))

Common mistakes and how to fix them

Broadcasting deserves special attention. Subtracting targets shaped (1200,) from predictions shaped (1200, 1) creates a (1200, 1200) result. Keeping both arrays shaped (1200, 1) preserves the intended one-target-per-row calculation.

Practice experiments

Experiment 1 — Inspect the learning rate

Run fresh sessions with learning rates 0.01, 0.1, and 0.5, keeping the data, initialization, and update budget fixed. Record training and validation loss at the same checkpoints.

Explain which learns fastest and whether its validation behaviour stays stable. Select using validation results and reserve the test set for the final selected experiment.

Experiment 2 — Remove temperature

Keep only moisture and change the first weight matrix to accept one input. Update the saved feature names and inference inputs. Retrain and compare validation performance.

Focus on observations near 32% moisture. The simulator can label these differently depending on temperature. A moisture-only model must give identical moisture values the same prediction, losing information that helps distinguish those cases.

Experiment 3 — Check a gradient numerically

Before the first update, choose W1[0, 0]. Calculate loss with that weight increased by 0.00001, then with it decreased by the same amount. Divide the loss difference by 0.00002 and restore the original weight.

Compare this finite-difference estimate with backward. In the supplied run, the analytical gradient is −0.215379589521 and the numerical estimate is −0.215379589502. Their agreement checks this specific derivative implementation.

Experiment 4 — Remove the hidden nonlinearity

Replace hidden = np.tanh(z1) with hidden = z1. In backward, replace the tanh derivative line with d_z1 = d_hidden. Change both so the gradient matches the forward computation.

The resulting model has an affine score and a straight classification boundary. Compare validation loss and boundary shape with the original. Many observations may still be classified correctly even if some curvature is missed.

Practice project — Learn from three sensor readings

Extend the simulator with sunlight_hours, sampled uniformly from 0 to 10. Add 0.8 * sunlight_hours to the moisture threshold as an explicit new synthetic assumption.

Create three-column inputs, split and scale all three features, and change the first weight matrix from (2, 8) to (3, 8). Its initialization scale becomes 1 / np.sqrt(3). The network now has 41 trainable parameters.

Train from a fresh initialization. Compare two observations with identical moisture and temperature but different sunlight hours. Check whether predictions reflect the added threshold contribution, then save and reload the model.

Your completion checks are consistent shapes, a baseline comparison, validation-based checkpoint selection, final test metrics, and matching predictions after reloading.

Frequently asked questions

What is a neural network in one sentence?

A neural network is a machine-learning model that transforms inputs through connected calculation units, using weights and biases that training adjusts to improve its outputs for an objective.

Is a neural network the same as machine learning?

No. Neural networks are one family of machine-learning models. Other families include decision trees and linear models. Deep learning uses neural networks with multiple learned layers.

Does a neural network keep learning whenever I use it?

Our model does not. Calling forward computes a prediction using the current parameters. Learning requires a separate training procedure that calculates and applies updates. Saving a model and loading it for prediction does not automatically enable further training.

Can I build a neural network using only NumPy?

Yes. NumPy supplies the array operations for the forward pass and parameter updates. This example implements derivatives explicitly in backward.

What exactly does the network learn?

It learns values for 33 weights and biases. The relationship between inputs and predictions is represented across those parameter arrays and the activation functions.

Does each hidden neuron represent a named concept?

We assigned no semantic labels to hidden neurons. Each calculates a learned response to inputs. Assigning a meaning such as “heat detector” requires evidence from its behaviour across observations.

Why does the network need labels?

Loss compares predictions with targets. That comparison supplies the signal used to calculate gradients in supervised learning.

Why can loss improve while accuracy stays the same?

Accuracy depends on which side of the threshold a prediction falls. Cross-entropy also responds to confidence. A positive prediction moving from 0.7 to 0.9 keeps its class and reduces its loss.

Why initialize weights randomly?

Different weights help neurons begin with different responses. Identically initialized neurons can receive identical updates and remain redundant. Zero biases work here because random weights already break that symmetry.

Do I need a GPU?

This network has 33 trainable parameters and 1,200 training rows. Its computation is small enough for a CPU. Much larger models and datasets can benefit substantially from GPUs.

What would PyTorch handle?

PyTorch can record differentiable tensor operations and calculate gradients automatically. Optimizers apply updates and modules manage parameters. The forward calculations, loss objective, and gradient-based learning principle still apply.

Can I predict outside the sampled ranges?

Evaluation covers moisture 10–70% and temperature 15–40°C. Behaviour beyond those ranges needs its own evaluation. Producing a number for an unfamiliar input does not establish reliable performance there.

What should I learn next?

Reimplement this network in PyTorch and compare predictions and gradients using the same parameters and data. Then explore mini-batches, optimizers, regularization, and evaluation across multiple data splits.

Glossary

Conclusion

A neural network connects simple calculations into a function whose weights and biases can be learned from examples. Our two inputs pass through eight hidden neurons to one output probability. Training adjusts that function; inference uses the resulting parameters to answer a new question.

For observation [32, 35], the initial network produced logit −0.3383 and probability 0.4162. After training, it produced logit 0.9225 and probability 0.7155.

You can trace that change through the code. The forward pass calculates activations and predictions. Loss measures agreement with targets. Backpropagation calculates gradients. Gradient descent changes weights and biases. Repeating these operations produces the trained arrays saved in plant_water_model.npz.

To check your understanding, stop after the first update and inspect W1[0, 0]. Reproduce its change from 0.00086985 to 0.02240781 using the gradient and learning rate. That calculation demonstrates the update principle applied to every trainable parameter in this network.