How can neural networks learn nonlinear relationships if their layers use linear operations?

Start with the precise meaning of linear

In casual neural-network language, a dense or fully connected layer is often called a linear layer. More precisely, a layer that includes a bias is affine. It has the form z = Wx + b, where x is the input vector, W is a matrix of weights, and b is a vector of biases. The matrix multiplication mixes and scales inputs. The bias shifts the result. PyTorch describes its Linear module in exactly this affine form. PyTorch Linear documentation

An affine map can shift a line or plane, but its decision boundary remains flat. In two input dimensions, a linear threshold model can separate classes with a straight line. In many dimensions, its boundary is a flat hyperplane. Adding a bias changes where the boundary sits, not its flat shape.

A common hidden layer has two steps:

1. It calculates a pre-activation value, z = Wx + b.
2. It applies an activation element by element, h = phi(z).

Here, phi denotes the activation function. For ReLU, phi(z) = max(0, z). For sigmoid, it is an S-shaped function that maps any real number to a value between 0 and 1. The second operation is not affine. A neuron is therefore usually better described as a weighted sum, bias, and activation, not only as a weighted sum.

Why stacking affine layers alone collapses

Consider a network with two layers and no activation between them. The first layer computes h = W1x + b1. The second computes y = W2h + b2. Substitute the first equation into the second:

y = W2(W1x + b1) + b2

Distributing the multiplication gives:

y = (W2W1)x + (W2b1 + b2)

Call W2W1 a new weight matrix and call W2b1 + b2 a new bias. The result has exactly the form of one affine layer, y = W*x + b*. The intermediate layer may have more numbers and a different width, but it has not created a new kind of input-output relationship.

The same substitution works repeatedly. Three affine layers become one affine layer, ten affine layers become one affine layer, and a very large stack remains affine. This is why a network with only matrix multiplications and bias additions does not gain nonlinear expressive power by becoming deeper. Google’s neural-network lessons make the same point: linear operations performed on linear operations are still linear. Google’s activation-functions guide

Adding biases does not change this result. They matter because they let a model fit a shifted line or plane instead of forcing it through the origin, but the algebra above shows that all the biases still combine into one final bias. The network may contain more parameters, but it still represents an affine transformation of the original input.

What an activation function changes

Put a nonlinear activation between the two affine maps and the calculation becomes y = W2 phi(W1x + b1) + b2. The earlier substitution now stops at phi(W1x + b1). Matrix multiplication cannot be distributed through a nonlinear function. In general, there is no one matrix and one bias that produce the same mapping for every possible input.

This is composition: one function’s output becomes the next function’s input. Composition by itself is not enough if every function is affine, because affine functions are closed under composition. Composition becomes powerful when the intermediate function changes shape nonlinearly. Each layer can take the features produced by the earlier layer, divide or bend their value ranges, and build new features from them.

An activation need not be complicated. ReLU returns zero for a negative input and returns the input itself for a nonnegative input. It creates a hinge at zero. Sigmoid smoothly compresses very negative values near zero and very positive values near one. Tanh is another smooth, S-shaped activation with outputs from minus one to one. Google’s overview of common activation functions

The table is about function shape, not a rule that one activation is always best. In hidden layers, ReLU and variants are common partly because they tend to avoid some gradient difficulties of sigmoid and tanh. In an output layer, the appropriate activation depends on the prediction task. For example, a sigmoid can turn a score into a value between zero and one for binary classification. Google’s activation guidance

A small numerical example with ReLU

Here is a one-dimensional network that is deliberately small enough to calculate by hand. Its input is one number, x. It has two hidden ReLU units and a linear output:

h1 = ReLU(x + 1)

h2 = ReLU(x - 1)

y = h1 - 2h2

The weights and biases in this example are chosen by hand to show the idea. A trained network would adjust such values from data. Now calculate several inputs.

The output is not a single straight line. For inputs at or below minus one, both hidden units are off and the output is zero. Between minus one and one, only the first unit is on, so the output rises with slope 1. Above one, both units are on and the output falls with slope minus 1. The function is a tent shape with bends at minus one and one.

No single affine rule of the form wx + b can make that tent shape. A straight line has one slope everywhere, while this network has different slopes in different input regions. ReLU has not made the individual weighted sums mysterious. It has made their combination conditional on the region of the input, which is exactly the source of the new expressiveness.

Example as a classifier

Setup: Use the output from the preceding hypothetical network and classify an input as positive when y is greater than 1. The positive inputs lie strictly between 0 and 2. Inputs far to either side are negative.

Action: A one-dimensional affine classifier has the form wx + b, followed by a threshold. Its positive region is one side of a cutoff, such as all values greater than 4, or all values less than minus 2. It cannot mark only a middle interval as positive while leaving both outer sides negative. The two ReLU hidden units create the interval by turning on and off at different input values, and the output layer combines their values.

The activated hidden units let the model represent a more complex decision boundary. In one dimension, a boundary is a point and the positive region can become an interval. In two or more dimensions, the same mechanism can create several line segments or regions rather than one flat separating line.

Decision boundaries in more than one dimension

For a binary classifier, a decision boundary is the set of inputs where the model is exactly at its switching threshold. A single affine score, w1x1 + w2x2 + b, equals zero along a straight line in two dimensions. Thresholding that score colors one side positive and the other negative. It cannot, by itself, isolate a ring, an island, a checkerboard pattern, or two separated positive areas.

Each hidden ReLU unit starts with its own affine score. The line where that score equals zero is a gate. On one side, the unit outputs zero. On the other side, it passes through a linear value. Several such gates cut the input plane into regions. Within one region, the network is still affine because the on or off state of every ReLU is fixed. Across a gate, the formula changes. Combining enough regions lets the final score bend or join boundaries.

That is why a ReLU network’s output is called piecewise linear. It is not a smooth curve everywhere, but with enough pieces it can approximate many curved-looking shapes. A finite ReLU network represents polygonal pieces exactly, not a mathematically perfect circle. For a classification task, that is often enough if the pieces fit the data and generalize to unseen cases.

Google illustrates the underlying problem with patterns that a line cannot separate, including a circular cluster surrounded by another class. It also contrasts hand-designed feature crosses with neural networks that learn useful nonlinear interactions during training. Google’s neural-networks introduction

What depth contributes after nonlinearities exist

Activations supply the essential break from affine closure. Depth supplies repeated opportunities to transform and recombine features. An early image-model layer might respond to simple local contrasts, a later layer might combine them into texture-like patterns, and later layers may use those patterns for a task-specific score. This is an intuition for representation building, not a claim that every hidden unit has a human-readable meaning.

Depth does not guarantee better results. A shallow network with enough hidden units and a suitable nonlinear activation can approximate a broad class of functions under mathematical conditions. Cybenko’s foundational result concerns approximating continuous functions on a bounded domain with a sigmoidal hidden layer. Cybenko’s 1989 paper That is an expressiveness result. It does not say a small network will learn the desired function, that training will find the useful parameters, that the model will need few examples, or that it will generalize well.

Deep networks can represent some functions more efficiently than a single extremely wide hidden layer, but architecture choices are practical trade-offs. More layers bring more parameters, computation, tuning choices, and potential optimization difficulties. The useful question is not "is deeper always more nonlinear?" It is "does this architecture, activation, data, and training setup capture the structure of this task without overfitting?"

The output activation is a separate choice

It is common to see a final sigmoid activation in a binary classifier. That makes the displayed output nonlinear as a number, because the sigmoid bends the final score into the range from zero to one. However, if every hidden layer before it is affine, the pre-sigmoid score still has the form w*x + b. Because sigmoid is monotonic, thresholding it produces the same straight decision boundary as thresholding that affine score.

So a final sigmoid alone is useful for output interpretation, but it does not give the hidden representation the ability to create nonlinear decision boundaries. For that, nonlinearity must occur before a later affine combination, usually in one or more hidden layers. This distinction resolves a frequent apparent contradiction: a model can have a nonlinear output function while still being a linear classifier in terms of its decision boundary.

The output activation should match the task, not be copied by habit. A regression model predicting an unrestricted real number may use a linear output. A multiclass classifier often uses a softmax output. A binary classifier often uses a sigmoid output or uses a raw score with an appropriate loss. Those output choices do not replace the role of hidden activations in feature construction.

Learning the pieces versus having the capacity

Activation functions are usually fixed choices made by the model designer. The trainable parts are typically the weights and biases. During a forward pass, the network turns inputs into predictions and calculates a loss by comparing predictions with labels. During a backward pass, backpropagation computes how a small change to each parameter would affect that loss. An optimizer then updates the parameters. Google’s backpropagation guide

In the ReLU example, training could move the gates by changing the hidden biases, rotate or reshape them in multiple dimensions by changing weights, and change how strongly later layers combine them. The activation does not itself search for a relationship. It supplies a class of bends or gates that the learned parameters can position and combine.

This is why nonlinear capacity is necessary but not sufficient. A model may still fail because it is too small, badly initialized, trained with unsuitable settings, shown too little or biased data, or evaluated on the wrong metric. It can also overfit, learning accidental shapes in its training set rather than a pattern that holds for new inputs. Expressive models need held-out evaluation and regularization, not merely more layers.

Common misconceptions

The comparison matters in practice. A linear model can capture a nonlinear relationship if someone supplies nonlinear features, such as a product of two inputs or a squared input. A neural network can learn useful nonlinear feature transformations from data, but it is not always the simplest, most reliable, or easiest-to-explain choice.