Activation Functions Explained: ReLU, Sigmoid, Tanh & Softmax

An activation function is the small piece of math applied after a layer's weighted sum that decides how strongly a neuron "fires." Without it, a neural network's layers could be collapsed into a single equivalent linear layer no matter how many were stacked; with it, a network can bend, curve, and carve up its decision space in ways a straight line never could. ReLU, Sigmoid, Tanh, and Softmax are the four activation functions almost every practitioner meets first, and each one exists to solve a different, specific problem. 🔥

This matters in production because the wrong activation choice, or the right one used carelessly, causes some of the most common training failures: gradients that vanish to nothing in deep networks, neurons that permanently "die" and stop learning, and probability outputs that silently stop summing to one. Knowing exactly what each function does to a number — and to a gradient — is what separates debugging a stalled model in minutes versus days. ⚙️

Figure 1. Original diagram: the same input range, three very different output shapes.

📑 In This Post

1. Foundations: what an activation function actually does
2. Sigmoid and Tanh: squashing functions and their cost
3. ReLU: simplicity, sparsity, and the dying neuron problem
4. Softmax: turning scores into probabilities
5. Real example: the 2010 paper that helped popularize ReLU
6. Implementation walkthrough with short code examples
7. Enterprise rollout: choosing, testing, and monitoring activations
8. Common mistakes and why they hurt production systems
9. FAQ
10. References & further reading
11. Summary

🔀 Quick Comparison: The Four Activation Functions

Property	Sigmoid	Tanh	ReLU	Softmax
Output range	(0, 1)	(-1, 1)	[0, ∞)	(0, 1) per entry, summing to 1
Applied to	One value at a time	One value at a time	One value at a time	A whole vector jointly
Typical use	Binary output gate, older hidden layers	Zero-centered hidden layers, gating in RNNs	Default hidden-layer choice in most modern networks	Final layer of a multi-class classifier
Main failure mode	Vanishing gradient when saturated	Vanishing gradient when saturated	Dying neurons stuck at zero	Numeric overflow on large logits if unguarded

1. Foundations: What an Activation Function Actually Does

🧠 Child-friendly analogy first: imagine a row of light dimmer switches, but each one is a plain on-or-proportional dimmer with no personality of its own — turning two of them in sequence is exactly the same as turning one dimmer to the combined setting. Now imagine replacing some of those plain dimmers with switches that behave strangely on purpose: one that only lets light through once it crosses a threshold, one that can flip from "warm" to "cool" light depending on the input. Suddenly, chaining them produces effects a single plain dimmer never could. That strange, deliberate non-plainness is what an activation function adds.

Formally, an activation function f takes the pre-activation value z (the weighted sum of inputs plus bias) and maps it to the unit's final output, y = f(z). The mathematical reason this step cannot be skipped or replaced with another linear operation is that the composition of any number of linear functions is itself always a single linear function — so a "deep" stack of purely linear layers has no more representational power than one shallow layer of the same overall shape.

Different activation functions exist because they make different trade-offs between three things: how easily gradients flow backward through them during training, how expensive they are to compute at scale, and what shape of output they naturally produce (a bounded probability-like value, an unbounded positive value, or a full probability distribution over several classes).

🎯 Use this when you need to explain to a teammate why "just remove the activation functions and keep more layers" would actually make a network weaker, not stronger.

2. Sigmoid and Tanh: Squashing Functions and Their Cost

🧠 Analogy: think of a volume knob on an old radio that gets harder and harder to turn the closer it gets to fully off or fully blasting — small twists near the middle change the volume a lot, but the same small twist near either end barely changes anything at all. Sigmoid and tanh behave exactly like that stiff knob: near their extremes, a real change in the input barely changes the output, which means barely any gradient signal survives to flow backward through them.

What they do: the sigmoid function squashes any real number into the open range between 0 and 1. PyTorch's own documentation defines it directly as sigma(x) = 1 / (1 + exp(-x)). (PyTorch Sigmoid documentation) The hyperbolic tangent, tanh, squashes into the range between -1 and 1 instead, which keeps its output centered around zero rather than around 0.5.

Why they were needed: early neural networks needed a smooth, differentiable stand-in for a hard on/off decision, since a true step function has a derivative of zero almost everywhere and cannot be trained with gradient-based methods. Sigmoid and tanh provided that smooth approximation while still resembling a "decision" shape.

What fails without understanding them: when the input to sigmoid or tanh is very large or very negative, the function's output flattens out — it "saturates" — and its local slope approaches zero. During backpropagation, that near-zero slope multiplies into the gradient flowing backward through every earlier layer, so stacking many saturating layers can shrink the gradient reaching early layers down to nearly nothing. This is the classic vanishing gradient problem, and it is a major reason sigmoid and tanh are used far more sparingly as hidden-layer activations in deep networks today than they were in earlier network designs.

💡 Trade-off: sigmoid's output can be directly interpreted as a probability, which is exactly why it remains the standard choice for a single-output binary classification head even in modern networks — the vanishing-gradient concern mainly applies to using it repeatedly across many stacked hidden layers, not to a single well-placed output layer.

🎯 Use this when deciding whether sigmoid or tanh belongs in a hidden layer versus an output layer, or when diagnosing a deep network whose early layers seem to stop learning.

3. ReLU: Simplicity, Sparsity, and the Dying Neuron Problem

🧠 Analogy: imagine a one-way turnstile at a train station: push it from the correct direction and it passes you through exactly as hard as you pushed; push it from the wrong direction and it simply doesn't move at all, no matter how hard you push. ReLU treats positive and negative inputs the same asymmetric way — it passes positive values through completely unchanged, and simply refuses to pass anything through at all for negative values.

What it does: PyTorch's documentation defines the rectified linear unit directly as ReLU(x) = max(0, x) — every negative input becomes exactly zero, and every positive input passes through completely unchanged. (PyTorch ReLU documentation)

Why it's needed: because ReLU's slope is exactly 1 for every positive input, it doesn't saturate on the positive side the way sigmoid and tanh do, which lets gradients flow backward through many stacked layers far more easily. It's also extremely cheap to compute — a single comparison against zero — compared to an exponential.

How it works, step by step, inside a layer:

1. The layer computes its usual weighted sum plus bias for every unit, producing a pre-activation vector.
2. ReLU is applied elementwise: every negative entry becomes 0, every non-negative entry is left unchanged.
3. During the backward pass, the local gradient of ReLU is exactly 1 wherever the pre-activation was positive, and exactly 0 wherever it was negative or zero.
4. That gradient multiplies into the chain rule exactly like any other layer's local gradient, but without the shrinking effect that a saturating function's near-zero slope would introduce.

What fails without understanding it: because ReLU's gradient is exactly zero for any negative pre-activation, a unit whose weights and bias happen to push it into permanently negative pre-activation values for every training example stops receiving any gradient signal at all — it can never update its weights again and effectively "dies," contributing nothing to the network from that point forward. This is known as the dying ReLU problem, and it's one reason variants like Leaky ReLU (which allows a small non-zero slope for negative inputs) exist, though this article focuses on the standard ReLU as documented above.

🎯 Use this when a network's training accuracy plateaus early and you suspect a large fraction of hidden units have gone permanently inactive.

4. Softmax: Turning Scores into Probabilities

🧠 Analogy: imagine three friends voting for a restaurant, each holding up a hand-drawn sign to show how much they want it, with bigger signs meaning stronger preference — but the signs are all different, unstandardized sizes. Softmax is the process of measuring every sign, exaggerating the size differences a bit, and then rescaling all three so their sizes add up to exactly 100%, giving you a clean, comparable percentage for each option instead of three arbitrary sign sizes.

What it does: unlike sigmoid, tanh, and ReLU, softmax is not applied to one number at a time — it's applied to a whole vector of raw scores (often called logits) jointly. PyTorch's documentation defines it as Softmax(x_i) = exp(x_i) / sum_j(exp(x_j)), and explicitly notes that it rescales the vector so every element lies in the range [0, 1] and the whole vector sums to 1. (PyTorch Softmax documentation)

Figure 2. Original diagram: exponentiate first, then normalize by the total — that order matters.

Why it's needed: a multi-class classifier needs to output something interpretable as "how confident am I in each possible class," and those confidences need to be non-negative and sum to exactly one to behave like a valid probability distribution. Raw logits from a final linear layer satisfy neither property on their own — softmax is what converts them into one.

What fails without understanding it: exponentiating a large positive logit can overflow standard floating-point range before the division step ever happens, producing an invalid result instead of a valid probability; PyTorch's functional softmax implementation documents an optional dtype argument specifically described as useful for preventing this kind of data type overflow. (PyTorch functional softmax documentation) Separately, PyTorch's documentation also notes that its softmax module should not be paired directly with its negative log-likelihood loss, since that loss expects the log of the softmax to be computed as one combined, more numerically stable step instead.

🎯 Use this when building the final layer of any multi-class classifier, and always check which loss function you're pairing it with.

5. Real Example: The 2010 Paper That Helped Popularize ReLU

Rectified linear units were formally studied and shown to improve results in Vinod Nair and Geoffrey Hinton's 2010 paper "Rectified Linear Units Improve Restricted Boltzmann Machines," presented at the International Conference on Machine Learning. The paper's own abstract describes generalizing binary stochastic hidden units into an approximation the authors call noisy, rectified linear units, and reports that the resulting features performed better than binary units for object recognition on the NORB dataset and for face verification on the Labeled Faces in the Wild dataset. (Nair & Hinton, "Rectified Linear Units Improve Restricted Boltzmann Machines," ICML 2010)

✅ Worked example: the paper's abstract specifically notes that, unlike binary units, rectified linear units preserve information about relative intensities as information travels through multiple layers of feature detectors — a property directly tied to the fact that ReLU passes positive values through completely unscaled, rather than compressing them the way a saturating function does.

This single, narrowly scoped result about restricted Boltzmann machines is one documented data point in ReLU's history, not a claim that this paper alone caused every later network to adopt it; ReLU's broad popularity across many later architectures reflects a longer body of subsequent work beyond the scope of this article.

🎯 Use this when you want a concrete, citable historical anchor for why ReLU became a serious alternative to saturating activation functions.

6. Implementation Walkthrough

The short, illustrative examples below show the concepts above as code. They are simplified teaching snippets, not production-ready modules.

# Illustrative example: the four activation functions, from scratch
import math

def sigmoid(x):
    return 1.0 / (1.0 + math.exp(-x))

def tanh(x):
    return math.tanh(x)

def relu(x):
    return max(0.0, x)

def softmax(scores):
    exp_scores = [math.exp(s) for s in scores]
    total = sum(exp_scores)
    return [e / total for e in exp_scores]

print(softmax([2.0, 1.0, 0.1]))  # -> roughly [0.66, 0.24, 0.10]

# Illustrative example: checking for dead ReLU units during training
pre_activations = get_layer_pre_activations(batch)  # shape (batch_size, num_units)
always_negative = all_batch_values_below_zero(pre_activations, axis="batch")
dead_unit_count = count_true(always_negative)
log_metric("dead_relu_units", dead_unit_count)

Best practices: when implementing softmax by hand, subtract the maximum logit from every logit before exponentiating (a mathematically equivalent operation that keeps the largest exponent at zero and avoids overflow), and log the fraction of ReLU units that stay inactive across an entire batch as a routine training-health metric rather than only investigating it after a model has already stalled.

🎯 Use this when writing a numerically stable softmax from scratch or adding early-warning training diagnostics to a new architecture.

7. Enterprise Rollout: Choosing, Testing, and Monitoring Activations

An activation function choice is a small architectural decision with outsized downstream consequences for training stability, so treat it with the same rigor as any other part of a production model's contract.

Ownership and governance: document the activation function used in every layer of a production architecture alongside its initialization scheme, since the two are usually chosen together (for example, initialization schemes designed for ReLU-based networks assume a specific expected fraction of zeroed activations) and changing one without revisiting the other can quietly destabilize training.

CI gates: add a regression test that runs a fixed, known input batch through the model and asserts that the fraction of permanently inactive ReLU units and the softmax output's sum-to-one property both stay within expected bounds, so a refactor that silently changes activation behavior is caught before deployment rather than discovered as a training slowdown weeks later.

Dataset and checkpoint versioning: record the exact activation function and any numerical-stability tricks (like the max-subtraction step in softmax) as part of a checkpoint's metadata, since silently swapping in a different but similarly named activation implementation between training and serving can produce subtly different outputs that are hard to trace back to their source.

Dashboards and alerts: track the distribution of pre-activation values per layer during training, and specifically alert on a rising fraction of permanently negative ReLU pre-activations, since that is one of the earliest reliable signals of the dying ReLU problem taking hold across a training run.

Incident response: when a classifier's softmax outputs stop summing to a value close to one, or when inference produces NaN probabilities, capture the exact logits that triggered the failure as part of the incident record, since an unguarded overflow in the exponentiation step is traceable to specific input logits far more often than it first appears.

🎯 Use this when a research architecture using non-default activation choices is being handed off to a team that will train and serve it at scale.

8. Common Mistakes

Using sigmoid or tanh across many stacked hidden layers by default. Because both functions saturate and their local slope approaches zero away from the origin, stacking many of them multiplies many near-zero slopes together during backpropagation, which is one of the classic causes of the vanishing gradient problem in deep networks. The production impact is early layers that receive almost no gradient signal and effectively stop learning, even though the loss curve might still show slow, confusing progress from later layers alone.

Ignoring a growing count of dead ReLU units. Because ReLU's gradient is exactly zero for negative pre-activations, a unit that drifts into permanently negative territory during training can never recover through gradient descent alone. The production impact is wasted model capacity that silently shrinks over the course of training, without any explicit error, until the effective network is smaller than the one that was designed and budgeted for.

Computing softmax without subtracting the maximum logit first. Directly exponentiating a large logit can overflow floating-point range before the normalizing division ever happens, and PyTorch's own functional softmax documentation notes a dtype option specifically meant to help prevent this class of overflow. The production impact is intermittent NaN outputs that appear to happen "randomly," when they're actually deterministic responses to specific large logit values.

Pairing softmax directly with a loss function that expects raw logits or log-probabilities. PyTorch's documentation explicitly warns that its softmax module doesn't work directly with its negative log-likelihood loss, which expects the log of the softmax computed as one combined step for better numerical properties. The production impact is either a training-time error or, worse, numerically unstable loss values that make debugging convergence problems much harder than necessary.

Treating softmax outputs as absolute, calibrated confidence. A softmax vector always sums to one and always assigns the largest share to the largest logit, even when the underlying model is genuinely unsure about all of its options — the shape of the distribution reflects the logits' relative differences, not necessarily true calibrated confidence, unless the model has been specifically evaluated and calibrated for that property.

❓ FAQ

Why is ReLU the default choice for hidden layers in most modern networks?

Mainly because it doesn't saturate for positive inputs, so it doesn't contribute to the vanishing gradient problem the way sigmoid and tanh can, and because computing max(0, x) is far cheaper than computing an exponential at the scale of millions or billions of activations per forward pass.

Is softmax applied to each number individually, like the other three functions?

No. Sigmoid, tanh, and ReLU are all applied elementwise, with each output depending only on its own input. Softmax's formula divides by a sum over every element in the vector, so every output value depends on every input value, not just its own.

What exactly is a "dead" ReLU unit?

It's a unit whose weights and bias have drifted such that its pre-activation value is negative for essentially every input the network sees, so ReLU always outputs zero for it and its gradient is always zero too — meaning gradient descent can never push its weights away from that state again.

Should I use sigmoid or softmax for a binary classification output?

A single sigmoid unit is the standard choice for binary classification, producing one probability for the positive class. Softmax is typically reserved for three or more mutually exclusive classes, where you need a full probability distribution across all of them at once.

Does tanh solve the vanishing gradient problem that sigmoid has?

Not fully. Tanh is zero-centered, which can help optimization compared to sigmoid, but it still saturates at both extremes just like sigmoid does, so stacking many tanh layers can still produce vanishing gradients in deep networks.

🔗 References & Further Reading

• PyTorch — torch.nn.ReLU (official documentation)
• PyTorch — torch.nn.Sigmoid (official documentation)
• PyTorch — torch.nn.Softmax (official documentation)
• PyTorch — torch.nn.functional.softmax (official documentation)
• Nair & Hinton — "Rectified Linear Units Improve Restricted Boltzmann Machines," ICML 2010 (original paper)

PyTorch is a trademark of the PyTorch Foundation. The 2010 paper is credited to its original authors, Vinod Nair and Geoffrey Hinton, and linked above. 

📝 Summary

• Activation functions add the non-linearity that lets stacked layers represent more than a single linear function ever could.
• Sigmoid and tanh squash values into a bounded range but saturate at the extremes, which can cause vanishing gradients in deep stacks.
• ReLU avoids saturation for positive inputs and is cheap to compute, but a unit stuck in negative territory can permanently "die."
• Softmax is a joint, vector-level operation that turns raw logits into a valid probability distribution summing to one.
• The 2010 Nair and Hinton paper is a documented, narrowly scoped early result showing rectified linear units outperforming binary units in specific tasks.
• Treat activation choice, initialization, and numerical-stability tricks as a versioned, tested, monitored part of a production model's contract.

From a stiff old radio knob to a clean turnstile to a room full of voters — that's the whole activation function lineup. Happy training! 🚀