An activation function is the non-linear rule applied after a layer’s linear computation.

For a layer:

$$z = Wx + b$$ $$a = f(z)$$

Where:

• z is the pre-activation or logit-like score.
• f is the activation function.
• a is the signal passed to the next layer.

The linear layer computes a score. The activation decides how that score should become signal.

Without activation functions, deep networks collapse into one linear transformation.

$$W_3(W_2(W_1x)) = W'x$$

Depth would add parameters, but not real expressive power.

Activation functions introduce non-linearity, which lets networks learn curves, thresholds, interactions, and hierarchical features.

This is the core point: linear layers mix information; activations bend the function.

Think of activations as gates.

Different gates pass information differently:

• Sigmoid: soft yes/no probability gate.
• Tanh: centered squashing gate.
• ReLU: hard positive-only gate.
• GELU / SiLU: smooth probabilistic gates.
• Softmax: competition gate across classes.

The activation shapes both the forward signal and the backward gradient.

Hidden-layer activations mainly affect optimization and representation.

They answer:

• Can gradients flow?
• Does the layer keep useful signal?
• Does training stay stable?
• Does the architecture expect this activation?

Output activations mainly affect prediction meaning.

They answer:

• Is this a probability?
• Are classes mutually exclusive?
• Can multiple labels be true?
• Is the output bounded or unconstrained?

During backpropagation, each activation contributes a local derivative.

If:

$$a = f(z)$$

then gradients flowing backward are multiplied by:

$$f'(z)$$

This is why activation shape matters. If f'(z) is often near zero, gradients shrink. If it is stable, gradients flow more easily.

In practice, activation functions are not just forward transformations; they are also gradient filters.

Saturation happens when large input ranges produce almost constant outputs.

Examples:

• Sigmoid saturates near 0 and 1.
• Tanh saturates near -1 and 1.

In saturated regions, derivatives are close to zero. During backpropagation, multiplying by many tiny derivatives can make gradients vanish.

That is why saturating activations are usually poor hidden-layer defaults in very deep networks.

A useful activation usually balances:

• Non-linearity
• Stable gradients
• Cheap computation
• Good behavior near zero
• Good behavior for large positive and negative inputs
• Suitable output range
• Compatibility with the architecture

There is no universally best activation. The right question is: what behavior do I need here?

Activation	Full form	Formula	Range
Sigmoid	Logistic sigmoid	$\sigma(x)=\frac{1}{1+e^{-x}}$	(0, 1)
Tanh	Hyperbolic tangent	$\tanh(x)$	(-1, 1)
ReLU	Rectified Linear Unit	$\max(0,x)$	[0, infinity)
Leaky ReLU	Leaky Rectified Linear Unit	$x$ if $x>0$, else $\alpha x$	(-infinity, infinity)
Softplus	Smooth ReLU-like function	$\log(1+e^x)$	(0, infinity)
SiLU	Sigmoid Linear Unit	$x\sigma(x)$	roughly (-0.28, infinity)
GELU	Gaussian Error Linear Unit	$x\Phi(x)$	roughly (-0.17, infinity)
Softmax	Vector probability map	$\frac{e^{z_i}}{\sum_j e^{z_j}}$	probabilities sum to 1

This table is the activation zoo in one card. Most practical decisions come from understanding the behavior, not memorizing every variant.

Sigmoid maps any real number to (0, 1).

$$\sigma(x)=\frac{1}{1+e^{-x}}$$

Its derivative is:

$$\sigma'(x)=\sigma(x)(1-\sigma(x))$$

Its maximum derivative is only 0.25, and it saturates for large positive or negative inputs.

Use sigmoid for binary or independent probabilities. Avoid it as a default hidden-layer activation.

Tanh maps values to (-1, 1) and is zero-centered.

$$\tanh(x)=2\sigma(2x)-1$$

Its derivative is:

$$\frac{d}{dx}\tanh(x)=1-\tanh^2(x)$$

Tanh is often better centered than sigmoid, but it still saturates.

Use it when bounded, symmetric output is useful, such as some recurrent states or action outputs scaled to [-1, 1].

ReLU stands for Rectified Linear Unit.

$$\operatorname{ReLU}(x)=\max(0,x)$$

For positive inputs, its derivative is 1. For negative inputs, its derivative is 0.

That makes it cheap and good for gradient flow on the positive side.

Its weakness is that negative signal is killed completely, which can cause dead neurons.

A ReLU unit can die when it outputs zero for almost all inputs.

If the pre-activation is always negative:

$$\operatorname{ReLU}(x)=0$$

and the gradient is also zero.

The unit may stop learning.

This is why variants like Leaky ReLU, PReLU, ELU, GELU, and SiLU preserve some negative-side signal.

Activation	Full form	Core idea
Leaky ReLU	Leaky Rectified Linear Unit	Keep a small fixed negative slope
PReLU	Parametric Rectified Linear Unit	Learn the negative slope
ELU	Exponential Linear Unit	Smoothly map negatives toward -\alpha
SELU	Scaled Exponential Linear Unit	ELU-like activation for self-normalizing networks

Use these when plain ReLU is too harsh, especially if dead neurons or negative-signal loss seem problematic.

SELU is special: it expects conditions like LeCun normal initialization and alpha dropout.

Softplus is a smooth approximation to ReLU.

$$\operatorname{Softplus}(x)=\log(1+e^x)$$

For large positive inputs, it behaves like x.

For large negative inputs, it approaches 0.

It is useful when you need a smooth positive output, such as variance, rate, scale, or another positive parameter.

SiLU stands for Sigmoid Linear Unit.

$$\operatorname{SiLU}(x)=x\sigma(x)$$

Swish is:

$$\operatorname{Swish}(x)=x\sigma(\beta x)$$

When beta = 1, Swish is SiLU.

The intuition: SiLU softly gates the input by its own sigmoid. It keeps large positive values, smoothly downweights negatives, and preserves some small negative signal.

GELU stands for Gaussian Error Linear Unit.

$$\operatorname{GELU}(x)=x\Phi(x)$$

Where Phi(x) is the standard normal CDF.

GELU is like a smooth, probabilistic ReLU: instead of hard-clipping negatives, it softly gates values based on magnitude.

It is common in transformer MLP blocks.

A common approximation is:

$$\operatorname{GELU}(x) \approx 0.5x\left(1+\tanh\left(\sqrt{\frac{2}{\pi}}(x+0.044715x^3)\right)\right)$$

This avoids computing the exact normal CDF.

You do not usually need to implement this by hand, but recognizing it helps when reading model code.

Softmax turns a vector of logits into a probability distribution.

$$\operatorname{softmax}(z_i)=\frac{e^{z_i}}{\sum_j e^{z_j}}$$

Each output is positive, and all outputs sum to 1.

Unlike sigmoid or ReLU, softmax is not elementwise. Each output depends on every logit.

Softmax makes classes compete.

Increasing one logit increases that class probability and lowers the relative probability of other classes.

Use softmax when exactly one class should be correct.

Examples:

• Digit classification
• Single-label image classification
• Next-token prediction
• Mutually exclusive classes

Exponentials can overflow for large logits.

Use:

$$\operatorname{softmax}(z_i)=\frac{e^{z_i-m}}{\sum_j e^{z_j-m}}$$

Where:

$$m=\max_j z_j$$

Subtracting the same constant from every logit does not change the probabilities, but it prevents very large exponentials.

Use sigmoid when outputs are independent.

Use softmax when outputs compete.

Situation	Activation
Is this email spam?	Sigmoid
Which digit is this?	Softmax
Which tags apply to this document?	Sigmoid per tag
What is the next token?	Softmax
Can this image be both indoor and blurry?	Sigmoid per label

For unconstrained regression, use a linear output.

For constrained regression:

Desired output	Activation
Any real number	Linear
Positive number	Softplus
Probability in (0, 1)	Sigmoid
Value in (-1, 1)	Tanh
Distribution over classes	Softmax

Hidden layers can still use ReLU, GELU, SiLU, or another hidden activation.

Many libraries combine the output activation and loss for numerical stability.

Examples:

• Binary cross-entropy with logits combines sigmoid and binary cross-entropy.
• Cross-entropy loss for multi-class classification often combines softmax and negative log likelihood.

This avoids unstable computations like taking log of probabilities that are extremely close to 0.

Practical rule: check whether the loss expects raw logits or already-activated probabilities. Passing both can silently hurt training.

GLU stands for Gated Linear Unit.

A GLU splits a projection into two parts: values and gates.

One common form is:

$$\operatorname{GLU}(x)=a \otimes \sigma(b)$$

Where a and b are learned projections, and \otimes means elementwise multiplication.

The model learns what information to pass through.

SwiGLU is a gated activation that uses a Swish or SiLU-style gate.

A simplified form is:

$$\operatorname{SwiGLU}(x)=a \otimes \operatorname{SiLU}(b)$$

It is common in modern transformer feedforward blocks.

For basics, learn ReLU, sigmoid, tanh, softmax, GELU, and SiLU first. Then treat GLU and SwiGLU as architecture refinements.

Practical defaults:

Architecture / use case	Good default
Simple MLP	ReLU or GELU
CNN	ReLU, Leaky ReLU, or SiLU
Transformer	GELU, SiLU, or SwiGLU
Recurrent gates	Sigmoid and tanh
Positive scalar output	Softplus

When in doubt, start with the activation used by the architecture you are implementing.

Use ReLU when you want a fast, simple baseline.

Use GELU when you are working with transformer-style architectures or want smooth probabilistic gating.

Use SiLU when you want a smooth ReLU-like activation that preserves small negative signal.

Activation	Main benefit	Main drawback
ReLU	Fast and simple	Can kill negative signal
GELU	Smooth and transformer-friendly	More expensive than ReLU
SiLU	Smooth and preserves small negative signal	More expensive than ReLU

The biggest beginner mistake is choosing activations only by habit.

Ask:

• What should the output mean?
• Are labels mutually exclusive or independent?
• Does the loss expect logits or probabilities?
• Does the activation saturate?
• Does it preserve useful gradients?
• Does the architecture expect a specific activation?

Output activations define prediction meaning. Hidden activations shape optimization.

Remember:

• Activations add non-linearity.
• Hidden activations shape learning dynamics.
• Output activations shape prediction meaning.
• Saturating activations can vanish gradients.
• ReLU is simple and strong.
• GELU and SiLU are smooth modern defaults.
• Sigmoid is for probabilities and gates.
• Softmax is for competing classes.
• There is no universally best activation.