AI, machine learning and deep learning · Introductory

Activation functions

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

An activation function is a nonlinear function applied to a neuron's weighted sum of inputs. Without it, stacked layers collapse into a single linear map. Common choices are sigmoid, tanh, and ReLU, each with different output ranges and gradient behavior.

Intuition

Think of a neuron as computing a score z from its inputs, then deciding how strongly to 'fire'. A linear neuron simply passes the score along, so a deep network of them can only draw straight lines or flat planes. An activation function bends the score: sigmoid squashes it to between 0 and 1 like a probability, tanh squashes it between -1 and 1, and ReLU switches off negative scores and passes positive ones unchanged. These bends let layers combine into curved, complex decision boundaries, much like a medical test threshold turning a continuous biomarker level into a graded response.

Key equations

z=𝐰⊤𝐱+b,a=f(z)
(1)

A neuron computes a weighted sum z, then applies activation f to get output a.

σ(z)=11+e−z,σ′(z)=σ(z)(1−σ(z))
(2)

Sigmoid squashes values to (0,1); its gradient is at most 0.25.

tanh⁡(z)=ez−e−zez+e−z,tanh′⁡(z)=1−tanh2⁡(z)
(3)

Tanh outputs values in (-1,1), is zero-centered, with maximum gradient 1.

ReLU(z)=max⁡(0,z)
(4)

Passes positive inputs unchanged and outputs zero for negative inputs; gradient is 1 for z>0 and 0 for z<0.

Derivation

  1. Take two layers with no activation: a1 = W1 x + b1, then a2 = W2 a1 + b2.
  2. Substitute: a2 = W2(W1 x + b1) + b2 = (W2 W1) x + (W2 b1 + b2).
  3. This is W' x + b', a single linear layer, so depth adds no expressive power.
  4. Insert a nonlinear f between layers: a2 = W2 f(W1 x + b1) + b2. This cannot be reduced to one linear map in general.
  5. Derivative of sigmoid: with s = 1/(1+e^{-z}), ds/dz = e^{-z}/(1+e^{-z})^2 = s(1-s).
  6. Since s(1-s) is at most 0.25 (at z=0) and approaches 0 for large |z|, gradients shrink across many sigmoid layers, causing vanishing gradients.
  7. ReLU has gradient 1 for positive z, so gradients do not shrink there, which is why it is the default in deep networks.

Worked example

Problem

A neuron has weights w = (2, -1), bias b = 0.5, and input x = (1, 3). Compute the output using ReLU, sigmoid, and tanh.

  1. Compute z = 2*1 + (-1)*3 + 0.5 = 2 - 3 + 0.5 = -0.5.
  2. ReLU: max(0, -0.5) = 0.
  3. Sigmoid: 1/(1+e^{0.5}) = 1/(1+1.6487) ≈ 0.378.
  4. Tanh: (e^{-0.5} - e^{0.5})/(e^{-0.5}+e^{0.5}) = (0.6065-1.6487)/(0.6065+1.6487) ≈ -0.462.

Answer z = -0.5; ReLU = 0, sigmoid ≈ 0.378, tanh ≈ -0.462.

Common mistakes

  • Believing deeper linear networks are more powerful than a single linear layer.
  • Using sigmoid in many hidden layers and being surprised by vanishing gradients.
  • Forgetting that ReLU gradient is zero for negative inputs, which can cause 'dead neurons'.
  • Confusing the output range: sigmoid is (0,1), tanh is (-1,1).
  • Applying a hidden-layer activation to the output layer when the task needs a different one, such as softmax for classes.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(123 lines)
import sys
import numpy as np
from manim import *
from blocks import *


def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


def relu(z):
    return np.maximum(0, z)


def tanh(z):
    return np.tanh(z)


def leaky_relu(z, alpha=0.01):
    return np.where(z > 0, z, alpha * z)


class Main(Scene):
    def construct(self):
        self.camera.background_color = "#111118"
        TitleCard(self, "Activation Functions", "Non-linearities for Neural Networks")

        # Define a grid of functions to plot
        # We will plot them in a 2x2 grid using CurvePlot blocks or manual axes
        # To keep it clean, let's use a single large plot area or separate small ones.
        # Let's do a 2x2 grid of small plots.

        # Helper to create a small plot
        def make_plot(func, name, x_range=(-4, 4), y_range=(-2, 2), color=C_MAIN):
            axes = Axes(
                x_range=x_range,
                y_range=y_range,
                x_length=3.5,
                y_length=2.5,
                axis_config={"color": C_DIM, "include_tip": True, "font_size": 20}
            )
            axes.to_edge(LEFT, buff=0.5).shift(UP * 0.5) # Placeholder, will position manually
            curve = axes.plot(func, x_range=x_range, color=color, use_smoothing=False)
            label = MathTex(name, font_size=24, color=C_ACCENT)
            label.next_to(axes, UP, buff=0.1)
            return VGroup(axes, curve, label)

        # Positions for 2x2 grid
        # Top-Left: Sigmoid
        # Top-Right: Tanh
        # Bottom-Left: ReLU
        # Bottom-Right: Leaky ReLU

        # Define positions manually to ensure no overlap and fit in frame
        # Frame is 14.2 x 8.0
        # Let's place them in a 2x2 grid centered
        
        pos_tl = np.array([-3.5, 1.5, 0])
        pos_tr = np.array([3.5, 1.5, 0])
        pos_bl = np.array([-3.5, -1.5, 0])
        pos_br = np.array([3.5, -1.5, 0])

        # Sigmoid
        sig_axes = Axes(x_range=(-6, 6, 2), y_range=(-0.1, 1.1, 0.5), x_length=4, y_length=2.5, axis_config={"color": C_DIM})
        sig_axes.move_to(pos_tl)
        sig_curve = sig_axes.plot(sigmoid, x_range=(-6, 6), color=C_MAIN)
        sig_label = MathTex(r"\sigma(z) = \frac{1}{1+e^{-z}}", font_size=20, color=C_ACCENT)
        sig_label.next_to(sig_axes, UP, buff=0.1)
        sig_group = VGroup(sig_axes, sig_curve, sig_label)

        # Tanh
        tanh_axes = Axes(x_range=(-6, 6, 2), y_range=(-1.5, 1.5, 1), x_length=4, y_length=2.5, axis_config={"color": C_DIM})
        tanh_axes.move_to(pos_tr)
        tanh_curve = tanh_axes.plot(np.tanh, x_range=(-6, 6), color=C_MAIN)
        tanh_label = MathTex(r"\tanh(z)", font_size=20, color=C_ACCENT)
        tanh_label.next_to(tanh_axes, UP, buff=0.1)
        tanh_group = VGroup(tanh_axes, tanh_curve, tanh_label)

        # ReLU
        relu_axes = Axes(x_range=(-6, 6, 2), y_range=(-1, 6, 1), x_length=4, y_length=2.5, axis_config={"color": C_DIM})
        relu_axes.move_to(pos_bl)
        relu_curve = relu_axes.plot(relu, x_range=(-6, 6), color=C_MAIN)
        relu_label = MathTex(r"\text{ReLU}(z) = \max(0, z)", font_size=20, color=C_ACCENT)
        relu_label.next_to(relu_axes, UP, buff=0.1)
        relu_group = VGroup(relu_axes, relu_curve, relu_label)

        # Leaky ReLU
        lrelu_axes = Axes(x_range=(-6, 6, 2), y_range=(-1, 6, 1), x_length=4, y_length=2.5, axis_config={"color": C_DIM})
        lrelu_axes.move_to(pos_br)
        lrelu_curve = lrelu_axes.plot(lambda z: leaky_relu(z, 0.1), x_range=(-6, 6), color=C_MAIN)
        lrelu_label = MathTex(r"\text{Leaky ReLU}(z)", font_size=20, color=C_ACCENT)
        lrelu_label.next_to(lrelu_axes, UP, buff=0.1)
        lrelu_group = VGroup(lrelu_axes, lrelu_curve, lrelu_label)

        # Animate them in
        self.play(Create(sig_group), run_time=1)
        self.play(Create(tanh_group), run_time=1)
        self.play(Create(relu_group), run_time=1)
        self.play(Create(lrelu_group), run_time=1)

        StepCaption(self, "Common non-linearities introduce expressivity to neural networks")
        self.wait(2)


if __name__ == "__main__" and "--selftest" in sys.argv:
    # Test sigmoid properties
    assert abs(sigmoid(0) - 0.5) < 1e-9
    assert abs(sigmoid(10) - 1.0) < 1e-4
    assert abs(sigmoid(-10) - 0.0) < 1e-4
    
    # Test ReLU properties
    assert relu(5) == 5
    assert relu(-5) == 0
    
    # Test Tanh properties
    assert abs(tanh(0) - 0) < 1e-9
    assert abs(tanh(10) - 1.0) < 1e-4
    
    # Test Leaky ReLU properties
    assert leaky_relu(5) == 5
    assert abs(leaky_relu(-5, 0.1) - (-0.5)) < 1e-9
    
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.