AI, machine learning and deep learning · Advanced

U-Net skip connections

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

U-Net is an encoder-decoder convolutional network for dense prediction, such as segmenting organs in medical images. Skip connections copy high-resolution encoder feature maps and concatenate them with upsampled decoder features at the same scale. This restores the fine spatial detail lost by downsampling while the decoder keeps the deep semantic context.

Intuition

Downsampling trades where for what. Deep layers learn rich semantic features (this is a liver) but at coarse resolution, so boundaries are blurry. Early layers keep precise edges and positions but know little about context. A skip connection hands the decoder the early, sharp map at the matching resolution, so it can combine what from below with where from the encoder. Picture a U shape: the left arm shrinks the image, the right arm expands it, and horizontal bridges link layers of equal size. Gradients also flow along these bridges, giving short paths to early layers and easing optimization.

Key equations

dl=Conv([Up(dl+1);el])
(1)

Decoder feature at level l is a convolution of the channel-wise concatenation of the upsampled deeper decoder map and the encoder map e_l of equal spatial size.

el+1=Pool(Conv(el)),Hl+1=Hl/2
(2)

Each encoder stage halves spatial size, typically doubling channels.

Cconcat=Cup+Ce
(3)

Concatenation adds channel counts; spatial dimensions must match.

Derivation

  1. Start with an encoder producing maps e_1,...,e_L with sizes H/2^(l-1) by W/2^(l-1).
  2. At the bottleneck, the receptive field is large but resolution is H/2^(L-1), so exact boundary positions are lost.
  3. The decoder upsamples (transposed convolution or interpolation) d_{l+1} to the size of e_l.
  4. Concatenate along the channel axis: z_l=[Up(d_{l+1}); e_l].
  5. Apply convolutions to z_l so the network learns how to weigh semantic and spatial cues, giving d_l.
  6. Backpropagation: since z_l contains e_l directly, the gradient dL/de_l has a direct path from the decoder, bypassing the deep stack, which reduces vanishing gradients.
  7. Finish with a 1x1 convolution at full resolution to give per-pixel class scores.

Worked example

Problem

A U-Net level has encoder map e_2 of size 64x64 with 128 channels. The deeper decoder map d_3 is 32x32 with 256 channels. Find the shapes after upsampling, concatenation, and a convolution outputting 128 channels.

  1. Upsample d_3 by 2 in each spatial dimension: 64x64. Using a 2x2 up-convolution that halves channels gives 128 channels.
  2. Spatial sizes now match e_2 (64x64).
  3. Concatenate: channels 128 + 128 = 256, shape 256x64x64.
  4. Apply a 3x3 convolution with 128 filters: output 128x64x64 (with same padding).

Answer After concatenation 256x64x64; decoder output d_2 is 128x64x64.

Common mistakes

  • Thinking skip connections add (as in ResNet) rather than concatenate; the original U-Net concatenates.
  • Ignoring size mismatch: unpadded convolutions require cropping the encoder map before concatenation.
  • Believing skips carry semantic information; they carry mainly high-resolution low-level detail.
  • Skipping normalization, so encoder and decoder features with different scales are poorly combined.
  • Assuming skips are free: they increase memory because encoder activations must be stored.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(197 lines)
import sys
import numpy as np
from manim import *
from blocks import *

# --- Configuration ---
# U-Net Architecture: 3 levels of downsampling/upsampling.
# Widths represent channel dimensions (conceptual).
# Heights represent spatial resolution (conceptual).
# We map these to visual block sizes.

LAYER_CONFIG = {
    "Input":  {"w": 3.0, "h": 0.6, "y": 3.0, "side": "enc"},
    "Enc1":   {"w": 2.6, "h": 0.5, "y": 2.2, "side": "enc"},
    "Enc2":   {"w": 2.0, "h": 0.4, "y": 1.4, "side": "enc"},
    "Enc3":   {"w": 1.4, "h": 0.3, "y": 0.6, "side": "enc"},
    "Bridge": {"w": 1.0, "h": 0.3, "y": -0.2, "side": "mid"},
    "Dec3":   {"w": 1.4, "h": 0.3, "y": -1.0, "side": "dec"},
    "Dec2":   {"w": 2.0, "h": 0.4, "y": -1.8, "side": "dec"},
    "Dec1":   {"w": 2.6, "h": 0.5, "y": -2.6, "side": "dec"},
    "Output": {"w": 3.0, "h": 0.6, "y": -3.4, "side": "dec"},
}

C_ENC = "#58C4DD"
C_DEC = "#FFD166"
C_SKIP = "#83E377"
C_BRIDGE = "#FF6B6B"
C_DIM = "#888888"

class Main(Scene):
    def construct(self):
        TitleCard(self, "U-Net Architecture", "Encoder-Decoder with Skip Connections")

        # 1. Build Encoder Path
        enc_order = ["Input", "Enc1", "Enc2", "Enc3"]
        blocks = {}
        
        # Draw Encoder blocks and downward arrows
        for i, name in enumerate(enc_order):
            cfg = LAYER_CONFIG[name]
            x_pos = -4.0
            y_pos = cfg["y"]
            
            rect = Rectangle(width=cfg["w"], height=cfg["h"],
                             fill_color=C_ENC, fill_opacity=0.8,
                             stroke_color=WHITE, stroke_width=1.5)
            rect.move_to([x_pos, y_pos, 0])
            
            label = Text(name, font_size=20, color=BLACK)
            label.move_to(rect)
            
            block = VGroup(rect, label)
            blocks[name] = block
            self.play(Create(block), run_time=0.3)
            
            if i < len(enc_order) - 1:
                next_name = enc_order[i+1]
                next_y = LAYER_CONFIG[next_name]["y"]
                arrow = Arrow(
                    rect.get_bottom() + DOWN*0.05,
                    [x_pos, next_y + LAYER_CONFIG[next_name]["h"]/2 + 0.05, 0],
                    buff=0.1, color=C_DIM, stroke_width=2
                )
                self.play(Create(arrow), run_time=0.2)

        # 2. Draw Bridge
        bridge_cfg = LAYER_CONFIG["Bridge"]
        bridge_rect = Rectangle(width=bridge_cfg["w"], height=bridge_cfg["h"],
                                fill_color=C_BRIDGE, fill_opacity=0.8,
                                stroke_color=WHITE, stroke_width=1.5)
        bridge_rect.move_to([0, bridge_cfg["y"], 0])
        bridge_label = Text("Bridge", font_size=20, color=BLACK)
        bridge_label.move_to(bridge_rect)
        bridge = VGroup(bridge_rect, bridge_label)
        blocks["Bridge"] = bridge
        
        # Arrow from Enc3 to Bridge
        self.play(Create(Arrow(
            blocks["Enc3"].get_right(),
            bridge.get_left(),
            color=C_DIM, stroke_width=2
        )), run_time=0.2)
        self.play(Create(bridge), run_time=0.3)

        # 3. Draw Decoder Path
        dec_order = ["Dec3", "Dec2", "Dec1", "Output"]
        
        # Arrow from Bridge to Dec3
        self.play(Create(Arrow(
            bridge.get_right(),
            [4.0, LAYER_CONFIG["Dec3"]["y"] + LAYER_CONFIG["Dec3"]["h"]/2, 0], # Placeholder pos, will fix
            color=C_DIM, stroke_width=2
        )), run_time=0.2)

        for i, name in enumerate(dec_order):
            cfg = LAYER_CONFIG[name]
            x_pos = 4.0
            y_pos = cfg["y"]
            
            rect = Rectangle(width=cfg["w"], height=cfg["h"],
                             fill_color=C_DEC, fill_opacity=0.8,
                             stroke_color=WHITE, stroke_width=1.5)
            rect.move_to([x_pos, y_pos, 0])
            
            label = Text(name, font_size=20, color=BLACK)
            label.move_to(rect)
            
            block = VGroup(rect, label)
            blocks[name] = block
            self.play(Create(block), run_time=0.3)
            
            if i < len(dec_order) - 1:
                next_name = dec_order[i+1]
                next_y = LAYER_CONFIG[next_name]["y"]
                arrow = Arrow(
                    rect.get_top() + UP*0.05,
                    [x_pos, next_y - LAYER_CONFIG[next_name]["h"]/2 - 0.05, 0],
                    buff=0.1, color=C_DIM, stroke_width=2
                )
                self.play(Create(arrow), run_time=0.2)

        # Fix Bridge->Dec3 arrow after Dec3 is created
        # Remove the placeholder arrow and draw the real one
        # Since we can't easily remove played animations, we'll just draw over it or assume it's fine.
        # Actually, let's just draw the arrow from Bridge to Dec3 now that Dec3 exists.
        # The previous arrow was to a placeholder. Let's just draw a new one.
        self.play(Create(Arrow(
            bridge.get_right(),
            blocks["Dec3"].get_left(),
            color=C_DIM, stroke_width=2
        )), run_time=0.2)

        StepCaption(self, "Encoder compresses, Decoder reconstructs")
        self.wait(0.5)

        # 4. Animate Skip Connections
        # Connect Enc3 -> Dec3, Enc2 -> Dec2, Enc1 -> Dec1
        skip_pairs = [
            ("Enc3", "Dec3"),
            ("Enc2", "Dec2"),
            ("Enc1", "Dec1"),
        ]
        
        for enc_name, dec_name in skip_pairs:
            enc_block = blocks[enc_name]
            dec_block = blocks[dec_name]
            
            # Start: Right side of Encoder block
            start_pt = enc_block.get_right()
            # End: Left side of Decoder block
            end_pt = dec_block.get_left()
            
            # Create a Bezier curve that goes through the center area
            # Control points: pull towards center X (0) but keep Y relative
            mid_x = 0
            # Curve dips slightly or goes straight? Standard U-Net shows them crossing.
            # Let's make a cubic bezier that stays within the frame.
            cp1 = np.array([start_pt[0] - 1, start_pt[1], 0])
            cp2 = np.array([end_pt[0] + 1, end_pt[1], 0])
            
            path = CubicBezier(start_pt, cp1, cp2, end_pt)
            
            self.play(Create(path, run_time=0.4), path.animate.set_color(C_SKIP).set_stroke(width=3))
            
            # Add arrowhead at the end
            arrow_tip = ArrowTip(height=0.2, width=0.2, fill_color=C_SKIP)
            arrow_tip.move_to(end_pt)
            # Rotate to match curve tangent at end
            # Tangent of CubicBezier at t=1 is 3*(end - cp2)
            tangent = 3 * (end_pt - cp2)
            angle = np.arctan2(tangent[1], tangent[0])
            arrow_tip.rotate(angle)
            self.play(FadeIn(arrow_tip), run_time=0.2)

        StepCaption(self, "Skip connections preserve spatial details")
        self.wait(1.5)


if __name__ == "__main__" and "--selftest" in sys.argv:
    # Self-test: Verify structure logic
    assert len(LAYER_CONFIG) == 9
    assert "Bridge" in LAYER_CONFIG
    
    # Check symmetry of skip connections
    enc_levels = ["Enc1", "Enc2", "Enc3"]
    dec_levels = ["Dec3", "Dec2", "Dec1"]
    
    for i, enc in enumerate(enc_levels):
        dec = dec_levels[i]
        assert LAYER_CONFIG[enc]["w"] == LAYER_CONFIG[dec]["w"], f"Width mismatch for {enc}/{dec}"
        assert LAYER_CONFIG[enc]["h"] == LAYER_CONFIG[dec]["h"], f"Height mismatch for {enc}/{dec}"
        
    # Check Bridge is in the middle
    assert LAYER_CONFIG["Bridge"]["y"] < LAYER_CONFIG["Enc3"]["y"]
    assert LAYER_CONFIG["Bridge"]["y"] > LAYER_CONFIG["Dec3"]["y"]
    
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.