Content
Summary
U-Net is an encoder-decoder convolutional network for dense prediction, such as segmenting organs in medical images. Skip connections copy high-resolution encoder feature maps and concatenate them with upsampled decoder features at the same scale. This restores the fine spatial detail lost by downsampling while the decoder keeps the deep semantic context.
Intuition
Downsampling trades where for what. Deep layers learn rich semantic features (this is a liver) but at coarse resolution, so boundaries are blurry. Early layers keep precise edges and positions but know little about context. A skip connection hands the decoder the early, sharp map at the matching resolution, so it can combine what from below with where from the encoder. Picture a U shape: the left arm shrinks the image, the right arm expands it, and horizontal bridges link layers of equal size. Gradients also flow along these bridges, giving short paths to early layers and easing optimization.
Key equations
Decoder feature at level l is a convolution of the channel-wise concatenation of the upsampled deeper decoder map and the encoder map e_l of equal spatial size.
Each encoder stage halves spatial size, typically doubling channels.
Concatenation adds channel counts; spatial dimensions must match.
Derivation
- Start with an encoder producing maps e_1,...,e_L with sizes H/2^(l-1) by W/2^(l-1).
- At the bottleneck, the receptive field is large but resolution is H/2^(L-1), so exact boundary positions are lost.
- The decoder upsamples (transposed convolution or interpolation) d_{l+1} to the size of e_l.
- Concatenate along the channel axis: z_l=[Up(d_{l+1}); e_l].
- Apply convolutions to z_l so the network learns how to weigh semantic and spatial cues, giving d_l.
- Backpropagation: since z_l contains e_l directly, the gradient dL/de_l has a direct path from the decoder, bypassing the deep stack, which reduces vanishing gradients.
- Finish with a 1x1 convolution at full resolution to give per-pixel class scores.
Worked example
A U-Net level has encoder map e_2 of size 64x64 with 128 channels. The deeper decoder map d_3 is 32x32 with 256 channels. Find the shapes after upsampling, concatenation, and a convolution outputting 128 channels.
- Upsample d_3 by 2 in each spatial dimension: 64x64. Using a 2x2 up-convolution that halves channels gives 128 channels.
- Spatial sizes now match e_2 (64x64).
- Concatenate: channels 128 + 128 = 256, shape 256x64x64.
- Apply a 3x3 convolution with 128 filters: output 128x64x64 (with same padding).
Answer After concatenation 256x64x64; decoder output d_2 is 128x64x64.
Common mistakes
- Thinking skip connections add (as in ResNet) rather than concatenate; the original U-Net concatenates.
- Ignoring size mismatch: unpadded convolutions require cropping the encoder map before concatenation.
- Believing skips carry semantic information; they carry mainly high-resolution low-level detail.
- Skipping normalization, so encoder and decoder features with different scales are poorly combined.
- Assuming skips are free: they increase memory because encoder activations must be stored.