AI, machine learning and deep learning · Advanced

Multi-task learning

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Multi-task learning is a meta-learning approach that simultaneously fits multiple related models to improve performance by sharing statistical strength through common parameters or priors. This technique addresses limited data challenges by leveraging similarities across tasks, often implemented via hierarchical Bayesian models or specific feature engineering tricks.

Intuition

Imagine you are trying to predict outcomes for several different groups, some with lots of data and others with very little. Instead of building a separate model for each group, which might overfit the small groups, or pooling all data into one model, which ignores group differences, multi-task learning finds a middle ground. It assumes that while each group has its own specific parameters, these parameters are drawn from a common underlying distribution. This allows groups with small sample sizes to 'borrow' statistical strength from those with larger sample sizes, resulting in better generalizability and reduced overfitting compared to fitting separate models.

Key equations

E[yij|xij]=g(xijTβj)
(1)

The expected output for task j given input x, where beta_j is the task-specific parameter vector.

βj∼N(β∗,σj2I)
(2)

Task-specific parameters βj are drawn from a normal distribution centered around a common latent parameter β∗ with variance σj2.

β∗∼N(μ,σ∗2I)
(3)

The common latent parameter β∗ is drawn from a normal distribution with mean μ and variance σ∗2.

Derivation

  1. Start with the assumption that input-output mappings are similar across different tasks.
  2. Define task-specific parameters beta_j for each task j.
  3. Assume these parameters are drawn from a common distribution centered at a latent parameter β∗.
  4. Assume the latent parameter β∗ is drawn from a global prior with mean μ.
  5. This hierarchical structure correlates βj's via the common parent β∗.
  6. Groups with small sample sizes benefit from the data of groups with larger sample sizes through this correlation.
  7. The method acts as a compromise between fitting separate models and using the same model for all groups.

Worked example

Problem

Consider a scenario where you are building classifiers for named entity recognition on two datasets: email text (large dataset) and newswire text (small dataset). How does multi-task learning using hierarchical Bayesian modeling improve performance compared to fitting separate models?

  1. Identify that the tasks are related (both are NER) but have different data sizes.
  2. Apply hierarchical Bayesian modeling where parameters for email and newswire are drawn from a common distribution.
  3. The newswire model, having little data, borrows statistical strength from the email model via the shared latent parameter.
  4. This reduces the chance of overfitting on the small newswire dataset.
  5. Performance is evaluated by comparing against separate models and a single pooled model.

Answer Multi-task learning improves performance on the small newswire dataset by allowing it to leverage the statistical strength from the large email dataset through shared parameters, resulting in better generalizability than fitting separate models.

Common mistakes

  • Assuming that all tasks are equally similar can lead to worse performance if the inductive bias of the prior is incorrect.
  • Requiring the variance parameters for individual tasks (σj2) to be equal to the global variance (σ∗2) may not yield the best performance.
  • Simply pooling all data and fitting a single model is not always optimal; multi-task learning can provide improvements over this approach.
  • Using different algorithms or hyperparameters to create diversity is a key aspect of ensemble methods, but multi-task learning specifically focuses on sharing parameters across related tasks.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.