AI, machine learning and deep learning · Intermediate

Learning

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

This lesson distinguishes between inference, which computes hidden variables given fixed parameters, and learning, which estimates parameters from data. It highlights that while complete data allows for factorized likelihoods and independent parameter updates, missing or latent variables destroy this structure, leading to non-convex optimization challenges.

Intuition

Think of inference as solving a puzzle where the rules (parameters) are known, and you are trying to figure out the missing pieces (hidden variables). Learning is the reverse: you see the completed puzzle (data) and must deduce the rules that generated it. In a Bayesian view, both the rules and the pieces are unknown variables to be inferred simultaneously. The key structural advantage of complete data in directed graphical models is that the problem breaks down into independent sub-problems for each node, making computation tractable. However, introducing missing data or latent variables couples these sub-problems together, creating a complex, non-convex landscape where finding the global optimum is difficult.

Key equations

θ^=arg⁡maxθ⁡∑i=1Nlog⁡p(xi,v|θ)+log⁡p(θ)
(1)

The MAP estimate for parameters, combining log-likelihood and log-prior.

p(θ,D)=p(θ)∏i=1Np(xi|θ)
(2)

Joint distribution under the i.i.d. assumption represented by plate notation.

p(D|θ)=∏t=1Vp(Dt|θt)
(3)

Factorization of likelihood for complete data in directed graphical models.

θtck=Ntck+αtck∑k′(Ntck′+αtck′)
(4)

Posterior mean for a tabular CPD parameter using Dirichlet priors.

Derivation

  1. 1. Define Inference vs. Learning: Inference computes functions of hidden variables xh given observed data xv and fixed parameters θ. Learning determines parameters θ given observed data.
  2. 2. Bayesian Unification: In a Bayesian perspective, parameters θ are treated as random variables. Inference and learning are unified as probabilistic inference over all nodes, including parameters.
  3. 3. Structural Scaling: Hidden variables scale linearly with dataset size N, requiring integration to avoid overfitting. Parameters are typically fixed in number regardless of N.
  4. 4. Plate Notation: Use plates to represent i.i.d. data cases. This implies conditional independence of data given parameters, but marginal dependence among data cases.
  5. 5. Complete Data Factorization: If data is complete (no missing/hidden variables), the likelihood factorizes by graph structure: p(D|θ)=∏t=1Vp(Dt|θt). This allows independent posterior computation for each CPD if the prior factorizes.
  6. 6. Dirichlet Posterior: For tabular CPDs, a Dirichlet prior Dir(α) leads to a Dirichlet posterior. The posterior mean is the empirical count Ntck plus pseudo-counts αtck, normalized by the sum.
  7. 7. MLE and Zero-Count Problem: MLE sets pseudo-counts to zero. If an event is unseen (Ntck=0), the probability becomes zero, causing overfitting. Priors regularize this.
  8. 8. Missing/Latent Variables: When data is incomplete or latent variables exist, the likelihood no longer factorizes. It becomes a sum of products, making the optimization non-convex and preventing global guarantees.

Worked example

Problem

Consider a simple binary node X with no parents. We observe 5 data points: D={1,1,0,0,0}. Calculate the MLE estimate for P(X=1) and the posterior mean estimate using a uniform Dirichlet prior with α1=1 and α0=1.

  1. 1. Count empirical frequencies: N1=2 (two 1s), N0=3 (three 0s). Total N=5.
  2. 2. MLE Calculation: θ^MLE=N1N1+N0=25=0.4.
  3. 3. Dirichlet Posterior Calculation: Add pseudo-counts α1=1,α0=1.
  4. 4. Numerator for X=1: N1+α1=2+1=3.
  5. 5. Denominator (sum of counts + pseudo-counts): (N1+N0)+(α1+α0)=5+2=7.
  6. 6. Posterior Mean: θpost=37≈0.428.

Answer MLE: 0.4; Posterior Mean: 3/7

Common mistakes

  • Confusing inference (finding hidden variables) with learning (finding parameters).
  • Assuming the likelihood factorizes when missing or latent variables are present.
  • Using MLE without smoothing, leading to zero probabilities for unseen events (zero-count problem).
  • Treating parameters as fixed constants in a Bayesian framework instead of random variables.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.