Classroom · AI
AI, machine learning and deep learning
From gradient descent to attention, drawn step by step with the real algorithm run.
144 lessons, text and equations; animations coming soon.
Early edition: these lessons are being reviewed.
Introductory level (81)
- Linear regression and least squaresLinear regression fits a straight line to data so that predictions are as close as possible to observed values.
- Gradient descentGradient descent is an iterative method for minimizing a function by repeatedly stepping in the direction opposite to the gradient.
- k-means clusteringk-means clustering partitions unlabeled data points into k groups by repeatedly assigning each point to its nearest cluster center and then moving each center to the…
- k-fold cross-validationk-fold cross-validation estimates how well a model will perform on unseen data by splitting the dataset into k equal parts, training on k-1 parts and testing on the…
- A neuron and a forward passAn artificial neuron takes several numerical inputs, multiplies each by a weight, adds a bias, and passes the result through a nonlinear activation function.
- Softmax and cross-entropySoftmax turns a vector of raw scores (logits) into a probability distribution that is positive and sums to one.
- Activation functionsAn activation function is a nonlinear function applied to a neuron's weighted sum of inputs.
- Perceptron learning ruleThe perceptron is a single artificial neuron that classifies inputs into two classes using a weighted sum and a threshold.
- The Machine Learning TsunamiDeep learning, a technique revived by Geoffrey Hinton et al.
- What Is MATLAB and Why Use It?MATLAB is presented not just as a software tool but as a collaborative culture essential for modern neuroscience and psychology research.
- Understanding Python codeIntegrated Development Environments (IDEs) such as IDLE, IPython Notebook, and Spyder provide essential tools for writing, testing, and debugging Python code.
- Python and its packages – download and installationPython environments are established either through the Anaconda distribution, which bundles over 195 packages and tools like Spyder and IPython Notebook, or via…
- WHAT IS AN EXAMPLE OF A DATA SET WITH A NON-GAUSSIAN DISTRIBUTION?Many machine learning datasets follow non-Gaussian distributions within the exponential family, such as Bernoulli, Binomial, Poisson, Gamma, and Beta, which are often…
- Python basics - Part 2This lesson covers Python's function definition syntax, emphasizing indentation-based scoping and first-class function capabilities.
- Big Data ParadigmThe big data paradigm is a system that observes an environment to learn characteristics and make accurate decisions by transforming raw data into knowledge.
- Keep It SimpleThe philosophy of data analysis prioritizes simplicity, advising researchers to use the simplest possible methods that adequately address their hypotheses.
- Use Simulations, but Trust Real DataUnderstanding data analysis requires explaining its mechanics, interpreting results, assessing parameter effects, and detecting errors, rather than necessarily…
- What Kinds of Data Can Be Mined?Data mining extracts patterns from diverse sources like databases and streams, categorized into descriptive tasks (characterization, discrimination) and predictive…
- WHAT IS SELECTION BIAS?Selection bias is an error arising from non-random participant selection, leading to inaccurate study conclusions.
- Which Technologies Are Used?Data mining is an interdisciplinary field that synthesizes statistics, machine learning, database systems, and information retrieval to extract patterns from large…
- Look at the Big PictureThis lesson introduces a machine learning project designed to predict California district median housing prices using census data, aiming to replace a costly and…
- Major Issues in Data MiningData mining faces major challenges across five domains: methodology, user interaction, efficiency, data diversity, and societal impact.
- Have a “Large Enough” NDetermining a sufficient sample size (N) is essential for statistical power, generalizability, and robustness against outliers, though no universal 'magic number' exists.
- Independent ReplicationsIndependent replications involve separate research groups using distinct equipment and locations to verify that scientific findings are robust and not artifacts of…
- WHAT IS P-VALUE?A p-value is a statistical metric used in hypothesis testing to determine the strength of results by indicating the minimum significance level at which the null…
- Explore and Visualize the Data to Gain InsightsData exploration requires isolating the training set and creating a copy to preserve the original state while experimenting.
- Write a Clear Methods SectionA methods section must provide sufficient detail, including specific function names and nondefault parameters, to allow other researchers to replicate scientific…
- The MATLAB ProgramThe MATLAB Graphical User Interface (GUI) serves as the central environment for interactive programming, typically opening with approximately six subwindows that users…
- R data structuresR data structures are specialized containers for statistical analysis, with vectors serving as the fundamental building block for ordered, homogeneous data.
- Finding the Maximum and Minimum ValuesNumPy provides max and min functions to identify extreme values in arrays, while broadcasting allows element-wise operations without explicit loops.
- Color-Coordinating MATLABMATLAB scripts are plain text.m files that store code for organization, requiring explicit evaluation in the Command window to run.
- Python and its packages for predictive modellingPython serves as the primary language for predictive modeling due to its ease of use and extensive ecosystem of libraries like scikit-learn, NumPy, pandas, and…
- Prepare the Data for Machine Learning AlgorithmsThis lesson outlines the essential steps for preparing data for machine learning, including establishing a standardized project structure, handling missing values via…
- Data Objects and Attribute TypesData objects are entities described by attributes, which are categorized into nominal, binary, ordinal, and numeric types based on their possible values.
- STATISTICSThe Central Limit Theorem states that the sampling distribution of the sample mean approaches a normal distribution as the sample size increases, regardless of the…
- Select and Train a ModelModel selection involves applying preprocessing pipelines to raw data and fitting algorithms like Linear Regression or Decision Trees.
- Coin Toss Probability After Ten HeadsThis lesson applies Bayes' theorem to update the probability of a coin's type after observing 10 heads, shifting belief from a strong prior favoring a fair coin to a…
- Changing Directories inside MATLABMATLAB operates within a specific current directory, which can be navigated using commands like cd, mkdir, and ls.
- Keyboard ShortcutsMATLAB keyboard shortcuts streamline coding by enabling efficient cell execution, code commenting, and autocompletion, thereby reducing reliance on the mouse.
- Basics – summary, dimensions, structureThe pandas library's read_json function loads JSON files into DataFrames, enabling data manipulation and analysis.
- Describing a MatrixA matrix is a geometrically organized list of numbers, typically structured as rows and columns, essential for storing and manipulating data in machine learning.
- Calculating the Average, Variance, and Standard DeviationVariance quantifies data spread by averaging squared differences from the mean, while standard deviation is its square root, restoring original units.
- Multiplying MatricesMatrix multiplication involves two distinct operations: the dot product, which transforms data spaces by combining rows and columns, and element-wise multiplication…
- Back Up Your Scripts, and Use Only One VersionResearchers must maintain a single, backed-up version of analysis scripts using cloud services to prevent version conflicts, while storing large experimental data in at…
- VariablesIn MATLAB, variables are placeholders for data created through assignment, requiring strict naming conventions and proper memory management.
- WHAT ARE CONFOUNDING VARIABLES?A confounding variable is a third factor that influences both the independent and dependent variables in a study, potentially creating a false association between them.
- Reading the data – variations and examplesData exploration involves loading tabular data into data frames and inspecting their structure, features, and summary statistics to understand the underlying examples…
- Handling Missing ValuesMissing values, denoted by NumPy's NaN in Pandas, are a critical data wrangling issue that can lead to incorrect interpretations if untreated.
- Deleting a ColumnTo delete a column from a pandas DataFrame, use the drop method with axis=1, passing the column name or a list of names.
- Data ReductionData reduction techniques minimize dataset volume while preserving analytical integrity through dimensionality, numerosity, and compression methods.
- Data Transformation and Data DiscretizationData transformation and discretization are preprocessing steps that consolidate raw data into forms suitable for efficient mining and easier interpretation.
- Grouping Rows by ValuesGrouping rows by values in pandas uses the groupby method to split data into DataFrameGroupBy objects based on categorical variables.
- Concatenating DataFramesConcatenation combines multiple DataFrames into a single object using the pandas concat function, which is the standard method for merging datasets.
- Merging DataFramesMerging DataFrames in pandas combines disparate datasets by matching values in specified columns, analogous to SQL joins or Excel VLOOKUP.
- Exporting Text FilesExporting and importing text files involves reading data from formats like CSV and ASCII using tools such as Python's pandas or MATLAB's built-in functions, while…
- Handling Numerical DataThis lesson covers the classification of numerical data types and essential preprocessing techniques like min-max scaling and standardization to ensure features are on…
- Functions Are FilesMATLAB functions are implemented as text files with.m extensions, where the initial comment blocks serve as official help documentation accessible via the help command.
- Interacting with Your Operating System via MATLABMATLAB utilizes specific native file formats like.m,.mat, and.fig for code, data, and graphics, while.mex files provide OS-specific compiled performance.
- DATA ANALYSISData analysis is the process of inspecting, cleaning, transforming, and modeling data to support decision-making, often utilizing the Star Schema for efficient…
- Calculating Dot ProductsThe dot product is a fundamental operation that yields a scalar by summing the products of corresponding vector elements or by multiplying vector lengths by the cosine…
- Think Global, Act LocalMATLAB manages variable scope through isolated execution environments called 'miniverses' or 'stacks', where variables are local by default and inaccessible to other…
- Importing and Exporting Microsoft Excel FilesThis lesson covers importing and exporting Excel files using Python's pandas library and MATLAB's xlsread/xlswrite functions.
- A/B Testing FundamentalA/B testing compares a control model against a new treatment model, but in Ads Marketplaces, budget competition can skew results by shifting funds rather than improving…
- First steps with scikit-learnScikit-learn is a prominent open-source Python library for machine learning that provides a consistent API for preprocessing, training, and evaluating models.
- Timing Functions Using the ProfilerThe MATLAB profiler is a diagnostic tool that tracks execution time for each line of code, helping users identify inefficient code, unnecessary loops, and computational…
- Data Warehouse Modeling: Data Cube and OLAPA data cube is an n-dimensional model that represents data through dimensions and facts, enabling multidimensional analysis.
- Data Warehouse ImplementationData warehouse implementation centers on efficient cube computation, indexing, and query processing to support fast OLAP queries on large datasets.
- Data Generalization by Attribute-Oriented InductionAttribute-oriented induction is a query-oriented, data-driven technique that generalizes low-level data values into higher-level concepts or removes irrelevant…
- Data Warehouse Design and UsageA data warehouse is a centralized repository integrating data from multiple operational sources to support business analysis and competitive advantage.
- Common Deployment PatternsML deployment often splits inference workloads across multiple servers using an aggregator or load balancer to manage high traffic.
- High-level DesignHigh-level ML system design employs a two-stage architecture of retrieval and ranking to balance coverage and precision.
- Control StatementsControl statements enable flexible program execution in MATLAB by directing flow based on logical conditions.
- CAN YOU CITE SOME EXAMPLES WHERE A FALSE POSITIVE IS IMPORTANT THAN A FALSE NEGATIVE?A false positive, or Type I error, occurs when a non-event is wrongly classified as an event, while a false negative, or Type II error, is the reverse.
- CAN YOU CITE SOME EXAMPLES WHERE BOTH FALSE POSITIVE AND FALSE NEGATIVES ARE EQUALLY IMPORTANT?In banking loan approval, both false positives and false negatives are equally critical because they result in significant financial losses.
- PauseThe pause function in MATLAB is a timing utility that temporarily halts script execution to allow users to observe intermediate results or intervene.
- Machine Learning ParadigmMachine learning is categorized into supervised learning, which uses labeled data for classification, and unsupervised learning, which identifies hidden structures in…
- Copy-PasteThis lesson covers importing data into MATLAB via copy-pasting for small datasets or loading.mat files for larger data.
- A Gentle StartStatistical learning formalizes deriving predictors from labeled data to minimize true error on unseen instances.
- Big Data EssentialsBig data is defined by three controllers: observations (n), features (p), and classes, which determine volume, dimensionality, and variety respectively.
- Confusion MatrixA confusion matrix is a table that counts how often instances of one class are classified as another, providing a detailed view of classifier performance beyond simple…
- Measures of PerformanceAccuracy is often misleading for skewed datasets, as a dummy classifier predicting the majority class can achieve high scores without learning useful patterns.
Intermediate level (51)
- Logistic regression and the decision boundaryLogistic regression models the probability of a binary outcome by passing a linear combination of features through the sigmoid function.
- Principal component analysisPrincipal component analysis (PCA) finds new orthogonal axes along which a centered dataset varies most.
- Decision tree splits and entropyDecision trees split data by asking questions that make the resulting groups as pure as possible.
- Bias-variance trade-offThe expected prediction error of a model on new data splits into squared bias, variance, and irreducible noise.
- Backpropagation step by stepBackpropagation computes the gradient of a loss with respect to every weight in a neural network by applying the chain rule backwards from the output.
- 2D convolution and feature mapsA 2D convolution slides a small grid of weights, called a kernel, across an image and computes a weighted sum at each position.
- SGD with momentumSGD with momentum keeps a running average of past gradients, called velocity, and uses it to update parameters.
- Adam optimizer pathAdam is a gradient-based optimizer that keeps running averages of the gradient (momentum) and of the squared gradient (scale), then takes steps normalized per parameter.
- L2 regularization and overfittingL2 regularization adds a penalty on the squared size of the weights to the loss function.
- Dropout maskingDropout randomly zeroes each hidden activation with probability p during training, forcing the network not to rely on any single unit.
- Batch normalizationBatch normalization standardizes each feature of a layer's pre-activations using the mean and variance computed over the current mini-batch, then applies a learned…
- CNN feature map pipelineA convolutional neural network turns an image into a stack of feature maps by sliding small learned filters across it, applying a nonlinearity, and pooling to shrink…
- RNN unrolled in timeA recurrent neural network applies the same cell repeatedly across a sequence, carrying a hidden state forward.
- RecursionRecursion in Prolog defines relations in terms of themselves, serving as the sole looping mechanism to describe arbitrarily large chains.
- Structured termsStructured terms in Prolog use functors to name complex, hierarchical data objects rather than compute values, enabling the representation of variable-length sequences…
- Full clausal logicFull clausal logic extends relational logic by introducing complex terms (functors), enabling reasoning over infinite domains like natural numbers.
- Pruning the search by means of cutThe cut operator (!) in Prolog prunes the SLD-tree to optimize search efficiency by eliminating unnecessary backtracking.
- Negation as failureNegation as failure is a procedural mechanism in logic programming where a goal is considered false if it cannot be proven true, implemented via the meta-predicate not.
- The beta-binomial modelThe beta-binomial model is a Bayesian framework for inferring the probability of success θ in binary trials by combining a binomial likelihood with a beta prior.
- The Dirichlet-multinomial modelThe Dirichlet-multinomial model generalizes binary inference to K mutually exclusive outcomes by placing a Dirichlet prior over the probability simplex.
- Shallow neural networks: general caseShallow neural networks with a single hidden layer map multi-dimensional inputs to outputs by combining linear transformations with non-linear activation functions…
- Uncertainty Estimates from ClassifiersScikit-learn classifiers provide uncertainty estimates via decision_function and predict_proba, which are vital when error costs are asymmetric.
- Identifying the Best Neighborhood SizeThe neighborhood size k in K-Nearest Neighbors is a critical hyperparameter that controls the bias-variance trade-off, where small k leads to high variance…
- Inference in jointly Gaussian distributionsInference in jointly Gaussian distributions allows for the exact computation of marginal and conditional probabilities, which are themselves Gaussian.
- Before we begin: the mathematical building blocks of neural networksThis lesson introduces the foundational mathematical concepts of deep learning, specifically tensors, differentiation, and gradient descent, using the MNIST dataset as…
- Measuring Data Similarity and DissimilarityMeasuring data similarity and dissimilarity is fundamental to data mining tasks like clustering, outlier analysis, and nearest-neighbor classification.
- Empirical BayesEmpirical Bayes (EB) is a computational shortcut for hierarchical Bayesian models that estimates hyperparameters by maximizing the marginal likelihood, effectively…
- Learning with Imbalanced DataImbalanced data arises when the underlying distribution is skewed, causing standard accuracy-optimizing algorithms to ignore the minority class.
- Interpreting Perceptron WeightsThe perceptron algorithm defines a decision boundary as a hyperplane in D-dimensional space, converging to a separating solution if and only if the data is linearly…
- Example – Diagnosing breast cancer with the k-NN algorithmThe k-NN algorithm is applied to the Wisconsin Breast Cancer Diagnostic dataset, which contains 569 examples with 30 numeric features, to automate the classification of…
- Solving Ax = bSolving the linear system Ax = b isolates the unknown vector x by left-multiplying both sides by the inverse of matrix A, resulting in x = A⁻¹b.
- WHAT ARE EIGENVECTORS AND EIGENVALUES?Eigenvectors are vectors that define the specific directions along which a linear transformation acts by flipping, compressing, or stretching.
- Shortlist Promising ModelsShortlisting promising models involves rapid experimentation with diverse algorithm categories using standard parameters to identify top candidates based on performance…
- Naive Bayes classifiersNaive Bayes classifiers are generative models that classify data by assuming features are conditionally independent given the class label, enabling efficient parameter…
- Transforming FeaturesFeature transformation involves applying custom functions to data, such as creating interaction features, to better capture relationships with the target variable.
- Estimating Class ProbabilitiesDecision trees estimate class probabilities by calculating the ratio of training instances for each class within the leaf node reached by an input instance.
- Encoding Nominal Categorical FeaturesNominal categorical features lack intrinsic ordering and require one-hot encoding to prevent false hierarchies, while ordinal features possess a natural order and are…
- ExamplesDirected graphical models (DGMs) factorize joint distributions into conditional probabilities to reduce parameter complexity from O(KV) to O(VKF).
- LearningThis lesson distinguishes between inference, which computes hidden variables given fixed parameters, and learning, which estimates parameters from data.
- Conditional independence properties of DGMsDirected Graphical Models encode conditional independence via d-separation, where a graph G is an I-map for distribution p if all graph-encoded independencies hold in p.
- Candidate GenerationCandidate generation is the initial phase in recommendation systems that narrows a vast item pool into a manageable subset for ranking, primarily using content-based or…
- RankingRanking in machine learning is categorized into point-wise, pairwise, and list-wise approaches, with pairwise methods like RankNet preferred to avoid the complexity of…
- CalibrationCalibration measures the alignment between predicted probabilities and actual observed frequencies, defined as the ratio of the sum of predictions to the sum of labels.
- The Bias-Complexity TradeoffThe No-Free-Lunch theorem proves that no universal learning algorithm exists, necessitating prior knowledge formalized through a hypothesis class H.
- MapReduce Programming PlatformMapReduce is a programming platform for big data scalability that combines parametrization and parallelization.
- Multilabel ClassificationMultilabel classification assigns multiple binary tags to a single instance, such as identifying several faces in one image.
- Error AnalysisError analysis systematically diagnoses model failures by decomposing total error into noise, bias, and variance.
- Evaluating Binary Classifier ThresholdsThe Receiver Operating Characteristic (ROC) curve evaluates binary classifiers by plotting the True Positive Rate against the False Positive Rate across various…
- Training a Random Forest ClassifierA Random Forest Classifier is an ensemble method that constructs multiple randomized decision trees using bootstrapped samples and random feature subsets to mitigate…
- Challenges in Unsupervised LearningUnsupervised learning lacks labels, making evaluation difficult and often requiring manual inspection to verify if clusters are semantically meaningful.
- Clustering with a Mixture of GaussiansA Gaussian Mixture Model (GMM) is a probabilistic density estimation technique that assumes data arises from a mixture of K Gaussian distributions, enabling soft…
Advanced level (12)
- Self-attention: queries, keys, valuesSelf-attention lets every token in a sequence build a new representation as a weighted average of all tokens' value vectors.
- U-Net skip connectionsU-Net is an encoder-decoder convolutional network for dense prediction, such as segmenting organs in medical images.
- Autoencoder latent spaceAn autoencoder compresses an input x into a low-dimensional latent vector z through an encoder, then reconstructs x from z with a decoder.
- Diffusion model noising scheduleA diffusion model gradually corrupts data with Gaussian noise over T steps according to a noising schedule that sets how much signal survives at each step.
- Meta-programsMeta-programs in Prolog manipulate object-level clauses by treating logical constructs like ':-' as both predicates and functors, enabling the creation of…
- Digression: The Wishart distribution *The Wishart distribution is the multivariate generalization of the Gamma distribution, primarily used in probabilistic machine learning to model uncertainty over…
- Generalized linear mixed models *Generalized linear mixed models (GLMMs) extend generalized linear models by incorporating both fixed effects, which are constant across groups, and random effects…
- Learning to rank *Learning to rank (LETOR) orders items by relevance to a query, primarily through pointwise, pairwise, or listwise approaches.
- Multi-task learningMulti-task learning is a meta-learning approach that simultaneously fits multiple related models to improve performance by sharing statistical strength through common…
- Influence (decision) diagrams *Influence diagrams extend Bayesian networks by adding decision nodes (rectangles) and utility nodes (diamonds) to model multi-stage decision problems under uncertainty.
- The VC-DimensionThe VC-dimension is a combinatorial measure of a hypothesis class's capacity, defined as the size of the largest set it can shatter.
- Nonuniform LearnabilityNonuniform learnability relaxes PAC learnability by allowing sample complexity to depend on the specific target hypothesis rather than being uniform across the entire…
ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.