AI, machine learning and deep learning · Introductory

High-level Design

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

High-level ML system design employs a two-stage architecture of retrieval and ranking to balance coverage and precision. This structure relies on specialized services like the Application Server, Feature Store, and Model Serving to handle large-scale data efficiently. Proper scaling strategies are essential to maintain low latency and high throughput for millions of users.

Intuition

Think of an ML system like a library search. You cannot read every book to find the best one, so you first use a quick filter (retrieval) to get a manageable list of candidates. Then, you spend more time carefully reading and scoring those few candidates (ranking) to pick the best result. This separation allows the system to handle massive amounts of data without slowing down. The infrastructure supports this by storing user features quickly in a Feature Store and using specialized services to orchestrate the flow from user request to final result.

Derivation

  1. The system starts with a user request at the Application Server.
  2. The request is forwarded to a specialized service (e.g., Search Service).
  3. The service retrieves candidate items from an Indexing Database or Item Store.
  4. The Ranking Service fetches features from the Feature Store.
  5. Model Serving provides real-time predictions (e.g., CTR) for the candidates.
  6. The Ranking Service scores and sorts the candidates by predicted probability.
  7. The final sorted list is returned to the user via the Application Server.

Worked example

Problem

A food delivery app needs to rank restaurants for a user. How does the system handle the data flow from request to result?

  1. The Application Server receives the user's location request.
  2. The Search Service retrieves candidate restaurants from the Indexing Database.
  3. The Feature Store provides real-time user preferences and restaurant stats.
  4. Model Serving predicts the probability of ordering from each restaurant.
  5. The Ranking Service sorts restaurants by these probabilities.
  6. The sorted list is sent back to the user.

Answer The system uses a two-stage architecture where retrieval filters candidates and ranking scores them using features from the Feature Store and predictions from Model Serving, ensuring a consistent and fast user experience.

Common mistakes

  • Starting with a simple architecture without considering scale for millions of users.
  • Ignoring the need for low-latency feature lookups in the Feature Store.
  • Failing to separate retrieval and ranking, leading to performance bottlenecks.
  • Not using a distributed Model Store for efficient model management.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.