Content
Summary
ML deployment often splits inference workloads across multiple servers using an aggregator or load balancer to manage high traffic. Systems may route requests to different models based on user cohorts, though this increases complexity and risks data scarcity. Embedding services frequently use offline training on YARN and pyramid two-tower architectures to handle high cardinality features efficiently.
Intuition
Think of an ML deployment as a busy restaurant kitchen. When traffic is high, you cannot rely on a single chef (server) to handle every order. Instead, a manager (Aggregator Service) takes the customer's request and distributes the tasks among a pool of chefs (workers) based on their current workload or a simple rotation. In more complex scenarios, different chefs might specialize in different cuisines (models) for different customer groups (cohorts). However, having too many specialists can be problematic if there isn't enough demand for each specific dish to keep them fully trained and effective. For specialized tasks like generating embeddings, the system might use a sophisticated architecture like a pyramid two-tower model, supported by a massive offline training pipeline that processes hundreds of millions of instances on YARN, ensuring the serving layer remains fast and flexible for experiments.
Derivation
- High traffic requires splitting inference workloads across multiple servers.
- An Aggregator Service or Load Balancer receives client requests.
- The Aggregator selects workers from a pool based on criteria like round-robin or current workload.
- Workers process tasks and return responses to the Aggregator.
- The Aggregator forwards the final response to the client.
- Business rules may dictate routing to different models based on user cohorts.
- Using multiple models increases system complexity and may lead to insufficient training data per model.
- Embedding services use an offline training pipeline on YARN for distributed training on large datasets.
- A pyramid two-tower architecture handles extremely high cardinality features by converting sparse features into embeddings.
- Serving components optimize for high write throughput and fast experimentation velocity.
Worked example
A company wants to deploy a recommendation system that serves different models to users in different regions. They plan to use an aggregator to route requests. What are the potential risks and architectural considerations mentioned in the source?
- Identify the routing mechanism: The aggregator will route requests based on user cohorts (regions).
- Assess the risk of multiple models: Using different models for different cohorts increases system complexity.
- Check data availability: Ensure there is sufficient data to train each distinct model effectively; otherwise, model performance may suffer.
- Consider architecture: If embeddings are involved, use a pyramid two-tower architecture to handle high cardinality features.
- Plan for scale: Utilize an offline training pipeline on YARN to support distributed training for hundreds of millions of instances.
Answer The primary risks are increased system complexity and potential data scarcity for individual models. Architecturally, the system should use an aggregator for routing, potentially employ a pyramid two-tower model for embeddings, and leverage YARN for scalable offline training.
Common mistakes
- Assuming that using multiple models for different cohorts is always beneficial without considering the increased complexity and data scarcity.