Content
Summary
Candidate generation is the initial phase in recommendation systems that narrows a vast item pool into a manageable subset for ranking, primarily using content-based or collaborative filtering. Content-based filtering relies on feature matrices and similarity metrics like the dot product to match user history, while collaborative filtering leverages collective user-item interaction data to suggest items based on similar users.
Intuition
Think of a recommendation system as a two-stage filter. First, you have millions of items, which is too many to process deeply. Candidate generation acts as the coarse sieve, quickly selecting a small, relevant subset of items to pass on. It does this either by looking at what an individual user has liked before (content-based) or by seeing what similar users have liked (collaborative). This step is crucial because it prevents the system from wasting computational resources on irrelevant items before the final ranking occurs.
Derivation
- Candidate generation is defined as the preliminary step occurring before the final ranking step.
- Its primary function is to narrow down a vast pool of items to a manageable subset.
- Two methodologies are employed: content-based filtering and collaborative filtering.
- Content-based filtering uses a feature matrix where rows are items and columns are features.
- A similarity metric, such as the dot product, scores items against the user's profile.
- Collaborative filtering analyzes similarities between users and items using interaction data.
- Content-based filtering scales well but cannot expand interests beyond existing history.
- Collaborative filtering mitigates content-based limitations by incorporating cross-user data.
Worked example
A user has previously liked two apps: App A (features: [1, 0, 1]) and App B (features: [1, 1, 0]). A new candidate, App C, has features [1, 0, 0]. Using a simplified binary content-based approach, determine if App C is a strong candidate based on the dot product similarity with the user's profile, assuming the user profile is the sum of liked items.
- Construct the user profile vector by summing the feature vectors of liked items: [1, 0, 1] + [1, 1, 0] =.
- Identify the feature vector for the candidate App C: [1, 0, 0].
- Calculate the dot product between the user profile and App C [1, 0, 0].
- Dot product = (2*1) + (1*0) + (1*0) = 2.
- A non-zero score indicates similarity; thus, App C is a candidate.
Answer App C is a candidate with a similarity score of 2.
Common mistakes
- Assuming content-based filtering can discover new interests outside a user's historical data.
- Confusing the candidate generation step with the final ranking step.
- Overlooking that content-based filtering does not leverage other users' features.