How I think about evals for content discovery

I want my team to be able to explain why a change to content discovery is worth shipping. That explanation should hold up when we look at what a person was trying to do, what the system returned, and what improved over the version we already have.

Imagine a viewer searching for a film by its exact title. The film is in the catalog and available to that profile. The current search puts it first. A proposed update puts it sixth, below titles that appear to match the viewer's usual interests.

For that request, I would expect the exact match to come first. The viewer has told us what they want. Personalization can help resolve ambiguity and make useful alternatives available, but it needs a good reason to get in the way of an explicit request.

We can turn that expectation into an eval. An eval is a structured assessment of a system's behavior against explicit criteria, using defined inputs and context. In this case, we have a query, an eligible catalog, an ordered result list, and a judgment about where the requested film belongs. We can run the same case against another version and inspect what changed.

That is one evaluation case. A collection of cases lets us compare behavior across more situations and keep known failures from quietly returning. The broader evaluation practice also includes human review, production monitoring, and experiments with users. Search and recommendation teams have been doing this work for years. Large language models (LLMs) add new behaviors to evaluate, including explanations and conversations, while the underlying question remains familiar: how well is this system serving the purpose we gave it?

I find evals useful because they force that purpose into the open. Before asking which version won, we have to agree on what the result should mean.

The experience determines the judgment

The exact-title example has a relatively clear expectation. A homepage row presents a harder question. Someone opening the app may want to continue a series, revisit something familiar, or find a film they would never have thought to search for. The product has to infer more of the intention, and the evaluation has to account for that uncertainty.

A row meant for discovery could produce more starts by showing familiar titles. That may be valuable to the viewer, but I would want to understand whether it fulfilled the purpose of that row. If the intended benefit is helping people find unfamiliar content they value, predicting another start is only part of the evidence we need. A continuation experience has a different purpose and should be judged accordingly.

This is a product choice that needs to be explicit before we ask Data Science (DS) to measure it. Novelty is useful when it creates worthwhile discovery. It can also produce a row of obscure titles someone has no interest in watching. I would want the team to examine which titles changed and for whom, rather than assume that more novelty means a better experience.

The same issue appears inside a search request. Consider a hypothetical request for a funny, nonanimated film under 90 minutes. Duration and media type can be checked with code against trusted catalog data. Whether the film fits the viewer's idea of funny requires a different judgment. A result can satisfy both constraints and still miss the mood completely. If the metadata is incomplete, the evaluation should preserve that uncertainty rather than count an unverified constraint as satisfied.

Human reviewers can help explain those differences, provided we are clear about what they know. A catalog specialist can assess whether a title's description is supported. A viewer can tell us whether it suits what they wanted tonight. Neither judgment automatically stands in for the other. Disagreement about taste may be legitimate, particularly across audiences and cultural contexts.

Viewing history adds evidence, but it is an incomplete account of preference. Someone may have watched a title because it was prominently displayed. They may have left another unwatched because they never saw it. A shared profile may combine several people's choices. I would be careful about turning those observations into a definitive account of what one person likes.

This also affects the unit we evaluate. A single title can fit while the whole row feels repetitive. In conversational discovery, the opening answer can be good while the follow-up forgets the viewer's constraints. The case needs to cover enough of the experience to reveal the failure we care about.

Looking at failures changes the work

Return to the film buried in search. The visible problem is clear; its cause still needs investigation.

In this example, the film reached the results. I would examine what determined its position, including the information used to rank it and any rules that changed the order. A related case in which the film disappears entirely would need a wider investigation. Was it in the candidate set, the pool the system considers before ordering results? Did an incorrect availability record remove it? A ranking change will not repair a title that an upstream service incorrectly excluded.

I want the team to be able to follow that case through the system. That requires a usable record of the request and the relevant decisions behind the output, often called a trace. For search, the useful record could show which candidates were retrieved, which were filtered, and how the remaining titles were ordered. The result tells us what went wrong for the viewer; the additional evidence helps us investigate why.

Looking at examples can also expose an incomplete expectation. Perhaps a query could name either a film or its remake. Perhaps a sequel keeps displacing the original. We need to decide what a good response to that ambiguity looks like before treating one ordering as correct. Reviewing cases is part of developing the criteria, not merely checking whether someone implemented them.

Known requirements do not need to wait for that discovery. Availability rules and explicit constraints can be checked from the start. The value of inspecting failures is finding where our existing account of quality is too vague, or where the system behaves in a way we had not considered.

There is a practical limit here. Maintaining evaluators takes work. I would prioritize a recurring failure that affects an important task over a large collection of checks nobody uses to make a decision. Severity matters too: a rare eligibility violation may deserve attention before a common but minor ordering preference. The purpose is to give the next investment a clearer basis.

A selected collection of difficult examples is useful for exposing weaknesses. To estimate how often those weaknesses affect customers, we need a sample that represents the population we are making claims about. A test set assembled from failed searches will make failure look common by construction. A set dominated by frequent viewers can leave the experience of new viewers largely unanswered.

An existing QA check might confirm that every returned title is eligible and still miss the problem with putting the requested film sixth. Looking at the case gives us a more precise expectation to test. Measuring it across appropriate queries helps us decide how much work it warrants. Where the team already has QA, analytics, and experimentation, I would connect that evidence around the decision rather than create another dashboard to maintain.

A comparison needs limits we can explain

For the proposed search update, I would compare the current experience with the candidate on stored cases, before exposing viewers to the change. That is an offline comparison. It needs the same relevant conditions, with the inputs and system versions recorded, so changes in the test environment do not get confused with changes in ranking behavior. When outputs vary between runs, the comparison also has to account for that variation. I would include unsuccessful requests; a version that times out more often should not benefit from having its missing results left out.

For recommendation comparisons built from viewing history, the test must respect what the system could have known at the time. Later viewing can be used as an outcome label, but it cannot leak into the inputs or the model's training data for that earlier prediction. Otherwise, the test gives the system knowledge it would not have had in the product.

Ranking metrics help summarize these comparisons. NDCG, or normalized discounted cumulative gain, gives more weight to relevant items near the top and compares the ordering with an ideal ranking under the same relevance labels. That makes it useful for examining ordering. The meaning of the gain still depends on what those labels reward. If they are derived from plays, the result concerns that behavioral signal; it does not independently establish satisfaction.

I would look at the aggregate alongside the changed cases and important audiences. A candidate might improve broad queries while making exact-title searches worse. Whether that tradeoff is acceptable depends on our product intent. Combining the results into one number can make the choice less visible without resolving it.

LLM judges can extend this work where a criterion requires contextual judgment. For a generated recommendation explanation, a judge might assess whether the stated reason follows from supplied catalog facts. That can make review more manageable, but the judge is another system whose behavior needs evaluation.

I would want to know which failures it misses and which acceptable answers it wrongly rejects, using human-labeled examples kept out of its development. Agreement alone can be misleading when most outputs pass. In comparisons, presentation order can influence a judge, and a changed judge or rubric can move the score even when the recommendations remain identical. The evaluation needs enough version history to distinguish those changes.

Offline evidence can justify taking a candidate into a randomized user experiment. The experiment addresses a further question: what effect does the change have on the outcomes we measure? It needs valid assignment and logging, enough observation for the claim, and uncertainty that we interpret against a worthwhile benefit and acceptable harm. A result that shows no statistically significant deterioration can still be too uncertain to rule out damage we would consider unacceptable.

For the search example, faster access to the intended film could be a meaningful benefit. For a discovery row, more starts may need to be considered alongside abandonment and direct feedback. Those signals can disagree. I would want to understand the disagreement before calling the experience more satisfying, especially if the claim extends beyond the observation period.

Production monitoring keeps the question open after release. It can reveal a rise in empty results, unsupported explanations, or response time as the catalog and audience change. A movement in those measures gives us a reason to investigate; a before-and-after chart alone does not establish what caused it. Cases from production can strengthen the next evaluation, while untouched test cases remain necessary for checking how well a change generalizes beyond its development examples.

Product, DS, and Engineering need to examine the same evidence

I think Product has to take responsibility for making the intended experience clear enough to evaluate. If we want discovery, we need to explain what worthwhile discovery means on that surface. If an exact-title request should take precedence over inferred interests, that expectation needs to be explicit. Leaving those choices unstated makes it too easy for each discipline to solve a different problem well.

That responsibility depends on technical partnership. DS can challenge whether the labels express the intended outcome, whether the sample supports the claim, and how much uncertainty remains. Engineering can establish whether the comparison reflects the serving system and whether the necessary context is observable. Both may identify a product assumption that needs to change. I want us to examine some of the same cases together while we build the evaluation.

The useful agreement is specific to the decision. For the search candidate, I would want us to agree on the queries and audiences it is meant to help, the evidence that would justify exposure, and the failures that would stop us. The expected benefit and acceptable deterioration should be settled before we interpret which version won. Cost and latency belong in that conversation too. A small ranking gain may be a poor purchase if it makes search noticeably slower or adds maintenance we cannot justify.

We also need room to discover that the evaluator is wrong. If people reviewing the results consistently find a problem the score rewards, we should examine the criteria and the judgments behind them. A metric should remain open to challenge. Changes to its definition need to be visible, with comparisons rerun where necessary, so an improved score does not merely reflect an easier test.

For the hypothetical film buried in search, I would hold the candidate while the team investigates the exact-title regression. The fix might belong in the ranking logic or in the data it relies on. I would want the revised ordering checked across the broader query set, including examples that did not guide the fix. A controlled experiment can then test whether the repaired candidate helps viewers under the conditions we agreed to measure. That is a release decision I could explain, including what the evaluation has left unresolved.

---

This writing reflects my personal perspectives on product management, AI, and content discovery. It does not represent the official position of my employer or any affiliated organization.