Following a Metric Into the Product

“Why did this metric move?” is a classic product management question. AI agents give us a new way to pursue it, carrying the investigation across data, code, and product context as each answer changes what we need to know next. We have a shopping app. The share of sessions in which a shopper saves an item for later has fallen. The dashboard shows the movement, but it leaves the cause open. Saving may be broken, the measurement may have changed, or shoppers may have less reason to save. We start with that question and follow it into the product.

An AI agent with configured access to the relevant data, source code, bug database, and internal wiki can help shorten the distance between a product question and evidence we can inspect. The value is in keeping the investigation moving as we learn. A finding in the data changes what we look for in the code; something in the code gives us a more precise question for the data.

The counts behind the rate are the first useful comparison. How has the number of sessions with a save changed relative to the number of eligible sessions? The agent prepares the query for complete, equivalent periods with a consistent definition of eligibility. An analyst checks the population and counting rules. A query can run successfully and still include the wrong population.

The decline is concentrated in a recent mobile version. That narrows the search, but the version's share of activity also matters. We compare the movement with similar sessions on the earlier version, checking whether the shoppers still using it differ from those on the newer version. Otherwise, a difference between populations can look like a difference between releases.

The customer consequence stays alongside the analysis. If people cannot save items, the responsible team can respond while the diagnosis continues. Restoring the save rate is a different decision. A flow that helps someone find and buy the right item could reduce saving without making shopping worse. The rate alone cannot tell us which experience people are having.

Into the released product

The agent carries the question from the chart into the implementation, tracing where the app records a save and comparing that path with the previous deployed version. This continuity is what makes the connected tools useful: the investigation can move beyond an explanation of the chart.

In the newer version, the event is recorded after a confirmation step. An engineer checks the change with us, including whether it reached the shoppers behind the decline. The current branch may contain unreleased work, and a feature flag may limit exposure within a released version. The comparison follows the behavior customers encountered.

That change gives us a possible measurement problem: a save may complete while one path through confirmation fails to emit the event. The proposed mechanism gives the next query a purpose. If an independent record of completed saves exists, we can compare the rate of sessions with successful saves against the rate with recorded save events for the exposed population. A widening gap supports further investigation of missing events, provided the session definitions and counting rules agree across the sources.

The confirmation step may also make saving harder. Now we need evidence about attempted saves and where people leave the flow. Successful-save records cannot reveal every attempt that failed. Where those attempts are not measured, a focused reproduction gives us another way to examine the behavior. The missing data remains a limit on the analysis.

Both explanations remain open, so the next checks test where each falls short. Does the decline also appear among people who never encountered the changed path? Does the apparent version effect survive a comparison of similar sessions? Evidence against the release explanation changes where we look, even while the code change remains interesting.

An analyst may catch a join that counts a session repeatedly. An engineer may show that the suspected path cannot run for the affected shoppers. Those corrections change the investigation while the explanation is still taking shape. Reading the implementation sharpens the question about the data; checking the data can send us back to a different part of the implementation.

The bug database and wiki add context, with their own limits. A report about saved items appearing only after a refresh gives us a behavior to reproduce and a previous change to inspect. A similar ticket title does not connect the incidents. Documentation that says confirmation follows a durable save gives us a promise to check against the released implementation. Someone familiar with the work can help explain any disagreement.

A working record

The agent keeps the relevant code revision beside the save-path claim, and the query and population beside the comparison. Checks that weaken an explanation belong in the record too. Another person can follow the work without reconstructing the conversation, including why an initially appealing explanation lost support.

The record also shows where the evidence runs out. Without reliable exposure information, we cannot confidently connect the suspicious code to the people behind the decline. More documentation about the design will not establish who encountered the change. That gap gives engineering a specific question to resolve. Reproducing the save behavior and validating the measurement can continue while we wait for the exposure record.

We began with a falling save rate. We now have a more precise question: on the path we have identified, are shoppers failing to save, are successful saves missing from the event stream, or are both happening? The cause remains open, but the next piece of evidence has a clear job. The query, released code, exposure information, and unresolved gaps give engineering and data something concrete to work through together.

This writing reflects my personal perspectives on product management, AI, and content discovery. It does not represent the official position of my employer or any affiliated organization.