One Task, Multiple Agents, One Continuous Loop

The simple project that wasn’t.

Building an agent loop that can observe, validate, and learn from a phone interface

Over the past week, I have been building a system to capture and analyze the recommendations appearing in a vertical video feed on my iPhone.

The initial task sounded straightforward: open the app, move through the feed, take screenshots, read the titles, and put the results into a dashboard.

The reality was far from straightforward. But the experience gave me the opportunity to work In an orchestrated loop, managing multiple agents completing different tasks.

A phone interface is not an API. It changes state, plays video, displays loading screens, repeats recommendations, loses focus, re-enables audio, and occasionally fails to respond to a gesture. A screenshot can contain a valid recommendation, a black frame, an authentication message, a half-completed transition, or a loading surface that looks superficially similar to the feed.

This project evolved into an orchestration effort. I am building a loop in which an orchestrator coordinates several specialized agents and workers:

one operates the iPhone through iPhone Mirroring and captures screenshots
one performs OCR and watches for evidence that the capture should stop
one retrieves information from TMDB to validate resolved titles and metadata
one evaluates data quality, including black frames, duplicates, repeated clips, and missing positions
one creates the analytics
one turns the validated output into a visual, reviewable experience

The important architectural decision is that these components do not simply run one after another. They participate in a control loop.

Capture is an active process, not a recording step

The capture worker is responsible for manipulating the phone, but its job is more complicated than issuing a swipe and saving an image.

Before each interaction, it has to locate the current iPhone Mirroring window, bring it to the foreground, calculate the phone’s live position, and confirm that the expected app surface is visible.

It then captures the current recommendation, verifies the screenshot, performs the swipe, saves a frame from the transition, waits for the next card to settle, and compares the new state with the previous one.

Video pixels can change even when the feed has not advanced. A recommendation can also repeat immediately, which means an unchanged title does not necessarily indicate a failed swipe. The system needs evidence that one card moved out and another moved in.

That turns a seemingly simple action into a state transition:

observe → capture → verify → move → prove movement → accept

If the transition cannot be proven, the orchestrator does not increment the recommendation position. It retries with a bounded recovery strategy. If it still cannot establish what happened, it stops before corrupting the sequence.

That is a much more important behavior than successfully swiping most of the time.

A hand-drawn state-transition loop showing a phone worker observe, capture, verify, move, prove movement, and accept, with retry or stop when the transition is uncertain. — A swipe is only an action. The position advances when the system can prove the interface moved from one recommendation to the next.

The OCR worker can interrupt the operator

OCR runs independently from the phone-driving loop.

Screenshots are preserved in durable batches, while smaller groups are sent to a background OCR worker. This allows phone capture to continue without waiting for every image to be fully interpreted.

But the OCR worker is not just an extraction service. It is also an observer.

It can recognize signals such as:

a blank or loading surface
an authentication overlay
an “iPhone in Use” interruption
missing title information
a possible end-of-feed condition

These signals are sent back to the orchestrator. Some trigger a retry. Some make the current row review-only. Some stop the profile. Some invalidate the entire run.

For example, several consecutive clear loading frames may indicate that the stream has ended. An authentication overlay is different: it is an operational interruption, not evidence that the recommendation feed ended normally.

The affects whether the session is completed, partial, blocked, failed, or excluded entirely.

This is where the loop comes in. A specialized worker is changing what the rest of the system is allowed to do next.

Retrieval comes after evidence

Once the screenshot has been read, another worker retrieves information from TMDB.

The tempting implementation would be to send partial OCR text to TMDB, take the closest result, and treat that result as the title. That would produce cleaner-looking data, but it would also create false confidence.

The system therefore follows a screenshot-first identity policy.

The visible title treatment in the screenshot is the primary evidence. OCR helps extract it. Repeated visual evidence can support it. TMDB is used afterward to validate the resolved entity, confirm media type and episode information, and retrieve artwork.

TMDB is not allowed to invent a title that the screenshot did not establish.

This separation between evidence and enrichment has become one of the most important principles in the project:

Retrieval can validate an identity, but it should not silently originate one.

When the available evidence is insufficient, the system preserves the recommendation and marks it for review. It does not drop the row, and it does not replace uncertainty with the most plausible database result.

That makes the final dataset less cosmetically perfect, but more trustworthy.

A hand-drawn evidence pathway in which screenshot evidence is supported by OCR, validated by TMDB, and routed to a supported match or unresolved review. — The screenshot establishes identity; OCR extracts it; TMDB validates and enriches it. Uncertainty stays visible.

Quality assessment is part of the loop

Another worker evaluates the resulting evidence for data-quality problems.

It looks for missing positions, duplicated position numbers, incomplete sessions, unchanged captures, adjacent repeats, exact clip duplication, and screenshots that begin with an essentially black playable scene.

These checks serve different purposes.

A missing position can indicate a broken evidence chain and prevent publication. A repeated title may be a legitimate recommendation and must remain in the dataset. An exact repeated clip is an analytical signal, not a reason to delete a row. A dark opening frame may be a real creative choice rather than a capture failure.

The system therefore does not reduce quality to a single pass-or-fail score. It classifies findings according to what they mean.

Some are integrity failures.
Some are product-experience signals.
Some require human review.
Some are expected characteristics of the recommendation system.

The orchestrator needs a shared vocabulary for state, evidence, severity, and allowable next actions.

The dashboard is downstream of trust

Once the run passes its readiness checks, the analytics worker creates per-profile summaries and historical comparisons.

It calculates how many clips were shown, how many titles appeared, how frequently each title repeated, which exact clips appeared more than once, which recommendations started on dark frames, and how the feed compares with previous days.

A final visualization worker turns that output into an interface where I can inspect the feed in order, switch between poster art and the original screen capture, open the underlying evidence, filter quality signals, and review historical overlap.

The dashboard is where the system’s reasoning becomes inspectable.

A polished chart without traceable screenshots would make the pipeline look more reliable than it is. Every important analytical claim therefore needs a path back to the captured evidence, the profile, the session, and the original recommendation position.

The visual layer is useful because the evidence lineage underneath it is intact.

Human judgment retrains the loop

Automation can identify anomalies, but it cannot always determine what those anomalies mean.

A repeated title might be a legitimate recommendation, a duplicate clip, or evidence that the phone never advanced. A dark frame might be a capture failure or the intentional opening of a scene. An OCR result might be wrong even when a TMDB search returns a convincing match.

We therefore created a human-in-the-loop review process.

When the system encounters uncertain evidence, it preserves the original screenshot, OCR output, proposed identity, retrieval results, and quality signals. A human reviewer can then inspect that evidence and record a judgment without rewriting the original history.

Those decisions become feedback for the orchestration loop. We use them to identify recurring failure patterns, refine capture and validation rules, add regression tests, adjust thresholds, and improve the evidence shown to later workers.

In that sense, human judgment retrains the system—not necessarily by fine-tuning an underlying model, but by improving how the complete loop behaves.

The feedback cycle looks like this:

capture → evaluate → escalate → review → learn → update the loop

Importantly, a human decision does not become an unchecked shortcut for future cases. It helps us improve the policy and create better evaluation evidence, but new recommendations still have to pass their own validation requirements.

This protects the system from turning one correct human judgment into a broad and potentially incorrect assumption.

The purpose of the human-in-the-loop process is therefore not to manually clean the output forever. It is to convert ambiguous cases into structured learning that makes the next run more reliable.

The automation handles scale and consistency. Human judgment helps the system understand where its rules are incomplete.

Together, they create a loop that can improve without hiding uncertainty.

A hand-drawn feedback loop moving from capture through evaluation, escalation, human review, learning, and updated rules while preserving an evidence ledger. — Human review improves policies, tests, and thresholds for the next run without rewriting the evidence from the last one.

Orchestration is more than parallelism

It is easy to describe a multi-agent architecture as several intelligent components working simultaneously. But the real leverage comes from defining the contracts between them:

What evidence does each worker receive?
What output is it allowed to produce?
Which worker can interrupt another?
Which failures are recoverable?
Which uncertainty must be preserved?
What conditions allow the loop to continue?
What conditions make publication unsafe?

The orchestrator is responsible for those contracts.

It discovers the account profiles, starts independent work where parallel execution is safe, keeps phone control responsive, receives OCR stop signals, waits for required evidence, evaluates session status, and joins the different workstreams before anything becomes canonical.

It also fails closed. If OCR results do not correspond exactly to the captured screenshot positions, the run cannot quietly continue into publishing. If an expected profile is missing or blocked, the system cannot describe the account-wide capture as complete. If the data fails readiness validation, the dashboard is not rebuilt from it.

That is what makes the loop accountable.

Not every worker needs to be an LLM

I use the language of agents because these components have bounded responsibilities, operate with some independence, and return findings to an orchestrator.

But the implementation intentionally mixes several kinds of workers.

Some tasks benefit from an agent that can reason about ambiguous visual or textual evidence. Others are better handled by deterministic code: checking contiguous positions, calculating image luminance, comparing hashes, validating schemas, or rebuilding a workbook.

The goal is not to maximize the number of LLMs in the architecture.

The goal is to assign each decision to the most reliable mechanism available and make every transition reviewable.

That is a broader lesson I am taking from the project. A strong agentic system is not a group chat between models. It is a designed operating system for work.

From prompting agents to designing control systems

I started this project trying to automate the capture of information from my phone.

What I am actually building is a small evidence-processing system.

The phone worker observes the interface. The OCR worker interprets and monitors it. The retrieval worker validates identity. The quality worker challenges the integrity of the result. The analytics worker turns accepted evidence into measures. The visualization worker makes the entire process reviewable.

The orchestrator keeps those roles aligned and determines whether the loop should continue, recover, stop, or publish.

The next stage of working with agents is not just asking several agents to do different things. It is designing the relationships between them: their responsibilities, evidence contracts, interruption rights, recovery paths, and stopping rules.

The orchestrator does not need to perform every task.

It needs to know what must be true before the system is allowed to take the next step.

That is how a collection of automated tasks becomes a trustworthy loop.

This writing reflects my personal perspectives on product management, AI, and content discovery. It does not represent the official position of my employer or any affiliated organization.