What Should an AI Be Allowed to Know?

Lessons from building an AI AMA with explicit boundaries for knowledge, privacy, and personal claims.

I began with a simple requirement for my website: a visitor should be able to ask about my writing, product thinking, career background, or interests and receive a useful answer grounded in material I was comfortable sharing.

As I worked through the questions the AMA might receive, that requirement became less simple.

“How should a product team evaluate AI-powered recommendations?” is a request for general expertise. “What does Cris believe about AI discovery?” asks the system to represent my point of view. “Which private file supports that answer?” is probing for information the visitor should never receive. A question about personal legal or employment status should be stopped before the model tries to answer it.

All four arrive through the same text box. They do not require the same evidence or the same product behavior.

That was when I realized that deciding what the system could know was only part of the work. I also had to decide what it could claim, which sources could support that claim, and what should happen when a question crossed a boundary.

A personal answer requires a different standard of evidence

The public part of the knowledge base came from material already on the site: articles, notes, and profile information. Instead of giving the model a collection of pages and asking it to make sense of them at runtime, I turned that material into structured records. Each record identifies the topics it supports, how strong the evidence is, which kinds of answers may use it, and which uses are prohibited.

Private material required a stricter process. Source candidates were screened before review so that obviously sensitive categories could be excluded without opening the underlying files. Some material was rejected. Some was usable only after narrowing or redaction. Approved information was paraphrased and separated from file names, URLs, senders, recipients, message identifiers, and raw document text before it could enter the runtime knowledge.

I deliberately kept the runtime knowledge smaller than the collection of information the system could theoretically access. Retrieval searches the reviewed claims, not my personal archive.

That constraint became important when I considered broad questions. I was comfortable letting the AMA draw on general product and technical knowledge to explain how a team might evaluate recommendations. I was not comfortable letting the same general knowledge stand in for my experience or beliefs. A personal claim needed support from my writing or another approved record.

This gave the AMA room to answer useful questions beyond the subjects I had already covered in an article, while keeping a higher bar for answers that represented me.

I did not want privacy to depend on the model remembering a rule

Prompt instructions are part of the protection, but I did not consider them sufficient. A privacy rule that exists only inside a prompt can fail when a question is phrased unexpectedly, when unsafe context is retrieved, or when a generated follow-up introduces a detail that was not in the original answer.

The AMA therefore checks the question before any model call. It checks the approved context selected for the answer. It scans the generated response and follow-up suggestions before returning them to the visitor. The interaction is checked again before storage so that privacy-risk text can be redacted and tagged. Stored transcripts also have a limited retention period.

The checks cover several distinct failure modes: direct personal information, private family references, credentials and account-security content, protected legal or employment topics, confidential business details, and metadata that could reveal where private knowledge came from.

The source-probing case was useful because it exposed a boundary that is easy to miss. An answer may be safe while the provenance behind it is not. If someone asks which private file supported a response, the system blocks the request before it reaches OpenAI. The same applies to questions about protected personal status. If prohibited material appears later in an answer or suggested follow-up, it is stopped before the visitor sees it.

I had initially thought about privacy mostly as an answer problem. Mapping the full runtime exposed other places where information could escape: the context sent to the model, a suggested follow-up, an analytics record, or a transcript opened for review. Each needed its own handling.

The evaluation lab

I built deterministic evaluations first. They check whether the AMA chose the expected outcome, selected required knowledge, stayed within the correct answer mode, avoided forbidden patterns, and refused privacy-sensitive questions. Those tests are well suited to decisions that should produce the same result every time.

Those checks do not tell me whether an otherwise valid answer is direct, useful, or specific enough. Approved evidence can still produce a generic response. A safe follow-up can still lead the conversation nowhere. The distinction between general guidance and a personal claim can be technically correct while the answer itself remains weak.

For that part of the evaluation, I added support for an optional LLM-as-judge pass. The judge receives the question, expected behavior, observed answer, follow-ups, approved knowledge summaries, and deterministic findings. Raw private sources and their metadata are excluded.

The judge can assess qualities such as directness, groundedness, usefulness, and follow-up relevance. It cannot erase a deterministic privacy failure. A low privacy or confidentiality score also fails the evaluation rather than being averaged into an otherwise acceptable result.

I did not want “the grader liked it” to become a sufficient explanation for releasing an answer. A qualitative score can help me review the experience. A deterministic privacy finding remains a release blocker.

The same principle shaped the admin experience. The AMA records category, answer mode, outcome, selected knowledge, latency, and refusal reason. Aggregate patterns can be reviewed separately from individual transcripts. Deterministic tests and judge runs can be inspected from an evaluation lab.

Unanswered questions appear as possible coverage gaps, but they do not automatically become knowledge.

When a question recurs, I still have to decide whether the public material is incomplete, the prompt or policy is getting in the way, or the question should remain outside the system. The review queue leaves that decision with a person.

Frontier AI tools accelerated the interface and implementation. They also brought the evidence, privacy, and evaluation questions into the work earlier, while I was still deciding what the product should be.

By the time the AMA could answer coherently, the question I kept returning to was no longer whether the response sounded good. It was whether I could explain why the system was allowed to say it. That is still the test I use when I review the product.

---

This writing reflects my personal perspectives on product management, AI, and content discovery. It does not represent the official position of my employer or any affiliated organization.