Retrieval is a product decision

Ask an assistant, “Can a contractor export customer data?”

The search layer returns three fragments:

1. an old onboarding guide saying contractors use the same tools as employees;
2. the current data-handling policy saying exports require a staff owner;
3. a product manual explaining where the export button lives.

The model has enough language to produce a convincing answer in either direction. The product has not yet done enough work to deserve one.

Retrieval is an evidence pipeline

OpenAI's retrieval guide describes semantic search as a way to surface similar results “even when they match few or no keywords”. That is useful, but similarity is not authority.

A production retrieval pipeline has at least four decisions:

source → index → retrieve/rank → answer

Debug them in that order. Prompt changes cannot recover a missing source. A better model cannot infer that one of two conflicting documents has been superseded unless the system provides that information.

Source: decide what is allowed to be true

Create a source register before an index:

• Which system is authoritative for each class of question?
• Who owns the source?
• How quickly must an update appear?
• Which version is effective now?
• Who is allowed to retrieve it?
• When should it expire or be deleted?

In the contractor example, the policy needs an effective date and higher authority than the onboarding guide. The guide should be updated or excluded, not left in the corpus with a hope that ranking will compensate.

Permissions belong at ingestion and retrieval. Filtering after generation risks exposing the fact or shape of information the user was never allowed to access.

Index: preserve the unit of meaning

Fixed-size chunks are convenient and often destructive. A table separated from its headings, an exception separated from the rule it modifies or a code example separated from its version can become misleading evidence.

Store useful structure with every chunk:

• document ID and canonical URL;
• heading path;
• version or effective date;
• source owner;
• access-control attributes;
• neighbouring section references;
• content type.

OpenAI's vector-store API supports file attributes, filtering, query rewriting, score thresholds and ranking options. Those controls are valuable only when the metadata carries product meaning.

Retrieve: test the awkward language

Users will not phrase questions like the policy document. They use acronyms, old product names and incomplete descriptions. Build a query set from real support language and include:

• the obvious wording;
• a colloquial version;
• a question containing an incorrect assumption;
• a request that spans two documents;
• a question the corpus cannot answer;
• the same question from users with different permissions.

For “Can a contractor export customer data?”, inspect the ranked evidence before looking at generated prose. Did the current policy appear first? Was the obsolete guide filtered or clearly marked? Did the product manual appear as procedure rather than permission evidence?

Hybrid lexical and semantic search can help when exact identifiers and conceptual meaning both matter. Reranking can improve order. Each additional stage should earn its latency and complexity against the evaluation set.

Answer: make uncertainty useful

A grounded answer should separate policy from procedure:

Contractors cannot export customer data independently. The current policy requires a staff owner to authorise the export. If that approval exists, follow the export procedure here.

The user should be able to open the supporting policy at the relevant section, see its effective date and inspect the procedural source separately.

When evidence conflicts, say so. When no authoritative source answers the question, refuse narrowly and route the gap to the source owner. “I could not find a current policy for this case” is more useful than a fluent average of two obsolete documents.

Evaluate evidence before eloquence

Score retrieval and generation separately:

1. Authority: did the correct source enter the result set?
2. Rank: did it appear before distracting or obsolete material?
3. Coverage: was enough context present to apply the rule?
4. Permission: was every retrieved fragment allowed for this user?
5. Grounding: did each material claim follow from the evidence?
6. Abstention: did the product decline when evidence was insufficient?

Only then assess clarity and tone.

This is the kind of product boundary we handle in AI engineering and work such as AMiSi.ai: the model is one component, while the reliability of the answer depends on the system that selects and exposes knowledge.

The best retrieval improvement is sometimes a reranker. Sometimes it is deleting an obsolete PDF. Product teams need to be able to tell which.

Sources

• Retrieval guide — OpenAI
• Vector stores API — OpenAI
• AI Risk Management Framework — NIST