AI systems need deterministic control points

An AI feature can produce a convincing answer and still be unsafe to trust with an action. The reason is architectural: language models generate probable outputs, while many business decisions require rules that are explicit, testable and repeatable.

This does not make generative AI unsuitable for operational systems. It means the model should sit inside a controlled workflow rather than become the workflow. NIST’s AI Risk Management Framework treats AI risk management as a lifecycle activity across governance, context mapping, measurement and management. That is a useful engineering reminder: reliability depends on the surrounding system, not only on the model.

Separate judgement from authority

A model may be well suited to summarising a document, suggesting a classification or drafting an explanation. It should not automatically inherit permission to publish, approve a payment, change a user’s access or delete a record simply because it can describe those actions.

The safest boundary is usually simple: the model proposes; deterministic software decides what is permitted. Authentication, role checks, data validation, rate limits and transaction rules should remain ordinary application logic. If human approval is required, the approval state should be stored as data and verified before execution, not inferred from conversational wording.

This distinction also limits excessive agency. OWASP’s guidance for large-language-model applications identifies both improper output handling and excessive agency as material risks. In practice, a model response must be treated as untrusted input, even when the response was produced by a model selected and configured by the organisation.

Design the control points

A useful AI workflow should expose several control points that engineers can inspect and test.

Input boundary: accept only the data needed for the task, and remove secrets or personal information that the model does not require.

Retrieval boundary: identify which sources may be used, record their versions or timestamps, and make missing evidence visible.

Output boundary: require a schema, validate types and allowed values, and reject incomplete or unexpected structures.

Permission boundary: calculate the user’s authority outside the model and give tools the minimum capability needed.

Approval boundary: route higher-impact proposals to a named human or a deterministic rule before execution.

Audit boundary: record the request, relevant source references, model version, decision, approver and final action without storing unnecessary sensitive content.

These controls are valuable even when the model performs well. They make failure observable. A system that quietly converts a malformed response into a default action is harder to govern than one that stops, records the reason and asks for review.

Test the system, not only the prompt

Prompt testing is useful, but prompts are only one component. A stronger test plan includes stale documents, conflicting sources, missing fields, hostile instructions inside retrieved content, malformed structured output, unavailable tools and users attempting actions beyond their role.

Consider a hypothetical research platform that uses AI to prepare a short description of a newly collected document. The model may draft the description, but publication should still depend on source provenance, rights status, required metadata and editorial approval. If rights information is absent, the correct outcome is not a more persuasive description. It is a blocked publication with a clear reason.

Evaluation should therefore measure more than writing quality. Useful measures include unsupported-claim rate, schema-validation failures, correct refusals, blocked permission violations, source coverage and the proportion of proposed actions that required correction. These measures connect model behaviour to operational risk.

Make failure states part of the product

Responsible AI design is often described in policy language, but some of its strongest controls are ordinary software-engineering practices: least privilege, explicit state transitions, idempotent operations, validation, audit logs and rollback.

The interface should also explain what happened. A user needs to know whether an answer was generated from current evidence, whether an action is awaiting approval, or whether a safeguard prevented completion. Hiding these states behind a smooth conversational response creates false confidence.

The goal is not to eliminate uncertainty from AI. That would be unrealistic. The goal is to prevent uncertain output from silently becoming authorised action. When deterministic control points surround the model, teams can gain the flexibility of AI without surrendering the discipline that reliable systems require.