Understanding Things — Kean Neng Lim
AI / Software Architecture

Designing Deterministic Boundaries Around Probabilistic AI

LLMs are probabilistic components. Reliable systems are built by deciding, explicitly, which parts of the system are allowed to be probabilistic and wrapping everything else in code that behaves the same way every time.

Updated July 5, 20264 min read

Every reliability problem in an AI-integrated system traces back to the same root cause: something that needed to be deterministic was left to the model. Not because the model is unreliable in some vague sense, but because an LLM is fundamentally a probability estimator — asking it to be reliable in the way a function call is reliable is asking it to be something it structurally isn’t.

The fix isn’t a better prompt. It’s an architectural boundary.

The boundary, stated precisely

A deterministic boundary is the line in your system on one side of which behavior is guaranteed by code — validated, type-checked, tested — and on the other side of which behavior is a probability distribution the model happened to sample from. Everything that crosses that boundary from the probabilistic side has to be treated as untrusted input, exactly the way you’d treat unvalidated data from an external API.

┌────────────────────────┐        ┌───────────────────────────┐
│   Probabilistic zone    │        │      Deterministic zone     │
│                          │        │                              │
│  LLM generation          │──────▶│  Schema validation           │
│  (creative, variable,    │ output │  Business rule enforcement   │
│   context-dependent)     │        │  State transitions           │
│                          │        │  Side effects (writes,       │
│                          │        │  payments, external calls)   │
└────────────────────────┘        └───────────────────────────┘

The failure mode this prevents is specific: a system where the model’s output is trusted to already satisfy a constraint — a valid enum value, a well-formed identifier, a business rule like “never approve a refund over $500 without a second check” — because the prompt asked for it. The prompt is a strong nudge on the probability distribution. It is not a guarantee, and treating it as one means the system is only as reliable as the model is at that specific request, which varies by input, by model version, and by nothing you can fully predict from the training data.

What belongs in each zone

Probabilistic zone — where you actually want the model’s flexibility:

  • Drafting language, summarizing, explaining
  • Proposing options, ranking candidates by fuzzy relevance
  • Interpreting ambiguous or unstructured input into a structured guess

Deterministic zone — where a guarantee, not a tendency, is required:

  • Anything that mutates state: writes, payments, sending a message, deleting a record
  • Enforcing an invariant the business depends on (limits, permissions, uniqueness)
  • Parsing the model’s output against a schema and rejecting or retrying on mismatch
  • Deciding what happens on failure — retry, fall back, escalate to a human

A concrete pattern that follows directly from this: never let model output execute a side effect directly. Have the model propose a structured action (a typed object, a function call with validated arguments), and have deterministic code decide whether to execute it. The model doesn’t get to “just do it” — it gets to make a well-formed request that the deterministic zone is free to reject.

A worked example

Consider a support system that lets an LLM draft a response and, based on the conversation, decide whether to issue a refund.

The naive version prompts the model to “issue refunds up to $50 automatically, and otherwise escalate,” and trusts whatever the model says. This puts a business-critical decision — moving money — entirely inside the probabilistic zone. It will work most of the time. “Most of the time” is not a property you can build a payments system on.

The bounded version looks different:

1. Model reads conversation, proposes: { action: "refund", amount: 32.00, reason: "..." }
2. Deterministic validator checks:
     - amount <= 50            (business rule, enforced in code)
     - conversation has an associated order       (data integrity check)
     - no refund already issued for this order     (idempotency check)
3. If all checks pass: deterministic code executes the refund.
4. If any check fails: deterministic code escalates to a human, regardless of
   what the model's proposal said.

The model still does the part it’s good at — reading unstructured conversation and producing a structured judgment call. The part that has to be reliable — enforcing the $50 limit, preventing double refunds — never depends on the model getting it right. It depends on code that behaves the same way on every input, which is the actual definition of “deterministic” being used here.

Human review as a deterministic state, not an escape hatch

This pattern generalizes: whenever a proposed action fails a deterministic check, the system needs a defined next state, not an implicit failure. “Escalate to a human” is itself a state transition that should be modeled explicitly — who gets notified, what they see, what happens if they don’t respond, whether the conversation is blocked or continues in parallel. Treating human review as a formal workflow state, with the same rigor as any other state transition, is what keeps “the model wasn’t sure” from turning into a silent stall.

Why this scales better than better prompting

Prompt engineering improves the probability that the model does the right thing. It does not change the fact that it’s still a probability. Deterministic boundaries change the failure mode from “occasionally wrong in production, discovered by a customer” to “occasionally rejected before it can matter, discovered by a validator.” The second failure mode is one you can log, alert on, and fix. The first one you often can’t even detect.

This is also why the boundary should sit as close to the model’s output as possible. Every layer of the system downstream of an unvalidated model output inherits its uncertainty. Validate once, immediately, at the seam — and let every deterministic layer after that seam operate on data it can actually trust. That same instinct — validate at the boundary, don’t let uncertainty propagate downstream — is the general systems principle explored in Caching Is a Consistency Problem, which looks at what happens when a different kind of boundary (between a cache and its source of truth) is left unguarded instead.

Related articles