Building Reliable LLM Applications

A language model can produce an excellent answer in one run and an unsafe, malformed, or subtly unsupported answer in the next. That variability is not a temporary defect waiting for a larger model. It follows from probabilistic decoding, ambiguous inputs, changing context, model revisions, and external tools that fail independently. Reliable LLM engineering therefore begins by refusing to treat a model call as an ordinary deterministic function.

The useful unit of design is a system around the model: contracts before and after generation, evidence retrieval, permissions, retry policy, evaluation, observability, and a defined path to a human. The model remains powerful, but it occupies one bounded stage in a pipeline whose behavior can be measured and controlled.

This article builds that pipeline for a support assistant that may answer questions, look up an order, or propose a refund. The same design applies to document extraction, coding agents, internal search, and workflow automation. The details differ, but the central rule survives: uncertainty must become explicit data, not an exception discovered by a customer.

Start With Nondeterminism and a Failure Taxonomy #

Traditional software fails too, but identical inputs usually exercise identical branches. An LLM request has more moving parts: prompt construction, retrieved documents, token limits, sampling, provider policy, model version, and tool results. Even with temperature set to zero, providers do not promise bit-for-bit determinism. A reliability target such as “99.9% of requests return HTTP 200” consequently says little about whether answers are useful or safe.

Define failures in terms of user and business outcomes. A practical taxonomy separates at least these classes:

This taxonomy prevents the universal and usually harmful response of “retry everything.” A timeout before any side effect is a retry candidate. A refund tool that may already have succeeded is not, unless the tool accepts an idempotency key. A policy violation should never be retried with slight wording in the hope that it slips through.

Reliability also needs explicit service-level indicators. Track task success, grounded-claim rate, refusal correctness, schema-valid rate, tool execution accuracy, human escalation rate, and latency. Aggregate success alone can hide severe regressions in a rare language or high-value workflow, so segment by intent, model, locale, tenant, and risk tier.

Put Contracts, Grounding, and Guardrails Around the Model #

A reliable request moves through independently testable stages. The model never receives unrestricted credentials and never writes directly to a system of record.

The input gate detects prompt injection patterns, unsupported file types, secrets, and requests outside the product’s purpose. It should not rely only on keyword matching; combine deterministic limits, a specialized classifier where appropriate, and tenant policy. Keep blocked content out of logs or redact it before persistence.

Grounding supplies a closed evidence set. Retrieval is not merely a vector search call: filter by authorization and freshness first, retrieve candidates, rerank them, and attach stable source identifiers. Instructions must distinguish data from commands because retrieved documents can contain adversarial text. The output validator then checks that every material claim cites a source present in the evidence set. Citation presence alone is insufficient; the cited passage must actually support the claim.

Structured output turns a fuzzy completion into a boundary that software can inspect. Use a JSON Schema or a runtime type library, reject unknown fields, constrain enums and numeric ranges, and represent abstention explicitly. Avoid parsing prose with regular expressions. A response such as { "kind": "needs_human", "reason": "conflicting refund policies" } is a successful controlled outcome, not a model failure.

Guardrails should be layered. Input checks protect the model and downstream systems; output checks protect the user. Deterministic checks cover permissions, monetary bounds, personally identifiable information, source membership, and forbidden tool names. Statistical classifiers can cover nuanced safety categories, but they need their own thresholds, eval sets, and monitoring. No single “safety model” is a complete boundary.

Implement a Typed, Retry-Safe Pipeline #

The following TypeScript sketch uses Zod to validate model output and tools. The provider is deliberately abstract: reliability policy should remain application-owned so a provider change does not rewrite business controls.

import { createHash, randomUUID } from 'node:crypto';
import { z } from 'zod';

const Answer = z.object({
  kind: z.literal('answer'),
  text: z.string().min(1).max(4_000),
  citations: z.array(z.string()).min(1),
  confidence: z.number().min(0).max(1),
}).strict();

const ToolProposal = z.object({
  kind: z.literal('tool'),
  name: z.enum(['lookup_order', 'propose_refund']),
  arguments: z.record(z.string(), z.unknown()),
  rationale: z.string().max(500),
}).strict();

const NeedsHuman = z.object({
  kind: z.literal('needs_human'),
  reason: z.string().min(1).max(500),
}).strict();

const ModelDecision = z.discriminatedUnion('kind', [
  Answer,
  ToolProposal,
  NeedsHuman,
]);

type Decision = z.infer<typeof ModelDecision>;
type Evidence = { id: string; text: string; updatedAt: string };

interface ModelClient {
  generate(input: {
    requestId: string;
    prompt: string;
    responseSchema: object;
    timeoutMs: number;
  }): Promise<unknown>;
}

class TransientModelError extends Error {}

async function withBackoff<T>(operation: () => Promise<T>): Promise<T> {
  const delays = [0, 250, 800];
  let lastError: unknown;

  for (const delay of delays) {
    if (delay > 0) await new Promise((resolve) => setTimeout(resolve, delay));
    try {
      return await operation();
    } catch (error) {
      lastError = error;
      if (!(error instanceof TransientModelError)) throw error;
    }
  }
  throw lastError;
}

function verifyCitations(decision: Decision, evidence: Evidence[]): void {
  if (decision.kind !== 'answer') return;
  const allowed = new Set(evidence.map((item) => item.id));
  if (decision.citations.some((id) => !allowed.has(id))) {
    throw new Error('Answer contains an unknown citation');
  }
}

function authorizeTool(
  actor: { id: string; roles: string[] },
  decision: z.infer<typeof ToolProposal>,
): void {
  if (decision.name === 'propose_refund' && !actor.roles.includes('refund-agent')) {
    throw new Error('Actor cannot propose refunds');
  }
  if (decision.name === 'propose_refund') {
    const args = z.object({ orderId: z.string(), amount: z.number().positive().max(200) })
      .strict()
      .parse(decision.arguments);
    decision.arguments = args;
  }
}

async function handleRequest(input: {
  actor: { id: string; roles: string[] };
  question: string;
  evidence: Evidence[];
  primary: ModelClient;
  fallback: ModelClient;
}): Promise<Decision> {
  const requestId = randomUUID();
  const prompt = buildPrompt(input.question, input.evidence);
  const invoke = (client: ModelClient) => withBackoff(async () => {
    const raw = await client.generate({
      requestId,
      prompt,
      responseSchema: ModelDecision.toJSONSchema(),
      timeoutMs: 8_000,
    });
    return ModelDecision.parse(raw);
  });

  let decision: Decision;
  try {
    decision = await invoke(input.primary);
  } catch (error) {
    if (!(error instanceof TransientModelError)) throw error;
    decision = await invoke(input.fallback);
  }

  verifyCitations(decision, input.evidence);
  if (decision.kind === 'tool') {
    authorizeTool(input.actor, decision);
    const key = createHash('sha256')
      .update(`${requestId}:${decision.name}:${JSON.stringify(decision.arguments)}`)
      .digest('hex');
    await executeToolOnce(decision.name, decision.arguments, key);
  }
  return decision;
}

withBackoff retries only an error classified as transient. Production code should add randomized jitter and honor the request’s total deadline rather than granting every attempt a fresh timeout. Retry budgets prevent an impaired provider from multiplying traffic. Circuit breakers stop calls when recent failure rates cross a threshold, while bulkheads keep one expensive tenant or workflow from consuming all concurrency.

The fallback should be meaningfully independent: another model, provider, region, or a deterministic retrieval-only response. A smaller fallback may be faster and cheaper, but it still needs the same output contract. For low-risk informational requests, a cached response can be an excellent degradation. For money movement or account changes, the safest fallback is often needs_human.

Tool execution has a separate transaction boundary. Persist the idempotency key and result atomically, then return the stored result for duplicates. Bind approval to the exact tool name and canonical arguments; changing the amount after approval must invalidate it. Short-lived capabilities are preferable to placing broad credentials in model context.

Build Evals and Human Review Into Delivery #

Unit tests cover prompt assembly, schema validation, authorization, citation membership, and idempotency. They cannot establish answer quality. Evals fill that gap with representative tasks and explicit scoring. Begin with production-shaped examples: common requests, policy boundaries, multilingual inputs, prompt injections, empty retrieval, conflicting sources, long context, and rare high-impact cases.

Keep a versioned dataset with the input, permitted evidence, expected behavior, risk tier, and rubric. Some checks are deterministic: valid schema, expected refusal, allowed citation, exact tool and arguments. Nuanced answers may use expert labels or an LLM judge, but calibrate that judge against human ratings and periodically measure disagreement. Never let one opaque score decide deployment.

from dataclasses import dataclass
from statistics import mean
from typing import Literal

@dataclass(frozen=True)
class EvalCase:
    case_id: str
    prompt: str
    expected: Literal["answer", "tool", "needs_human"]
    allowed_sources: frozenset[str]
    risk: Literal["low", "high"]

@dataclass(frozen=True)
class EvalResult:
    case_id: str
    correct_route: bool
    grounded: bool
    latency_ms: int
    cost_usd: float

def release_gate(results: list[EvalResult]) -> None:
    route_rate = mean(r.correct_route for r in results)
    grounded_rate = mean(r.grounded for r in results)
    p95_latency = sorted(r.latency_ms for r in results)[int(len(results) * 0.95) - 1]
    average_cost = mean(r.cost_usd for r in results)

    failures = []
    if route_rate < 0.98:
        failures.append(f"route accuracy {route_rate:.2%} < 98%")
    if grounded_rate < 0.995:
        failures.append(f"grounded rate {grounded_rate:.2%} < 99.5%")
    if p95_latency > 5_000:
        failures.append(f"p95 latency {p95_latency}ms > 5000ms")
    if average_cost > 0.025:
        failures.append(f"average cost ${average_cost:.4f} > $0.025")
    if failures:
        raise RuntimeError("release blocked: " + "; ".join(failures))

Run offline evals on every prompt, model, retriever, or policy change. Compare against the current production configuration, not an arbitrary absolute target. Maintain slices so an overall improvement cannot conceal a regression in Vietnamese requests or refund handling. Add failures from incidents and human review back to the set, while keeping a hidden holdout to discourage tuning directly to the test cases.

Human review is a control plane, not an apology. Route requests when policy demands approval, confidence is low, evidence conflicts, a high-value action is proposed, or automated checks disagree. Reviewers need the original request, retrieved evidence, model decision, validation results, and a clear action, with sensitive data minimized. Their overrides become labeled examples, but avoid learning blindly from hurried or inconsistent decisions.

Observe, Budget, and Roll Out the Whole System #

An LLM trace should connect request classification, retrieval query and source IDs, prompt template version, model and provider version, token counts, validation outcomes, retry attempts, tool calls, final route, latency, and estimated cost. Store hashes or redacted representations when prompts contain sensitive material. Logs must answer “what changed?” without becoming a new data leak.

Measure stage latency rather than only end-to-end time. Retrieval, queueing, time to first token, generation, guardrails, and tools have different owners and remedies. Track percentiles, because averages hide the users trapped behind retries. Cost should be attributed per successful task, not merely per token: a cheap model that causes repeated calls and human escalations may be the expensive option.

Set budgets before optimization. Route simple classifications to a small model, cap retrieved context, cache stable evidence and safe deterministic responses, stream only after checks compatible with streaming, and run independent retrieval operations concurrently. Do not compress prompts until evals show quality remains stable. Latency and quality are a portfolio decision: high-risk actions can justify slower validation, while autocomplete cannot.

Deploy changes through shadow traffic and canaries. Shadowing sends sampled production inputs to the candidate without using its outputs; canaries expose a small, representative cohort and watch both quality and operational indicators. Pin model versions where possible. Record prompt, retriever, policy, and model as one deployable configuration so rollback restores a known combination.

A rollout gate might require no critical policy failures, non-inferior task success within a confidence interval, bounded p95 latency, and cost below budget. Increase traffic gradually and preserve a kill switch that disables tools independently of answers. A model provider’s successful rollout does not establish that your prompts, evidence, and user population are safe.

Respond to Incidents and Keep the Takeaways #

Prepare runbooks before the first incident. Alerts should map to user impact: schema-valid rate collapse, unsupported claims rising, a tool denial spike, retrieval freshness lag, provider saturation, or cost per task jumping. The first response is containment: disable the affected tool, route to a fallback, force retrieval-only answers, narrow traffic, or require human approval. Preserve trace identifiers and configuration versions without copying sensitive prompts into chat channels.

Then determine the failure class and blast radius. Query by model version, prompt version, intent, locale, tenant, source corpus, and tool. Replay a sanitized sample against the previous configuration. If an external model changed silently, the stored traces and eval baseline make that visible. Communicate which outcomes may be wrong, not only that latency increased.

Recovery includes more than a prompt patch. Add the incident cases to regression evals, repair the missing deterministic control, review whether monitoring detected the problem early enough, and test the rollback path. Prompt wording can reduce frequency, but authorization, idempotency, schema validation, and source checks must enforce invariants even when generation is adversarial.

The durable takeaways are straightforward:

• Treat model output as untrusted input and validate it with strict schemas.
• Ground consequential claims in an authorized, inspectable evidence set.
• Gate every tool with deterministic authorization and idempotent execution.
• Retry only classified transient failures within a total deadline; prefer safe degradation elsewhere.
• Evaluate behavior by risk and slice, with human review for ambiguity and high-impact actions.
• Observe versions, quality, latency, and cost across the entire pipeline.
• Roll out model-system configurations gradually and rehearse containment and rollback.

Reliable LLM applications are not deterministic, but they can be disciplined. The goal is not to eliminate every uncertain answer. It is to detect uncertainty, constrain what it can affect, and give the system a tested way to abstain, degrade, or ask a person before uncertainty becomes damage.