Balancing Determinism and Stochasticity in a Public-Facing Immigration-Law Chatbot
A Guardrail Architecture for Natural-yet-Predictable Conversational AI
Large language models feel natural precisely because they sample from probability distributions, but in a regulated, public-facing channel that same property becomes a reliability problem. This paper argues that prompt engineering is necessary but structurally insufficient, and proposes a layered architecture that keeps the model for language judgment while moving compliance-critical facts into deterministic code, validation, and canonical resource resolution.
.png)
.png)
.png)
PDFBalancing Determinism and Stochasticity in a Public-Facing Immigration-Law Chatbot
A Guardrail Architecture for Natural-yet-Predictable Conversational AI
.png)
Anonymized operational contextAn intentional division of labor between the model and the system around it
Deterministic guardrails
Mechanically checkable rules such as approved links, mandatory disclosures, and banned phrasings are enforced in code, validation, and allowlists, not left to the model.
.png)
Probabilistic language generation
The model is reserved for judgment-bound work: recognizing intent, adapting tone, and composing natural, human-feeling responses.
Compliance-critical validation
A post-generation validation pass checks required disclosures and canonical values, correcting or regenerating before anything reaches the user.
Human oversight and escalation
The public assistant stays on the information side of the advice boundary. Anything case-specific escalates to a licensed professional.
Why this matters for regulated AI systems
In a regulated domain, fluency is not enough. A public assistant operating in a legal-adjacent context cannot be judged only on whether its answers read well. It must not provide individualized legal advice, predict a user's likelihood of success, invent procedural requirements, or obscure the role of licensed professionals. These are boundary conditions for safe deployment, not tone preferences.
The exposure is sharpest for a non-licensee operator whose product sits adjacent to licensed practice. A rule that holds ninety percent of the time still fails roughly one interaction in ten, and the system gives no advance warning which interaction that will be. For revenue-critical calls to action, mandatory disclosures, link accuracy, and legal-scope boundaries, that residual variance is unacceptable.
The paper's central, practical claim: reliability in regulated conversational AI comes from separating the tasks that require language judgment from the tasks that require exactness, and building the exact ones into the system architecture rather than the prompt.
Compliance can be verified by checking the output string, structure, metadata, or selected resource. Enforce deterministically: validation, substitution, fixed rendering, allowlist checks.
Compliance requires semantic interpretation of meaning, audience, context, or intent. Must remain model-mediated, supported by classifiers, second-pass review, and human escalation.
What the paper covers
Introduction
A public, pre-authentication assistant for a legal technology platform must triage and route visitors without crossing into individualized legal advice, eligibility determinations, or outcome predictions.
Why LLM output is stochastic
Transformer models generate autoregressively and sampling preserves randomness. Instruction tuning shifts the probability distribution but does not convert prompt instructions into hard constraints.
The determinism-naturalness trade-off
Fully templated systems are predictable but brittle. Unconstrained generation is fluent but too variable for a regulated context. The useful question is where determinism belongs.
Mechanizable versus judgment-bound constraints
Constraints that can be verified after generation belong in deterministic code. Constraints that require semantic interpretation must remain model-mediated.
Layered guardrail architecture
Input guardrails, grounding and retrieval, deterministic resolution of canonical resources, output validation with bounded correction, and human oversight and escalation.
Regulated-domain considerations
Unauthorized practice of law, accuracy and hallucination, overconfident generation, privacy under PIPEDA, and adversarial robustness for public endpoints.
Evaluation methodology for stochastic systems
Multi-sample evaluation, paraphrase robustness, dimension-level scoring, adversarial and red-team suites, and production telemetry rather than single-pass testing.
Conclusion
A deliberate division of labor: the model performs language judgment while mechanically checkable facts and rules move beyond the reach of sampling through deterministic resolution, validation, and fallback logic.
The same rigor that governs regulated AI is how Envoy is built.
Draft-only by default. Deterministic action logic the AI cannot reach. You decide what sends. See how the principles in this paper shape the Envoy product.
.png)
.webp)
