Artificial Intelligence · Systems Architecture

Designing Bounded Multi-Agent Systems for Enterprise Operations

1. Why unconstrained autonomous agents fail in production

Over the past three years, thousands of organisations have built prototype AI agents that appear impressive in a five-minute demonstration. A single large language model (LLM) is given a system prompt, a list of twenty tools (database queries, email senders, CRM mutators, web scrapers), and a `while` loop that continues until the model decides its task is complete. When tested on clean, happy-path inputs, the prototype succeeds.

When moved into live operations, however, unconstrained single-loop agents exhibit four recurring failure modes that make them unacceptable for regulated enterprise workloads:

• Context window pollution and goal drift: As intermediate tool outputs (such as large JSON payloads or verbose error stack traces) accumulate in the conversation history, the model’s attention degrades. By step eight or nine, the agent frequently forgets constraints stated in the original system prompt.
• Non-terminating retry storms: When an downstream API returns an unexpected `422 Unprocessable Entity` or a malformed record, an unconstrained agent will often guess variations of the same tool call in a loop, burning tens of thousands of tokens and occasionally triggering rate limits across production services.
• Privilege escalation via ambient authority: If a single agent holds credentials to both read untrusted external text (such as an incoming support email or a uploaded PDF contract) and execute state-changing tools (such as issuing a refund or updating an employee record), any prompt injection embedded in the untrusted document can hijack the agent’s control flow.
• Unverifiable intermediate reasoning: When a monolithic agent produces an incorrect final action after fifteen internal steps, operations teams cannot easily isolate whether the fault lay in document retrieval, data extraction, policy interpretation, or tool parameter formatting.

2. Directed state-machine orchestration vs. free-form chat loops

At TICNES AI, our core architectural principle is that control flow belongs in deterministic application code, not inside a probabilistic language model. Rather than allowing an LLM to decide which arbitrary step to take next across an entire business process, we model workflows as explicit directed graphs (using frameworks such as LangGraph or custom TypeScript/Python state machines).

Each node in the graph represents a specialised worker agent or a deterministic validation function with a single, narrowly defined responsibility:

By decomposing the workflow into discrete nodes, each worker agent operates with a minimal, clean context window containing only the exact inputs required for its step. Furthermore, transitions between nodes are enforced by deterministic edge conditions written in standard code. If the Guardrail Verifier detects that a cited legal article does not exist in the retrieved source passages, the state machine routes execution back to the Domain Analyst at most twice before escalating the ticket to a human operator.

3. Bounded tool contracts and schema validation

Giving an agent raw SQL access or generic HTTP client capabilities is an invitation to production outages. In our architectures, every tool exposed to an agent must satisfy three engineering constraints:

1. Narrow, domain-specific parameterization: Instead of a generic `run_sql(query: string)` tool, we expose strongly typed functions such as `lookup_contract_clauses(contract_id: UUID, clause_category: ClauseEnum)`. Parameters are validated against strict JSON Schemas (via Pydantic or Zod) before the underlying function ever executes.
2. Read/Write separation at the identity layer: Agents that process untrusted user input run under service identities that possess zero write permissions at the database and API gateway level. Even if a novel prompt-injection technique bypasses input filters, the execution runtime physically cannot mutate records.
3. Hard step and token budgets: Every workflow invocation is initialized with a strict execution envelope — for example, a maximum of `4` LLM calls, `12,000` total input/output tokens, and a wall-clock timeout of `15 seconds`. If any budget is exceeded, execution halts cleanly and logs an alert.

// Example: Deterministic state-transition guard in a TICNES AI workflow
export function evaluateVerifierTransition(state: WorkflowState): NextNode {
  if (state.stepCount >= state.maxAllowedSteps) {
    return "escalate_to_human_queue";
  }
  if (!state.verificationResult.citationsValid) {
    return state.retryCount < 2 ? "regenerate_with_citation_feedback" : "escalate_to_human_queue";
  }
  if (state.requiresConsequentialAction) {
    return "await_named_human_authorisation";
  }
  return "deliver_verified_response";
}

4. Designing human-in-the-loop authorisation gates

Across every system TICNES AI has deployed since 2015, our standing operational rule is that automation prepares; humans decide. However, poorly designed human-in-the-loop systems suffer from “automation bias” — if a human reviewer is presented with a plausible-looking two-page AI summary and an “Approve” button, they will eventually click “Approve” without checking the underlying facts.

To make human oversight genuine rather than ceremonial, we design our review interfaces around side-by-side provenance:

• Every factual claim or legal obligation in the agent’s draft is hyperlinked directly to the exact highlighted paragraph in the source document.
• Uncertainty flags (such as ambiguous dates, missing contractual annexes, or low retrieval similarity scores) are surfaced at the top of the review card in high-contrast callouts.
• Consequential actions (such as disbursing a scholarship award, filing a compliance notice, or modifying an enterprise schedule) require the authenticated human reviewer to confirm specific checklist items, recording their user ID, timestamp, and diff in an append-only audit log.

5. Pre-deployment evaluation harnesses and regression testing

No multi-agent system reaches production at TICNES AI without passing an automated evaluation harness. Before writing prompt templates or selecting model weights, our engineers collaborate with the client’s domain experts to curate a golden evaluation dataset of `200` to `500` real-world scenarios — explicitly weighted toward edge cases, adversarial inputs, incomplete records, and bilingual (Portuguese/English) queries.

Every pull request that modifies a prompt, chunking strategy, tool schema, or model version runs against this evaluation suite, measuring:

• Citation faithfulness: Does every statement in the output strictly follow from the retrieved context without extrinsic hallucination?
• Tool selection precision: Did the agent invoke the minimal necessary set of tools with valid arguments?
• Latency and cost p95: Does the end-to-end execution remain within our target latency envelope (typically under `240ms` for classification steps and under `3.5s` for full retrieval-augmented synthesis)?

When a model upgrade or prompt edit causes a regression on any golden test case, the deployment pipeline blocks automatically. This discipline transforms AI engineering from guesswork into repeatable systems engineering.