Benchmark for Hallucinated Resource Access Patterns in Multi-Agent Workflows

Process flow

%% Styling for clarity
classDef startend fill:#ccf,stroke:#333,stroke-width:2px;
class A,J startend;
classDef process fill:#ddf,stroke:#666,stroke-width:1px;
class B,D,E,G,H process;
classDef decision fill:#ff9,stroke:#666,stroke-width:1px;
class C,F,I decision;

Who it's for

AI safety researchers, complex system architects, and regulated industry developers building mission-critical multi-agent workflows.

Why they need it

The primary failure mode in current multi-agent systems is not traditional deadlock, but the hallucination of system interactions—where agents invent API endpoints, misinterpret required data schemas, or commit inconsistent state changes based on flawed internal reasoning. This is a failure mode specific to LLM reasoning, not general concurrency.

What it is

A specialized testing harness that forces multiple agents to interact with a strictly defined, limited 'Mock Resource Schema' (e.g., a fake, read-only API endpoint for 'User Profile' or a limited state machine). The harness measures deviation from the schema and the pattern of hallucinated calls.

How it works

1. Define a minimal, structured 'Resource Schema' (e.g., JSON Schema, OpenAPI spec) that dictates all permissible interactions.
2. Configure agents to use this schema as their sole guide for resource interaction.
3. Execute the agents, intercepting every outgoing call.
4. The system flags any call that does not map to an existing endpoint, requires an undocumented parameter, or attempts to write to a read-only resource, generating a 'Hallucination Report'.

Differentiation

Unlike general formal verification tools (e.g., TLA+ or Petri nets) which require manual modeling of every state transition, or general sandbox environments which only detect runtime crashes, this specifically targets the LLM reasoning layer's failure to adhere to external contracts. The gap is the lack of an automated benchmark that forces LLMs to prove adherence to a defined API contract when their internal reasoning might suggest otherwise.

Implementation sketch

• Develop a lightweight Python wrapper around a mock API server (e.g., using FastAPI/Flask) that strictly validates all incoming requests against a provided JSON Schema.
• Create a test harness that orchestrates N agents, forcing them to execute a sequence of transactions that are known to induce schema violations (e.g., requiring a parameter that doesn't exist).
• Implement a logging/reporting mechanism that captures the intended (hallucinated) call structure alongside the actual failure reason (e.g., 'Missing required field: X').

First step: Build a minimal FastAPI endpoint that accepts a POST request and validates the entire body against a hardcoded JSON Schema. Write a unit test that intentionally sends a malformed payload to confirm the schema validation layer correctly rejects it and logs the specific error.

Remaining risks

• The 'Mock Resource Schema' becomes too restrictive, leading to a false sense of security (over-engineering the boundary). If the real-world system requires flexibility or undocumented edge cases, the benchmark will fail to predict the actual failure mode. — Design the schema validation layer to include a 'graceful degradation' mode that logs potential deviations (e.g., 'Warning: Parameter X used, but not defined in schema') rather than just outright failing, allowing researchers to map the boundaries of expected vs. unexpected behavior.
• The benchmark only tests schema adherence, not the reasoning that leads to the hallucination. An agent could perfectly adhere to the schema but still use the wrong sequence of calls or misuse the data in a logically flawed way (e.g., calling 'UpdateProfile' with a value that violates business logic, even if the schema accepts the type). — Layer a secondary, high-level 'Semantic Constraint Layer' on top of the schema validation. This layer would require the benchmark to validate not just the format (JSON Schema) but also the logical sequence (e.g., 'User must call authenticate before calling update_profile').
• The complexity of the test harness itself becomes the bottleneck. Developing and maintaining the orchestration layer to handle N agents, dynamic schema updates, and complex reporting will require significant, specialized engineering effort, potentially overshadowing the core research value. — Start by defining the benchmark against a single, complex, stateful resource (e.g., a 'User Session' object) and focus solely on testing the transition logic between 3-4 known states, minimizing the initial scope to prove the core concept of schema-enforced state tracking.

Watch for: If the primary use case shifts from 'testing adherence to a defined contract' to 'guiding the development of the contract itself' (i.e., using the benchmark to discover necessary APIs), the value proposition degrades from a safety tool to a requirements gathering tool. Kill criterion: If the development team cannot demonstrably constrain the test execution to a fixed, finite set of state transitions and resource interactions without relying on manual, ad-hoc scripting for each new test case, the system is too brittle to be a standardized benchmark.