AI Agent Evaluation, Guardrails and Human Approval Gates in Production

Why agents need a different kind of testing

Google's ADK documentation explains the core issue: agents are probabilistic, so classic pass or fail unit tests do not fit them well. The same input can produce different but acceptable answers, and an agent can reach a right answer by a wrong path. ADK therefore evaluates two things. The trajectory is the sequence of steps and tool calls compared with an expected sequence. The final response is judged for quality, relevance and correctness.

This idea applies whichever framework you use. For a clinic appointment agent in Chennai, a correct trajectory might be check availability, then book the slot, then send a confirmation by SMS. If the agent books before checking, the final message may look perfectly polite while the calendar now holds a double booking. Testing the path catches that, and it also shows you where the agent wastes steps and tokens.

Building an evaluation set

ADK stores evaluation cases in two formats. Test files hold one session each and suit fast, unit style checks during development. Evalsets hold many, possibly multi turn, sessions and suit integration tests that run less often. Both record the user query, expected tool calls, expected intermediate responses and a reference final response. You can run them from the adk web interface, from pytest, or with the adk eval command in a build pipeline.

Two default metrics show the pattern. tool_trajectory_avg_score checks an exact match of the tool call trajectory, with a default threshold of 1.0. response_match_score measures ROUGE-1 similarity to the reference answer, with a default threshold of 0.8. ADK also offers model judged metrics, and efficiency metrics such as token usage that report values without passing or failing. Whatever tool you use, the habits below are the same.

• Collect 20 to 50 real user requests, including awkward and multilingual ones.
• Write the expected tool path and a reference answer for each.
• Add adversarial cases, such as prompt injection hidden in a document.
• Run the set on every prompt, tool or model change, and track scores over time.

Guardrails on inputs, outputs and tools

Guardrails are checks that run around the agent. In the OpenAI Agents SDK, input guardrails check the user's first input, output guardrails check the final output, and tool guardrails wrap individual function tools. When a check sets tripwire_triggered, the SDK stops the run and raises an exception such as InputGuardrailTripwireTriggered. Input guardrails run in parallel with the agent by default for speed; set run_in_parallel to False if the agent must never start, so no tokens are spent and no tools run.

The example below, adapted from the SDK docs, uses a small checking agent with a structured output type to decide whether a message asks for a refund above a set limit. If it does, the tripwire fires, the support agent stops, and the application routes the customer to a person instead. In a real system you would log the guardrail result so you can review false positives later.

# Illustrative, check the official docs for the current API
from pydantic import BaseModel
from agents import (Agent, GuardrailFunctionOutput, InputGuardrailTripwireTriggered,
                    RunContextWrapper, Runner, TResponseInputItem)
from agents.decorators import input_guardrail

class RefundCheck(BaseModel):
    over_limit: bool
    reasoning: str

checker = Agent(name="Refund check", output_type=RefundCheck,
                instructions="Does the user ask for a refund above 10,000 INR?")

@input_guardrail
async def refund_guardrail(ctx: RunContextWrapper[None], agent: Agent,
                           input: str | list[TResponseInputItem]) -> GuardrailFunctionOutput:
    result = await Runner.run(checker, input, context=ctx.context)
    return GuardrailFunctionOutput(output_info=result.final_output,
                                   tripwire_triggered=result.final_output.over_limit)

support = Agent(name="Support agent", instructions="Help customers with orders.",
                input_guardrails=[refund_guardrail])

async def handle(message: str) -> str:
    try:
        return (await Runner.run(support, message)).final_output
    except InputGuardrailTripwireTriggered:
        return "I am passing this to a colleague who can help."

Human approval gates for risky actions

Some actions should never run without a person saying yes: payments, refunds, emails to clients, production deployments. The OpenAI Agents SDK marks such a tool with needs_approval. When the model calls it, the run pauses and result.interruptions lists the pending calls. You convert the result to a RunState, call approve or reject for each item, and resume with Runner.run. The state can be serialised, so the reviewer can answer hours later from a different process.

The SDK docs add advice that applies to every framework. Keep paused state on the server and send reviewers only what they may see. Authenticate reviewers and prevent the same approval being replayed. Approval does not guarantee a tool runs exactly once after a crash, so use idempotency keys for consequential actions such as bank transfers. Store a version marker with saved state, so a pause that spans a deployment resumes with matching code.

# Illustrative, check the official docs for the current API
from agents import Agent, Runner
from agents.decorators import tool


@tool(needs_approval=True)
async def issue_refund(order_id: str, amount_inr: int) -> str:
    return f"Refund of {amount_inr} INR queued for {order_id}"


agent = Agent(name="Refund assistant", tools=[issue_refund],
              instructions="Use the tools to process refund requests.")


async def run_with_approval(request: str, ask_human) -> str:
    result = await Runner.run(agent, request)
    while result.interruptions:
        state = result.to_state()  # state.to_string() lets you pause for hours
        for item in result.interruptions:
            if ask_human(item.name, item.arguments):
                state.approve(item)
            else:
                state.reject(item)
        result = await Runner.run(agent, state)
    return result.final_output

Permissions, tracing and a production checklist

Other frameworks offer the same building blocks under different names. LangGraph uses interrupt() for approval pauses. The Claude Agent SDK combines allow and deny rules with permission modes and hooks that run before or after tool calls. Microsoft Agent Framework and the OpenAI Agents SDK both include tracing. Whatever you pick, make every run observable: log the prompt version, model, each tool call with arguments and the final answer, with personal data masked. XDQ Labs teams review a sample of traces each week alongside the evaluation scores.

• Separate read and write tools; gate every write tool that moves money or contacts people.
• Set step, time and spending limits per run.
• Treat documents, web pages and tool outputs as untrusted input.
• Version prompts and tools, and rerun the evaluation set on every change.
• Give each agent the narrowest credentials it needs.

Check your understanding