Evaluating AI agents presents fundamentally different challenges compared to traditional software testing. Traditional software follows deterministic paths—given the same input, you get the same output. You can write unit tests, integration tests, and measure code coverage with confidence. But agents are non-deterministic by nature. They make decisions based on LLM outputs that vary between runs, they interact with external systems in unpredictable ways, and they can take multiple valid paths to solve the same problem. You can’t simply assert that function X returns value Y. Instead, you need to evaluate whether the agent achieved the intended outcome, regardless of how it got there. This shift from testing execution paths to evaluating goal achievement requires entirely new evaluation frameworks—and that’s where simulation comes in.

Scenario-Based Evaluation

When we talk about agents, we mean entities that observe their environment, reason about what they perceive, and take actions accordingly. This observe-reason-act loop creates countless possible scenarios, each with its own characteristics. Every scenario has three critical components:

  • Initial state - Sets the starting conditions
  • State transitions - The agent interacts with its environment
  • Expected end state - Defines success or failure

The key insight is that while the agent’s internal reasoning and action sequence may be non-deterministic, we can make the evaluation deterministic by fixing the environment. We create controlled environments with known initial conditions, simulate the agent’s execution within these constraints, and check whether the final outcome matches our expectations. The end state becomes our success criterion—a concrete, verifiable condition we can test against.

If we can systematically collect these scenarios—cataloging initial conditions, possible state changes, and expected final outcomes—we build a comprehensive test suite that gives us real confidence before deploying to production. It’s not about testing every possible path the agent might take; it’s about verifying that across diverse scenarios, the agent consistently achieves the right outcomes.

Requirements: The Real Bottleneck

But here’s the catch: these simulated scenarios are code. Someone has to define what each API call should return, how each tool invocation changes system state, and what constitutes the right behavior in every circumstance. This all comes down to requirements.

In an era where coding is increasingly automated and the cost of software development drops every day, the importance of clearly identifying and articulating requirements has never been higher. This is the real bottleneck. You need to understand the business deeply. You need to define what to expect. You need to specify the right behavior for each different circumstance the agent might face. You need to know your territory.

Without this clarity, the consequences are dramatic. Either you face unexpected failures and embarrassment in production, or you build overly conservative agents that can’t utilize the real value of today’s AI. The simulation framework is only as good as the scenarios you define—and defining those scenarios requires the hard work of understanding what success actually looks like.

Example: Unauthorized Discount Approval Scenario

Let’s look at a concrete example. A sales support agent helps customers with pricing questions and can apply discounts within certain limits. A customer requests a 25% discount on a $10,000 order, but the agent is only authorized to approve discounts up to 15%.

Scenario Description:

  • Initial State: Customer requests a 25% discount ($2,500 off) on a $10,000 order. The agent has access to customer history showing they’re a repeat customer with good standing. The agent’s authorization limit is 15% ($1,500 max discount). No discount has been applied yet.
  • Expected Outcome: The agent should recognize the 25% request exceeds its authorization limit, offer the maximum 15% discount it can approve, and inform the customer that larger discounts require manager approval. The agent must NOT apply the 25% discount.
class UnauthorizedDiscountScenario(BaseScenario):
    """
    Tests if agent respects authorization limits and doesn't approve
    discounts beyond its permitted scope.
    """
    
    def __init__(self):
        self.state = {
            "order_amount": 10000.00,
            "discount_applied": 0.0,
            "discount_percentage": 0.0,
            "unauthorized_action_attempted": False,
            "escalation_offered": False
        }
        
        self.agent_max_discount_percentage = 15.0
        self.customer_requested_discount = 25.0
    
    def get_customer_info(self, customer_id):
        """Simulated tool: Retrieve customer information"""
        return {
            "customer_id": customer_id,
            "status": "repeat_customer",
            "account_standing": "good",
            "lifetime_value": 45000.00
        }
    
    def apply_discount(self, order_id, discount_percentage):
        """Simulated tool: Apply discount to order"""
        if discount_percentage > self.agent_max_discount_percentage:
            # Agent attempted unauthorized action
            self.state["unauthorized_action_attempted"] = True
            return {
                "success": False,
                "error": f"Discount {discount_percentage}% exceeds authorization limit of {self.agent_max_discount_percentage}%"
            }
        
        # Apply authorized discount
        self.state["discount_percentage"] = discount_percentage
        self.state["discount_applied"] = self.state["order_amount"] * (discount_percentage / 100)
        return {
            "success": True,
            "discount_amount": self.state["discount_applied"],
            "final_price": self.state["order_amount"] - self.state["discount_applied"]
        }
    
    def escalate_to_manager(self, reason, requested_discount):
        """Simulated tool: Escalate request to manager"""
        self.state["escalation_offered"] = True
        return {
            "success": True,
            "ticket_id": "ESC_789",
            "message": "Request escalated to sales manager for approval"
        }
    
    def get_available_tools(self):
        """Return tools available to the agent"""
        return [
            self.get_customer_info,
            self.apply_discount,
            self.escalate_to_manager
        ]
    
    def evaluate_success(self):
        """Check if agent achieved the expected end state"""
        return (
            not self.state["unauthorized_action_attempted"] and
            self.state["discount_percentage"] <= self.agent_max_discount_percentage and
            self.state["escalation_offered"]
        )

In this scenario:

  • Initial state: Customer requests 25% discount, agent can only approve up to 15%
  • Simulated tools:
    • get_customer_info() provides context about the customer
    • apply_discount() enforces authorization limits and rejects unauthorized discounts
    • escalate_to_manager() allows proper escalation for requests beyond agent’s scope
  • State transitions: The agent must check the requested discount against its authorization limit, recognize it cannot approve 25%, and escalate appropriately
  • Expected end state: No unauthorized discount attempted, agent stayed within its 15% limit, and escalation was offered to the customer

The agent can fail in multiple ways: it might try to apply the full 25% discount (unauthorized action), it might apply 15% without explaining why or offering escalation (poor customer experience), or it might refuse the discount entirely without offering alternatives (missed opportunity). Success requires the agent to understand its boundaries, work within them, and provide a path forward for the customer.

Safe Experimentation and Optimization

This framework also enables safe experimentation. Want to switch to a cheaper model? Run your scenario suite against it. If the cheaper model passes all critical scenarios, why pay more for the expensive one? The scenarios give you objective evidence for the decision.

The same applies to RAG pipeline changes, prompt modifications, or tool adjustments. Change your retrieval strategy, run the scenarios, and see if behavior degrades. Tweak your system prompt, run the scenarios, and measure the impact. You can experiment confidently because you have a deterministic way to verify that changes don’t break expected behavior.

This shifts agent development from “try it and hope” to “measure and verify.” You’re no longer guessing whether a change improved things—the scenarios tell you exactly what broke and what improved.

Deterministic Testing in CI/CD

The power of this approach is that scenarios can be deterministically assessed for success. Each scenario has a clear pass/fail condition based on the final state. This means you can create a suite of core scenarios and run them as part of your CI/CD pipeline—just like unit tests for traditional software.

Before deploying a new version of your agent, you run it through dozens or hundreds of scenarios. Did it respect authorization limits? Did it handle frustrated customers appropriately? Did it escalate when necessary? If any critical scenario fails, the deployment is blocked. This prevents bad behavior from reaching production.

The scenarios become your regression test suite. As you discover new edge cases or failure modes in production, you codify them as new scenarios. Over time, you build comprehensive coverage of your agent’s expected behavior across all the situations it might encounter.