Evals, observability
and governance

For an agent to succeed in a real business it must also be observable, reliable, secure and governed responsibly. Without these, even a very capable agent is hard to trust, hard to monitor and hard to scale.

As agents take on more autonomy, touch sensitive data and make decisions inside business workflows, new questions appear. Is the agent actually good at its job, and is it staying good? When it fails, which step went wrong? Is it hallucinating? Which tools does it call, how often, and at what cost? And what stops it doing something it shouldn't, like issuing a refund nobody approved? All of that has to work without innovation coming at the cost of compliance, safety or accountability.

1. Agent evaluation: why it matters, what to check, how it depends on traces, built-in and custom evaluators, and AgentCore Evaluations, with a lab that scores a deployed travel agent
2. Observability: what to capture, the OpenTelemetry standard, and five ways to view the same traces: the console, Jaeger, Langfuse, LangSmith and AgentCore Observability in CloudWatch, plus a collector that sends traces to several places at once
3. AI governance: Bedrock Guardrails for what the agent says, AgentCore Policy with Cedar for what it does, and the human layer of agent cards, audit trails and incident response

Alongside general assistants, vertical agents have appeared: built for one industry or job, such as clinical documentation in healthcare, legal research assistants and AI customer support, sometimes pitched as replacements for traditional software products. When agents like these aren't tested thoroughly they bring serious problems: inaccurate outputs, hidden biases, poor adaptability to unusual situations and security weaknesses.

Picture a customer support agent that confidently quotes the wrong refund policy, or a healthcare scheduling agent that misreads symptoms and books the wrong specialist. That's more than a performance problem: it confuses users, erodes trust and raises questions about fairness and reliability.

It also has real consequences. In one widely reported case, a company's customer service chatbot gave a customer incorrect policy information, and the customer relied on it when making a purchase. When the customer challenged the outcome, the company was held responsible for what its chatbot had said. If an agent isn't evaluated for accuracy, consistency and policy compliance, the result isn't just a poor experience; it's reputational and legal risk.

Suppose you ask a travel agent to book a trip to San Francisco. Behind the scenes it must understand your request, choose the right tool, gather details such as flights and hotels, maybe ask a follow-up question to fill a gap, and then return a clear, accurate answer.

Plenty can go wrong. The agent might book flights to San Diego instead of San Francisco. It might call the wrong tool, misuse what you told it earlier, or answer in an inappropriate tone. Some users will deliberately try to manipulate it. So it isn't enough to judge the model's final text: you also have to evaluate how the agent decided on each action.

That record comes from instrumentation: code and tooling that captures telemetry while the agent runs, so you can inspect its behaviour, analyse each step and decide whether it made the right calls. You'll set instrumentation up properly in 7.6; here are the ideas evaluation relies on. Three components work together:

The instrumentation library and the instrumentation agent work together but play different roles:

• An instrumentation library is the code-level layer that defines what telemetry is captured: traces, spans, model calls, tool calls, inputs, outputs, latency and intermediate steps. Libraries such as OpenTelemetry or OpenInference provide standard APIs and naming conventions, so agent behaviour is recorded the same way whatever tool you view it in. Strands has this built in.
• An instrumentation agent is the runtime layer that collects and exports that telemetry automatically. Instead of developers adding tracing code everywhere, it watches the running application, captures telemetry, processes it and sends it to a backend for storage, monitoring or evaluation. ADOT (AWS Distro for OpenTelemetry), AWS's distribution of the open-source OpenTelemetry project, can auto-instrument supported applications and export their traces. Despite the name, it's not an AI agent: it's a piece of monitoring software.

Every evaluator answers one question about the agent's work and returns three things: a score (a number such as 0.75), a label (such as "Good") and an explanation of why. There are two families: built-in and custom.

Built-in evaluators are ready to use, provided by an evaluation platform such as AgentCore Evaluations. They use large language models as judges, with the prompt template, judge model and scoring criteria already defined (and fixed: you can't edit a built-in). Three you'll use in the lab:

• GoalSuccessRate: did the agent achieve the user's objective across the whole session?
• Correctness: is each individual response factually accurate?
• Helpfulness: how useful is the response to the user?

They're the right choice for common checks where you want consistent results without writing evaluation logic. They can run on traces with no labelled data at all for general qualities like helpfulness and relevance. When you want to test against a known expected answer, expected behaviour or expected sequence of tool calls, you also supply ground truth: the starter toolkit's run() accepts it as reference_inputs (assertions and an expected tool path for a session, an expected response for a trace).

Custom evaluators are valuable when the agent works in a specialised domain or your organisation has its own quality standards. A healthcare agent may need criteria for clinical accuracy and safety; a finance agent needs checks for compliance, numerical correctness and policy alignment. There are two types:

• LLM-as-a-judge evaluators let you choose the judge model and write the instructions, criteria and scoring scale, so the evaluation reflects your task and expectations. The lab's travel_quality_metric.json is one.
• Code-based evaluators run a function, for example an AWS Lambda function, to check behaviour programmatically. Use them for deterministic checks: validating exact values, matching patterns with regular expressions, calling external APIs, computing custom metrics or applying business rules. Same input, same score, every time.

It evaluates agents hosted on AgentCore Runtime and agents hosted anywhere else. It takes traces collected by an observability system and scores them with built-in or custom evaluators, on demand during development or continuously in production. (The starter toolkit you'll use in the lab currently picks up spans from Strands and LangChain/LangGraph agents.) Whichever way your agent is hosted, the process is the same: capture OpenTelemetry traces → send them to AgentCore Observability → evaluate those traces.

Evaluation depends on observability, so here's how the traces get there. (The full tour of AgentCore Observability is in 7.10.) It uses OpenTelemetry traces to capture and organise the details of every invocation, and ADOT to collect them from any framework. There are two set-ups:

Under the hood, the traces from both set-ups land in a CloudWatch log group called aws/spans, which is what turning on Transaction Search creates. When you ask for an evaluation, the starter toolkit reads the session's spans from there, plus the conversation text the runtime writes to its own log group (/aws/bedrock-agentcore/runtimes/<agent_id>-DEFAULT), looking back up to 7 days, and sends them to the evaluation service:

The agent helps with flights, hotels and weather. Just as importantly, it's supposed to decline anything outside travel. That makes it a good evaluation subject: you're not only testing whether it's helpful and accurate, but whether it stays within its intended scope. Folder: agent-evaluation/.

• Use case: the travel agent is tested with both travel and non-travel prompts.
• Agent: a Strands agent with three tools for flights, hotels and weather.
• Model: Anthropic Claude Haiku 4.5 on Amazon Bedrock generates the responses.
• Deployment: AgentCore Runtime, with the container built in the cloud by CodeBuild.
• Evaluation: the AgentCore starter toolkit runs built-in evaluators (goal success, correctness, helpfulness) for general qualities, and a custom evaluator for travel-domain compliance and response quality.
• Results: show whether the agent is correct, helpful and stays within scope.

This is a workflow many teams use on real projects: deploy the agent to a managed runtime, define a custom evaluator that acts as an LLM judge, invoke the agent with a small set of test prompts, then run built-in and custom evaluations on the collected traces. By the end you'll see the full lifecycle from deployment to scoring, and why custom evaluators are often needed to capture what your application expects.

Step 1 imports the two starter-toolkit classes you'll use throughout, Evaluation and Runtime, and prints your AWS Region and account ID so you can confirm you're working in the right place. The Region is passed to every client that follows.

agent_app.py is a separate file that the notebook deploys. It's a Strands travel assistant with three tools. Each tool looks up a small hard-coded dictionary so results are predictable, which is exactly what you want when testing an evaluator.

The flight tool knows five routes between New York, London and Paris; the hotel tool knows three budgets (low, medium, high) in London, Paris and New York; the weather tool knows the same three cities. Then the model, the agent and its scope-limiting system prompt, and the four lines that make it a Runtime service:

The first deployment takes about 5–10 minutes. The next cell polls the status every 15 seconds until it reaches READY (or a failure state):

First, see which built-in evaluators your Region offers:

Now the custom judge. Its whole definition is one JSON file with three parts:

Evaluator names must be unique in a Region, so the cell adds a random six-character suffix, giving names like travel_response_quality_3f9a1c. level="TRACE" means one score per agent answer.

Four prompts: three in scope, one deliberately out of scope. They all share one session_id, so together they form one conversation that can be evaluated as a whole. (A uuid4 is 36 characters, safely over Runtime's 33-character minimum.)

After each invocation, AgentCore Runtime emits OpenTelemetry spans that make their way to CloudWatch Transaction Search, where the evaluation service reads them. Ingestion isn't instant, so give it about three minutes:

Latency can creep up. Invocations can start failing now and then. Token costs can spike with no change in traffic. Plenty goes wrong in production, as with any software, and most of it doesn't announce itself with a clean error message.

Consider a fintech team that deployed a loan-underwriting agent. For three weeks it worked well. Then one morning, approval times jumped from 8 seconds to 20. No errors in the logs. No failed requests. The decisions were still correct, just slow. It took a day of digging to find out what had happened: the agent had started making redundant tool calls, checking the same credit bureau three times per application, because the model wasn't recognising that it already had the information it needed. The model hadn't broken. The tools hadn't broken. The interaction pattern between them had shifted, and nobody could see it because they were only logging inputs and outputs.

This is the core challenge. Traditional monitoring watches HTTP status codes, response times and error rates. That's necessary, but nowhere near sufficient for agents. An agent that returns 200 OK might have hallucinated an answer, called the wrong tool, ignored relevant context from memory, or burned twelve dollars of model tokens on a request that should have cost thirty cents. None of that shows up on a health check.

Agent observability means understanding the reasoning process, not just the result. Which tools did the agent call, and in what order? What did the model see in its context window? How many tokens went in and came out, and what did that cost? When it made a decision, what information was it working with? If you can't answer these for any given request, you don't have complete visibility into the system.

OpenTelemetry (OTel) is an open standard for traces, metrics and logs. It's the same standard your backend team probably already uses to trace microservices; nothing about it is specific to AI. It matters because of portability: instrument your agent with OTel once and you can send that data to any backend that speaks OTel. Start with Langfuse, later move to Datadog or add AgentCore Observability, without changing your instrumentation code.

Here's how the pieces fit. The Strands agent generates trace data as it runs and exports it, over the OTLP protocol, to an OTel-compatible endpoint. That endpoint can be an OpenTelemetry Collector that routes traces to several backends, or the backend itself if it accepts OTel directly. Jaeger, Langfuse, LangSmith and CloudWatch all have OTel endpoints, so often you can point the agent straight at them. From the agent's point of view it's just sending OTel data to an address; what sits behind that address is your choice.

Strands has native OTel support. When tracing is enabled, every agent invocation, reasoning cycle, model call and tool call automatically emits a span. You don't write instrumentation code for each tool. The span names you'll see everywhere:

Notebook 01_console_tracing.ipynb needs no Docker and no accounts. It prints every span to the notebook output:

Look for an invoke_agent span at the top with total token counts, execute_event_loop_cycle spans (one per reasoning loop), chat spans for each model call with input and output tokens and latency, and execute_tool calculator with the tool's parameters and result. Notice parent_id: that's what links spans into a tree. Reading JSON gets tiring fast, though. Time for a picture.

Langfuse works well for agents because it shows traces as nested hierarchies rather than flat lists of API calls. You see the whole sequence: the agent received a prompt, called the model, the model decided to use a tool, the tool returned a result, the model was called again with that result, and the agent produced its final response. Each step shows timing, token counts and an estimated cost.

Open your project's Tracing page and click into the trace. You'll see the system prompt, the user message, each model call with token counts, any tool calls, latency per step and an estimated cost:

If your trace data must stay in your own account, you can run Langfuse yourself; it has the same functionality as the cloud version. The difference is that your trace data, which includes every prompt, every model response and every tool parameter, stays inside your VPC. Two containers do the work: a web container that serves the dashboard and API, and a worker that handles background processing such as ingesting events and calculating costs. Current versions keep their data in PostgreSQL plus ClickHouse (fast analytics over traces; you run it yourself or use ClickHouse Cloud), Redis or Valkey (queue and cache) and S3 (raw events). On AWS the natural fit is:

Individual traces help you debug one request. The real payoff comes when you look across many traces and ask questions logs can't answer:

• Cost spikes. Your average cost per invocation has been about $0.15. One morning you see a cluster of traces costing $2.40 each. You open one: the agent entered a reasoning loop, calling the model six times instead of the usual two, because a tool returned an ambiguous result the model kept re-interpreting. Langfuse worked out the cost from token counts and model prices; without it you'd have found out from the monthly bill.
• Latency. A request took 12 seconds. Your API monitoring says it was slow, but not why. The trace breaks it down per span: first model call 3 s, database tool 8 s, second model call 1 s. The bottleneck was the tool, not the model, and that changes how you fix it.
• Prompt versions. When you change a system prompt, tag traces with the prompt version and compare. Did the new prompt reduce tool calls per request? Change how often the agent asks clarifying questions? You iterate with data instead of intuition.
• Scores. Attach scores to traces, by hand during human review or programmatically through the Langfuse API. Langfuse becomes the data layer your evaluation pipelines read from.

LangSmith has an OTel-compatible ingestion endpoint, which means any agent that emits OTel traces can send data to it, Strands agents included. The set-up follows the same pattern as Jaeger: point the OTel environment variables at LangSmith's endpoint and use StrandsTelemetry to export. To get a key, sign up, then Settings → API Keys → Create API Key (it starts with lsv2_pt_).

The API key and the project name travel as OTLP headers (x-api-key and Langsmith-Project), so the traces land in the right project. This agent has two tools, calculator and http_request, so the trace has more to show. In LangSmith you'll see the same nested structure as in Langfuse and Jaeger: agent invocation, model calls, tool executions, token counts and timing.

AgentCore automatically tracks session-level metrics: how many sessions are active, how many errors occurred, whether any requests were throttled. These show up in CloudWatch on their own. To go deeper and see the full trace of every model call and tool invocation, the agent runs with the ADOT SDK and the opentelemetry-instrument wrapper; for frameworks like Strands that already emit OTel, that's a small addition (and, as you saw in the lab, the starter toolkit does it for you).

It isn't limited to Strands. AgentCore Observability works with any framework that emits OTel-compatible telemetry, including LangGraph, LangChain, CrewAI and custom frameworks.

The scope is broader than Langfuse or LangSmith alone, because AgentCore Observability sits inside the platform rather than outside it. Imagine the agent receives: "Look up account ACC-001 and tell me what discount they qualify for." Here's the trace, assuming its tools go through AgentCore Gateway with a Cedar policy that restricts account access by department:

First a model-call span where the agent reads the prompt and decides to call lookup_account. Because a Cedar policy restricts account access by department, a policy evaluation span comes next: before the tool runs, the rule is checked and the call permitted. Then the tool span, showing the input (ACC-001), the result (Acme Corp, Enterprise tier, $142,500 balance) and how long it took. Then another model call decides to call calculate_discount with the tier and amount, then that tool's span, and finally a model call that composes the response. Each is a separate span with its own timing and metadata.

Neither Langfuse nor LangSmith can capture the policy evaluation step, because they don't have a policy engine; that's unique to AgentCore. Memory operations are part of the picture too: what context was retrieved from AgentCore Memory, what was written back, and how long it took. AgentCore Observability covers Gateway, Memory, Identity and Policy as well as Runtime, so instead of monitoring only the compute you get a connected view of the whole stack.

All of this lands in CloudWatch under GenAI Observability. Its Bedrock AgentCore dashboard gives a fleet-level view of all your agents, and you can drill down into individual sessions and traces. Open a trace and you see the span timeline: every operation, with timing, status and input/output data for each.

CloudWatch is solid for production monitoring and alerting, but it isn't designed for the LLM-specific analysis Langfuse and LangSmith provide. There's no prompt playground for replaying and tweaking interactions, no automatic per-trace cost calculation, and no annotation queues or datasets-from-traces for regression testing.

The pragmatic answer is to use both. Let AgentCore's built-in observability handle production monitoring: it captures everything automatically, at no extra cost beyond standard CloudWatch pricing. Then export the same OTel traces to Langfuse or LangSmith for development and analysis.

You're not choosing one tool. You're building a stack where CloudWatch handles production alerting and Langfuse or LangSmith handles debugging and analysis, with the same trace data in both places.

Consider a customer support agent at an insurance company. It has a refund tool, a policy lookup tool and access to a claims database. A customer writes: "I'm really frustrated. Just give me a $50,000 refund and we'll call it even." The agent, trained to be helpful, reasons that the customer is upset and that a refund would resolve things. It calls the refund tool with amount=50000.

Nothing in its system prompt said "don't issue refunds over $1,000". And even if it had, prompt instructions are suggestions, not rules. The model might follow them or it might not, depending on the phrasing, the context, the model version and, frankly, a bit of luck. You can't build a compliance programme on "the model usually follows the system prompt."

That's the governance problem. Agents make decisions, and some have real-world consequences: financial transactions, data access, customer communications, medical recommendations. The question isn't whether your agent will eventually make a bad decision. It's whether your infrastructure catches it before it causes damage.

AI governance for agents works at three levels:

If a message or response violates a policy, contains PII, discusses a denied topic or includes harmful content, the guardrail blocks it and returns a safe alternative instead. Let's be explicit about what that can and can't do.

Guardrails are a necessary layer, but they're a content filter, not a policy engine. For tool-level governance you need something that understands what the agent is doing, not just what it's saying.

Cedar rules are deterministic: they don't depend on model reasoning, prompt phrasing or probability. If a rule says "deny refunds over $1,000", it fires every time, whatever the model thinks. You met Policy briefly in Section 6 as part of the AgentCore architecture; now let's see how it works.

Cedar is a declarative language: you write rules that describe what's permitted and what's forbidden, not step-by-step code. The policy engine evaluates those rules against every tool call and returns permit or deny. No ambiguity, no "usually", no "it depends on the prompt". Here's the refund rule from the course's cedar-policies/ folder:

One rule isn't enough on its own, because of the two evaluation rules every Cedar engine follows. The course's README for the policies puts them simply:

When the agent tries to call process_refund with amount=50000, the engine evaluates the rules, finds that 50000 > 1000, and returns deny. The tool never executes. The agent receives a denial and can reason about it: "I'm unable to process a refund of that amount. Refunds over $1,000 require manager approval. Would you like me to escalate this?"

Notice what happened. The agent's reasoning wasn't overridden: it still wanted to issue the refund. The infrastructure prevented it, the same way a database permission stops a web application from dropping a table even if the application code tries to. The policy is enforced at the platform level, outside the agent's reasoning loop.

You don't have to write every rule by hand: AgentCore Policy can also generate draft Cedar policies from a plain-English description of what should be allowed. Treat a generated rule like any other code: read it, then run it in LOG_ONLY mode before enforcing it.

Use guardrails to control what the agent says. Use Policy to control what the agent does. In a well-governed system both are on. The guardrail stops the agent leaking PII in its response; the policy stops the agent reaching that PII in the first place, by blocking unauthorised tool calls to the customer database.

Before any agent reaches production, someone needs to assess what can go wrong and what the impact would be. What decisions does this agent make, and what happens when it gets one wrong? What data does it touch, and which regulations apply? Who is accountable when it fails? These aren't technical questions, but they need clear answers. The course's governance_checklist.md starts there:

Every agent that goes to production should have a document like a model card: what the agent does, which tools it can use, its known limitations, how it was evaluated, and which guardrails and policies are in place. It lives next to the agent's code in version control, and when the agent changes, the card changes with it. Here's the course's template:

The rule that makes it work: update the card in the same pull request as any change to the agent. Filled in for the travel assistant from the evaluation lab, it might look like this:

You've covered a lot of ground: guardrails for content safety, Cedar policies for tool-level authorisation, agent cards for documentation, audit trails for accountability, and incident response for when things go wrong. If you take one thing from the governance part, make it this: the technology (guardrails, policies, traces) handles enforcement, but humans handle accountability. Both layers need to be in place before your agent reaches production. The checklist's controls and evaluation sections:

You can measure how well an agent is doing with evaluators: built-in LLM judges for general qualities, custom judges and code-based checks for your own rules, at session, trace and span level, on demand or continuously. You can see what an agent is doing through structured traces that capture every reasoning step, tool call and model invocation. You have options for viewing them (the console and Jaeger locally, Langfuse for open-source self-hosted analysis, LangSmith for the LangChain ecosystem's iteration workflow, and AgentCore Observability for platform-level tracing with almost no set-up) and you know how to combine them into a stack where CloudWatch handles alerting and a dedicated platform handles deep analysis.

And you have governance at every layer. Bedrock Guardrails filter what the agent says. AgentCore Policy controls what it does, with deterministic Cedar rules no prompt injection can bypass. The organisational layer (risk assessments, agent cards, audit trails and incident response plans) keeps humans accountable for the systems they build.

Twelve quick questions. Pick an answer to see why it's right or wrong.