Evaluating Production AI Agents with Microsoft Foundry

A supply chain director is about to let an agent answer stock questions for store managers. Before release, she asks two things: how do we know it answers correctly, and how will we know when it stops? A fluent reply answers neither: the agent chooses a tool, passes it an item number, reads the result and explains it, and each step can fail separately.

Microsoft Foundry is Microsoft's Azure platform for building and operating AI applications and agents. Within it, Foundry Agent Service is the managed runtime that hosts agents, manages their conversations and runs their tool calls. Foundry evaluation tests an agent's outputs and behavior against prepared cases, while Foundry observability traces live runs and monitors production quality.

Those capabilities generate evidence; the organization supplies the business expectations and decides what the evidence means for a release. For where Foundry sits among Microsoft's other agent products, start with the Microsoft Agentic AI Stack.

What Foundry Evaluation Is

Foundry evaluation is a project capability, not another agent, model or runtime. A team chooses a target, attaches test data, selects evaluators and inspects a run report, in the Foundry portal or through SDKs and APIs.

The target can be a Foundry agent or model, a dataset of responses already produced, or collected traces. Stored outputs can be graded without calling the live agent again; available evaluators depend on the target.

A run returns aggregate results for comparing versions and case-level results showing each prompt, response and evaluator feedback. A score without that case context is a weak release argument.

What Foundry Can Evaluate About an Agent

Foundry's evaluator catalog answers several different questions, which should not be collapsed into one quality score:

Answer and retrieval quality: Is the response relevant and coherent? Is each claim supported by the context supplied to the evaluator?
Task behavior: Did the agent complete the task or follow its instructions, including a required refusal or clarification? Some task evaluators are in preview.
Tool behavior: Did it choose an appropriate tool, pass accurate inputs and use the returned result? Support varies by tool and evaluator.
Risk and safety: Does the output show content risks, prohibited actions or sensitive-data leakage? The AI Red Teaming Agent can also generate adversarial probes.
Business-specific criteria: Custom evaluators, code-based or rubric-based, apply rules the team defines. Some mechanisms are in preview.

The boundary sits in the inputs. Groundedness measures support from the supplied context, not whether that source is current, and a custom grader applies a rule or reference answer only if the team provides one. For consequential claims or actions, domain review and deterministic permission checks remain necessary alongside evaluator scores.

How a Foundry Evaluation Run Works

A useful test dataset contains a request and enough context to judge the expected behavior. Cases should cover ordinary work, ambiguous questions, missing data, unauthorized requests and tool errors, and the expected outcome may be an answer, a clarification, a refusal or a tool-use path. Foundry accepts CSV or JSONL data and can generate synthetic cases, which still need review.

Model-based graders can disagree with a domain expert, so reviewers should check that evaluators measured the intended behavior on high-impact cases.

Consider a fictional inventory agent asked why an item is out of stock at the Dallas warehouse. Its inventory lookup confirms zero units on hand but returns no supplier or purchase-order data. If the agent blames a supplier delay, a groundedness or custom evidence check can flag the unsupported cause, while tool inspection shows the lookup itself was correct.

Three panels for a fictional stock-out question. The request expects the agent to report stock and open orders and to name a cause only with supplier or order evidence. The tool call completed and returned zero units on hand with no supplier records. The agent answered that a supplier delay caused the stock-out, which fails as an unsupported claim. — The example is fictional. The inventory record supports the shortage, not its cause, so the supplier explanation fails even though the tool call succeeded.

How Is Success Evaluated?

Foundry's Task Completion evaluator (preview) judges whether the agent's messages appear to satisfy a request. Customer Satisfaction (preview) estimates how satisfied a user would be from the conversation. That estimate is a model's inference, not a rating from a real user.

For a business agent, success has another layer: did the intended outcome actually happen, correctly and with authorization? If an agent says it created a replenishment case, the case system must confirm it. For the stock-out question without supplier evidence, the correct answer reports the shortage and says the cause is not yet known.

Verified outcomes, corrections, escalations and real user ratings complete the picture, compared with the pre-agent baseline. That measurement is application-owned, and when a judge grade, a tool result and a user rating disagree, the case needs review.

From Repeatable Tests to a Release Decision

Evaluation becomes more valuable when the same cases run against every candidate version. SDK or API runs make that reproducible, and preview GitHub Actions and Azure DevOps integrations place results in continuous integration.

The pipeline enforces whatever criteria the team chooses. Those team-owned acceptance criteria might require correct tool inputs, supported claims, expected refusals and human review of costly or irreversible cases. No universal passing percentage establishes that an agent is safe for its intended use.

When instructions, models, tools or data interfaces change, rerun the regression set; a new judge-model configuration can also change grades without any change in the agent. The release and change controls around this are covered in agentic development in Microsoft Fabric.

What Changes After Deployment

Production brings requests, data changes and tool outages the test set did not anticipate.

Tracing records an execution path through OpenTelemetry into an Azure Monitor Application Insights resource connected to the Foundry project. A trace can show model calls, tool invocations, timing, errors and selected content, which helps distinguish a retrieval problem from a wrong tool argument or an answer that ignored missing evidence.

The Agent Monitoring Dashboard, on an agent's Monitor tab, shows token usage, latency, run success rates and evaluation results. Its settings add scheduled and continuous evaluation of sampled live responses, red-team scans and alerts, several in preview. This is after-response monitoring. If an answer must be rejected before display or an action prevented, the application needs an inline validation or approval control, as described in governing agents in the Microsoft stack.

Traces may hold prompts, tool arguments, customer data or outputs, so access, retention, redaction and content capture need deliberate configuration before production monitoring is enabled.

Does Foundry Keep Every Conversation?

Agent Service conversation state and observability traces are different records. Agent Service persists a conversation's messages and tool items when an application uses a Conversation object, in Microsoft-managed resources (basic setup) or customer-owned resources such as Cosmos DB (standard setup). Responses can also run without one, and the hosted-agent Invocations protocol leaves history to the application.

Trace content and retention follow the Application Insights configuration. Neither record is a complete transcript of every interaction. A deployment that needs an audit record must decide which interactions to store, how to join conversation, response and trace IDs, and who can access or delete the data.

Can Reviewers and Users Give Feedback?

Yes, through different paths. Foundry's human evaluation (preview) gives reviewers question templates, such as ratings and free text, in an agent's preview experience. Builders can also annotate individual traces in the Foundry portal (preview).

For a deployed agent, the application presents its own feedback control to end users. Microsoft's preview feedback-logging pattern emits a correlated OpenTelemetry event, so the rating appears with the relevant trace in Application Insights and Foundry. A thumbs-up count says nothing about users who gave no rating, or about whether a downstream action was correct.

When a Production Failure Becomes a Better Test

The loop closes when production evidence becomes a test case. Foundry can evaluate historical traces directly and, in preview, convert a sample of an agent's traces into a versioned evaluation dataset. A reviewer then labels the failure and records the expected response, and the next candidate runs against the updated set.

Four-stage flow. Production traces in Application Insights record the request, tool calls and response. A preview Foundry job samples those traces into a versioned dataset of query and response rows. A reviewer marks failures and adds the expected response and criteria. The next candidate version is evaluated on the new cases alongside previously accepted cases. — Foundry supplies the trace and the dataset job; a reviewer supplies the failure label and expected answer before the set is used to test the next version.

Questions to Settle Before Release

Before relying on Foundry evidence for a release, confirm:

which evaluators, targets and features you use are generally available or in preview;
which business rules and reference answers your custom evaluators apply;
which acceptance criteria block a release, and who owns them;
what tracing captures, where it is stored and who can read it;
how end-user feedback and reviewed production failures reach the regression set.

Reliancy helps teams connect Foundry's evaluation and observability capabilities to business test cases, tool contracts and release ownership. Talk to Reliancy about evaluating a production agent.

Sources and Product Documentation

All articles