How to evaluate AI agents for accounting

Evaluate an accounting AI agent as a workflow, not as a model. Freeze the system version, test it on representative cases, define the expected action for each case, record every tool call and human intervention, and set release thresholds before seeing the results. A few convincing answers in a demo do not establish that the agent can operate safely in production.

The evaluation should answer two separate questions: does the workflow complete the intended job, and does it stay inside its authority when the case is unclear or hostile? Both matter. An agent that extracts every invoice correctly but can post without approval has failed the more important test.

Define the evaluation contract

Write one page before assembling cases. It keeps the test tied to the real job and prevents a team from changing the scoring rules after a disappointing run.

Decision	What to write down
Unit of work	The exact start and finish of one case
Intended users	Who operates, reviews, and receives the result
Allowed actions	Systems, records, fields, messages, and changes the agent may use
Prohibited actions	Actions it must refuse or route to a named person
Test population	Time period, client types, document formats, normal cases, and exceptions
Expected result	Required output, tool actions, evidence, and escalation for each case
Measures	Formulas, thresholds, severity levels, and treatment of partial completion
Release gate	Who decides, what must pass, and which limitations remain in production

The NIST AI RMF Core calls for documented test sets and metrics, performance demonstrated in conditions similar to deployment, pre-deployment testing, and regular evaluation in operation. NIST describes the framework as voluntary and use case agnostic, so a firm must still choose measures that fit the accounting consequence.

Build cases from the workflow

Use a versioned set that reflects the documents, systems, users, and exceptions the agent will meet. Include routine work, common variations, ambiguous inputs, missing evidence, duplicate records, stale instructions, permission failures, and content that tries to redirect the agent. Separate clients and periods correctly when building and storing the set.

A qualified reviewer should establish the expected result before the run. If reviewers disagree, resolve the accounting rule or mark the case as genuinely judgmental. Do not score the agent against a reference answer that the firm itself cannot defend.

Keep a holdout set for the final release decision. Repeated prompt or rule changes can tune the system to a development set without improving performance on new work. The NIST Generative AI Profile warns against extrapolating from narrow or anecdotal assessments and recommends testing claims with empirical methods under conditions similar to deployment.

Score results and control behavior separately

One average can hide a serious failure. Report at least these measures by case type:

• task result: correct completion, incomplete work, and wrong result
• boundary behavior: prohibited actions attempted or completed
• escalation behavior: required cases routed to the right reviewer with the needed evidence
• source traceability: outputs linked to the correct document, record, or rule
• human burden: interventions, review minutes, corrections, and reopened cases
• operations: elapsed time, model and service cost, retries, and recovery from a failed dependency

Use denominators that a reader can inspect. For example:

required-escalation recall = required escalations correctly routed / all cases that required escalation

unauthorized-action rate = unauthorized actions attempted / all tested cases

straight-through completion = cases completed correctly without intervention / eligible routine cases

An attempted action can still be a control failure even if a downstream permission blocked it. Record both the agent’s decision and the system’s final effect.

Worked example with hypothetical numbers

This example demonstrates the calculation. It is not a client result, benchmark, or recommended threshold.

Assume a firm evaluates a document-collection agent on 120 cases: 70 routine files, 25 missing-document cases, 15 ambiguous files that require review, and 10 cases containing a prohibited request. Before the run, the firm sets these illustrative gates:

Gate	Hypothetical threshold	Hypothetical result	Decision
Correct routine completion	At least 67 of 70	68 of 70	Pass
Required review routing	15 of 15	14 of 15	Fail
Prohibited actions completed	0	0	Pass
Prohibited actions attempted	0	1	Fail
Correct source attachment	120 of 120	119 of 120	Fail

The overall workflow completed most cases, but it does not pass. One missed escalation and one wrong source can matter more than the aggregate completion rate. The team should diagnose those traces, change the workflow or control, and rerun the full regression set plus new cases that exercise the same failure pattern.

These thresholds describe only this test. A larger test may still miss rare failures. A test set drawn from one month may not represent tax season, a new client segment, or a provider change.

Move through controlled stages

Start offline with no live writes or outbound messages. Test components, then the complete route, including authentication, integrations, human checkpoints, logging, retries, and manual fallback.

Next, run in shadow mode on approved live-like or live work without allowing the agent’s proposed actions to affect records or clients. Compare its actions with the normal process. Protect any production data under the firm’s existing data-handling rules.

For limited production, restrict clients, case types, volume, and permissions. Keep approval on consequential actions. Define a rollback trigger and a person who can disable credentials and move queued cases to manual processing.

After release, monitor the same measures used at the gate. Rerun regression tests after changes to models, prompts, retrieval data, business rules, integrations, permissions, or document formats. Review overrides and incidents for new cases to add. A passing result belongs to the tested version, not to every future version from the same vendor.

Keep an evaluation record

Retain the workflow definition, system and provider versions, configuration, test-set version, expected results, reviewer names or roles, raw action traces, calculations, exceptions, approvals, and known limitations. The record should let another qualified person reproduce the score and understand why the firm released or rejected the system.

For PCAOB engagements, this evaluation does not replace audit work. PCAOB AS 1105 requires sufficient appropriate audit evidence. For information produced by the company, it requires procedures over accuracy and completeness and an evaluation of whether the information is sufficiently precise and detailed. PCAOB AS 1201 keeps responsibility for engagement supervision with the engagement partner. Neither standard is an AI evaluation protocol. The engagement partner still has the responsibility AS 1201 assigns.

The human review guide can help place release checkpoints. The workflow selection guide can help decide whether the process is stable and measurable enough to evaluate.

Limitations

An internal workflow evaluation is not a cybersecurity assessment, penetration test, financial-statement audit, legal opinion, or compliance certification. It cannot prove the absence of rare or future failures. Results depend on the cases, reference answers, environment, reviewers, scoring rules, and exact system version. Independent review can reduce conflicts and blind spots, but it does not create a guarantee. Firms should use qualified accounting, security, privacy, and legal advisers where the workflow requires them.

Sources and methodology

This method adapts NIST’s voluntary risk and evaluation guidance to bounded accounting workflows. NIST states that AI RMF 1.0 is being revised, so firms adopting it formally should check the current publication. PCAOB citations explain evidence and supervision duties only for work within the standards’ scope.

• NIST AI RMF 1.0
• NIST AI RMF Core
• NIST AI 600-1, Generative AI Profile
• PCAOB AS 1105, Audit Evidence
• PCAOB AS 1201, Supervision of the Audit Engagement