Model interpretation
Extract, classify, compare, summarize and draft. Its output is always a proposal.
Guide
Evaluate a construction AI agent before you build it.
An AI agent demo on a few clean documents is easy to produce. A workflow that holds up on real project inputs, with exceptions, permissions and accountability, is not. Evaluating before building is how to tell the two apart without paying for the second one first.
This guide sets out the questions State Method uses to evaluate one construction workflow: its boundary, baseline, test cases, acceptance criteria, authority and failure behavior. The answer can be to build, to change the design, to gather more evidence, to stay manual or to stop.
- Author
- Hugo Sequier
- Published
- 10 October 2026
- Reading time
- 4 min
- Format
- Practical guide
01 / Bound
Define one workflow with a start, an end and an owner.
“Use AI for document control” is a programme, not a workflow. Evaluation needs one path that can be described completely.
- Trigger: what starts a case, such as an email received or a package issued.
- Inputs: the documents and data the workflow may use.
- Output: the artifact a person reviews, such as a register entry or review record.
- Owner: the named role accountable for the outcome.
- Out of scope: the cases the workflow must refuse or hand back.
02 / Baseline
Measure the manual baseline first.
Without a baseline, no result can be called an improvement. Record time per case, weekly volume, how often outputs are corrected later and what a mistake costs.
If the baseline has not been measured, say so plainly and measure it on a sample. An unknown baseline is a legitimate starting point; an assumed one is not.
03 / Value
Check the workflow is worth improving.
Some workflows are slow but rare, and some are fast but expensive when they go wrong. The case for an AI-assisted version depends on volume, time per case, the cost of an error and who carries that cost. A workflow that runs a few times a year rarely justifies a custom build, however frustrating it is.
State the expected benefit as a hypothesis with its assumptions, such as cases per week and review time saved, so the evaluation can confirm or reject it.
04 / Data access
Confirm the inputs can actually be used.
Many evaluations stall on access rather than on the model. Check early that the inputs can be obtained, read and used under the project’s confidentiality terms.
- Who owns the documents and who can authorize their use.
- Which formats arrive in practice: vector PDF, scans, spreadsheets, email bodies.
- Whether the systems involved offer API access, exports or neither.
- Confidentiality, retention and hosting constraints for project data.
- Whether representative past cases with known outcomes can be retrieved.
05 / Cases
Build a case set that includes the hard cases.
Agents tend to do well on typical inputs and fail on the exceptions that cost the most. The case set should reflect that.
- Representative normal cases.
- Known exceptions from past projects.
- Edge cases, such as poor scans or conflicting documents.
- Cases the workflow should block or hand back.
- A recorded correct outcome for each case, taken from past decisions.
Synthetic cases are useful for testing controls and failure paths. Claims about accuracy need representative real cases.
06 / Criteria
Write pass and fail criteria before seeing results.
Criteria written after the run tend to describe whatever happened. Set them in advance, and separate zero-tolerance rules from targets.
- Zero tolerance: no register record is created without a valid project reference.
- Zero tolerance: no consequential action runs without the named approval.
- Target: missed critical items stay below an agreed threshold on the case set.
- Target: reviewer time per case falls below the measured baseline.
- Requirement: every finding cites its source.
07 / Authority
Map who may do what: model, software, person.
Software authority
Validate inputs, enforce rules and permissions, control state and record events.
Human responsibility
Approve, correct, reject or stop consequential actions, with the decision recorded.
Tool access
List every system the agent may read or write, and what it may never do.
08 / Failure
Test failure on purpose.
An agent that is right 95 times and silently wrong 5 times is harder to operate than one that is right 85 times and says when it cannot decide. Failure behavior is part of the evaluation, not an afterthought.
- Missing inputs.
- Conflicting documents.
- An integration that is down or slow.
- Low model confidence.
- A duplicate submission of the same case.
For each, the expected behavior is defined in advance: stop, log, route to a person, retry safely or fall back to the manual process.
09 / Decide
Choose between build, change, more evidence, manual or stop.
The evaluation ends with a decision, not a slide. Proceed to a Controlled Pilot only when the criteria are met on representative cases and the authority and failure paths are clear. Changing the design, gathering more evidence, staying manual or stopping are equally valid outcomes.
FAQ
Before the next decision.
How long does an evaluation take?
The State Method Diagnostic evaluates one workflow in 10 business days. A Controlled Pilot that tests real tools is scoped separately, only when the evidence supports it.
How many test cases are needed?
There is no universal number. The set must cover the normal cases, the known exceptions and the cases that should be blocked, and any result should be reported with the size of the set it came from.
Can the evaluation use synthetic data?
Synthetic data is useful for testing controls, permissions and failure paths. Accuracy claims need representative real cases.
Is existing software a better choice?
Often, yes. If an existing product covers the workflow with acceptable controls, configure it. A custom build is worth evaluating when a specific gap remains after that comparison.
Continue with the evidence
Review the next relevant boundary.
Assessment / one workflow
Bring one workflow to evaluate.
Describe the workflow, its owner, its inputs and how it is measured today. A person reviews every assessment.