Companies Need To Stop Testing AI With Questions Nobody Asks
Enterprise AI evaluation often begins with benchmarks because they offer an apparently objective way to compare models. One system scores higher in mathematical reasoning, another leads a coding benchmark and a third performs better across a broad academic test, allowing procurement teams to place numbers beside models whose outputs would otherwise be difficult to compare. The problem begins when companies assume that a benchmark advantage predicts performance inside their own business.
An insurance company does not need a model to excel at every graduate-level science question if its main job involves extracting information from claims documents. A manufacturer deploying an agent to diagnose equipment problems cares about whether the system follows maintenance procedures and recognises uncertainty, while a bank may prioritise reliable handling of policy and compliance information over general conversational ability.
Useful evaluation therefore begins with work rather than the model. Teams need to identify the tasks employees actually perform, the mistakes that carry consequences and the level of quality required before automation produces more value than supervision costs.
Real examples provide better test material than invented prompts because organisational language contains ambiguities that generic benchmarks miss. Customer names resemble product codes, abbreviations mean different things between departments and internal policies rely on assumptions employees learned through experience rather than documentation.
A test set built from actual work exposes those conditions early. Companies can remove sensitive information, preserve the structure of the problem and evaluate whether different models produce outputs employees would genuinely use.
Correctness also needs a definition appropriate to the workflow. A numerical extraction can often be scored exactly, while a strategic summary may contain several acceptable answers. Treating both tasks through the same evaluation metric creates apparent precision without measuring the quality employees care about.
Human evaluation remains useful for subjective work, although organisations need clear criteria if several reviewers are involved. Asking whether an answer is “good” produces inconsistent judgements; asking whether it omitted a material risk, used unsupported information or failed to follow the required format produces more comparable evidence.
Pairwise comparisons can help when absolute scoring becomes difficult. Reviewers can examine two anonymised outputs and decide which would be more useful in the actual workflow, allowing companies to compare systems without pretending that writing quality can always be reduced to one objective percentage.
Cost belongs inside evaluation because an improvement has economic value only in relation to what it requires. A model that produces slightly stronger summaries at five times the inference cost may remain worthwhile for investment research while being difficult to justify for millions of routine customer interactions.
Latency matters for similar reasons. Employees may accept a minute of reasoning when the system analyses a complex contract, whereas a customer-service assistant becomes frustrating if every straightforward request takes several seconds longer than the user expects.
Agentic systems require evaluation of the process as well as the final answer. An agent can reach the correct result after making unnecessary tool calls, accessing information outside its assigned scope or taking an action that happened to be harmless in the test environment.
Companies therefore need traces showing which steps occurred. The final output can look excellent while the path towards it reveals a security, cost or reliability problem that will become more serious once the system operates at scale.
Failure tests deserve greater weight than average performance. A system that succeeds on 98 percent of invoices may still be unsuitable if the remaining 2 percent involve precisely the documents containing the largest payments, while a medical or compliance workflow can carry risks concentrated in rare cases.
Testing should deliberately include poor scans, contradictory documents, missing information and malicious instructions because production environments contain all of them. Models evaluated only on clean examples will appear more reliable than the systems users eventually encounter.
Uncertainty needs measurement too. An AI system that admits it lacks enough information can be more useful than one with higher average accuracy that answers confidently when evidence is missing, particularly in workflows where a human can resolve uncertain cases.
Companies can route those cases rather than demanding perfect automation. The system handles routine work automatically, escalates ambiguous examples and gradually expands its scope as evaluation demonstrates that additional categories perform reliably.
Model updates make the process continuous. Providers change systems, companies alter prompts and internal data evolves, meaning an application that passed evaluation six months ago may behave differently without anyone changing its visible interface.
Regression testing can catch those shifts by running representative cases whenever an underlying component changes. The principle is ordinary software engineering even though the outputs are probabilistic.
Executive teams should also resist searching for one enterprise-wide AI score. Different applications require different thresholds because summarising an internal meeting and approving a financial transaction do not deserve identical tolerance for error.
The model with the strongest public benchmark can still be the wrong choice when another system performs the company’s narrow task more reliably, quickly or cheaply. Once enterprise AI moves beyond experimentation, evaluation needs to answer a less glamorous question than which model is smartest: which one consistently performs the work the organisation actually needs done.
