AI procurement checklist. Last reviewed:
An enterprise AI proof of concept should answer a buying question: can this system complete our work, within our controls, at an acceptable cost? A polished demonstration cannot answer that. Give shortlisted suppliers the same tasks, set acceptance criteria before testing, and keep the outputs and failures.
These ten tests cover quality, security, operations and commercial fit. The order is an editorial checklist, not a measured ranking. Adapt the thresholds to the consequences of a wrong answer or action. A tool drafting internal notes needs different controls from one changing customer records.
The National AI Centre’s implementation guidance recommends aligning supplier testing with your acceptance criteria and keeping governance and testing records. The checks below turn that principle into a practical procurement exercise.
1. Can it complete a real task from start to finish?
Select a bounded workflow your team already performs: summarising an approved policy, extracting contract dates, or preparing a draft response. Specify the input, expected output and acceptance criteria. Include the hand-offs and checks needed after the model responds. A correct paragraph is not a completed workflow if someone must spend another twenty minutes finding its sources.
Evidence to retain: the task brief, original inputs, complete output and reviewer decision. Compare total staff time with the current process, including correction time. Reject demonstrations that measure only how quickly the first answer appears.
2. Does it perform on examples the supplier has not rehearsed?
Use a small development set to configure the solution, then a separate evaluation set that was not used for tuning. Include ordinary cases, awkward documents, missing fields and unfamiliar wording. Keep task difficulty comparable across suppliers. Do not let a supplier silently replace difficult cases with cleaner ones.
Evidence to retain: the evaluation-set version, scoring rubric and per-case results. Separate material errors from cosmetic ones. Where judgement is involved, use a second reviewer for disputed cases. A single overall accuracy figure can conceal a poor result on the one task your business most needs.
3. Are its answers supported by the right sources?
For a system that answers questions from company documents, ask questions with known answers and questions the documents cannot answer. Check that each citation opens the right document and supports the nearby statement. Include an outdated policy alongside its replacement and test whether the system distinguishes them.
Evidence to retain: the answer, cited passage, document version and any unsupported claim. Require the system to flag missing evidence or escalate when appropriate. A plausible answer with an unrelated citation is a failure, even when the writing sounds confident.
4. Do permissions survive retrieval and tool use?
Test with separate users who have different access rights. Ask the same question as an authorised employee and as someone who cannot read the underlying record. Check search results, summaries, attachments and tool responses. For an agent, also attempt an action outside its approved scope.
Evidence to retain: the test identity, permissions, retrieved records and denied actions. ASD’s secure AI deployment guidance calls for strict access controls and least privilege. A reassuring chat response is insufficient if a connector still retrieves restricted information in the background.
5. Can untrusted content redirect the system?
In an isolated test environment, put a harmless conflicting instruction inside a document or email the system must process. For example, ask the system to ignore its task and output a fictional marker. Then test whether it treats the text as source material or as an instruction. For connected agents, include attempts to trigger an unauthorised action using synthetic records.
Evidence to retain: the supplied content, attempted redirection, output and tool activity. One successful defence does not establish complete protection. Record which attack classes were tested, unresolved failures and the controls that limit their impact.
6. Can you account for the data it receives and retains?
Start with synthetic or appropriately de-identified data where possible. Ask the supplier to map where prompts, files, outputs, logs and support copies go. Identify subprocessors, retention periods, deletion processes and any use for model training. Verify the relevant configuration and contract terms; an answer in a sales questionnaire is only a starting point.
Evidence to retain: the data-flow map, settings, contractual commitments and a deletion test where available. The OAIC’s commercial AI guidance explains privacy considerations for personal information in both inputs and outputs. Assess the proposed use before introducing real personal information.
7. Can a person review, stop and correct its work?
Give the reviewer a deliberately flawed draft. Can they see the evidence, change the result, reject it and stop a queued action? Test what happens after rejection. For agents, confirm that approval applies to the exact action and parameters the reviewer saw, and that changing them requires a fresh review where your policy demands it.
Evidence to retain: the approval record, rejected action and corrected output. Measure whether review is practical at expected volumes. If the reviewer must reconstruct the entire task from scratch, include that effort in the business case. See CYBORIUM’s AI agent procurement guide for wider buying considerations.
8. What happens when a dependency fails?
Simulate an unavailable connector, a timeout, a missing document and a partial write in the test environment. Check whether the system reports uncertainty accurately, retries within a limit and avoids duplicate actions. Verify recovery after access is restored. A workflow that says “complete” after a failed write creates extra operational risk.
Evidence to retain: timestamps, error records, retry behaviour and the destination record after recovery. Define the fallback owner and acceptable interruption. Test realistic concurrency and longer inputs as well as isolated happy-path requests.
9. What does an accepted result actually cost?
Measure the whole workflow: licences, model usage, retrieval, connectors, storage, monitoring and human review. Count retries and rejected outputs. Divide the measured cost of the test batch by the number of accepted tasks, then model ordinary and peak demand. Keep one-off implementation costs separate from ongoing operating costs.
Evidence to retain: usage records, pricing assumptions and a reconciliation to the billable units. Treat a free trial allowance separately from production pricing. CYBORIUM’s enterprise AI pricing guide explains the questions to resolve when seats, credits and consumption charges overlap.
10. Can you repeat the result after a change?
Save the test set, prompts, model or service version where exposed, configuration and scoring rules. Re-run the agreed checks after a model update, connector change or new document source. Ask what advance notice the supplier provides and whether you can defer changes or roll back. Document any version information the supplier cannot expose.
Evidence to retain: a repeatable test pack, change log, comparison results and an accountable owner. The NIST AI Risk Management Framework offers voluntary guidance for managing AI risk. Procurement should turn the relevant controls into responsibilities that continue after the pilot.
Turn the results into a buying decision
Use one record per test: requirement, expected result, observed result, evidence link, owner and decision. Agree which failures block deployment before seeing the results. A high average score must not cancel a critical access-control failure.
- Pass: the agreed requirement was met and the evidence is retained.
- Conditional: a specific limitation has an owner, a control and a deadline for retesting.
- Fail: the requirement was missed or the evidence was not sufficient to assess it.
These are suggested decision labels, not a certification scheme. Keep critical conditions in the implementation scope and acceptance terms. A pilot pass supports the next decision; it does not prove performance at every production volume or certify legal compliance.
Frequently asked questions
How large should an AI proof of concept be?
Large enough to cover the intended tasks, meaningful variations and important failure cases. Choose the sample with the people who own the work. A convenient round number of prompts is not evidence that the sample is representative.
Should every supplier use the same model?
Usually compare complete solutions against the same requirements. Record model and configuration differences so you can explain the results. Forcing identical components can hide the real strengths and limits of each offer.
Can a successful pilot replace security and contract review?
No. The pilot tests observed behaviour under defined conditions. Security review, privacy assessment, commercial terms and operational readiness address questions a limited trial cannot settle.
Build your next AI evaluation around evidence
CYBORIUM helps Australian organisations define requirements and compare technology providers. If you are preparing an AI shortlist, discuss your evaluation with CYBORIUM. Bring the workflow, its risks and the decision you need to make. The enterprise technology RFP template is a useful place to start documenting requirements.
