AI agent procurement guide. Last reviewed:
AI agent procurement is different from buying software that helps a person, because an agent does not wait to be asked at every step. It plans, calls tools, reads your data and takes actions, so what you are really buying is delegated authority. This guide sets out how to tell a real agent from a rebranded chatbot, nine areas to test before you sign, the contract terms specific to agents, how agents are priced and how to cap the bill, and how to run a pilot that can be stopped. It follows the Five Eyes guidance on agentic AI that ASD’s Australian Cyber Security Centre co-authored in May 2026.
What is an AI agent, and how is it different from an AI assistant?
The joint guidance from ASD’s ACSC, the United States’ CISA and NSA and the Canadian, New Zealand and United Kingdom cyber agencies describes agentic AI as systems that autonomously reason, plan and execute actions by combining large language models with external tools and data sources. An assistant answers a question or drafts text and a person decides what happens next. An agent decides the next step itself and carries it out, such as sending an email, changing a record, raising a ticket or calling another system.
That difference moves the risk. A wrong answer from an assistant is a bad draft. A wrong action from an agent can change a customer record, spend money or expose data before anyone notices.
How do you tell a real agent from agent washing?
Gartner predicted in June 2025 that over 40 per cent of agentic AI projects will be cancelled by the end of 2027, because of rising costs, unclear business value or inadequate risk controls. It also warned of “agent washing”, where vendors rebrand assistants, robotic process automation and chatbots as agents, and estimated that only about 130 of the thousands of vendors claiming agentic AI offer real capability. Before you compare agents, check that the product is one.
- Does it plan across several steps without a person approving each one?
- Does it call tools such as your email, ticketing or finance systems, and can you see which ones?
- Does it keep state across a task, so that a later step depends on an earlier result?
- Can it show a log of what it did, step by step, on your own workflow?
- Does the demo run on your data and process, not a scripted example?
If the answers are mostly no, you are evaluating an assistant or a workflow tool. That can be perfectly useful, but it should be priced and governed as one.
What should you test before you buy an AI agent?
Test the nine areas below. The right-hand column is the evidence to ask for, because a statement in a sales deck is not evidence.
| Area | What to test | Evidence to ask for |
|---|---|---|
| 1. Scope of actions | Exactly what the agent can and cannot do, written as a list. | An action schedule agreed in the contract. |
| 2. Identity and permissions | Whether the agent has its own identity and the least access it needs. | The permission model, and a demonstration that it cannot exceed it. |
| 3. Tool and data access | Which systems, data stores and external services it can reach. | A list of every integration and the data each can touch. |
| 4. Human approval points | Which actions need a person to approve first, and who. | Configurable thresholds, with a record of each approval. |
| 5. Action logs and audit | Whether every step, tool call and decision is recorded. | A sample log from a real task, and how long logs are kept and exported. |
| 6. Stop and rollback | Whether you can halt it immediately and undo what it did. | A live demonstration of the stop control and of reversing an action. |
| 7. Testing and red-teaming | Whether it has been attacked, including with prompt injection, and what was found. | Test reports, and the right to run your own tests. |
| 8. Model and tool dependencies | Which models, plug-ins and subcontractors sit behind it, and how changes are handled. | A dependency list with notice of change. |
| 9. Cost behaviour | How cost grows with steps, retries and volume. | A cost model on your workflow, with spend caps you can set. |
Which risks should drive the evaluation?
The joint guidance groups the risks into five: weaknesses inherited from the underlying language model, such as prompt injection and data poisoning; a larger attack surface through the tools and integrations an agent uses; complexity that lets one failure cascade through several steps; misuse of privilege and identity; and reduced accountability because decisions are hard to trace. The OWASP Top 10 for Agentic Applications for 2026 gives a more detailed threat list, numbered ASI01 to ASI10, covering risks such as goal hijacking, tool misuse and rogue agents. Use both as a checklist for your security team when you review a shortlist.
The practical point for procurement is that an agent’s exposure is everything it can reach. An agent with broad access to email, files and finance is as risky as a person with those keys, and it works faster and does not tire. That is why permissions and logs come before features. Our guide to information security in procurement shows where to place these requirements in a tender.
What contract terms matter most for an AI agent?
The general AI terms in our guide to AI contract clauses in Australia apply to agents: no mining of your data, notification of AI incidents, human oversight, a circuit breaker you control, and data handover and destruction at exit. Agents need five more.
- An action schedule. A list of what the agent may do, with anything outside the list prohibited unless you approve a change.
- Approval thresholds. Actions above an agreed value or risk need a named person to approve before they run.
- Exportable action logs. Logs of each step and tool call are kept for an agreed period and handed to you on request and at exit.
- Responsibility for actions. The contract says who bears the cost of an action the agent takes outside its schedule, and whether liability limits apply to it.
- Change control. Changes to the underlying model, tools or sub-agents need notice, and you can test before they reach your environment.
At exit, ask for the agent’s configuration, prompts, tool connections and any memory it holds about your business, in a form you can use elsewhere. Our guide to pricing the exit before you sign applies in full.
How are AI agents priced, and how do you cap the bill?
Agents are sold in several ways, and each one hides a different risk.
| Pricing model | How you pay | What to watch |
|---|---|---|
| Per seat or per agent | A fixed fee for each user or each agent. | Limits on usage inside the fee, and what happens when you exceed them. |
| Consumption | By model use, such as tokens or calls. | An agent makes many calls per task, so cost grows with steps and retries. |
| Per outcome or action | By completed task or resolved case. | A clear definition of “completed”, and who decides. |
| Hybrid | A base fee plus variable use. | Whether the base fee covers anything real, and how the variable part is capped. |
Estimate cost per task as the number of steps, multiplied by the model calls in each step, multiplied by the cost of each call, then add retries and background runs. A cheap demo task can be an expensive production task once the agent loops, retries or runs unattended. Ask the supplier to model cost on your own workflow, and ask for:
- A hard spend cap per agent and per workflow, with alerts before the cap is reached.
- Rate limits, so a fault cannot run up a bill overnight.
- Price protection when the underlying model changes.
- No minimum commitment that outlasts your pilot.
- Usage reporting you can read without the supplier’s help.
Our five-year total cost of ownership model shows how to put these lines side by side.
How do you run an AI agent pilot that can be stopped?
The joint guidance advises introducing agentic AI in a measured way, starting with clearly defined lower-risk use cases before expanding autonomy, privileges and scope. Prefer reversibility and containment over speed. The Government’s Guidance for AI Adoption, released in October 2025, points the same way with six practices that include deciding who is accountable, testing and monitoring, and maintaining human control.
- Pick a task that is low risk and reversible, where a mistake costs time and not money or reputation.
- Write success and failure triggers first, including the point at which you stop the pilot.
- Run in shadow mode. The agent proposes, a person acts, and you compare results before the agent acts alone.
- Grant the least access that works, and widen it only after review.
- Read the logs weekly, and test the stop control during the pilot, not after.
- Decide before you start what evidence will justify widening the scope, and who signs it off.
What are the red flags in an AI agent proposal?
- The demo uses a scripted example and will not run on your workflow.
- No list of actions, or the answer to “what can it do?” is “anything you connect”.
- Logs are available “on request” with no sample.
- No way for you to stop the agent yourself.
- Pricing is described only as “per outcome” with no definition of an outcome.
- Security testing is described but no report can be shared.
- The supplier will not name the models and sub-agents behind the product.
What changes if you are regulated?
APRA-regulated entities should check whether an agent supplier counts as a material service provider under CPS 230, which brings audit access, subcontracting notice and exit terms into scope. See our CPS 230 due diligence guide. If the agent makes or supports decisions that significantly affect individuals, the transparency duties in our guide to APP 1.7 and vendor contracts apply. Have your privacy, security and legal teams review the terms before signature.
What is CYBORIUM’s role?
CYBORIUM helps Australian organisations compare AI providers against their requirements, test claims with evidence and prepare the commercial terms before signature. The selected provider pays CYBORIUM a capped fee at an agreed percentage, providers that are shortlisted but not selected pay nothing, and you contract directly with the provider. The decision is always yours. See our AI vendor evaluation page for how an evaluation runs.
AI agent procurement in short
Confirm it is a real agent, then test nine things: scope, identity and permissions, tool access, approval points, logs, stop and rollback, red-team evidence, dependencies and cost behaviour. Add agent-specific terms for actions, approvals, logs, responsibility and change. Cap the bill before you start, and run a shadow-mode pilot that you can stop.
Evaluating an AI agent? Get the Procurement Readiness Brief or tell us about your project. The client pays CYBORIUM nothing and chooses the provider.
Frequently asked questions
What is AI agent procurement?
It is buying AI that plans and carries out actions with tools and data, so the contract and evaluation cover delegated authority as well as accuracy: what the agent may do, how it is controlled, logged and stopped, and what it costs as it runs.
How do I know if a product is a real AI agent?
Check whether it plans across several steps, calls tools you can see, keeps state across a task, shows a step-by-step log, and runs on your own workflow in the demo. If not, it is probably an assistant or automation tool.
What should an AI agent contract include beyond normal AI clauses?
An action schedule, approval thresholds, exportable action logs, a statement of responsibility for actions outside the schedule, and change control over models, tools and sub-agents.
How do I stop an AI agent from running up costs?
Ask for a hard spend cap per agent and workflow with alerts, rate limits, price protection when models change, no long minimum commitment and usage reporting you can read yourself.
Where does Australian guidance on agentic AI come from?
ASD’s Australian Cyber Security Centre co-authored joint guidance with CISA, NSA and the Canadian, New Zealand and United Kingdom agencies in May 2026, titled Careful Adoption of Agentic AI Services. The National AI Centre’s Guidance for AI Adoption sets six practices for responsible adoption.
For the pricing models, forecasting and contract caps in more depth, see enterprise AI pricing.
Related guidance: AI contract clauses in Australia, AI procurement guardrails in Australia, enterprise AI vendor selection criteria.
Sources
- ASD’s ACSC with CISA, NSA and partner agencies, Careful adoption of agentic AI services, released 1 May 2026: definition, risk categories and mitigations.
- OWASP, Top 10 for Agentic Applications for 2026: threat list for agentic systems.
- Gartner, Over 40% of agentic AI projects will be canceled by end of 2027, 25 June 2025: cancellation forecast and agent washing.
- HLC, Australia’s new Guidance for AI Adoption: summary of the National AI Centre’s six practices, October 2025.
- APRA, Prudential Standard CPS 230 Operational Risk Management: service provider requirements for regulated entities.
