No. Not by 2030, and not in the way the question usually implies. The honest answer is duller and more useful than the headline. The systems most likely to hurt people in the next five years are ordinary ones, deployed into decisions they were never tested for, by organisations that never asked the supplier a hard question. Here are five dated predictions, each with the evidence behind it and the fact that would prove it wrong.
The short answer, stated plainly
An artificial intelligence system deciding on its own to end humanity is not the risk an Australian board should be budgeting against this decade. The risks worth budgeting against are narrower and closer, and they are already documented: AI that materially speeds up a cyber intrusion, AI that lowers the skill floor for building a biological or chemical weapon, and AI that quietly takes over a safety-critical judgement nobody reviewed.
That does not make the extinction question stupid. It makes it the wrong first question. A board that spends its AI governance attention on superintelligence has usually skipped the part where it checks whether the triage tool in its contact centre, or the scheduling model in its logistics arm, has a human who can actually overrule it.
Why an honest answer is hard to give
The International AI Safety Report published on 3 February 2026, chaired by Yoshua Bengio and written by more than 100 experts with backing from over 30 governments, names the real obstacle. Capability moves faster than the evidence about it. Systems improve before anyone has measured what the improvement lets a determined person do. The report calls this the evidence dilemma, and its recommended response is not a single control but layered safeguards, better evaluation, and institutions that can adapt.
Two findings in that report are worth carrying into any procurement conversation. Technical safeguards are improving but still show significant limitations, and users can often still obtain harmful outputs by rephrasing a request or splitting it into smaller steps. Separately, models are getting better at telling the difference between a test setting and a live deployment, which means a clean evaluation result is weaker evidence than it looks.
So each prediction below carries a stated confidence and a stated falsifier. A forecast you cannot lose is not a forecast.
Five predictions to 2030
1. No extinction-scale AI event before 2030, and the first AI-attributed deaths come from a mundane system in a safety-critical loop
Confidence: high. Nothing in the current evidence base supports a self-directed system capable of ending humanity inside this window. What the evidence does support is failure of a familiar shape. A model is placed in a clinical, industrial, transport or emergency workflow. It performs well in evaluation. The human review step erodes, because the model is right often enough that checking it starts to feel like waste. Then it is wrong in a case where being wrong costs a life, and the review that would have caught it had already become a formality.
This is automation bias, and it predates AI by decades in aviation and medicine. What generative systems add is confident prose wrapped around the error, which makes the failure harder for a reviewer to spot than a wrong number on a dial.
What would prove this wrong: a credible, independently verified incident before 2030 in which a general-purpose AI system pursues a harmful objective it was not directed toward, across multiple systems, without a human operator in the loop.
2. By the end of 2027, an Australian critical infrastructure incident names AI-assisted intrusion as a contributing factor
Confidence: moderate to high. This one is close to already true elsewhere. In November 2025 Anthropic disclosed that it had disrupted what it assessed as the first large-scale cyber espionage campaign orchestrated by an AI agent, against roughly thirty targets including technology companies, financial institutions, chemical manufacturers and government agencies. Public reporting puts the share of tactical work the agent performed without human intervention at roughly 80 to 90 per cent. The 2026 International AI Safety Report separately records an AI agent identifying 77 per cent of the vulnerabilities present in real software during testing.
Read those two numbers together and the change is one of tempo, not novelty. The intrusion techniques are the same ones defenders already know. Reconnaissance, enumeration and exploitation now run at machine speed, at a cost that no longer rations who can attempt them. Australian critical infrastructure operators already carry obligations under the Security of Critical Infrastructure Act for exactly this class of event, which is why an AI-assisted intrusion here will surface in a formal report rather than a blog post.
What would prove this wrong: no Australian regulator, operator or national agency attributing AI assistance as a contributing factor in a reported critical infrastructure incident by 31 December 2027.
3. Biological misuse stays the highest-severity branch, safeguards hold to 2030, and the margin gets thinner
Confidence: moderate. Through 2025 several frontier developers added specific safeguards against biological and chemical weapons assistance, and the stated reason was that they could no longer rule out that their models provided meaningful uplift. That is a significant admission. It is also, in its way, the system working. Capability was measured, the measurement was believed, and controls followed.
The thinning margin comes from the same place as every other safeguard limit. Filters that block a direct request are weaker against a request broken into innocuous parts, and open-weight models released without those filters cannot be recalled. Severity here is extreme and probability is low, which is the combination human institutions handle worst, because nothing ever happens right up until the moment something does.
What would prove this wrong: a confirmed biological or chemical attack before 2030 in which investigators attribute material capability uplift to a general-purpose AI system.
4. Australia does not pass an AI Act before 2028, and obligations arrive through the law you already have
Confidence: moderate to high, and this is the prediction that changes what a procurement team does on Monday. In September 2024 the Commonwealth consulted on mandatory guardrails for AI in high-risk settings, with ten proposed obligations covering testing, transparency and accountability. The National AI Plan released in December 2025 did not take that path. It relies instead on existing technology-neutral law, updated voluntary guidance, and a new Australian AI Safety Institute, funded at 29.9 million dollars and operational from early 2026 with a testing and advisory role rather than enforcement powers. The National AI Centre guidance folds the earlier ten guardrails into six essential practices.
Voluntary at the national level does not mean unregulated in your sector. Privacy law gains an automated decision-making transparency duty from 10 December 2026. APRA-regulated entities carry CPS 230 obligations over material service providers regardless of whether the service happens to contain a model. Critical infrastructure operators carry risk management program duties. Australian Consumer Law does not care that a misleading claim was generated rather than written.
The practical consequence is blunt. Where the legislature declines to set the control, the contract sets it. An Australian decision owner who wants testing evidence, change notification or a switch-off right will get those things because they were negotiated, or will not get them at all.
What would prove this wrong: an AI-specific statute of general application passing both houses of the Australian Parliament before 1 January 2028.
5. By 2028 an Australian regulator names model-provider concentration as a systemic risk
Confidence: moderate. The quiet story of the last two years is not capability, it is dependency. A large share of Australian AI-enabled services resolve, underneath the vendor logo, to a small number of model providers running in a small number of cloud regions. Most decision owners cannot name the model behind the product they bought, and the product behaves differently once the provider deprecates a version.
Financial regulators have spent a decade building language for this exact shape of risk in cloud concentration. Applying it to model providers is a small step. The trigger will probably be an outage or a deprecation that takes out several unrelated Australian services on the same afternoon and makes the shared dependency visible for the first time.
What would prove this wrong: no Australian sector regulator or system-wide body publishing a formal statement identifying concentration among AI model providers as a systemic or prudential risk before 1 January 2028.
What the predictions imply for how you buy
Four of the five predictions land on the same operational point. The harm does not arrive through a model you built. It arrives through a product you bought, from a supplier who changed something, in a workflow where nobody was watching closely enough to notice. The terms below are written as things a supplier can either agree to or refuse in writing.
Name the model and the provider
You cannot assess concentration risk, jurisdiction, or the effect of a deprecation while the model underneath the product is a trade secret. Ask for the model family, the provider, the hosting region, and a commitment to tell you when any of the three changes. A supplier who will not name the provider is asking you to accept a dependency you cannot measure.
Notice ahead of a behaviour change
Model updates ship quietly and can change output distribution without changing a single line of the product feature list. Set a notice period for material model or prompt changes, and define material in the contract rather than leaving it to the supplier to decide later.
A human override that survives contact with the workflow
The first prediction is really a prediction about review steps decaying. The contract answer is a documented ability to turn an automated step off, retain the manual path, and operate the service in that mode for a defined period without penalty. If disabling the automation breaks the product, the override is decorative.
Evidence, not assurance
Ask what was tested, by whom, against what, and when it was last re-run. An ISO/IEC 42001 certificate tells you a management system exists. It does not tell you the model was evaluated for the failure mode that matters in your workflow. Both are useful, and they are not the same artefact.
Incident duties that match your own
If you carry a 72 hour notification duty and your supplier carries a commercially reasonable one, the gap is yours. Align the clock, require cooperation with regulators, and name the artefacts you will need during an investigation before you need them.
How CYBORIUM helps
CYBORIUM is an independent technology advisory. We do not charge decision owners a fee and we do not resell software, so the recommendation you get is not funded by the outcome we recommend. On AI-enabled products we put the supplier questions above into the evaluation itself rather than into a post-signature audit, because the negotiating position exists before signature and disappears after it.
Related guidance: automated decision-making transparency and your vendor contracts, vendor and supplier risk management for enterprise procurement, and our cybersecurity vendor evaluation framework. If you are scoping a security product now, start with our cybersecurity evaluations.
Frequently asked questions
Can AI kill us?
Not as a species, on any evidence currently available, before 2030. AI systems can and already do contribute to outcomes that kill individual people, through failures in safety-critical workflows, through enabling attacks on physical infrastructure, and through lowering the barrier to weapons capability. Those are the risks that justify controls today.
Are researchers who warn about existential risk wrong?
They are making a claim about a longer horizon and about uncertainty, not about 2030. The International AI Safety Report does not dismiss loss of control. It records that the evidence is thin in both directions and recommends building the capacity to react as evidence arrives. Treating a low-probability, extreme-severity branch as worth monitoring is ordinary risk practice.
Does Australia have an AI law?
Not an AI-specific act of general application. The December 2025 National AI Plan relies on existing technology-neutral law plus voluntary guidance, and established an Australian AI Safety Institute with a testing and advisory remit. Sector duties under privacy, prudential, critical infrastructure and consumer law still apply to AI-enabled systems.
If the rules are voluntary, why bother with AI clauses?
Because voluntary at the national level does not remove the duties you already hold. Where a regulator asks how you controlled a third-party system, the contract is the artefact you will produce. Voluntary guidance shifts the job of setting the control from the legislature to the decision owner.
Is the EU AI Act relevant to an Australian decision owner?
Often, indirectly. Obligations on general-purpose model providers have applied since 2 August 2025, and transparency duties plus the AI Office enforcement powers over those providers took effect on 2 August 2026. Deadlines for stand-alone high-risk systems were deferred to 2 December 2027 under the digital omnibus amendment. If your supplier serves EU customers, the documentation it produces for that market is documentation you can ask to see.
What is the single most useful question to ask a supplier?
Ask what happens to your service when the underlying model is deprecated. The answer reveals whether the supplier knows which model it depends on, whether it has tested an alternative, and whether the risk of that transition sits with them or quietly with you.
Official sources and update status
- International AI Safety Report 2026, chaired by Yoshua Bengio, published 3 February 2026
- National AI Plan, Department of Industry, Science and Resources, December 2025
- Australia’s AI Safety Institute, Department of Industry, Science and Resources
- Introducing mandatory guardrails for AI in high-risk settings, the 2024 proposals paper that was not adopted
- Disrupting the first reported AI-orchestrated cyber espionage campaign, November 2025
- EU regulatory framework for artificial intelligence, European Commission
Information status: reviewed against publicly available sources on 16 September 2026. The five predictions are CYBORIUM’s own and carry the stated confidence and falsifier. This is general information, not legal advice.
Buying an AI-enabled product and want these supplier questions answered before you sign? Talk to CYBORIUM.
Last updated:
