Security Awareness Training Vendor Evaluation Guide for Australian Organisations

Security awareness training is easy to buy and hard to evaluate. Most programmes report completion rates and phishing simulation click rates, and both can be improved without making the organisation safer. Completion shows that people watched the modules. A falling click rate can simply mean the simulations became easier to spot. A useful evaluation looks for evidence that people report suspicious activity faster and make fewer risky decisions in the situations that matter to the organisation.

CYBORIUM helps Australian organisations compare security awareness training providers through a structured vendor-neutral sourcing process. The selected provider pays CYBORIUM a capped fee. The client contracts directly with that provider, and CYBORIUM does not sell, deliver, operate or invoice the service.

Start with the behaviours you want to change

List the human actions behind recent incidents and near misses in your organisation. Common examples are approving a payment change after a convincing email, entering credentials on a fake login page, approving an unexpected multi-factor authentication prompt, and sharing sensitive files through personal accounts. Different roles face different risks. Finance staff, executives, IT administrators and front-line staff need different content.

Training works alongside technical controls and does not replace them. The ASD Essential Eight focuses on technical mitigations, and phishing-resistant multi-factor authentication will reduce credential theft more reliably than any course. Buy training to cover the gaps technical controls leave.

Five areas that decide training value

1. Role-based content

Ask vendors to show modules for your highest-risk roles, and check that scenarios reflect Australian conditions, such as local payment practices and common local scams. Generic content produced for another market is noticeably less persuasive.

2. Simulation realism and fairness

Phishing simulations should test the lures your organisation actually receives, at a difficulty that changes over time. Ask how the vendor avoids simulations that damage trust, for example fake bonus or redundancy notices, and how results are used. Programmes that punish clicks tend to discourage reporting.

3. Reporting behaviour

The most useful measure is how quickly people report a suspicious message, real or simulated. Check whether the product includes a report button that works with your email platform and whether reports from real attacks can flow to the security team.

4. Measurement

Ask what the vendor measures beyond completion and click rate: time to report, repeat behaviour by individuals, and trends by role. Request an example of a board-level report and judge whether it would support a decision or just fill a slide.

5. Administration and cost

Pricing is usually per user per year, with tiers for content libraries and simulations. Confirm integration with your identity provider so joiners and leavers are handled automatically, and ask how much administrator time a quarterly campaign takes.

A comparable evaluation

Run a short pilot with two or three shortlisted providers across comparable groups of staff, using the same simulation difficulty and the same reporting process. Compare reporting rates and times as well as click rates, and ask participants whether the content was relevant to their jobs. Add written evidence on content coverage, privacy and data handling for employee records, accessibility, and three-year cost.

Contract points to settle

The agreement should state what employee data the provider holds, where it is stored and how it is deleted, because training records and simulation results are personal information. Confirm content licensing if the subscription ends, the right to customise modules, accessibility standards and how price changes as headcount moves.

CYBORIUM’s role in the decision

CYBORIUM can map the behaviours and roles that matter, design the pilot, coordinate providers, evaluate results and support negotiation. The organisation keeps the decision and contracts directly with the selected provider. The guide to email security procurement covers the technical controls that reduce how many phishing messages reach staff at all.

Updated 15 September 2026. This guide explains CYBORIUM’s procurement evaluation method. It does not provide legal or regulatory advice.

Standards and further reading

Awareness training is measured by completion rates far more often than by behaviour change, which is why programmes survive without working.

• ASD Annual Cyber Threat Report. Reported incident causes in Australia, including how often the initial access route is a person rather than a system.
• ASD Essential Eight explained. A reminder that user education sits alongside technical controls rather than substituting for them when the controls are incomplete.
• ASD Information Security Manual guidelines. The guidelines a training programme should reflect if your organisation reports against the ISM.

Also relevant: email security procurement.

Last updated: 22 October 2026