Last reviewed:
An independent IT vendor evaluation defines scored criteria before any vendor is approached, gathers evidence against those criteria rather than accepting sales claims, and normalises commercial proposals onto one comparable baseline. The evaluator holds no reseller agreements, so the shortlist is not bounded by what they are able to sell.
The sequence matters more than any individual step. Criteria written after seeing proposals tend to describe the proposal somebody already preferred.
What makes an evaluation independent?
Independence is structural, not a statement of intent. Test it with three questions: does the evaluator sell, resell or implement any option being assessed, do they hold vendor quotas or accreditation tiers, and does their fee change depending on which vendor is selected. If any answer is yes, the evaluation carries an interest.
Independence is only useful if it is documented well enough to survive review. Auditing Commonwealth procurement, the Australian National Audit Office has found that where request documentation did not clearly articulate the evaluation criteria, or where records did not adequately explain how assessment outcomes informed the decision, entities could not demonstrate how a value for money conclusion was reached (ANAO procurement insights). Value for money is the core principle of the Commonwealth Procurement Rules, and the same standard is what a board or audit committee applies in the private sector. An evaluation that cannot be reconstructed from its own records is not independent in any way that counts.
What criteria should an IT vendor evaluation use?
Criteria should be agreed and weighted before the market is approached, and normally cover functional fit, security posture and certifications, implementation risk, data handling and residency, interoperability, support and escalation, total cost across the full term, and exit conditions including data return.
Weighting is where most of the argument belongs, and it is much easier to have that argument before anyone has a preferred answer.
How are vendor proposals made comparable?
Proposals arrive on different pricing units, term lengths and inclusions. Normalising means rebuilding each onto a single baseline: same term, same user count, same assumptions about overages and support tiers. Until that is done, comparing headline prices compares nothing.
What should an IT vendor evaluation checklist include?
A usable checklist follows the order of the evaluation rather than listing everything at once.
- Before approaching the market: requirements sorted into mandatory and weighted, weights agreed and recorded, evaluators named and conflicts of interest declared.
- Security and supply chain: current certifications and the scope they actually cover, where data is held and who can reach it, subcontractors, and incident notification terms.
- Delivery risk: the implementation plan, named resources, and reference customers of a similar size who can be contacted.
- Commercials: proposals rebuilt onto one baseline, overage and price review clauses, and exit costs including data return.
- Decision record: scores tied to evidence, the reason each unsuccessful vendor lost, and sign-off.
For the supply chain items, NIST SP 800-161 Rev. 1 on cybersecurity supply chain risk management is a detailed reference, and APRA-regulated entities will also test against CPS 230 and CPS 234. CYBORIUM evaluations follow this sequence, and the decision record stays with the client.
Sources
- Procurement and Contract Management, Australian National Audit Office.
- Commonwealth Procurement Rules, Australian Government Department of Finance.
- SP 800-161 Rev. 1, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, National Institute of Standards and Technology.
Where to read more
See guided vendor evaluations, structured vendor comparison methods, and what is vendor due diligence.
Also relevant: what cybersecurity procurement services cover, and worked examples of these evaluations.
