Last reviewed:
Structured vendor comparison methods score shortlisted vendors against criteria agreed in advance rather than assessing each proposal on its own terms. The common approaches are weighted scoring, a comparison matrix, pass or fail gating on mandatory requirements, and normalising commercial proposals onto a single baseline before any price is compared.
The reason to impose structure is that unstructured evaluation cannot be defended afterwards. Reviewing Commonwealth procurement, the Australian National Audit Office has found that where request documentation did not clearly articulate the evaluation criteria, or where records did not adequately explain how assessment outcomes informed the final decision, entities could not demonstrate how they reached a value for money conclusion. Value for money is the core principle of the Commonwealth Procurement Rules, and the ANAO’s procurement insights set out what adequate documentation looks like. A structured comparison is what makes the decision explainable months later, which is the test that actually gets applied.
What is a vendor comparison matrix?
A comparison matrix is a grid with criteria as rows and shortlisted vendors as columns, scored on a consistent scale with weights applied. Its value is less the arithmetic than the fact that it makes gaps visible: a vendor that scores well overall while failing one weighted criterion is easy to spot.
How does weighted scoring work in vendor selection?
Each criterion receives a weight reflecting its importance, agreed before proposals are seen. Vendors are scored on each criterion and the score is multiplied by the weight. The totals rank the shortlist. Agreeing weights in advance is what stops the model being tuned afterwards to justify a preferred outcome.
What is the difference between gating and scoring?
Gating is pass or fail on mandatory requirements and happens first, removing vendors that cannot meet a non-negotiable. Scoring then differentiates between the vendors that remain. Mixing the two lets a vendor compensate for a genuine deal-breaker by scoring highly elsewhere.
Why normalise commercials before comparing price?
Proposals rarely arrive on the same pricing unit, term or set of inclusions. Rebuilding each onto one baseline, with the same term length, same volumes and the same treatment of overages and support tiers, is what makes a price comparison mean anything.
How does AHP compare with a weighted scoring matrix?
Weighted scoring is one of a family of multi-criteria decision analysis (MCDA) methods. The best-known alternative is the Analytic Hierarchy Process (AHP), developed by Thomas Saaty. Instead of assigning weights directly, evaluators compare every pair of criteria and state how much more one matters than the other, and the method checks those judgements for consistency before it calculates the weights.
AHP is worth the extra effort when stakeholders cannot agree on weights, because the pairwise format turns a vague disagreement into a specific one. It gets unwieldy as criteria are added: ten criteria need 45 pairwise comparisons. For most technology evaluations a weighted scoring matrix, with weights agreed in a workshop, is easier to explain to a board, and being able to explain the decision is the test it will face.
What goes in an RFP evaluation scorecard?
An RFP scorecard is the working form of the matrix. For each criterion it records the weight, the scoring scale with a written description of what each score means, the evidence behind the score, and who gave it. The descriptions do most of the work. A scale where 3 means “meets the requirement, with evidence supplied” produces scores that two evaluators can agree on. A bare 1 to 5 scale does not.
In a CYBORIUM evaluation, gates and weights are agreed with stakeholders before any proposal is opened, and every score is recorded against the evidence that supports it. That record is what lets the decision be reconstructed months later.
Sources
- Procurement and Contract Management, Australian National Audit Office. Audit findings on evaluation criteria and value for money documentation.
- Commonwealth Procurement Rules, Australian Government Department of Finance.
Where to read more
See how independent IT vendor evaluation works, MoSCoW prioritisation for purchasing, and guided vendor evaluations.
Also relevant: bringing in an independent procurement service and market analysis tools for IT procurement.
