This section considers the different approaches to evaluating suppliers' responses to questionnaires in order to select the most appropriate suppliers. It reviews the two most common means of evaluating responses, to give a sense of how these can be incorporated into the tendering process.
Please note as with the entire course that what’s covered in this lesson is not exhaustive and is only offered as a guide. You are encouraged to work with your IT or cyber security teams as well as other stakeholders to develop an approach that’s most suitable for your council.
Different approaches to supplier evaluation
The two common approaches to evaluating suppliers’ responses are either by technical assessment, or a ‘pass or fail’ approach.
Technical Assessment
In a technical assessment, suppliers are given a score that forms a part of their overall assessment. The supplier or suppliers with the highest score, which includes cyber, would be selected to proceed in the procurement process.
To give scores in each question category, you may have to decide on a scoring scale which could be on a scale of nought to 10, as shown on the table. You will also have to clearly decide how you award each score on the scale. You can see here in the example above that the total cyber resilience score of 10 per cent contributes to the total quality score of 80 per cent. You may want to assign the bulk of the cyber resilience score to the control questions given that most of the questions would be from this category.
Pass or fail
In a pass or fail approach, you have a binary outcome where suppliers either do not meet your requirements and are disqualified from the procurement process or they meet your requirements and can proceed in the procurement process.
For example, you can decide that for a supplier to pass, the supplier must provide acceptable responses to questions like those featured above surrounding background, control and certification requirements for suppliers. It is up to each individual council to determine which questions to ask and, if using a pass or fail approach, what is considered an acceptable response and what is not. If a supplier does not provide acceptable responses to the questions, they may be failed and disqualified from the procurement process.
With this approach, there’s a risk of setting the bar too high so no-one passes, or too low so everyone does. Carrying out market engagement in the prior planning phase to understand the cyber security maturity of the market might be very useful here. Market engagement will also help to identify any potential pass or fail questions that all potential suppliers fail, and adjust accordingly.
Multi- vs single-stage procurements
Multi vs. single stage approaches to supplier evaluation can be conducted together in a multi-stage procurement, or alone as a single stage procurement. Both approaches are illustrated in the graphic below. Not all multi-stage procurements are pass or fail -some can incorporate a combination of pass or fail, and assessment, at both stages.
Multi-stage procurement
Multi-stage procurements provide the opportunity to use a combined evaluation approach. At the first or an earlier stage, you can use the pass or fail approach to sift out suppliers with unacceptable cyber security arrangements. In the final or later stage, you can use the technical assessment to choose the best or most appropriate supplier.
Single-stage procurement
A single stage procurement can use either the pass or fail approach or the cyber technical assessment in addition to other non-cyber security quality criteria such as social value to determine the right supplier to award the contract.