IIIOrganization & Project ManagementGuide
Evaluating Suppliers: Quality, On-Time Delivery, and Collaboration
Suppliers get talked about a lot in a company, usually after the latest mishap. Whether one delivers well or badly overall, however, hardly anyone knows precisely. An evaluation with a few figures changes that, provided it covers the right suppliers.
Key takeaways
- A supplier evaluation regularly assesses the performance of existing suppliers against fixed criteria. For small companies, three are enough: quality, on-time delivery, and collaboration.
- According to the Kraljic matrix, the depth of evaluation depends on two questions: how strongly does the purchase affect results, and how large is the supply risk?
- As early as 1966, Dickson found that buyers weight quality, delivery, and past performance more heavily than price. The weights should nevertheless be set in advance.
- A single poor value colors the judgment of all the others. Experiments show that simple counter-questions weaken this anchoring effect.
Why are suppliers usually judged by gut feeling?
Most companies have a clear opinion about their suppliers. One is reliable, another expensive, a third delivered a wrong batch six months ago and has been considered shaky ever since. These opinions arise from memories, and memories are unevenly distributed. The mishap in spring is vivid; the 40 flawless deliveries after it are not.
That has consequences. A decent supplier is dropped over a slip, or a weak one is kept because working together is pleasant. This leads to the guiding question of this guide: How does a small company evaluate its suppliers so that the evaluation is fair and can support decisions about the relationship?
To answer it, we first clarify the terms and classify suppliers with the Kraljic matrix. Then we examine which criteria research suggests and which judgment errors distort the evaluation, weigh figures against experience, and describe a three-criterion scorecard and a process that fits into one afternoon every six months.
A supplier is not as good as its best delivery and not as bad as its worst, but as good as its last fifty.
What is a supplier evaluation, and what is it not?
A supplier evaluation is the regular assessment of the performance of existing suppliers against criteria set in advance. It differs from two related steps. Supplier selection takes place before a relationship begins and compares offers. Vetting a business partner asks whether the company exists, how it stands financially, and whether there are warning signs. How this basic check with public registers can be done in about 20 minutes is described in the guide to due diligence on business partners.
The evaluation comes afterward. It does not ask whether a supplier is trustworthy but how well it actually delivers. Companies working to ISO 9001 must in any case define criteria for the evaluation, selection, and monitoring of external providers. But even without a certificate, it is the basis for any supplier conversation that goes beyond price.
Three criteria suffice for most small companies. Quality means that the goods or service meet the requirements, measured for example by the complaint rate. On-time delivery here means the share of deliveries that arrive complete on the promised date. Collaboration covers availability, response to problems, and willingness to raise mistakes openly.
The thinking model: the Kraljic matrix
Not every supplier deserves the same effort. In 1983, Peter Kraljic proposed classifying purchased items by two attributes (Kraljic 1983). The first is profit impact: how strongly does the purchase affect cost, margin, or quality of one’s own offering? The second is supply risk: how hard is it to find another supplier if this one fails? This produces four quadrants.
| Quadrant | Profit impact | Supply risk | Example | Evaluation |
|---|---|---|---|---|
| non-critical items | low | low | office supplies, cleaning | none, an annual price comparison is enough |
| leverage items | high | low | standard sheet metal, packaging | price and quality, every six months |
| bottleneck items | low | high | special part from a single manufacturer | on-time delivery and collaboration, every six months, fallback plan |
| strategic items | high | high | key component, core subcontractor | all three criteria, every six months, annual review meeting |
Worked example
For the evaluation, this implies a division of labor. Non-critical items need no evaluation; it only costs time. Leverage items are managed mainly through price and quality because there are enough providers. For bottleneck items, security of supply matters more than price. Strategic suppliers deserve the full evaluation and a conversation. How to trace the origin of bottleneck items back to the supplier’s supplier is shown in the analysis of geopolitics as a business risk.
As with a home medicine cabinet: you do not check every pack of bandages, but the medication someone depends on must always be in stock and not expired.
Which criteria matter? What research shows
The question of how suppliers should be judged is one of the oldest in purchasing research. In 1966, Gary Dickson asked buyers in the United States about the importance of 23 criteria (Dickson 1966). At the top were quality, delivery, and the supplier’s past performance. Price came only sixth. Later work shifted the weights by industry but confirmed the basic pattern: whoever compares only price overlooks what buyers themselves consider more important.
How to combine several criteria into a decision is a research field of its own. Weber, Current, and Desai (2000) describe a structured approach in which suppliers’ performance on several criteria is first measured and then weighed against each other, also as a basis for negotiations. For a small company, the message is simpler than the mathematics behind it: the weights must be set before the evaluation, not afterward.
A study from India shows that small companies have ground to make up here. Mohite, Patil, Pisal, and colleagues (2026) surveyed 275 small and medium-sized manufacturers in three states and conducted 20 in-depth interviews. They found weaknesses in purchasing that stemmed from scarce resources, informal decision-making, and little digital support. The environment differs from Germany’s Mittelstand, but the pattern will look familiar to many owners.
The anchor in your head: how one poor value colors everything
Even with fixed criteria, the evaluation remains a human judgment. Ricky Wong (2024) examined how strongly past performance distorts this judgment. In two online experiments with 104 and 408 participants from the United Kingdom, participants rated fictitious suppliers on several criteria. Beforehand they saw the value of one criterion from an earlier evaluation, either high or low.
The result is clear. An earlier low value on one criterion led to lower ratings on all other criteria compared with an earlier high value. This anchoring effect carried through to the decision: it changed how likely participants were to choose the same supplier again. Wong also points out that this makes it hard to recognize suppliers that have improved significantly.
Wong tested two simple remedies that small companies can use too. With consider-the-opposite, you explicitly look for reasons that speak against the anchor value. With mental mapping, you make yourself aware of the full range of possible values before rating. The effect was uneven: according to his recommendations, considering the opposite helps more against high anchors, mental mapping more against low ones. The participants were not observed buyers making real decisions but people on an online platform. That limits transferability but does not change the direction.
Figures or experience? Two perspectives
The first view says: only figures are fair. Complaint rate and on-time delivery can be read from goods receipt and documents, and sympathy then plays no role. This view has a lot going for it, especially given the findings on the anchoring effect.
The second view says: a supplier’s most important qualities appear in no statistic. Whether it delivers on a Saturday in an emergency, whether it reports a mistake itself before the customer finds it, whether it allocates fairly during shortages: only those who work with it know. That is true as well.
I consider a combination right. Quality and on-time delivery are measured, not estimated. Collaboration is assessed, but with fixed guiding questions, with the two-question rule, and by at least two people. The result is not a verdict but the start of a conversation with the supplier.
The scorecard: three criteria, five levels
| Criterion | Weight | Measurement | 5 points | 3 points | 1 point |
|---|---|---|---|---|---|
| Quality | 40% | share of deliveries with complaints in the half-year | below 1% | 2 to 4% | above 6% |
| On-time delivery | 35% | share of deliveries arriving complete and on time | above 97% | 90 to 95% | below 85% |
| Collaboration | 25% | assessment by two people using fixed guiding questions | reports problems itself, solves them quickly | responds when asked | hard to reach, shifts blame |
Worked example
The data for quality and on-time delivery arises in the company anyway. Whoever matches the delivery note and purchase order at goods receipt, as the three-way match in the guide to purchasing and approvals provides, needs only two additional checkmarks: complete yes or no, on time yes or no. How to define and evaluate on-time delivery properly is described in the guide to measuring and improving on-time delivery. Complaints come from the complaint log described in the guide to handling complaints, provided supplier errors are marked in it.
Worked example with freely chosen values. Below 3 points, a meeting with measures follows; below 2.5 points, also the search for an alternative.
Classify suppliers
Assign every supplier with meaningful spend to a quadrant of the Kraljic matrix. Only bottleneck and strategic suppliers and the largest leverage suppliers are evaluated.
Set the weights
Set the weights of the three criteria before the first evaluation and write them down. They apply to all suppliers in the same quadrant.
Collect data
Evaluate complaints and deliveries of the last six months. Where data is missing, start counting at goods receipt from now on.
Assess collaboration
Two people who work with the supplier rate independently using the guiding questions and the two-question rule. If they differ widely, they talk.
Hold the conversation
The result is discussed with strategic suppliers, with figures, examples, and a joint measure. Suppliers that perform well hear that too.
What follows beyond the evaluation
An evaluation kept hidden from the supplier achieves little. Its greatest benefit arises in conversation: the supplier learns what it is measured against and can give reasons the company does not see. Sometimes the cause of late deliveries lies in one’s own short-notice orders. Transparency works in both directions here.
Suppliers also affect one’s own reputation beyond their performance. If a supplier violates environmental or labor standards, it reflects on the buyer. How to assess this risk is covered in the analysis of reputational risks from suppliers and partners. For larger customers, obligations under Germany’s Supply Chain Act may also play a role and reach small suppliers as questionnaires.
The limits are clear. The studies cited come from the United States, the United Kingdom, and India, and the experiments work with fictitious suppliers. The point values on the scorecard are judgments that can feign precision. And a good evaluation does not replace the question of whether one is too dependent on a single supplier.
Conclusion: how does a small company evaluate suppliers fairly?
The question at the start was a small company evaluates its suppliers so that the evaluation is fair and can support decisions. The answer begins with a selection: following the Kraljic matrix, only suppliers with high profit impact or high supply risk are evaluated. For them, three criteria suffice, quality, on-time delivery, and collaboration, with weights set in advance.
The evaluation becomes fair when quality and on-time delivery are measured rather than remembered and the assessment of collaboration is protected against the anchoring effect. As Wong’s (2024) experiments show, simple counter-questions are enough for that. The sober observation is: a supplier evaluation is worth as much as the conversation it triggers.
How to get started
- List your ten suppliers with the highest spend and assign each to a quadrant of the Kraljic matrix.
- Select the three strategic or bottleneck suppliers whose failure would do the most damage.
- Set the weights for quality, on-time delivery, and collaboration before you rate.
- From Monday, introduce two checkmarks at goods receipt: complete and on time.
- Plan a meeting with your most important supplier next quarter in which you look at the figures together.
Frequently asked questions
Which criteria belong in a supplier evaluation?
For small companies, three are enough: quality, measured for example by the complaint rate, on-time delivery as the share of deliveries arriving complete and on time, and collaboration, assessed with fixed guiding questions. Price is negotiated separately.
How often should suppliers be evaluated?
Strategic and bottleneck suppliers every six months, the most important leverage suppliers as well. Non-critical suppliers need no regular evaluation; an annual price comparison is enough.
What is the Kraljic matrix?
A model by Peter Kraljic from 1983 that sorts purchased items by profit impact and supply risk into four quadrants: non-critical, leverage, bottleneck, and strategic items. A different purchasing strategy fits each quadrant.
How do you avoid unfair evaluations?
By rating quality and on-time delivery from data rather than memory, setting the weights in advance, and having collaboration assessed independently by two people. Counter-questions such as what speaks against my first impression help against the anchoring effect.
Does ISO 9001 require a supplier evaluation?
ISO 9001 requires a company to define and apply criteria for the evaluation, selection, and monitoring of external providers. The standard does not specify how elaborate this must be.
Sources
- Kraljic, P. (1983): Purchasing must become supply management. Harvard Business Review, 61(5), 109-117.
- Dickson, G. W. (1966): An Analysis of Vendor Selection Systems and Decisions. Journal of Purchasing, 2(1), 5-17.
- Weber, C. A.; Current, J. R.; Desai, A. (2000): VENDOR: A Structured Approach to Vendor Selection and Negotiation. Journal of Business Logistics, 21(1), 135-167.
- Wong, R. S. (2024): The power of past performance in multidimensional supplier evaluation and supplier selection: Debiasing anchoring bias and its knock-on effects. PLoS ONE, 19(5).
- Mohite, R.; Patil, R.; Pisal, R. et al. (2026): Strategic Vendor Selection and Purchasing Challenges in India’s Manufacturing MSMEs: An Empirical Perspective. South Asian Journal of Management, 33(2), 100-127.
