Building a software scoring matrix that survives the sales demo

Step 1: write requirements as must, should and could

Start with requirements, because the matrix can only score what you have written down. Sort each requirement into one of three groups.

• Must: without it, the system is unusable for you. Examples: works in Canadian dollars with GST/HST handling, imports your existing customer list, connects to your accounting system.
• Should: clearly valuable, but you could work around it for a while. Examples: mobile quoting, a customer portal, role-based permissions.
• Could: nice to have. Examples: built-in texting, a particular dashboard style.

Write each requirement as something you can test in a demo: "A dispatcher can reschedule a job and the customer gets a notice, in under one minute" is testable. "Easy scheduling" is not.

Must-haves are gates, not scores. A vendor that fails one is out before scoring starts. Mixing must-haves into weighted scores lets a vendor with flashy extras outscore one that actually meets your basic needs.

Step 2: choose criteria and weight them to 100

Group the should and could requirements into six to ten criteria, then share 100 points among them according to what matters to your business. Agree the weights before you see any demo, so no one adjusts them to suit a favourite.

Step 3: score 1 to 5, and write down why

Each score needs a reason someone else could check. Use a fixed scale so a 4 means the same thing for every vendor.

• 5: meets the criterion fully, shown in the demo or trial.
• 4: meets it with a small workaround.
• 3: meets it partly; real workarounds needed.
• 2: weak; significant gaps.
• 1: does not meet it, or the vendor could not show it.

A claim the vendor did not demonstrate scores no higher than 3 until it is shown. "It's on the roadmap" scores as if it does not exist.

Step 4: the worked matrix for vendors A, B and C

Here is the arithmetic. Multiply each weight by the score, add the column, then divide by 5 to get a result out of 100.

Vendor B has the strongest workflow fit, but weak integration, cost and export pull it down to 71. A and C tie at 77, which is common and useful: it tells you the decision now turns on risk, not on features.

Breaking the tie

Look at where each finalist is weakest on a heavily weighted criterion. C scored 3 on workflow fit, the criterion worth 30 points, so its strength in cost and ease of use is covering a gap in the thing your team does all day. A's weakest point is ease of use, worth 15.

Then test the weights. Move 5 points from cost to workflow fit (35 and 10):

• A: 385 − 120 − 45 + (35 × 4) + (10 × 3) = 390, or 78.
• C: 385 − 90 − 75 + (35 × 3) + (10 × 5) = 375, or 75.

A modest, defensible change separates them, and in A's favour. If the same small shift had swung the result back and forth, you would treat them as equal and decide on references and trial results.

Step 5: scripted demos, trials and reference checks

The quality of the scores depends on the quality of the evidence. Three habits make a large difference.

1. Script the demo. Send each vendor the same five to eight scenarios a week in advance, written from your real work: "Create a quote for a repeat customer with two line items and a deposit, convert it to a job, invoice it." Watch them do it in their system, not in slides.
2. Time the tasks. If one system takes four clicks and another takes eleven for a task done 40 times a day, that is a measurable ease-of-use difference.
3. Call references who match you. Ask for a customer of similar size and industry, ideally in Canada. Ask what took longer than expected, what they would configure differently and how support responded the last time something broke.

Step 6: questions to get answered in writing

Before signing, get written answers to the commercial questions that matter after the honeymoon. These are questions, not legal advice; the agreement itself should be reviewed by your lawyer.

• How do we export all our data, including history and attachments, in what format, and is there a fee?
• How is the price set at renewal, and is any increase capped?
• What are support hours, response targets and escalation steps, and are they in Pacific, Central or Eastern time?
• Where is our data stored and who can access it?
• What happens to our data and access if we stop paying, and for how long?
• Which features in the demo are included in the plan we are quoting, and which are add-ons?

A vendor that answers these clearly and quickly is telling you something about how the next three years will go.

Mistakes that quietly bias the result

A matrix can still mislead if the process around it is loose. Watch for these.

• Weights set after the demos. Weights must be fixed before anyone sees a product.
• One enthusiastic scorer. Average several independent scorers rather than adopting the loudest view.
• Demo-day scoring only. Re-score after the trial and reference calls; first impressions fade.
• Price used twice. If total cost is a weighted criterion, don't also knock points off "viability" because a vendor is expensive.
• Shortlist too long. More than four finalists means the requirements were not specific enough.

How we run this with you

Our software and vendor selection service delivers the requirements list, a shortlist, the scored matrix and the written questions, with your team doing the scoring and us keeping the process fair. We don't resell software, so you buy directly from the vendor you pick. Engagement sizes are on our pricing page, and if you'd rather start small, a text to (604) 632-4959 or a note through contact is enough to begin.