Selecting a technology vendor without relying on the demo

Define criteria before you look

Write and weight your criteria before seeing any product. Doing it afterwards means the criteria are shaped by what you have already been impressed by.

Separate must-have from desirable honestly. Long must-have lists usually contain preferences, and they eliminate good options for no real reason.

Make them demonstrate your cases

Provide your two or three most awkward scenarios and ask each vendor to show them. The standard demonstration is always the one that works well.

• Use your own data shapes, not their sample set.
• Include the exception your current process struggles with.
• Ask what happens when an integration is unavailable.
• Ask to see the administrative interface, not only the polished front end.

Reference calls are worth the effort

Ask for references in a similar sector and at a similar size, and ask them specific questions: what took longer than expected, what would you do differently, how responsive is support when something is genuinely broken.

Ask what they wish they had known. Reference customers are usually candid when asked precisely rather than generally.

Model the total cost and the exit

Licence or subscription, implementation, integration, training, and the internal effort to run it. A cheaper licence with expensive integration is not cheaper.

Then ask how you would leave: can you export your data in a usable, documented format? Establishing that before signing is straightforward; establishing it afterwards is not.

Running a fair evaluation

Evaluations drift towards whoever presented most impressively rather than whoever scored best, and a little structure prevents that.

Score independently before discussing, so the room does not converge on the first strong opinion. Record the reasoning behind each score, not just the number. Keep the weighting fixed once agreed rather than adjusting it after seeing results.

Where a strong preference emerges that the scoring does not support, that is worth examining honestly. Sometimes the criteria missed something real; sometimes it is simply the better demonstration.