Before the demo
What decision does this system own, stated as a decision and not a feature? What does it do when it is uncertain, and how is 'uncertain' defined in code rather than in the deck?
During the pilot
Can you reconstruct any single action end to end, six weeks later, without vendor assistance? When the system was wrong, how was that detected: by the system, by a human, or by a customer complaint?
Before you widen scope
What is the worst plausible action inside the current envelope, and what would it cost? And who, by name and role, owns the outcomes of this system on the day something goes wrong?
A good system makes these questions easy to answer. That ease is itself the strongest signal in the evaluation.
“If a vendor cannot describe the blast radius of their system's worst plausible action, they have not thought about it. That is the finding.”