Reading a Vendor's Productivity Claims

The marketing in this field carries percentage improvements. Understanding how they are generated makes the demonstration a different conversation.

The evaluation discipline in “Reading a Vendor's Productivity Claims” also applies to workforce software: begin with a named decision and test it in a bounded pilot. For teams considering productivity software for business, this detailed guide belongs in that comparison only with written criteria for notice, access, correction, retention and a dated review.

Where the figures come from

Customer self-reports, collected by the vendor after purchase.

For an independent reference relevant to “Reading a Vendor's Productivity Claims”, consult the European Commission data-protection resources; it provides a useful external check on scope, terminology, governance and the claims made during procurement or review.

Before-and-after comparisons with no control and no correction for anything else that changed.

Measurements of the vendor's own metric: "productive time increased", where productive is defined by the vendor's categorisation.

And occasionally a commissioned study with methodology that is not published.

The circularity problem

The most common claim is that productive hours rose.

Productive is defined by which applications the software counts as productive.

So the claim is that after deployment, people spent more time in applications the vendor's categorisation approves of.

That may be true and it is not a statement about output, which is the thing anybody buying it wants.

The observation effect

Behaviour changes when measurement starts, in any direction, temporarily.

A rise in the first weeks is expected and says nothing about the following year.

Ask for figures at twelve months rather than at three, and watch whether the answer becomes vaguer.

What is omitted

Attrition during the period.

Whether output actually changed, as the business measures output.

How many customers stopped using it.

What the deployment cost beyond the licence.

The omissions are consistent across the industry, which is itself informative.

Questions that change the conversation

"How is productive time defined, and can we see the category list?"

"Was there a control group?"

"What happened to output, measured by the customer's own business metrics?"

"What was staff turnover before and after?"

"Can we speak to a customer who deployed more than two years ago?"

That last one is the most useful and the least expected.

The reference call

Ask them: what surprised you, what did you turn off, did anybody leave over it, and does anybody still look at the dashboard.

The final question is the one that distinguishes a live deployment from a renewed licence.

Making your own case instead

Decide what you would measure, before buying, using your own business figures.

Then measure it.

A deployment that cannot show movement in your own numbers after a year has not worked, whatever the vendor's percentage says.

What to check

Can the vendor define productive time, with the list?

Do they have twelve-month figures, with a control?

Have you asked about turnover?

And have you written down what you would measure yourself?