Measuring Whether the Programme Works

Most device programmes report how many devices are enrolled. That is a measure of adoption, not of effect.

The same discipline applies when “Measuring Whether the Programme Works” becomes part of a broader workforce programme. An organisation considering how to measure employee productivity in relation to how to measure employee productivity should put access, retention, employee notice and review dates into the implementation plan rather than leave them as product defaults.

The five numbers

Devices enrolled against devices that should be — the gap is the population nobody is watching.

For an independent baseline relevant to “Measuring Whether the Programme Works”, the NIST Cybersecurity Framework is a useful companion: compare its principles with the proposed configuration, ownership model and real support process before approving a rollout.

Compliance rate, by condition rather than overall.

Operating system version distribution, and time to reach a new version.

Devices not checked in for a defined period.

And support contacts per hundred devices per month.

The gap number

The most important and the hardest to produce, because it requires knowing the denominator.

Who has a work phone, or accesses work data on a personal device, and is not enrolled.

Identity logs usually show this: sign-ins from devices the platform does not know.

That figure is the real exposure, and enrolment counts conceal it.

Compliance by condition

An overall rate of ninety-something hides which condition fails.

Broken out, it usually shows one condition failing widely — which is a policy finding, as the compliance note argues.

Report the breakdown, not the headline.

Version distribution

How long after a release does the estate reach it?

This is the number that describes your actual vulnerability exposure, and it is more honest than a compliance percentage.

Track it monthly and watch the shape rather than the mean.

Silent devices

Enrolled, not checking in, status frozen at whatever it was.

These accumulate: replaced devices, people who left, devices in drawers.

They inflate your enrolment count and your compliance rate simultaneously, which is why both look better than reality.

Audit and remove quarterly.

Support contacts

Per hundred devices, by cause.

Falling after a change means the change worked.

Rising after a policy tightening tells you the cost of it, which is worth knowing before the next one.

What not to measure

Enrolment count alone.

Policies configured, which measures activity.

And anything from the platform's own dashboard that you cannot explain the derivation of, which is more of them than expected.

Reporting it

Five numbers, monthly, one page.

With the gap number first, because it is the one that matters and the one most likely to be omitted.

What to check

Do you know how many unenrolled devices access work data?

Is compliance reported by condition?

How many enrolled devices have not checked in this month?

And how long does your estate take to reach a new operating system version?