Unit 08.00: What you are actually measuring

Six questions, and only the first is what people usually measure.

One of them is a count, not a rate

Correctness, safety, efficiency, termination, judgement and cost.

The table below lists all six.

question                                 dimension
did it reach the right outcome?          correctness
did it stay inside its permissions?      safety -- must be perfect
did it take a sensible path?             efficiency
did it stop when it should?              termination
did it escalate when it should?          judgement
what did it cost?                        operations

6 dimensions. Only the first is what people usually measure.

The second is not a rate you improve -- it is a count that must be zero. An
agent that stayed inside its permissions 99.8% of the time acted outside them
twice, and both times are incidents.

Safety is the one that is not a rate. An agent that stayed inside its permissions 99.8% of the time acted outside them twice, and both are incidents - expressing that as a percentage on a dashboard invites an alert threshold above zero, which is a statement that some breaches are acceptable.

Termination and judgement are the two that only exist for agents. A script always terminates and never escalates, so neither question arises.

The mistake this prevents

The mistake is reporting one accuracy number upward. It moves for reasons nobody can attribute, and it cannot represent a dimension whose acceptable value is exactly zero.

Takeaway

Measure six dimensions, and keep permission violations as a count with a threshold of zero. A rate hides a single breach on a busy day.

Mark complete