Unit 13.03: Testing whether readers reach the takeaway

Three readers and four questions takes ten minutes and finds what another hour of polishing would not.

Agreement, not opinions

Three readers answering the four questions the dashboard should answer.

The code compares their answers.

QUESTIONS = ["What is the finding?", "Which two suppliers?",
             "What period is this?", "What would you do next?"]
READERS = {
    "reader 1": ["two suppliers cost more", "Acme and Borex", "not sure",
                 "renegotiate"],
    "reader 2": ["spend is concentrated", "Acme and Borex", "not sure",
                 "look at freight"],
    "reader 3": ["Acme is expensive", "Acme", "not sure", "unclear"],
}
print(f"{'question':28} agreement")
for i, q in enumerate(QUESTIONS):
    answers = [r[i] for r in READERS.values()]
    print(f"{q:28} {'all agree' if len(set(answers)) == 1 else 'DISAGREE: ' + str(answers)}")

print("""
All three failed the period question, which means the period is not on the
chart. Two of three reached the intended finding; the third read it as being
about one supplier.

Three readers and four questions takes ten minutes and finds the defects that
another hour of polishing would not.
""")

All three failed the period question, which names a missing label precisely. Two of three reached the intended finding and the third read it as being about one supplier rather than two - which points at the highlighting rather than at the reader.

Both defects are specific and cheap to fix. Neither would have come out of asking what people thought of it.

The mistake this prevents

The mistake is testing with people who already know the analysis. They supply the missing context from memory and report that everything is clear. Test with someone who has not seen it before.

Takeaway

Ask three unfamiliar readers the questions the dashboard should answer. Disagreement names the defect; a question everyone fails names a missing element.

Mark complete