Unit 01.03: Measuring whether a chart was read correctly

Whether a chart works is testable, and the test takes ten minutes.

Three readers, four questions

Ask three people the questions the chart is supposed to answer, and compare their answers rather than their opinions.

The code shows what that produced for one chart.

QUESTIONS = [
    "Which region had the highest revenue?",
    "By roughly how much did the top region beat the second?",
    "Is the overall trend up or down?",
    "What period does this cover?",
]
ANSWERS = [
    ("reader A", ["North", "about 20%", "up", "not sure"]),
    ("reader B", ["North", "about 5%", "up", "not sure"]),
    ("reader C", ["Northeast", "double", "flat", "not sure"]),
]
print(f"{'question':46} agreement")
for i, q in enumerate(QUESTIONS):
    given = [a[1][i] for a in ANSWERS]
    agree = len(set(given)) == 1
    print(f"{q:46} {'all agree' if agree else 'DISAGREE: ' + str(given)}")

print("\nEvery reader failed the date question, which means the chart has no date.")

# Show the chart to three people and ask them the questions it is supposed to
# answer. Disagreement is a defect in the chart, not in the readers -- and the
# question everyone fails identifies the missing label.

The readers agree on the leader and disagree wildly on the margin - 20%, 5%, "double" - which means the chart communicates rank and not magnitude. That points at a truncated axis.

All three failed the date question, which identifies a missing label precisely. Disagreement localises the defect in a way "what do you think of this chart?" never does.

The mistake this prevents

The mistake is asking for opinions. People will tell you the colours are nice and will not tell you they misread the margin by a factor of four. Ask the questions the chart exists to answer and score the answers.

Takeaway

Test a chart by asking readers the questions it should answer. Disagreement is a defect in the chart, and the question everyone fails names the missing element.

Mark complete