Data & Metrics
Aggregation: combining sources multiplies their errors as well as their coverage
Combining sources improves coverage and imports every problem each one has. Both effects are real and only one is intended.
#data and metrics
Data & Metrics
Data & Metrics
Counting rules: the rule is part of the number
Two analysts with the same data and two defensible rules produce different totals. Publishing one without the other publishes half a finding.
Data & Metrics
Reproducibility: could someone else get your number from your data?
Most published analyses cannot be reproduced from what was shared — usually for want of documentation rather than secrecy.
Data & Metrics
Measurement design: what you measure becomes what people do
Choosing indicators is not a technical decision. It determines behaviour, whether or not that was intended.
Data & Metrics
Metrics: every number answers a narrower question than it appears to
Research metrics are not wrong. They answer narrower questions than the ones people use them to answer.