Choosing indicators looks like a technical decision. It is not.
What gets measured becomes what people do — whether or not that was the intention, and usually faster than anyone expects.
How indicators change behaviour
- Counting outputs produces more, smaller outputs.
- Counting venues moves work toward venues rather than readers.
- Counting citations creates incentives around citation practice.
- Anything unmeasured stops being done — this is the effect people notice last and regret most.
- The response is rational, not dishonest.
What output measures miss
- Training and supervision.
- Review and editorial work.
- Data and infrastructure that others build on.
- Applied work with no publication.
- Work published in unindexed venues.
- Systems that measure only output eventually lose all of these.
Indicators worth having
- Retention of trained researchers.
- Supervision capacity relative to need.
- Domestic first-authorship in international collaborations.
- Whether collaborations persist beyond single outputs.
- Whether infrastructure is used and maintained.
- These are harder to collect and describe the system better.
Designing measurement responsibly
- Ask what behaviour each indicator will produce before adopting it.
- Use several indicators, since any single one will be optimised toward.
- Include something that captures unmeasured contributions.
- Publish the definitions.
- Review for distortion and act on what you find.
What to avoid
- Single composite scores — they hide exactly what matters.
- Rankings, where the data does not support ranking.
- Comparing across fields without normalisation.
- Individual assessment based on aggregate indicators.
- Changing indicators frequently, which makes long-term work irrational.
Being honest about limits
- State what each indicator does not capture.
- State coverage limitations.
- Show uncertainty.
- Say when an indicator is a proxy rather than a direct measure.
- Most indicators in research are proxies, and saying so costs nothing.
One thing worth remembering
Before adopting any indicator, ask: what will people do to score well on this, and do we want that?
The answer arrives regardless of intent. Asking in advance is the only moment at which the design can still be changed.
Câu hỏi thường gặp
How do indicators change behaviour?
Counting outputs produces more and smaller outputs, counting venues moves work toward venues rather than readers, and anything unmeasured stops being done.
What do output measures miss?
Training and supervision, review and editorial work, data and infrastructure others build on, applied work with no publication, and work in unindexed venues.
Which indicators are worth having?
Retention of trained researchers, supervision capacity relative to need, domestic first-authorship, whether collaborations persist, and whether infrastructure is used and maintained.
What should be avoided?
Single composite scores, rankings the data does not support, cross-field comparison without normalisation, individual assessment on aggregate indicators, and frequent indicator changes.
What question should precede adopting an indicator?
What will people do to score well on this, and do we want that — the answer arrives regardless of intent.
