Metrics: every number answers a narrower question than it appears to

Research metrics are not wrong.

They answer narrower questions than the ones people use them to answer — and the gap between the two is where nearly all misuse lives.

Publication counts

• Measure: indexed output volume.
• Do not measure: quality, importance, or total output.
• Sensitive to publication practice — the same work split into more papers counts higher.
• Sensitive to indexing coverage, which varies by language and venue type.
• Not comparable across fields with different norms.

Citation counts

• Measure: how often something was formally referenced in indexed sources.
• Do not measure: why, or whether the citing work agreed.
• Vary enormously by field, making raw cross-field comparison meaningless.
• Accumulate over time, disadvantaging recent work.
• Are a property of the database as much as of the work.

Composite indicators

• Compress a distribution into one number, hiding the distribution.
• Cannot be compared across career stages.
• Inherit every limitation of the underlying database.
• Differ between databases for the same person.

Collaboration measures

• Measure: co-authorship recorded in indexed metadata.
• Do not measure: collaboration that has not yet produced a paper, or contribution below authorship threshold.
• Depend heavily on affiliation data quality.
• Depend on the counting rule, which changes the totals substantially.

Using metrics without overreaching

• State source and date always.
• Use field-normalised measures for any comparison.
• Show distributions rather than single figures where possible.
• Use metrics as context, never as a decision about a person.
• Report coverage limitations with the same prominence as the numbers.

Questions to ask of any metric

• What exactly does it count?
• What does it not count?
• Which database, and from when?
• Is it comparable to what it is being compared with?
• Would it change if coverage were equal across the comparison?

One thing worth remembering

Any metric is as much a property of the database it came from as of the research it describes.

Change the index and the number changes without anything about the work changing at all — which means using a metric without naming its source is reporting a number that cannot be checked or reproduced.

Câu hỏi thường gặp