Using academic data: what the figures can and cannot support

Academic data supports some claims well and others not at all.

The difference is not visible in the data itself — which is why the same figure can be used correctly by one analyst and misleadingly by another with no change to the number.

What it supports well

• Descriptive statements about indexed output, with the source and date stated.
• Trends within one system using consistent definitions.
• Collaboration patterns where affiliations are resolved to identifiers.
• Identifying who works on a topic.
• Institutional aggregates where the counting rule is stated.

What it does not support

• Judgements about the quality of individual researchers.
• Cross-country comparison without normalisation and coverage caveats.
• Claims about total research activity, since much is unindexed.
• Rankings based on a single indicator.
• Any conclusion about an individual that they cannot see and contest.

Questions to ask before using a figure

• What is the source and when was it extracted?
• What is the counting rule?
• What is not covered?
• Has it been normalised, and how?
• Would the figure change if coverage were equal across the things compared?

For media

• State the source and date.
• Include coverage limitations — a figure without them overstates itself.
• Avoid ranking language where the data does not support ranking.
• Ask the data provider what the figure cannot show — most will tell you.
• Give the affected institution or person the chance to respond before publishing anything critical.

For institutions

• Verify figures against your own records before using them.
• State the counting rule with every published figure.
• Do not use external data for individual assessment.
• Report errors to the source rather than only correcting locally.
• Keep your own affiliation data accurate — it feeds everyone's figures.

For researchers using it in studies

• Document the extraction date and query.
• Report coverage limitations in the methods.
• Note that databases update retroactively — figures change.
• Share the extracted dataset where licensing permits.
• State the counting rule explicitly.

One thing worth remembering

Before using any academic data figure, ask what the number cannot show.

Most misuse comes from treating a figure as answering a question it was never capable of answering — and asking that one question in advance prevents nearly all of it.

Câu hỏi thường gặp