Methodology that cannot be examined cannot be challenged.
And data that nobody can challenge cannot be trusted — which makes documentation the mechanism by which a data source becomes accountable rather than merely assertive.
What it must contain
- Sources, and what each is used for.
- Precedence rules where sources disagree.
- Matching and disambiguation approach, including thresholds.
- Counting rules for every aggregate published.
- Coverage by field.
- Known limitations, stated plainly.
Level of detail
- Enough that someone could disagree specifically with a decision.
- Enough that a figure could be reproduced given the same data.
- Include the reasoning, not only the rule — that is what allows informed disagreement.
- Include what was considered and rejected.
- Vague methodology is functionally the same as none.
Keeping it current
- Version it alongside the data model.
- Update when rules change, not afterwards.
- Keep old versions available so past figures remain interpretable.
- Date every section.
- Documentation that has drifted from practice is worse than none, because it misleads.
Communicating limitations
- State coverage gaps by category.
- State where inference is used and how confident it is.
- State what the data should not be used for.
- Put these with the data, not only in a separate document.
- Users assume completeness unless told otherwise.
Making it usable
- Summary first, detail behind it.
- Plain language for the summary.
- Worked examples of how a figure is derived.
- Searchable and linked from the data itself.
- Documentation nobody can find serves no purpose.
Why it matters
- Users can judge whether the data suits their purpose.
- Errors can be identified by people outside the organisation.
- Figures become reproducible.
- Disagreement becomes specific rather than general.
- It is the difference between a data source and an assertion.
One thing worth remembering
Write the methodology so that someone could disagree with a specific decision in it.
Documentation general enough that no particular objection is possible is documentation that cannot be checked — which means the data behind it cannot be either.
Câu hỏi thường gặp
What must methodology documentation contain?
Sources and their uses, precedence rules where sources disagree, matching approach including thresholds, counting rules for every aggregate, coverage by field, and known limitations.
What level of detail is needed?
Enough that someone could disagree specifically with a decision and that a figure could be reproduced — including the reasoning and what was considered and rejected.
How is documentation kept current?
Version it alongside the data model, update when rules change rather than afterwards, keep old versions available, and date every section.
How should limitations be communicated?
State coverage gaps by category, where inference is used and how confident, and what the data should not be used for — placed with the data since users assume completeness.
What is the test of good methodology documentation?
That someone could disagree with a specific decision in it — documentation too general for particular objection cannot be checked.
