Automated checks confirm that data is well-formed. They cannot confirm that it is right.
A record with a valid identifier, a plausible date and a correctly formatted affiliation can still describe the wrong person entirely — and nothing in the pipeline will say so.
What automation catches
- Format and validity errors.
- Missing required fields.
- Referential integrity failures.
- Implausible values.
- Duplicates by exact match.
- Necessary, and not sufficient.
What only manual audit catches
- Correct-looking values that are wrong.
- Incorrect merges.
- Systematic bias in a matching rule.
- Coverage gaps in particular categories.
- Misinterpretation of a source field's meaning.
Designing the audit
- Sample across categories, not only recent additions.
- Include high-impact records — heavily cited works, large institutions.
- Include records the automation flagged as low confidence.
- Include records it flagged as high confidence — this is where overconfidence hides.
- Verify against the original source by hand.
What to record
- Error rate by field and by category.
- Error type, not only count.
- Whether the automation had flagged it.
- The apparent cause.
- Cause is more useful than incidence.
Acting on findings
- Fix the cause before the instances.
- Adjust thresholds where confidence is miscalibrated.
- Add automated checks for error types found repeatedly.
- Reprocess the affected records.
- Publish the audit summary — it builds trust rather than undermining it.
Frequency
- Regular sampling, not only when something is reported.
- After any pipeline change.
- After adding or changing a source.
- Track error rates over time — the trend matters more than any single figure.
One thing worth remembering
Sample records the automation marked as high confidence.
Low-confidence records are already being reviewed. Overconfident errors are the ones nobody is looking at — which is exactly why they are the ones still in the data.
Câu hỏi thường gặp
What does automation catch?
Format and validity errors, missing required fields, referential integrity failures, implausible values and exact duplicates — necessary but not sufficient.
What only manual audit catches?
Correct-looking values that are wrong, incorrect merges, systematic bias in matching rules, coverage gaps in particular categories, and misinterpreted source fields.
How should an audit be designed?
Sample across categories not only recent additions, include high-impact records, include both low and high confidence flagged records, and verify against original sources by hand.
What should be recorded?
Error rate by field and category, error type rather than only count, whether automation had flagged it, and the apparent cause — which is more useful than incidence.
Which records should be sampled deliberately?
Those the automation marked as high confidence — low-confidence records are already reviewed, so overconfident errors are the ones nobody is looking at.
