The community question: data quality in action

A member of Apolitical’s Data Skills Network asked the difficult "now what?" of starting a quality practice from scratch in a small agency. Community members from Australia, Canada, Europe, and beyond contributed real-world templates and strategies for tackling decades of legacy records. This article shares the key takeaways and resources from the conversation.

I'm setting up from scratch a data governance practice and one of the things I'm struggling a bit with is the implementation of data quality as a practice. Whilst we have set up metrics and obviously the quality dimensions, it is with the "now what?" type of questions that I am grappling with. For context, I am in a small state government agency where governance wasn't really a thing until recently so even if we had the budget (which we don't, albeit we could) I am not sure the current state of play would benefit from a data cataloguing tool yet.

1. Starting small and staying proactive

The transition from defining quality dimensions (like accuracy and completeness) to actually implementing them is often the hardest hurdle. Community members suggested that for small agencies, governance should be built organically rather than imposed from the top down. Members emphasised that a 'Start, Stop, Continue' exercise with stakeholders to identify which existing data collection processes are working and which are simply creating "mud" for analysts to wade through.

2. Managing the "Inflow" vs. the "Archive"

Community members highlighted a dual challenge: managing the daily arrival of new data while cleaning historical records that, in some cases, date back to the 19th century. To tackle the former, members suggested enforcing quality at the point of entry through strict templates and public-facing forms with limited free-text fields. For the latter, some agencies are experimenting with automated bots that suggest corrections to historical data, though always with a human in the loop to make the final decision.

3. The human element: stewards and stakeholders

Data quality is not just a technical task but a cultural one. Community members emphasised that while not everyone is responsible for maintaining a dataset, everyone should be concerned with its quality. To gain buy-in from 'foot soldiers,' public servants are encouraged to celebrate small wins loudly. This builds the necessary momentum to overcome the discouragement that often comes when staff are faced with massive amounts of poor-quality legacy data.

4. Setting thresholds and rules

Before investing in expensive software, community members suggested establishing ground rules for what good data looks like. Examples include ensuring postcodes are valid, names are not missing, and IDs are not duplicated. By setting clear thresholds (for example, 99% completeness for a specific field), organisations can run profile jobs on their data to determine how closely it meets the required standards.

Resources shared by community members

Community members shared several high-level resources to help align local practices with international standards:

• Statistics Canada Quality Guidelines: A comprehensive Quality Assurance Framework and Quality Guidelines focusing on six dimensions: relevance, accuracy, timeliness, accessibility, interpretability, and coherence.
• FAIR Principles: A framework designed to ensure data is Findable, Accessible, Interoperable, and Reusable.
• Global Social Protection Data Standard: A Data Governance Framework specifically for the social protection sector.

Join the conversation

To participate in conversations like this, join the Data Skills Network.

Join the community

This article was written with the help of AI using anonymised conversation text and edited by a human before publishing.