PDF Crawler: Finding and checking the accessibility of PDF documents across government websites

The problem

Government websites host large numbers of PDF documents — forms, guidance, reports, policy papers, meeting minutes. For many people, these documents are their primary point of access to important government information.

But PDFs are often created without considering whether they will work for someone using assistive technology. A PDF document that looks clear on screen can still be unreadable for a screen reader. The document may lack a title, making it hard to identify. It may not specify what language it is written in, which prevents screen readers from pronouncing words correctly. For longer documents, the absence of bookmarks means there is no way for someone using assistive technology to navigate between sections. Any of these issues can make a document that is perfectly usable for one person completely inaccessible for another.

PDFs were initially a simplified version of what would be sent to a printer via PostScript. It was never intended to carry semantic meaning, unlike formats like HTML. PDFs have become too easy to produce and very hard to make accessible.

When government teams check their websites for accessibility, the focus tends to be on the web pages themselves. The PDF files and other documents that those pages link to are often overlooked — and most web accessibility scanning tools are not designed to find or check them. A website can pass many accessibility checks while the documents it hosts remain inaccessible. While there are proprietary tools that provide more in-depth scanning of accessibility issues, they generally need to be run on one document at a time.

For a team managing a site that hosts hundreds or thousands of PDFs, working out which ones have problems is a substantial task. Unless someone specifically opens and tests each file, there is no straightforward way to know which documents are accessible and which are not. The UK government is the first one to have actively discouraged employees from publishing PDFs to the web, but for most governments, PDFs remain a key way documents reach the public — and there is still no simple way to check them at scale.

The solution

The PDF Crawler was built to address this gap. It is a free, open-source tool that automatically searches through a website, finds every PDF file it contains, and produces a report on each document's accessibility.

The process starts with a user submitting a web address — either through a simple online form or by creating an issue request directly on GitHub, the platform where the tool is hosted. The tool then crawls the website, systematically visiting pages and following links until it has located the PDF files. This is a free tool, so large sites may hit limits on the number of URLs or PDFs. Users can choose to set a limit on how many pages the tool visits. If a scan runs out of time, the tool automatically reduces the scope and explains what happened, so the user can retry with a smaller batch.

Once the PDFs are found, the tool checks each one against a set of accessibility criteria based on internationally recognised standards (WCAG and EN 301 549). It checks whether the document is tagged (a technical requirement that allows assistive technology to understand the document's structure), whether it contains real text rather than just images of text, whether it has a title, whether it specifies a language, and whether longer documents include bookmarks for navigation. The tool also tracks each document by a unique fingerprint, so that files that have not changed since the last scan are skipped automatically — saving time on repeat scans.

The results are published as a public report in both a readable summary and a structured data format that can be used for further analysis. The reports are available online, so anyone can view them without needing to request access or receive an attachment. After analysis, the PDF files themselves are deleted to keep storage requirements low — only the results are kept.

Passing every automated check is a good starting point, but it is not a complete accessibility audit. Automated tools cannot evaluate reading order, whether link text is meaningful, whether heading levels are used appropriately, whether table headers are correctly associated with their data, or whether form fields are accessible. A document that passes all the automated checks may still have accessibility problems that only a person familiar with accessibility requirements would notice. The tool's reports make this clear, distinguishing between automated checks, deeper technical validation, and the manual review that is still needed for a full assessment.

The PDF Crawler runs entirely on free, publicly available infrastructure — GitHub Actions for running the scans and GitHub Pages for publishing the submission form and results. Tools like GitHub Copilot and Claude were used for writing code, improving workflows, and drafting documentation. No software needs to be purchased, no servers need to be maintained, and no external provider needs to be contracted, which means that the government team can set it up and run it using tools they may already have access to, at no cost.

The PDF Crawler is in active development. Core functionality — submitting a scan, crawling a website for PDFs, running accessibility checks, and publishing public reports — is working. Current limitations include the handling of very large sites or very large PDF files, which can cause scans to run out of time. The tool addresses this by automatically adjusting the scope when a timeout occurs. The tool is open-source and available for any government team or organisation to use, adapt, or build on.