AI Answers: Canada’s public trial of an AI assistant that draws only on official federal pages

The Problem

A single web address, Canada.ca, is the Government of Canada's online front door, and Service Canada maintains it. Service Canada is the arm of Employment and Social Development Canada (ESDC) through which people reach many federal programmes and payments in one place. Behind that one address sits material from many separate federal institutions, on subjects as different as immigration, taxes, benefits and health.

Feedback indicates that some visitors come away without the answers they were looking for. Every day, a comment box on Canada.ca pages receives upwards of 3,000 questions, many about immigration applications, benefit claims or tax accounts. The Canada.ca Experience Office, a team within Service Canada, traces much of this to the scale and intricacy of what is published there.

Self-service matters because government offices keep set opening hours, which many people cannot fit around their own commitments. When the website fails them, the Experience Office reports that people fall back on ringing or visiting in person, routes that are more expensive for government to run. The Experience Office has also seen people misread who qualifies for a programme or when a deadline falls, with consequences that can be expensive for them, and it connects these experiences with weaker public confidence in government.

Responsibility for public services in Canada is split between the federal government, provinces and territories, and municipalities. According to the Experience Office, people frequently cannot tell which government is responsible for a service, such as a driving licence, which adds to the difficulty of finding the right page.

Departments keep revising their pages, and the government has said that page-by-page improvement will not be enough on its own to help people through so much material.

The Solution

With page-by-page improvements unable to close the gap on their own, the Canada.ca Experience Office looked for a way to take people from a question straight to the information they need. In summer 2025, it began public testing of AI Answers, an assistant that covers content from across federal government websites. A visitor asks a question in their own words and receives a brief, plainly worded reply, written only from federal web content and ending with a single link onward to the page where they can take their next step. Behind the scenes, the assistant runs on a large language model (LLM), a type of AI trained on very large amounts of text so that it can interpret questions and write answers. The trials used GPT-4.1 hosted through Azure Canada, and the service moved to the newer GPT-5.1 family in March 2026. The tool is not locked to one supplier, and the Experience Office has tried models from more than one company.

Accuracy was the starting point: as its product manager has noted, federal communications policy makes each department's head of communications answer for published information being correct. On a fixed list of typical Canada.ca questions, the Experience Office judged about 56% of ChatGPT's replies correct, and it set its own target of above 90% of answers correct in a sample.

Before a question reaches the AI model, a pattern filter checks it for personal details such as phone numbers, email addresses and ID numbers, along with swearing and threats, and a second AI check, run in Canada, catches what the filter missed. A flagged question is discarded without being logged, and the user is asked to reword it.

The system converts each question into English and translates the reply back, which gave the best accuracy in testing. For every question, it searches government sites and reads the matching pages afresh, so replies reflect the latest published wording. Separate AI "agents", each an AI component with one job such as translation, search or drafting, handle these steps, which the Experience Office reports gives steadier answers at lower cost.

The system identifies which federal institution a question concerns and adds guidance written for it, and 10 institutions had tailored prompts during the trials. Departments can add their own guidance and resources, for example to route certain questions to an online form. Since August 2026, the model has also been shown up to three past answers experts have rated for similar questions, including ones marked wrong, to help it avoid repeating mistakes.

Replies run to no more than four sentences and one tested citation link. Sessions allow three questions, as accuracy tends to slip in longer conversations, and each question is capped at 260 characters, which makes prompt injection, where someone hides instructions in a question to make an AI misbehave, harder. Provincial questions, such as about driving licences, are pointed to provincial websites, and a notice explains that replies do not constitute official advice.

Subject experts from partner departments review answers inside the tool, scoring each sentence and the citation, with a target of a quarter of each institution's answers. Automated AI scoring is kept for reporting only, so the system learns only from human judgements.

The first trial ran on the Canada.ca sign-in page in June and July 2025. The second invited random visitors on 12 Canada.ca pages from 22 October to 7 November 2025. In the third, from 2 December 2025 to 9 January 2026, four partner departments placed invitation banners on 112 pages in both languages, covering Employment Insurance, public service pay and pensions, Indigenous Services Canada content, and flu and measles.

The Impact

1. Accuracy, as measured by department experts
Accuracy rose across the three documented trials, from 94% to 96.7%. For the third trial, experts, supported by AI-assisted checks, reviewed 1,155 sampled questions and found almost identical results in the two official languages: 96.9% in English and 96.1% in French. "Golden" answers, the Experience Office's label for replies that were entirely right and helpful, grew from 66% of those reviewed in the first trial to 80.5% in the second. Taking all the public trials together, the system card reports 96% accuracy across 2,500 questions reviewed by experts.

2. Use and user feedback
Across the three trials, people asked 7,851 questions in total. In the first trial, 88% of feedback was positive. Of those who left feedback, 22% reported that a phone call or office visit was no longer necessary, and 52% reported spending less time searching and reading. Positive feedback stood at 77% in the second trial and 65% in the third, by which point partner topics had made the questions more complex.

3. Questions across the whole of government
Although the first trial was offered only from the sign-in page, half of its questions concerned the 20 most-used government services on Canada.ca. Immigration was the most common subject in both of the first two trials. In the second trial, the content behind the answers came from 60 federal institutions.

4. Languages beyond English and French
The first trial received questions in 19 languages besides English and French, 8.4% of the total, and these answers were less accurate. The Experience Office then upgraded its translation and search. In the second trial, scores for 94 answers given in 22 such languages followed much the same pattern as scores for English and French.

5. Problems found in government web content
Reviewing the tool's answers led departments to problems in their own web pages. ESDC came across out-of-date guidance on submitting forms to Service Canada, along with missing information for employees. The Treasury Board Secretariat identified gaps in its content and instructions that needed plainer wording. Several teams then started work to fix these issues.

6. Influence on wider federal guidance
Guidance for departments designing AI help tools on Canada.ca, published in February 2026, says most teams should be able to serve their users through AI Answers, while leaving room for dedicated tools on topics outside its current reach.

7. Current status
The service is not yet in full public use. Public trials ran between June 2025 and January 2026, public testing paused in February 2026, and public access is currently restricted during infrastructure work. The project's code repository names the next step as a public pilot, and the Experience Office has said it plans a pilot, then wider rollout on Canada.ca.

Lessons Learned

• Measure accuracy with the people accountable for the content. Experts who know the programmes can judge whether an answer is right in ways a general reviewer cannot. Reviewers from partner departments grade answers line by line, and only their verdicts shape future answers.
• Narrow the tool deliberately. Tight limits on length, sources and conversation reduce the room for error. The service gives four sentences and one link and allows three questions per session. Even so, in the third trial, answers to follow-up questions were 2% more likely to include a mistake, and the Experience Office is looking into the cause.
• Stop personal information before it reaches the AI model, and tell people why. Blocking a question works better when users understand what happened. AI Answers combines a rule-based filter with an AI check hosted in Canada, keeps nothing from blocked questions, and shows people which words were flagged. In usability sessions, testers grasped why and resubmitted their questions.
• Expect an AI assistant to expose weaknesses in your website. An assistant is only as accurate as the pages it reads. Partner departments found outdated and unclear content through evaluation, and the Experience Office found that unclear source pages produced unclear answers. The Experience Office describes content and AI as needing to improve together over time.
• Give departments a direct role in a shared service. A single government-wide assistant can still reflect each department's expertise. Departments can add their own instructions, tools and files, and the partner trials were partly about learning how departments would run it day to day. When Canada's Budget 2025 was announced on 4 November 2025, the Experience Office set up Department of Finance instructions in under an hour.
• Test where and how people are invited to use the tool. Placement shapes who tries it. On the Contact page, banners at the bottom drew 2.5 times more clicks than banners at the top, and click-through on health pages stayed between 0.03% and 0.04%. The Experience Office also found that some users never saw the AI's clarifying questions further down the screen, and added a scroll prompt.
• Plan for people to ask about the AI itself. Curiosity about a new tool generates its own questions. In the second trial, 12% of questions asked about AI Answers itself, so the Experience Office wrote an "About" page and let the tool consult it when answering.
• Be open about known weaknesses. Publishing limitations helps others judge what would transfer. The project's own documentation notes that the system will often still reply when its search turns up weak results, and that it sometimes links a question to the wrong institution. Partners also asked for more precise citations, because links sometimes led to general topic pages when users needed a specific one.
• Keep the AI model replaceable. A design that is not tied to one provider makes upgrades easier to manage. AI Answers can run on models from several companies, and it moved from GPT-4.1 to the GPT-5.1 family in March 2026. Keeping a large bank of test questions allows the Experience Office to spot any drop in quality whenever prompts or models are updated.