Text Depot: A centralized AI search solution for city records and documents

The Problem

Public sector organisations generate an immense, continuously expanding volume of text-based information across multiple layers of operation. For years, the City of Edmonton faced severe operational friction due to high-value data being trapped inside disparate corporate silos. Critical organisational resources—ranging from Council report files and media releases to historic survey responses, bylaws, and memos—were stored independently across different software tools and web pages.

The baseline status quo forced policy analysts, city planners, and administrative leaders to spend an excessive amount of time manually sorting through separate systems like Google or the internal corporate intranet to track down historical precedents or public feedback. Under financial constraints that barred the procurement of expensive proprietary enterprise search licenses, the city needed a method to consolidate millions of records. The core structural challenge was to design a platform capable of handling unstructured text documents while maintaining strict data governance, local context relevance, and zero licensing fees.

The Solution

The Data Science and Research team engineered and scaled Text Depot—a centralised, context-aware semantic search platform tailored specifically for internal municipal operations.

Part 1: Technology Stack and Open Source Infrastructure

To ensure cost efficiency, the platform was constructed entirely from open-source technologies:

• Database Engine (Elasticsearch): Serves as the high-availability, distributed core optimised specifically for fast querying, text aggregations, and rapid document parsing across millions of entries.
• Web Framework (R/Shiny & ShinyProxy): The entire front-end user interface was coded using the R programming language and deployed via Shiny, bridging mathematical data analysis capabilities with a clean web environment. ShinyProxy containerises individual sessions to safely allocate resources for active internal users.
• Containerization & Operations: Managed through a localised Docker Swarm cluster running cron jobs for nightly automated parsing across municipal data endpoints.

Part 2: Implementation Mechanics, AI Annotations, and Security

To transform raw documents into searchable intelligence, the team built a uniform ingestion pipeline where every record is stored in a standardised format, including the full document text, source titles, dates, parent document items (where available), and URLs. As data flows into the Elasticsearch index, the system attaches automated metadata layer annotations using advanced NLP models:

• AI-Enabled Conceptual Search: To ensure broad semantic accuracy, our system currently utilises a generalised pretrained embedding model. This foundational approach ensures that standard concepts, such as identifying synonyms like "computer" and "laptop," are accurately interpreted during search. In addition, the roadmap includes implementing a hybrid search architecture. This future update will fuse results from the pretrained models with specialised local corpora, allowing the system to blend high-level accuracy with the ability to parse Edmonton-specific context, such as recognising operational similarities between municipal facilities like "Terwillegar" and "Clareview” (neighbourhoods in Edmonton).
• AI Feature Suite: Integrates automated sentiment mapping (tracking public tone over timelines), geographic extraction (neighbourhood tagging), and a Large Language Model (LLM) abstraction plugin that lets staff generate automated short summaries of massive reports at the click of a button.
• Enterprise Governance & Keycloak Security: To protect corporate information, Text Depot sits securely behind the city firewall, backed by Cloudflare WAF and uses Keycloak to handle granular, role-based dataset access control. No personal communication data (such as internal employee emails) is loaded into the platform.

The Impact

Text Depot successfully converted isolated data sources into an accessible centralised repository for city staff and senior administration.

1. Massive Scale Integration

Centralised over 1.4 million corporate data records, including approximately 40,000 Council reports, 30,000 daily media monitoring articles, thousands of media releases, active bylaws, carried Council motions, and historic engagement surveys.

2. Widespread User Adoption

Scaled to whole-organisation utility, reaching an active milestone of 1,373 unique internal users and logging over 9,500 total user sessions to date—with more than 5,500 sessions occurring since 2024 alone.

3. Informed Governance & Safeguards

Adopted directly by executive leadership and City Councillors to navigate contextual histories instantly. The internal tool architecture also successfully caught public-facing data leaks (re-securing legacy public tools like eScribe when sensitive fields inadvertently surfaced).

Lessons Learned

• Prioritise Internal Product Stewardship and Strict Governance. When handling data containing potentially identifiable public notes or historic surveys, organisations must institute granular, role-based access restrictions that undergo formal evaluation on an annual basis to maintain strict privacy compliance.

• Maximise Capital Efficiency with Built-in Tool Ecosystems. Edmonton demonstrated that an elite digital environment can be established using a completely open-source, non-proprietary tech stack (Elasticsearch, R/Shiny, Docker Swarm) without buying costly commercial end-to-end software suite licenses. Leveraging open-source code allows internal development teams to stay agile and continuously update analytical functions dynamically.

• Localise NLP Models to Understand Community Semantics. Standard, out-of-the-box AI tools fail to capture local institutional memory and context. Supplementing generic vector databases with custom models trained directly on specialised local vocabularies guarantees that regional concepts, neighbourhood groupings, and idiosyncratic civic terminology are successfully parsed to return hyper-accurate results.

Launch year: 2021