I recently had the opportunity to chair an AI Working Group focused on developing policy within a large organisation. It was a genuinely interesting experience, not just because of the topic, but because of what it revealed.

We had around 20 people in the group. Smart, experienced, highly capable professionals from across the organisation. People who understand risk, governance, operations, and strategy deeply within their domains.

But here’s the reality.

Only about six of those twenty actively contributed to the discussion in a meaningful way.

The rest weren’t disengaged. They weren’t uninterested. They simply didn’t understand how AI actually works well enough to contribute with confidence.

And that’s not a criticism. It’s a signal.

The uncomfortable gap

AI policy sounds like something that can be written from first principles. Governance frameworks, ethical guidelines, accountability structures. These are familiar territories for many organisations.

But AI is not a traditional technology.

It behaves differently. It fails differently. It evolves quickly. And increasingly, it acts.

When you don’t understand those differences, policy starts to drift into abstraction:

  • "AI must be fair"

  • "AI must be transparent"

  • "AI must be used responsibly"

All valid. All important.

But none of them tell a team what to actually do when deploying a model, building an agent, or integrating automation into workflows.

The shift most organisations are missing

AI is no longer just a tool.

It is becoming a virtual employee.

It can act, perform tasks, interact with systems, and influence outcomes.

And yet, we are not managing it like one.

We don’t:

  • onboard it

  • define its permissions clearly

  • supervise its actions in structured ways

  • or hold it accountable through measurable controls

This is where governance starts to break down.

The new structure… and the same risk

Before the current AI policy has even been rolled out, there is now a move to restructure the original working group into:

  • an AI Governance Group

  • an AI Champions Group to uplift capability across staff

On the surface, both are positive steps.

But there is a risk hiding in plain sight.

If the governance group itself lacks a foundational understanding of how AI systems actually operate, then decisions will be made without visibility of:

  • how agents execute actions

  • how data flows through systems

  • where security risks emerge

  • how models fail in real-world conditions

This is not hypothetical.

We are already seeing AI systems that:

  • call APIs

  • write and execute code

  • interact with internal systems

  • automate workflows end-to-end

At that point, governance is no longer about reviewing a tool.

It is about governing a system actor.

Why technical insight matters

This doesn’t mean policymakers need to become engineers.

But governance cannot operate in isolation from technical reality.

Without engineering input, it is very easy to:

  • approve architectures that expose sensitive data

  • overlook risks like prompt injection or agent misuse

  • apply controls in the wrong places

  • or create policies that are impossible to implement

Understanding concepts like:

  • where an LLM runs (local vs external)

  • what access it has (files, APIs, systems)

  • how outputs can trigger actions

  • and how it can be manipulated

…is no longer optional for effective governance.

The role of AI Champions

The idea of an AI Champions Group is a strong one.

Upskilling the broader workforce is essential.

But there is a difference between:

  • helping staff use AI

  • and ensuring the organisation can govern AI safely

Champions can drive adoption.

They cannot replace the need for deep technical insight in governance decisions.

What needs to change

If we want AI governance to be effective, not just well-intentioned, a few things need to happen:

  1. Embed technical expertise into governance
    Engineers and practitioners need a seat at the table, not as advisors on the side, but as core contributors.

  2. Lift baseline AI literacy for decision-makers\ Not deep technical training, but enough understanding to ask the right questions and recognise real risks.

  3. Shift the mindset\ From governing tools to governing system actors.

  4. Test policy against real systems\ This was one of the most surprising gaps. During the working group, there was no clear push to test the policy against real-world tools or implementations to expose where it might break.

    If policy can’t be applied to an actual system, it isn’t ready. Full stop.

    Real systems reveal things policy alone cannot:

    • hidden failure modes

    • gaps between intent and implementation

    • security and data risks that only appear in practice

    Without this step, policy risks becoming theoretical rather than operational.

Final thought

We are moving quickly into a world where AI doesn’t just assist work, it performs it.

If governance structures don’t evolve alongside that reality, we risk building frameworks that look strong on paper but fail in practice.

AI governance is not just a policy problem.

It is a collaboration problem between policy, domain expertise, and engineering.

And right now, that collaboration gap is where the real risk sits.