The gap between AI governance on paper and AI governance in systems

Why your AI governance program should focus on engineering-embedded controls.

Over the past two years many organisations have made progress establishing AI governance and risk management aligned to frameworks such as the DISR AI Ethics Principles, Voluntary AI Safety Standard or ISO 42001 AI Management System. Policies have been drafted, responsible AI principles published, impact assessment templates introduced, and inventories of AI systems created. Boards and executive teams are increasingly aware of the strategic, ethical and operational risks associated with AI technologies.

This work is valuable. Paper-based governance frameworks are particularly effective at clarifying roles, responsibilities and accountability. They establish who is responsible for approving AI systems, who owns the associated risks, and what processes must be followed before deployment. In large organisations, that clarity is important.

However, there is an emerging realisation across cyber security, technology and data teams: much of this governance activity merely documents AI risks after the fact rather than materially reducing them.

Policies, assessment forms and governance committees tend to operate around AI systems rather than within them. They describe acceptable use, identify potential harms, and assign oversight responsibilities. But they do not directly measure or change how models are built, how data flows through systems, or how users interact with AI tools in day-to-day work.

This creates a familiar pattern seen in other technology domains. Compliance artefacts accumulate, yet the underlying systems and behaviours remain largely unchanged. This is the AI equivalent of having a detailed password policy that nobody enforces technically; the policy document may exist, but weak passwords persist.

Paper-based governance can therefore struggle to achieve what most organisations actually want: safer, higher-quality and more reliable AI systems.

The shift towards engineering-embedded governance

AI risk is ultimately created through technical design decisions: how models are trained, what data they use, how they are integrated into operational processes and decision making, and how users interact with them. As a result, many of the most effective AI risk controls are not policy statements but engineering controls embedded directly in systems and processes.

This mirrors a shift that occurred in cyber security over the past decade. Security once relied heavily on policies and periodic audits. Today, leading organisations embed security controls into development pipelines, infrastructure and platforms. Security is enforced automatically through technology rather than relying solely on human compliance.

AI governance is beginning to follow the same trajectory.

Controls that are embedded into systems change behaviour by default. They prevent risky activity from occurring in the first place or detect problems early in the lifecycle. In contrast, controls that rely solely on documentation and process reviews often depend on individuals remembering to follow procedures under time pressure. For example, while a risk assessment template may anecdotally ask a developer whether they’ve checked for training data leakage, an automated testing pipeline check confirms this prior to deployment.

Embedding governance into technology therefore tends to produce greater and more consistent risk reduction.

Where engineering controls can make an immediate practical difference

Two areas stand out where organisations can introduce stronger system-level AI controls.

1. Browser, endpoint and network controls

One of the most common AI risks today arises not from formal AI projects, but from employees informally using public AI tools. Netskope’s 2026 Cloud & Threat Report data shows that at the end of 2025, roughly half of all enterprise network traffic to third party AI services was via non-enterprise accounts, meaning employees were using personal logins rather than company-managed accounts and taking data outside the control of their organisation. Staff may paste sensitive documents into chatbots, use external tools to generate code that is later deployed internally, or rely on AI-generated analysis without validation.

Many organisations address these risks through policies that prohibit unauthorised AI use. However, policies alone rarely prevent undesirable behaviour.

Technical controls can help enforce these expectations more reliably. For example:

• Web proxies, secure access service edge (SASE) and/or cloud access security brokers (CASB) can restrict access to unapproved AI platforms.
• Endpoint controls can prevent sensitive data from being uploaded to external AI services.
• Browser extensions can warn users when they attempt to paste corporate data into public AI tools.
• Network monitoring can identify high-volume AI tool usage that may indicate shadow AI adoption.

These measures do not eliminate all risk, but they significantly reduce the likelihood of uncontrolled AI usage across the organisation. It is important that these controls are implemented with a thorough understanding of how business workflows function, to avoid unintentionally creating friction and employee frustration.

2. ModelOps integrated into technology delivery pipelines

The second opportunity lies within the software development and technology change lifecycle itself.

AI systems introduce risks that traditional software governance processes are not always designed to address. These include issues such as poor data quality, biased training datasets, unpredictable behaviour from large language models, and vulnerabilities such as arbitrary execution via prompt injection attacks. Mistakes such as reusing training data in testing data sets can result in over-optimistic accuracy results and models that are prone to misclassify. Small variations in system instructions or LLM temperature adjustment can produce very different user experiences when applied to customer-facing chat agents.

ModelOps practices integrate AI validation directly into engineering pipelines so that models are tested and evaluated before they are deployed into production systems.

Examples of embedded controls include:

• Data validation pipelines that automatically detect anomalies, missing values or sensitive information in training datasets,
• Model evaluation tests that measure accuracy, fairness and robustness across different scenarios,
• Behavioural testing for large language models, checking for hallucinations, unsafe outputs or failures to follow system instructions,
• LLM safety and security testing, including automated attempts to bypass guardrails or exploit prompt injection vulnerabilities,
• Container image validation, static and dynamic source code analysis, dependency checks for vulnerable libraries and checks for hardcoded secrets, and
• Model integrity and provenance checks to tie a packaged model to a known training run and model version, and functional smoke testing from hold-out data to compare accuracy, latency and output characteristics to previously-collected evaluation stage results.

When integrated into CI/CD pipelines, these tests act as automated quality gates. Models that fail defined thresholds cannot progress into production environments until issues are addressed.

The Voluntary AI Safety Standard Guardrail 4 and Australian Government AI Technical Standard (Statements 26-32, in particular) provide helpful guidance on establishing testing and continuous integration practices for AI deployments.

The importance of testing and monitoring

In practice, the areas where organisations most often struggle are pre-deployment testing and ongoing monitoring.

AI systems behave differently from traditional software. They operate probabilistically, meaning outputs are not always deterministic. Their performance also depends heavily on the data they encounter in real-world environments.

As a result, organisations deploying AI into operational processes need to understand how models, data and integrated applications interact to influence performance outcomes, typical error rates and their underlying causes, the edge cases where models perform poorly, and the situations where safety guardrails fail or produce unintended outputs.

Testing methodologies provide the most direct insight into these behaviours.

For example, organisations can run structured test suites against AI systems to measure how often a model produces incorrect responses, how it handles ambiguous inputs, or whether certain prompt strategies can bypass safety controls. Security teams can simulate adversarial inputs to identify vulnerabilities before attackers do.

Integration testing between applications and AI models (e.g. frontend web applications, or applications accessible via MCP interfaces) is also essential to identify a range of risks that arise from the way AI models interact with other components of the technology stack. These scenarios can include input mishandling, truncation and other data transformations that degrade model performance, or access control logic flaws that enable RAG-systems to retrieve data they shouldn’t have access to. 

Organisations should make similar enquiries of vendors of proprietary AI solutions. An informed consumer of AI software-as-a-service should be making enquiries about the testing methodologies the vendor followed and the metrics vendors use to assess functional performance. Industry-standard benchmarks exist for evaluating different types of AI capability, buyers should ask vendors which ones they’ve tested against and what the results showed. By their non-deterministic nature, no AI solution performs perfectly, so responsible vendors should be able to disclose scenarios where their solutions reach performance limitations and failure modes.

Equally important is continuous monitoring after deployment. AI systems can degrade over time as data distributions change or as users interact with them in ways designers did not anticipate, a phenomenon known as model drift.

Monitoring systems can detect patterns such as rising error rates, unexpected output types, or repeated attempts to bypass safeguards. These signals provide early warning that systems may require retraining, adjustment or additional controls.

Most importantly, testing and monitoring help organisations develop a realistic understanding of how their AI systems actually behave. This insight enables two critical responses: improving the systems themselves, or putting in place operational safeguards to escalate and address issues when they occur.

Moving from governance on paper to governance in systems

None of this means policies and governance frameworks are unnecessary. Organisations still need clear accountability structures, ethical guidelines and risk management processes.

However, the organisations that manage AI risk most effectively will be those that move beyond governance on paper and begin embedding governance directly into their engineering platforms, development pipelines and operational controls.

One practical starting point to consider: pick one of the engineering controls described above and implement it over the next three months to start narrowing the gap between your AI policy and your AI reality.