Auditing LLMs — A Three-Layered Approach

Why it matters for NETEVO

NETEVO cites Auditing Large Language Models: A Three-Layered Approach (Mökander, Schuett, Kirk and Floridi, 2024) as the structural artefact behind the forthcoming AI-Washing Audit whitepaper. The paper proposes three coordinated audit layers — governance audits of provider accountability structures and quality-management systems, model audits of LLM capabilities and limitations between pre-training and release, and application audits of products built on LLMs — and argues that the three layers must operate as a single instrumented loop rather than as independent activities.

The layers are interconnected, not siloed. Output from one audit becomes input to the next: governance findings inform model audits, model-limitation reports shape application audits, and application-layer operational logs feed back into the governance layer. This is the audit-side mirror of the integrated-management-system thesis the Integrated Management Systems Practical Guide establishes — one engineered system, multiple normative regimes, observed end-to-end.

Governance audits live at the management-system level. The paper places software development processes and quality management systems squarely inside the governance-audit layer, which is the exact surface the NETEVO Law-to-Code Methodology engineers. AS ISO/IEC 42001 provides the management-system shell; this paper supplies the audit-side application.

Ex-ante and ex-post assessment together. The model argues against audit-as-snapshot and for audit as a continuous, telemetry-driven activity — the position the forthcoming AI-Washing Audit whitepaper takes in favour of executable controls over paper policies. The five institutional-arrangement archetypes the paper proposes for who audits whom also map cleanly onto Australia's emerging AI assurance landscape, anchoring NETEVO's AI Governance in ANZ framing.