Auditing LLMs — A Three-Layered Approach a board-ready LLM audit checklist

Why it matters for NETEVO

Auditing Large Language Models: A Three-Layered Approach (Mökander, Schuett, Kirk and Floridi, 2024) proposes three coordinated audit layers — governance audits of provider accountability structures and quality-management systems, model audits of LLM capabilities and limitations between pre-training and release, and application audits of products built on LLMs, covering legal compliance and impact — and argues that the three layers must operate as a single instrumented loop rather than as independent activities.

The layers are interconnected, not siloed. Output from one audit becomes input to the next: governance findings inform model audits, model-limitation reports shape application audits, and application-layer operational logs feed back into the governance layer. This is the audit-side mirror of the integrated-management-system thesis the Integrated Management Systems Practical Guide establishes — one engineered system, multiple normative regimes, observed end-to-end.

Governance audits live at the management-system level. The paper places software development processes and quality management systems squarely inside the governance-audit layer — the same surface on which the Law-to-Code Methodology operates. AS ISO/IEC 42001 provides the management-system shell; this paper supplies the audit-side application.

Ex-ante and ex-post assessment together. The paper argues against audit-as-snapshot and for audit as a continuous, telemetry-driven activity. Continuous audit presupposes controls that emit evidence: an executable control produces the telemetry a continuous audit consumes, where a paper policy does not. The five institutional-arrangement archetypes the paper proposes for who audits whom also map cleanly onto Australia's emerging AI assurance landscape.