NIST Report on Challenges in Monitoring Deployed AI Systems

NIST Publications

Published March 6th, 2026

Detected March 7th, 2026

Summary

The National Institute of Standards and Technology (NIST) has published a report detailing challenges in monitoring deployed artificial intelligence (AI) systems. The report, based on practitioner workshops and literature review, identifies gaps and opportunities for innovation in post-deployment AI monitoring methodologies.

View original document View source feed page

What changed

NIST has released a report titled "Challenges to the monitoring of deployed AI systems: Center for AI Standards and Innovation." This publication highlights the critical need for robust post-deployment monitoring of AI systems, which are increasingly integrated into commercial and government applications. The report identifies that while pre-deployment evaluations are common, post-deployment monitoring is essential for validating real-world performance, tracking unforeseen outputs, and understanding unexpected consequences. It proposes monitoring categories and outlines challenges, noting a lack of best practices, validated methodologies, and common terminology across the field.

The report's findings are rooted in practitioner workshops and a literature review, emphasizing that stakeholders across the AI ecosystem agree on the necessity of post-deployment monitoring. The identified gaps and barriers point to opportunities for further investigation and innovation, with practitioners repeatedly calling for guidance on monitoring methods. While this document is a report and not a regulation, it signals a growing focus on AI governance and may inform future regulatory or guidance development, requiring technology companies and government agencies to consider their current AI monitoring practices.

What to do next

Review NIST report on AI system monitoring challenges.
Assess current post-deployment AI monitoring practices against identified gaps.
Stay informed on potential future guidance or standards related to AI monitoring.

Source document (simplified)

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock () or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

PUBLICATIONS

Challenges to the monitoring of deployed AI systems: Center for AI Standards and Innovation

Published

March 6, 2026

Author(s)

Anita Rao, Andrew Keller, Neha Kalra, Ryan Steed, Kweku Kwegyir-Aggrey, Kevin Klyman, Diane Staheli, Amanda Bergman

Abstract

As artificial intelligence (AI) systems are increasingly integrated into commercial and government applications, there is a growing need to monitor these systems in real-world settings. Although pre-deployment evaluations are valuable for assessing AI system capabilities at multiple points prior to release, they are predominantly conducted in controlled testing environments. Post-deployment monitoring is crucial for (1) validating that AI systems operate reliably as expected in real-world scenarios, (2) tracking unforeseen outputs that occur due to, e.g., model non-determinism or dynamic input conditions, and (3) visibility into unexpected consequences of AI systems in deployment contexts. Stakeholders across the AI ecosystem agree on the need for post-deployment monitoring; however, monitoring best practices, validated methodologies, and common terminology are still nascent and scattered across the field. This report proposes monitoring categories and surfaces challenges to robust post-deployment AI system monitoring, rooted in practitioner workshops and a literature review. The identified gaps, barriers, and open questions highlight opportunities for further investigation and innovation. Notably, this report quotes practitioners' repeated calls for guidance on post-deployment AI system monitoring methods. Citation NIST Trustworthy and Responsible AI - 800-4 Report Number 800-4 Pub Type NIST Pubs

Download Paper

https://doi.org/10.6028/NIST.AI.800-4 Local Download

Keywords

post-deployment monitoring, continuous monitoring, artificial intelligence Artificial intelligence

Citation

Rao, A.
, Keller, A.
, Kalra, N.
, Steed, R.
, Kwegyir-Aggrey, K.
, Klyman, K.
, Staheli, D.
and Bergman, A.

(2026),
Challenges to the monitoring of deployed AI systems: Center for AI Standards and Innovation, NIST Trustworthy and Responsible AI, National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.AI.800-4, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=961461       
  (Accessed March 7, 2026)