Chapter 3: AI Safety
IN THIS LESSON
To make AI safer for everyone, policy has to work at more than one point in the pipeline. It is not enough to regulate only the model after it is released, and it is not enough to rely only on voluntary promises from companies. The most effective approach is layered: set rules for how AI is designed, tested, deployed, monitored, and enforced after harm occurs. That basic logic is reflected in the NIST AI Risk Management Framework, the OECD AI Principles, and the EU AI Act, all of which treat AI safety as an ongoing governance problem rather than a one-time technical fix.
A good course-ready way to teach this is to start with the question: what kind of harm are we trying to prevent? Policy gets clearer once students see that AI can create different kinds of risk. Some systems produce false or dangerous information. Some discriminate in hiring, lending, housing, or education. Some invade privacy, manipulate users, or make decisions people cannot challenge. The OECD Principles frame trustworthy AI around human rights, transparency, robustness, accountability, and democratic values, while NIST’s generative AI profile highlights risks such as confabulation, harmful content, privacy problems, and loss of human control.
1) Design & development policy
This is where governments and institutions require companies to build safety in from the start. That includes documented risk assessments, secure data practices, testing for bias and dangerous failure modes, and clear internal responsibility for safety decisions.
NIST’s framework is especially useful here because it organizes governance around mapping risks, measuring them, managing them, and establishing accountability inside organizations. For a course, this is the layer where students can see that “safer AI” begins before a product reaches the public.
2) Pre-release testing & evaluation policy
One of the clearest policy recommendations is that powerful AI systems should be evaluated before deployment, not only after problems appear. That can include red-teaming, external audits, benchmarking for bias and reliability, and testing whether a system can help produce fraud, cyber abuse, or other harms.
NIST’s generative AI profile points to structured testing actions, and the EU AI Act builds this idea into a legal framework by imposing obligations tied to risk level, especially for higher-risk systems.
3) Deployment regs for high-risk uses
Not every AI application needs the same level of oversight. A homework chatbot and an AI system used in hiring or healthcare should not be treated the same way.
The EU AI Act is useful to teach here because it uses a risk-based model: some uses are prohibited, some are tightly regulated, and some face lighter transparency duties.
4) Transparency & user protection
Even when a system is allowed, users need to know what they are dealing with. Policy can require notice when people are interacting with AI, disclosure when content is AI-generated, explanation rights when important decisions are automated, and channels to contest errors.
The White House’s Blueprint for an AI Bill of Rights and related U.S. governance materials emphasize protections such as notice, explanation, human alternatives, and protection from algorithmic discrimination. These measures matter because safety is not only about preventing catastrophic failure. It is also about making ordinary people less vulnerable to hidden automation.
5) Post-release monitoring & enforcement
Even careful design and testing will miss some harms. That is why policy also needs incident reporting, whistleblower protections, recall authority, penalties for noncompliance, and agencies that can investigate complaints.
The EU has created an enforcement structure involving the AI Office and national authorities, which is a reminder that rules without oversight do not do much. OECD guidance published in 2026 also stresses due diligence across the life cycle, not just at launch.
-
Add a short summary or a list of helpful resources here.