A key component of ISO 42001 certification is conducting an Artificial Intelligence Impact Assessment (AIIA). This assessment helps your organization identify how your AI program creates both opportunities and risks to relevant stakeholders and society at large. This assessment is vital to determine what resources are needed to address negative impacts to direct stakeholders and offers a fundamental mechanism to identify how your AIMS could create excessive issues for society in general.
The AI Impact Assessment is not optional. It is a core requirement of the AI governance program if ISO 42001 certification is your goal. We recommend performing the impact assessment as a fundamental first step (even before policy building). How exactly does one go about performing an AIIA?
Step 1: System Overview
This first step involves answering the “What” questions. What is the purpose of the AI system in question? What tasks does it perform? What data sources are involved? These foundational questions guide the process going forward.
Here is an example of common considerations for System Overview:
- AI System Name
- Owner / Department
- Primary Use Case
- AI Type / Technology (e.g., GenAI, machine learning, NLP, rule-based system)
- Deployment Stage
- Data Types Used (be especially diligent with sensitive data such as PHI/PII, PCI, credentials, etc)
- Dependencies (Consider vendors and tools)
Step 2: Stakeholder Impact
Once you understand the AI Management Systems and their purposes, it’s time to consider who is using these systems. There are two questions to answer here:
- Who is affected by the AIMS?
- What are the risks/benefits for each group?
Examples of considerations for Stakeholder Impact:
- Directly Affected Users: (e.g., clinicians, patients, coders)
- Indirectly Affected Groups: (e.g., IT, billing, compliance, patients, patient families)
- Potential Benefits: (e.g., faster diagnosis, improved billing accuracy)
- Potential Harms: (e.g., misclassification, data leakage, user overreliance)
Step 3: Risk Identification
There are a few core types of risk involved in the use of AI at scale: legal, ethical, operational, and cybersecurity. Each of these areas come with their own sets of challenges and potential consequences. Identifying these risks upfront allows you to develop effective mitigation plans.
Quick overview of risk types:
- Legal Risk: Usage of AI must remain within boundaries of regulations such as HIPAA, EU AI Act, GDPR, and other requirements.
- Ethical Risk: Biases, discrimination, autonomy and lack of transparency are all ethical risks that arise with AI use.
- Operational Risk: inefficient token usage, lack of privacy safeguards, opportunity cost of not using AI
- Cybersecurity risk: data leakage, shadow IT, availability, data integrity all need to be considered.
Step 4: Risk Evaluation
After identifying potential risks, the next step is to evaluate their severity. This step helps you prioritize and create actionable plans.
Consider the following elements:
- The Potential Threat Scenario (Model bias against age groups or genders)
- The Likelihood of the Threat Occurring (High/Medium/Low)
- The Impact of the Threat Occurring (High/Medium/Low)
- Treatment Plan (Accept, Avoid, Mitigate, Transfer)
Step 5: Mitigation & Controls
Having outlined the “what” and “who,” now it’s time to define the “how.” What strategies are you going to implement to be on guard? What controls can you put in place?
Examples of safeguards:
- Human-in-the-loop evals
- Regular model validation
- Explainability tools for AI outputs
- Robust Prompt Management
Step 6: Oversight & Accountability
A key requirement is identifying who is going to be responsible for managing AI risks. Develop a clear plan to assign accountability across departments such as legal, compliance, IT and clinical.
Create a regular cadence of time set aside to review timelines and check in with one another. This sounds obvious, but communication is vital for completing a successful AI Impact Assessment.
Step 7: Documentation and Reporting
The beauty of an AI Impact Assessment is that you are always stronger than the last one you conducted. By performing at least annually and at major architectural changes (i.e. design, implementation, and after major updates), you will be able to track your organization’s progress over time. You’ll also be able to monitor changes in model behavior, data sources, and risk ratings.
Conclusion
Ready to take control of your organization’s AI strategy the right way? Start your path to ISO 42001 certification by allowing Genius GRC to guide you in conducting an AI Impact Assessment and move forward with confidence.
