Capability records, perspective 7: Trust, security, and resilience
Part of the SEI AI Adoption Framework (AIAF).
7.1 AI threat modeling
Purpose
Prevent AI designs and operations from being protected against ordinary threats only, while threats specific to models, data, prompts, retrieval, tools, agents, users, providers, and external parties are left unconsidered.
Intended outcome
Design and operations address threats arising from models, data, prompts, retrieval, tools, agents, users, providers, and external parties.
Decisions enabled
- which threats apply to a use;
- which mitigations are required;
- which residual risks are accepted and by whom; and
- when the threat model must be reviewed.
Applicability
Workloads: All
Delivery modes: All
Typical state: Required
Key activities
- describe the system, its data flows, and trust boundaries, including retrieval, tools, and agents;
- identify threats such as prompt injection, data poisoning, leakage, abuse of tools, impersonation, and provider compromise;
- assess likelihood and consequence;
- select mitigations and assign owners;
- record residual risks and acceptance; and
- review the model when the system, threats, or incidents change.
Required evidence
- threat model with trust boundaries;
- mitigations and owners;
- residual-risk decisions;
- review records; and
- link to adversarial testing and incidents.
Common failure patterns
- threat modeling done once at the start;
- models that omit agents and retrieved content;
- mitigations listed without owners;
- residual risk accepted by the builder; and
- findings from incidents not fed back.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Threats are considered informally by individual security staff or builders. |
| Defined | A threat model with mitigations and owners exists for in-scope systems. |
| Managed | Models cover AI-specific threats, residual risks are accepted by accountable owners, and reviews occur on change. |
| Adaptive | Threat modeling adapts to new attack techniques and incident findings, and the organization uses it to shape design and testing early. |
Review and exit triggers
- system, data flow, tool, or authority changes;
- a new class of attack becomes known;
- an incident or test reveals an unmodeled threat;
- a provider changes architecture; or
- a residual-risk decision expires.
7.2 Agent identity, authority, and tool access
Full record: worked example 4 of 5 in Part 3, Capability model. The catalog entry in Part 4 gives the intended outcome, workloads, delivery modes, and condition.
7.3 AI interaction and action records
Purpose
Prevent investigation, accountability, and dispute handling from failing because records of AI interactions and actions are missing, unreliable, or unprotected, or conversely kept so broadly that they create privacy and security exposure.
Intended outcome
Records support investigation, evaluation, accountability, dispute handling, and improvement while respecting privacy and retention rules.
Decisions enabled
- which interactions and actions are recorded and how much detail is kept;
- who may access the records and under what conditions;
- how long they are kept; and
- how integrity is protected.
Applicability
Workloads: All
Delivery modes: All
Condition: Applied proportionately to consequence
Typical state: Required, with depth based on consequence
Key activities
- decide logging scope and detail in proportion to consequence;
- include inputs, outputs, tool calls, actions, approvals, and the configuration in force where needed;
- protect records from alteration and limit access;
- apply privacy, minimization, and retention rules;
- provide the means to search and retrieve for investigation and disputes; and
- test integrity and access controls.
Required evidence
- logging policy and scope;
- protected records;
- access records;
- integrity and retrieval tests; and
- retention and disposal records.
Common failure patterns
- logging nothing for agent actions;
- logging everything with no access controls;
- logs that can be edited by those they concern;
- records that cannot be matched to a person or decision; and
- retention conflicts unresolved.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Records are whatever the tool produces, and access and retention are not decided. |
| Defined | Logging scope, access, and retention are defined for the uses where consequence calls for records. |
| Managed | Records are protected, searchable, and tested, and privacy limits are applied. |
| Adaptive | Logging adapts to investigation and dispute experience, and the organization reconstructs events promptly with the least data necessary. |
Review and exit triggers
- an investigation cannot be completed for lack of records;
- an integrity or access test fails;
- authority or consequence of the AI increases;
- privacy or retention requirements change; or
- log volume or cost becomes unmanageable.
7.4 Model, data, and component supply-chain assurance
Purpose
Prevent the organization from using models, data, and components whose origin, terms, dependencies, or vulnerabilities it does not know.
Intended outcome
The organization understands the origin, terms, dependencies, vulnerabilities, and changes of important AI components.
Decisions enabled
- whether a component may be used;
- what verification it needs;
- what to do about a vulnerability or a change of terms; and
- which components to replace.
Applicability
Workloads: All
Delivery modes: All
Condition: Modes A and D directly; Modes B and C when they depend on a provider
Typical state: Conditional
Key activities
- inventory models, datasets, libraries, tools, and other components with versions and origins;
- check licenses and terms, including restrictions on use and redistribution;
- assess provenance and integrity, for example through signatures and trusted sources;
- review vulnerabilities and respond to advisories;
- track changes in components and their dependencies; and
- where components are supplied by a provider, obtain evidence about their supply chain.
Required evidence
- component inventory with versions;
- provenance and integrity checks;
- license and terms review;
- vulnerability review results; and
- provider supply-chain evidence where relevant.
Common failure patterns
- models downloaded from unverified sources;
- license terms for weights or data ignored;
- a plugin or tool added with unknown dependencies;
- vulnerabilities recorded and not fixed; and
- no record of what a provider's service relies upon.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Components are chosen by builders, and origin and terms are not recorded. |
| Defined | Components are inventoried with source, version, and license for in-scope uses. |
| Managed | Provenance and vulnerabilities are checked and tracked, and changes lead to review. |
| Adaptive | Assurance adapts to emerging supply-chain attacks, and the organization verifies components continuously and replaces risky ones early. |
Review and exit triggers
- a vulnerability or compromise is disclosed;
- license or terms change;
- a component loses support;
- provenance cannot be verified; or
- a provider changes its dependencies.
7.5 Shared-responsibility definition
Purpose
Prevent gaps and overlaps in responsibility between the organization, providers, integrators, users, and external parties from being discovered during an incident.
Intended outcome
Responsibilities across the organization, providers, integrators, users, and external parties are explicit and tested.
Decisions enabled
- who is responsible for each control and decision;
- what the organization must do in addition to what the provider does;
- how responsibility is shown to be met; and
- how disagreements are resolved.
Applicability
Workloads: All
Delivery modes: All
Typical state: Shared
Key activities
- map responsibilities across all parties for each in-scope use and delivery mode;
- write them into contracts and operating agreements;
- identify what remains with the organization even where a provider delivers the service;
- test the arrangements in exercises and incidents;
- define escalation across party boundaries; and
- update the map when parties, scope, or terms change.
Required evidence
- responsibility map;
- contracts and operating agreements;
- operating and escalation tests;
- records of disputes and resolution; and
- update records.
Provider evidence
Ask the provider for:
- provider statements of the controls it operates;
- contract terms assigning responsibility; and
- assurance reports or other evidence that the provider's controls operate.
Common failure patterns
- assuming the provider handles security and compliance;
- responsibilities defined in contracts but unknown to operators;
- no one responsible for the seams between parties;
- escalation contacts out of date; and
- map never tested.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Responsibilities are assumed from marketing materials or contract summaries. |
| Defined | A responsibility map exists for in-scope uses and is reflected in contracts. |
| Managed | Responsibilities are tested through exercises and incidents, and gaps are resolved. |
| Adaptive | The map adapts to changes in providers and delivery modes, and the organization reshapes agreements and operating practice on evidence. |
Review and exit triggers
- a gap or overlap is found in an incident or test;
- a provider, integrator, or contract changes;
- delivery mode or scope changes;
- escalation contacts fail; or
- a regulator or counterparty asks who is responsible.
7.6 AI incident response
Purpose
Prevent AI-related incidents from being handled by improvisation, which lets harm continue, evidence be lost, and lessons go unlearned.
Intended outcome
The organization can detect, contain, investigate, communicate, recover from, and learn from AI-related incidents.
Decisions enabled
- whether an AI event is an incident and how severe it is, judged by consequence and reach rather than by the AI label;
- when to contain, suspend, or stop an AI;
- who must be notified, including affected people and regulators; and
- what corrective action follows.
Applicability
Workloads: All
Delivery modes: All
Typical state: Required
Key activities
- define incident types specific to AI, such as harmful output, data leakage, unauthorized action, and manipulation;
- extend the existing incident route, such as support, critical-situation, and security teams, with AI-specific signals, triage, and playbooks instead of creating a separate organization;
- prepare playbooks with roles, contacts, and decision authority;
- provide the means to suspend or restrict AI quickly;
- preserve evidence and communicate with affected people and parties;
- exercise the playbooks, including scenarios involving providers; and
- review each incident and track corrective actions to closure.
Required evidence
- playbooks and roles;
- exercise results;
- incident records and communications;
- corrective actions with closure; and
- review findings.
Common failure patterns
- AI incidents handled through a general IT process that has no AI-specific signals or playbooks;
- every AI error treated as a critical incident, or a parallel AI team that bypasses existing escalation;
- no way to switch the AI off;
- provider incident not visible to the organization;
- communication delayed while facts are checked; and
- corrective actions left open.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Incidents are handled by whoever notices them, and records are inconsistent. |
| Defined | AI-specific signals and playbooks are added to the existing incident route, with roles and contacts named for in-scope uses. |
| Managed | Playbooks are exercised and used, evidence is kept, and corrective actions are tracked. |
| Adaptive | Response adapts to incidents and exercises across the organization and providers, and the organization detects and contains incidents sooner over time. |
Review and exit triggers
- an incident or near miss occurs;
- an exercise reveals a gap;
- a new AI capability or authority is introduced;
- a provider incident affects the organization; or
- a corrective action is overdue.
7.7 Adversarial evaluation and resilience
Purpose
Prevent AI that works for ordinary users from failing under manipulation, abuse, or misuse that determined or careless people will attempt.
Intended outcome
The organization tests manipulation, abuse, unsafe behavior, boundary failures, and likely misuse, then verifies that corrective changes improve resistance without unacceptable effects on usefulness or access.
Decisions enabled
- whether the AI resists the attacks that matter;
- which findings must be corrected before release or continued use;
- whether a fix reduces risk without harming usefulness or access; and
- how often to retest.
Applicability
Workloads: All
Delivery modes: All
Condition: Applied proportionately to consequence
Typical state: Required, with depth based on consequence
Key activities
- define test scope from the threat model and consequence, including prompt manipulation, data extraction, unsafe behavior, tool misuse, and boundary failures;
- use testers who bring skills the builders do not, internal or external;
- record findings with severity and owners;
- apply corrective changes and retest;
- check that fixes do not produce unacceptable losses in usefulness or in access for legitimate users; and
- repeat tests after changes to the model, data, tools, or provider.
Required evidence
- test scope and method;
- findings and severity;
- corrective changes;
- resilience and regression results; and
- retest schedule.
Common failure patterns
- testing only by the team that built the AI;
- fixes that block legitimate users;
- findings not retested;
- tests that cover single prompts but not multi-step attacks on tools; and
- no retest after a model update.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Testing is informal, and findings depend on the interest of individuals. |
| Defined | Test scope, method, and owners are defined for the uses where consequence calls for it. |
| Managed | Independent testers are used for consequential uses. Findings are corrected and retested, with regression and usefulness checked. |
| Adaptive | Methods evolve with attacker techniques and incident findings, and the organization tests continuously and measures improvement. |
Review and exit triggers
- a finding of high severity is open;
- the model, data, tools, or provider changes;
- a new attack technique emerges;
- an incident shows an untested path; or
- a fix degrades usefulness or access.
7.8 AI provider and vendor risk
Purpose
Prevent providers from being selected or renewed on reputation and demonstration alone, with data, security, reliability, location, subcontractors, terms, evidence, cost, change, continuity, and exit left unexamined.
Intended outcome
Provider selection and renewal address data, security, reliability, location, subcontractors, terms, evidence, cost, change, continuity, and exit.
Decisions enabled
- whether to select, renew, condition, or exit a provider;
- what contract terms to require;
- what independent evidence is sufficient; and
- what residual provider risk is accepted.
Applicability
Workloads: All
Delivery modes: All
Condition: Modes B, C, and D directly; Mode A when the solution is purchased
Typical state: Shared
Key activities
- set due-diligence depth in proportion to consequence and dependence;
- examine security, privacy, reliability, locations, subcontractors, financial standing, and terms;
- obtain and assess independent evidence rather than self-declaration only;
- negotiate terms on data use, notice, audit, incident response, continuity, and exit;
- score and record the risk and decision; and
- review at renewal and when circumstances change.
Required evidence
- due-diligence results;
- contracts and negotiated terms;
- scorecards;
- renewal and exit decisions; and
- residual-risk acceptance.
Provider evidence
Ask the provider for:
- security and assurance reports;
- documentation of data use, locations, and subcontractors;
- reliability and incident history; and
- commitments on notice, audit, continuity, and exit.
Common failure patterns
- due diligence applied equally to low and high consequence;
- a questionnaire answered and not verified;
- terms accepted as standard when they conflict with obligations;
- renewals approved without review; and
- subcontractors not examined.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Providers are chosen on reputation or price, and review depends on who handles the purchase. |
| Defined | Due diligence is defined for in-scope providers, with contracts and records. |
| Managed | Depth scales with consequence, evidence is verified, and renewals are reviewed. |
| Adaptive | Provider risk practice adapts to market and incident evidence, and the organization shares findings across the portfolio and acts on weak signals. |
Review and exit triggers
- a provider incident, acquisition, or financial change occurs;
- an assurance report is qualified or missing;
- renewal approaches;
- the use becomes more consequential; or
- terms or subcontractors change.
Boundary
Assesses provider risk and decides on contracts at selection and renewal. 1.5 chooses the pattern, 5.4 designs for switching, and 6.8 monitors change between reviews.
7.9 External representation and transaction accountability
Purpose
Prevent AI from communicating or committing the organization to outside parties without authority, limits, confirmation, attribution, protection from fraud, and a way to reverse or dispute what it did.
Intended outcome
AI communicates or commits externally only with defined authority, counterparties, limits, confirmation, attribution, fraud controls, reversal, and dispute handling.
Decisions enabled
- what the AI may say or commit to, and to whom;
- what limits apply by value, type, and counterparty;
- when a person must confirm; and
- how the organization responds to disputes and mistaken transactions.
Applicability
Workloads: W4
Delivery modes: All
Condition: In any mode where AI is delegated Communicate or Commit authority
Typical state: Conditional
Key activities
- define the authority held for Communicate and Commit as an explicit matrix;
- set transaction limits by value, type, and counterparty;
- require confirmation steps proportionate to consequence;
- identify the AI to counterparties and attribute actions to it and to the organization;
- apply controls against impersonation, fraud, and manipulation of the AI by counterparties; and
- provide ways to reverse transactions and handle disputes, and test them.
Required evidence
- authority matrix;
- transaction limits;
- confirmations and attribution;
- action evidence; and
- reversal and dispute tests.
Common failure patterns
- an AI that quotes, promises, or concedes beyond its authority;
- counterparties unable to tell they are dealing with AI;
- limits enforced by prompt instructions alone;
- transactions that cannot be reversed; and
- disputes handled without the action record.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | The AI communicates or acts externally under the settings its builders chose, with no stated limits. |
| Defined | An authority matrix with limits and confirmation requirements exists for in-scope external uses. |
| Managed | Limits are enforced by controls, actions are attributed and recorded, and reversal and dispute paths are tested. |
| Adaptive | Authority and controls adapt to fraud patterns, disputes, and counterparty behavior, and the organization widens authority only where evidence supports it. |
Review and exit triggers
- a transaction exceeds limits or is disputed;
- fraud or manipulation is attempted or succeeds;
- new counterparties, channels, or authority are added;
- a reversal test fails; or
- law or contract on automated commitments changes.
