Capability records, perspective 7: Trust, security, and resilience

Part of the SEI AI Adoption Framework (AIAF).

7.1 AI threat modeling

Purpose

Prevent AI designs and operations from being protected against ordinary threats only, while threats specific to models, data, prompts, retrieval, tools, agents, users, providers, and external parties are left unconsidered.

Intended outcome

Design and operations address threats arising from models, data, prompts, retrieval, tools, agents, users, providers, and external parties.

Decisions enabled

  • which threats apply to a use;
  • which mitigations are required;
  • which residual risks are accepted and by whom; and
  • when the threat model must be reviewed.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • describe the system, its data flows, and trust boundaries, including retrieval, tools, and agents;
  • identify threats such as prompt injection, data poisoning, leakage, abuse of tools, impersonation, and provider compromise;
  • assess likelihood and consequence;
  • select mitigations and assign owners;
  • record residual risks and acceptance; and
  • review the model when the system, threats, or incidents change.

Required evidence

  • threat model with trust boundaries;
  • mitigations and owners;
  • residual-risk decisions;
  • review records; and
  • link to adversarial testing and incidents.

Common failure patterns

  • threat modeling done once at the start;
  • models that omit agents and retrieved content;
  • mitigations listed without owners;
  • residual risk accepted by the builder; and
  • findings from incidents not fed back.

Capability levels

Level Observable condition
Emerging Threats are considered informally by individual security staff or builders.
Defined A threat model with mitigations and owners exists for in-scope systems.
Managed Models cover AI-specific threats, residual risks are accepted by accountable owners, and reviews occur on change.
Adaptive Threat modeling adapts to new attack techniques and incident findings, and the organization uses it to shape design and testing early.

Review and exit triggers

  • system, data flow, tool, or authority changes;
  • a new class of attack becomes known;
  • an incident or test reveals an unmodeled threat;
  • a provider changes architecture; or
  • a residual-risk decision expires.

7.2 Agent identity, authority, and tool access

Full record: worked example 4 of 5 in Part 3, Capability model. The catalog entry in Part 4 gives the intended outcome, workloads, delivery modes, and condition.

7.3 AI interaction and action records

Purpose

Prevent investigation, accountability, and dispute handling from failing because records of AI interactions and actions are missing, unreliable, or unprotected, or conversely kept so broadly that they create privacy and security exposure.

Intended outcome

Records support investigation, evaluation, accountability, dispute handling, and improvement while respecting privacy and retention rules.

Decisions enabled

  • which interactions and actions are recorded and how much detail is kept;
  • who may access the records and under what conditions;
  • how long they are kept; and
  • how integrity is protected.

Applicability

Workloads: All
Delivery modes: All
Condition: Applied proportionately to consequence
Typical state: Required, with depth based on consequence

Key activities

  • decide logging scope and detail in proportion to consequence;
  • include inputs, outputs, tool calls, actions, approvals, and the configuration in force where needed;
  • protect records from alteration and limit access;
  • apply privacy, minimization, and retention rules;
  • provide the means to search and retrieve for investigation and disputes; and
  • test integrity and access controls.

Required evidence

  • logging policy and scope;
  • protected records;
  • access records;
  • integrity and retrieval tests; and
  • retention and disposal records.

Common failure patterns

  • logging nothing for agent actions;
  • logging everything with no access controls;
  • logs that can be edited by those they concern;
  • records that cannot be matched to a person or decision; and
  • retention conflicts unresolved.

Capability levels

Level Observable condition
Emerging Records are whatever the tool produces, and access and retention are not decided.
Defined Logging scope, access, and retention are defined for the uses where consequence calls for records.
Managed Records are protected, searchable, and tested, and privacy limits are applied.
Adaptive Logging adapts to investigation and dispute experience, and the organization reconstructs events promptly with the least data necessary.

Review and exit triggers

  • an investigation cannot be completed for lack of records;
  • an integrity or access test fails;
  • authority or consequence of the AI increases;
  • privacy or retention requirements change; or
  • log volume or cost becomes unmanageable.

7.4 Model, data, and component supply-chain assurance

Purpose

Prevent the organization from using models, data, and components whose origin, terms, dependencies, or vulnerabilities it does not know.

Intended outcome

The organization understands the origin, terms, dependencies, vulnerabilities, and changes of important AI components.

Decisions enabled

  • whether a component may be used;
  • what verification it needs;
  • what to do about a vulnerability or a change of terms; and
  • which components to replace.

Applicability

Workloads: All
Delivery modes: All
Condition: Modes A and D directly; Modes B and C when they depend on a provider
Typical state: Conditional

Key activities

  • inventory models, datasets, libraries, tools, and other components with versions and origins;
  • check licenses and terms, including restrictions on use and redistribution;
  • assess provenance and integrity, for example through signatures and trusted sources;
  • review vulnerabilities and respond to advisories;
  • track changes in components and their dependencies; and
  • where components are supplied by a provider, obtain evidence about their supply chain.

Required evidence

  • component inventory with versions;
  • provenance and integrity checks;
  • license and terms review;
  • vulnerability review results; and
  • provider supply-chain evidence where relevant.

Common failure patterns

  • models downloaded from unverified sources;
  • license terms for weights or data ignored;
  • a plugin or tool added with unknown dependencies;
  • vulnerabilities recorded and not fixed; and
  • no record of what a provider's service relies upon.

Capability levels

Level Observable condition
Emerging Components are chosen by builders, and origin and terms are not recorded.
Defined Components are inventoried with source, version, and license for in-scope uses.
Managed Provenance and vulnerabilities are checked and tracked, and changes lead to review.
Adaptive Assurance adapts to emerging supply-chain attacks, and the organization verifies components continuously and replaces risky ones early.

Review and exit triggers

  • a vulnerability or compromise is disclosed;
  • license or terms change;
  • a component loses support;
  • provenance cannot be verified; or
  • a provider changes its dependencies.

7.5 Shared-responsibility definition

Purpose

Prevent gaps and overlaps in responsibility between the organization, providers, integrators, users, and external parties from being discovered during an incident.

Intended outcome

Responsibilities across the organization, providers, integrators, users, and external parties are explicit and tested.

Decisions enabled

  • who is responsible for each control and decision;
  • what the organization must do in addition to what the provider does;
  • how responsibility is shown to be met; and
  • how disagreements are resolved.

Applicability

Workloads: All
Delivery modes: All
Typical state: Shared

Key activities

  • map responsibilities across all parties for each in-scope use and delivery mode;
  • write them into contracts and operating agreements;
  • identify what remains with the organization even where a provider delivers the service;
  • test the arrangements in exercises and incidents;
  • define escalation across party boundaries; and
  • update the map when parties, scope, or terms change.

Required evidence

  • responsibility map;
  • contracts and operating agreements;
  • operating and escalation tests;
  • records of disputes and resolution; and
  • update records.

Provider evidence

Ask the provider for:

  • provider statements of the controls it operates;
  • contract terms assigning responsibility; and
  • assurance reports or other evidence that the provider's controls operate.

Common failure patterns

  • assuming the provider handles security and compliance;
  • responsibilities defined in contracts but unknown to operators;
  • no one responsible for the seams between parties;
  • escalation contacts out of date; and
  • map never tested.

Capability levels

Level Observable condition
Emerging Responsibilities are assumed from marketing materials or contract summaries.
Defined A responsibility map exists for in-scope uses and is reflected in contracts.
Managed Responsibilities are tested through exercises and incidents, and gaps are resolved.
Adaptive The map adapts to changes in providers and delivery modes, and the organization reshapes agreements and operating practice on evidence.

Review and exit triggers

  • a gap or overlap is found in an incident or test;
  • a provider, integrator, or contract changes;
  • delivery mode or scope changes;
  • escalation contacts fail; or
  • a regulator or counterparty asks who is responsible.

7.6 AI incident response

Purpose

Prevent AI-related incidents from being handled by improvisation, which lets harm continue, evidence be lost, and lessons go unlearned.

Intended outcome

The organization can detect, contain, investigate, communicate, recover from, and learn from AI-related incidents.

Decisions enabled

  • whether an AI event is an incident and how severe it is, judged by consequence and reach rather than by the AI label;
  • when to contain, suspend, or stop an AI;
  • who must be notified, including affected people and regulators; and
  • what corrective action follows.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • define incident types specific to AI, such as harmful output, data leakage, unauthorized action, and manipulation;
  • extend the existing incident route, such as support, critical-situation, and security teams, with AI-specific signals, triage, and playbooks instead of creating a separate organization;
  • prepare playbooks with roles, contacts, and decision authority;
  • provide the means to suspend or restrict AI quickly;
  • preserve evidence and communicate with affected people and parties;
  • exercise the playbooks, including scenarios involving providers; and
  • review each incident and track corrective actions to closure.

Required evidence

  • playbooks and roles;
  • exercise results;
  • incident records and communications;
  • corrective actions with closure; and
  • review findings.

Common failure patterns

  • AI incidents handled through a general IT process that has no AI-specific signals or playbooks;
  • every AI error treated as a critical incident, or a parallel AI team that bypasses existing escalation;
  • no way to switch the AI off;
  • provider incident not visible to the organization;
  • communication delayed while facts are checked; and
  • corrective actions left open.

Capability levels

Level Observable condition
Emerging Incidents are handled by whoever notices them, and records are inconsistent.
Defined AI-specific signals and playbooks are added to the existing incident route, with roles and contacts named for in-scope uses.
Managed Playbooks are exercised and used, evidence is kept, and corrective actions are tracked.
Adaptive Response adapts to incidents and exercises across the organization and providers, and the organization detects and contains incidents sooner over time.

Review and exit triggers

  • an incident or near miss occurs;
  • an exercise reveals a gap;
  • a new AI capability or authority is introduced;
  • a provider incident affects the organization; or
  • a corrective action is overdue.

7.7 Adversarial evaluation and resilience

Purpose

Prevent AI that works for ordinary users from failing under manipulation, abuse, or misuse that determined or careless people will attempt.

Intended outcome

The organization tests manipulation, abuse, unsafe behavior, boundary failures, and likely misuse, then verifies that corrective changes improve resistance without unacceptable effects on usefulness or access.

Decisions enabled

  • whether the AI resists the attacks that matter;
  • which findings must be corrected before release or continued use;
  • whether a fix reduces risk without harming usefulness or access; and
  • how often to retest.

Applicability

Workloads: All
Delivery modes: All
Condition: Applied proportionately to consequence
Typical state: Required, with depth based on consequence

Key activities

  • define test scope from the threat model and consequence, including prompt manipulation, data extraction, unsafe behavior, tool misuse, and boundary failures;
  • use testers who bring skills the builders do not, internal or external;
  • record findings with severity and owners;
  • apply corrective changes and retest;
  • check that fixes do not produce unacceptable losses in usefulness or in access for legitimate users; and
  • repeat tests after changes to the model, data, tools, or provider.

Required evidence

  • test scope and method;
  • findings and severity;
  • corrective changes;
  • resilience and regression results; and
  • retest schedule.

Common failure patterns

  • testing only by the team that built the AI;
  • fixes that block legitimate users;
  • findings not retested;
  • tests that cover single prompts but not multi-step attacks on tools; and
  • no retest after a model update.

Capability levels

Level Observable condition
Emerging Testing is informal, and findings depend on the interest of individuals.
Defined Test scope, method, and owners are defined for the uses where consequence calls for it.
Managed Independent testers are used for consequential uses. Findings are corrected and retested, with regression and usefulness checked.
Adaptive Methods evolve with attacker techniques and incident findings, and the organization tests continuously and measures improvement.

Review and exit triggers

  • a finding of high severity is open;
  • the model, data, tools, or provider changes;
  • a new attack technique emerges;
  • an incident shows an untested path; or
  • a fix degrades usefulness or access.

7.8 AI provider and vendor risk

Purpose

Prevent providers from being selected or renewed on reputation and demonstration alone, with data, security, reliability, location, subcontractors, terms, evidence, cost, change, continuity, and exit left unexamined.

Intended outcome

Provider selection and renewal address data, security, reliability, location, subcontractors, terms, evidence, cost, change, continuity, and exit.

Decisions enabled

  • whether to select, renew, condition, or exit a provider;
  • what contract terms to require;
  • what independent evidence is sufficient; and
  • what residual provider risk is accepted.

Applicability

Workloads: All
Delivery modes: All
Condition: Modes B, C, and D directly; Mode A when the solution is purchased
Typical state: Shared

Key activities

  • set due-diligence depth in proportion to consequence and dependence;
  • examine security, privacy, reliability, locations, subcontractors, financial standing, and terms;
  • obtain and assess independent evidence rather than self-declaration only;
  • negotiate terms on data use, notice, audit, incident response, continuity, and exit;
  • score and record the risk and decision; and
  • review at renewal and when circumstances change.

Required evidence

  • due-diligence results;
  • contracts and negotiated terms;
  • scorecards;
  • renewal and exit decisions; and
  • residual-risk acceptance.

Provider evidence

Ask the provider for:

  • security and assurance reports;
  • documentation of data use, locations, and subcontractors;
  • reliability and incident history; and
  • commitments on notice, audit, continuity, and exit.

Common failure patterns

  • due diligence applied equally to low and high consequence;
  • a questionnaire answered and not verified;
  • terms accepted as standard when they conflict with obligations;
  • renewals approved without review; and
  • subcontractors not examined.

Capability levels

Level Observable condition
Emerging Providers are chosen on reputation or price, and review depends on who handles the purchase.
Defined Due diligence is defined for in-scope providers, with contracts and records.
Managed Depth scales with consequence, evidence is verified, and renewals are reviewed.
Adaptive Provider risk practice adapts to market and incident evidence, and the organization shares findings across the portfolio and acts on weak signals.

Review and exit triggers

  • a provider incident, acquisition, or financial change occurs;
  • an assurance report is qualified or missing;
  • renewal approaches;
  • the use becomes more consequential; or
  • terms or subcontractors change.

Boundary

Assesses provider risk and decides on contracts at selection and renewal. 1.5 chooses the pattern, 5.4 designs for switching, and 6.8 monitors change between reviews.

7.9 External representation and transaction accountability

Purpose

Prevent AI from communicating or committing the organization to outside parties without authority, limits, confirmation, attribution, protection from fraud, and a way to reverse or dispute what it did.

Intended outcome

AI communicates or commits externally only with defined authority, counterparties, limits, confirmation, attribution, fraud controls, reversal, and dispute handling.

Decisions enabled

  • what the AI may say or commit to, and to whom;
  • what limits apply by value, type, and counterparty;
  • when a person must confirm; and
  • how the organization responds to disputes and mistaken transactions.

Applicability

Workloads: W4
Delivery modes: All
Condition: In any mode where AI is delegated Communicate or Commit authority
Typical state: Conditional

Key activities

  • define the authority held for Communicate and Commit as an explicit matrix;
  • set transaction limits by value, type, and counterparty;
  • require confirmation steps proportionate to consequence;
  • identify the AI to counterparties and attribute actions to it and to the organization;
  • apply controls against impersonation, fraud, and manipulation of the AI by counterparties; and
  • provide ways to reverse transactions and handle disputes, and test them.

Required evidence

  • authority matrix;
  • transaction limits;
  • confirmations and attribution;
  • action evidence; and
  • reversal and dispute tests.

Common failure patterns

  • an AI that quotes, promises, or concedes beyond its authority;
  • counterparties unable to tell they are dealing with AI;
  • limits enforced by prompt instructions alone;
  • transactions that cannot be reversed; and
  • disputes handled without the action record.

Capability levels

Level Observable condition
Emerging The AI communicates or acts externally under the settings its builders chose, with no stated limits.
Defined An authority matrix with limits and confirmation requirements exists for in-scope external uses.
Managed Limits are enforced by controls, actions are attributed and recorded, and reversal and dispute paths are tested.
Adaptive Authority and controls adapt to fraud patterns, disputes, and counterparty behavior, and the organization widens authority only where evidence supports it.

Review and exit triggers

  • a transaction exceeds limits or is disputed;
  • fraud or manipulation is attempted or succeeds;
  • new counterparties, channels, or authority are added;
  • a reversal test fails; or
  • law or contract on automated commitments changes.