Capability records, perspective 6: Lifecycle, adoption, and operations

Part of the SEI AI Adoption Framework (AIAF).

6.1 Feasibility and organizational readiness

Purpose

Prevent initiatives from starting or scaling on the assumption that they are feasible, when data, skills, workflow, provider support, cost, governance, or adoption readiness is missing.

Intended outcome

Decisions reflect technical feasibility, data, skills, workflow, provider, cost, governance, support, and adoption readiness.

Decisions enabled

  • whether to proceed, prepare first, narrow the scope, or stop;
  • which gaps must close before commitment;
  • what the organization must change to be ready; and
  • which assumptions need testing.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • assess technical feasibility with the actual data and workflow, not a demonstration;
  • assess readiness in data, skills, process, provider, cost, governance, support, and adoption;
  • record assumptions, risks, and the evidence for each;
  • run small tests of the riskiest assumptions;
  • decide with named owners and conditions; and
  • revisit the assessment when facts change.

Required evidence

  • feasibility and readiness assessment;
  • assumptions and evidence;
  • results of tests of key assumptions;
  • decision with conditions; and
  • revisit records.

Common failure patterns

  • feasibility judged from a vendor demonstration;
  • readiness limited to technology;
  • assumptions never tested;
  • gaps noted and then ignored; and
  • assessment repeated only after failure.

Capability levels

Level Observable condition
Emerging Feasibility is judged by sponsors or builders from experience and demonstrations.
Defined A feasibility and readiness assessment with assumptions exists for in-scope initiatives.
Managed Key assumptions are tested, gaps drive decisions, and assessments are revisited on change.
Adaptive Assessment methods improve through comparison of predictions with outcomes, and readiness is built ahead of the portfolio's needs.

Review and exit triggers

  • a key assumption fails a test;
  • scope, data, provider, or workflow changes;
  • a readiness gap persists past its date;
  • cost or timing departs from the assessment; or
  • sponsors or priorities change.

6.2 Requirements and quality attributes

Purpose

Prevent AI capabilities from being judged by impressions because usefulness and qualities such as safety, security, accessibility, privacy, performance, and cost were never stated in testable form.

Intended outcome

The capability has testable requirements for usefulness and qualities such as reliability, safety, security, accessibility, privacy, performance, cost, and maintainability.

Decisions enabled

  • what the capability must achieve and under what limits;
  • which qualities matter and at what level;
  • what counts as acceptable for release; and
  • whether a requirement has been met.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • describe the work, users, and context of use;
  • state usefulness requirements that can be tested;
  • state quality requirements for reliability, safety, security, accessibility, privacy, performance, cost, and maintainability;
  • set acceptance criteria and thresholds by consequence;
  • include requirements from affected people and from obligations; and
  • keep requirements under change control.

Required evidence

  • requirements and quality measures;
  • acceptance criteria;
  • trace between requirements and tests;
  • sources of requirements; and
  • change records.

Common failure patterns

  • requirements that say the AI should be helpful;
  • qualities treated as someone else's concern;
  • thresholds set after results are known;
  • accessibility and affected users omitted; and
  • requirements that drift without record.

Capability levels

Level Observable condition
Emerging Requirements are expressed as goals or examples, and quality is judged by impression.
Defined Testable usefulness and quality requirements with acceptance criteria exist for in-scope capabilities.
Managed Requirements trace to tests and obligations, thresholds reflect consequence, and changes are controlled.
Adaptive Requirements evolve based on operating evidence, and the organization improves its specification practice from escapes and complaints.

Review and exit triggers

  • a defect or complaint shows an unstated requirement;
  • obligations or user needs change;
  • the AI is used for different work;
  • thresholds prove too lax or too strict; or
  • a quality attribute cannot be measured.

6.3 Evaluation methodology

Full record: worked example 3 of 5 in Part 3, Capability model. The catalog entry in Part 4 gives the intended outcome, workloads, delivery modes, and condition.

6.4 Deployment and authority patterns

Purpose

Prevent AI from being given more autonomy, tools, access, and authority than evidence and consequence justify.

Intended outcome

AI receives only the autonomy, tools, access, and authority justified by evidence and consequence.

Decisions enabled

  • which deployment pattern applies;
  • what authority level the AI holds;
  • what evidence is needed to promote it to a higher level; and
  • when to restrict or withdraw authority.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • define patterns from assistance to supervised action to delegated action;
  • assign tools, data, and authority per pattern, using the least needed;
  • set controls suited to each pattern, including approval and limits;
  • define promotion criteria that rely on evidence from operation;
  • define demotion triggers; and
  • record every change in pattern or authority with its approver.

Required evidence

  • pattern decision and authority level;
  • controls by pattern;
  • promotion criteria and evidence;
  • demotion triggers; and
  • records of changes.

Common failure patterns

  • authority granted because it was technically possible;
  • a pilot pattern becoming permanent;
  • promotion based on enthusiasm;
  • no mechanism for demotion; and
  • tools added without reviewing authority.

Capability levels

Level Observable condition
Emerging Autonomy and access are decided by builders as convenient.
Defined Patterns and authority levels are defined, with approval for each in-scope deployment.
Managed Authority is based on evidence, promotion and demotion criteria are used, and changes are recorded.
Adaptive Patterns adapt to results across deployments, and the organization extends authority only where evidence justifies and withdraws it quickly when not.

Review and exit triggers

  • an error shows authority exceeded competence;
  • tools or access are added;
  • evidence for promotion is weakened;
  • consequence of the use rises; or
  • demotion criteria are met.

6.5 Staged deployment and adoption

Purpose

Prevent wide exposure of AI before it has been shown to work for real people, by expanding in controlled stages with the means to stop.

Intended outcome

Exposure expands in controlled stages with monitoring, support, feedback, rollback, and explicit promotion decisions.

Decisions enabled

  • who receives access at each stage;
  • what evidence promotes to the next stage;
  • when to hold, roll back, or stop; and
  • what support each stage needs.

Applicability

Workloads: All
Delivery modes: A, B, C
Typical state: Required

Key activities

  • define stages, cohorts, and entry criteria;
  • monitor behavior, outcomes, support demand, and feedback in each stage;
  • set explicit promotion decisions with owners;
  • prepare rollback and support before each stage;
  • include cohorts that represent the range of users and conditions; and
  • record decisions and lessons.

Required evidence

  • cohort and stage plan;
  • monitoring results per stage;
  • promotion and rollback records;
  • support and feedback logs; and
  • lessons recorded.

Common failure patterns

  • stages defined by calendar rather than evidence;
  • early cohorts of enthusiasts only;
  • rollback not rehearsed;
  • promotion by default; and
  • support unprepared for the next stage.

Capability levels

Level Observable condition
Emerging Release goes to whoever asks, and expansion follows enthusiasm or deadline.
Defined Stages, cohorts, and promotion owners are defined for in-scope deployments.
Managed Promotion depends on evidence, rollback is ready, and decisions and lessons are recorded.
Adaptive Staging practice adapts to what earlier releases revealed, and the organization sizes stages by risk and evidence.

Review and exit triggers

  • monitoring crosses a hold or rollback threshold;
  • support demand exceeds capacity;
  • a cohort differs from the expected population;
  • a promotion decision lacks evidence; or
  • the plan is changed under schedule pressure.

6.6 Continuous outcome, behavior, and change evaluation

Purpose

Prevent AI that was good at release from quietly declining in value, drifting in behavior, or changing underneath its users.

Intended outcome

Operation detects declining value, behavior change, provider change, drift, misuse, unexpected effects, and changing assumptions.

Decisions enabled

  • whether the AI still delivers its intended outcome;
  • whether behavior has changed in ways that matter;
  • whether to retrain, reconfigure, restrict, or retire; and
  • whether assumptions made at approval still hold.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • measure outcomes, not only technical performance;
  • set behavior thresholds and test sets that repeat on a schedule and after change;
  • watch for drift in data, users, provider behavior, and context;
  • detect misuse and unexpected effects;
  • compare current results with the baseline and expected value; and
  • decide and act on the findings.

Required evidence

  • outcome reviews;
  • behavior thresholds and test results;
  • change and drift alerts;
  • misuse and unexpected-effect findings; and
  • decisions taken.

Common failure patterns

  • monitoring uptime while quality falls;
  • reviews that happen only after complaints;
  • provider model updates that go unnoticed;
  • thresholds that never trigger; and
  • findings with no decision.

Capability levels

Level Observable condition
Emerging Problems are noticed when users complain or when someone happens to compare results.
Defined Outcome and behavior measures with thresholds and owners exist for in-scope uses.
Managed Reviews repeat on schedule and after change, drift and misuse are detected, and findings lead to recorded decisions.
Adaptive Evaluation adapts to the changing context, and the organization detects declining value early and redirects or retires capabilities on evidence.

Review and exit triggers

  • outcomes or behavior cross a threshold;
  • the provider changes the model or terms;
  • data or user population shifts;
  • misuse or unexpected effects appear; or
  • assumptions in the approval no longer hold.

6.7 Configuration, version, and decision lineage

Purpose

Prevent consequential outputs and actions from being impossible to explain, reproduce, or dispute because nobody can tell which model, data, prompt, tools, policies, and configuration produced them.

Intended outcome

Consequential outputs and actions can be connected to the relevant model, data, prompt, tools, policies, integrations, identity, and approved configuration.

Decisions enabled

  • whether a result can be explained and defended;
  • which configuration was in force at a given time;
  • whether a change caused a problem; and
  • how much lineage a use needs.

Applicability

Workloads: All
Delivery modes: All
Condition: Applied proportionately to consequence
Typical state: Required, with depth based on consequence

Key activities

  • decide the lineage needed in proportion to consequence;
  • record versions of models, prompts, data, tools, policies, integrations, and identities;
  • link outputs and actions to the configuration in force;
  • keep approved configurations and detect departures;
  • protect records from alteration; and
  • test that a past decision can be reconstructed.

Required evidence

  • configuration records and versions;
  • decision and action lineage;
  • approved-configuration baseline;
  • departure detection results; and
  • reconstruction tests.

Common failure patterns

  • prompts and settings changed without a record;
  • provider model updates not captured;
  • lineage kept for all uses equally or for none;
  • records that cannot be matched to a decision; and
  • reconstruction never tested.

Capability levels

Level Observable condition
Emerging Configuration is known to the people who set it, and past results cannot be reconstructed.
Defined Versions of key components are recorded for uses where consequence calls for it.
Managed Outputs and actions link to configuration, departures are detected, and reconstruction is tested.
Adaptive Lineage practice adapts to consequence and dispute patterns, and the organization reconstructs decisions quickly and with confidence.

Review and exit triggers

  • a result cannot be reconstructed;
  • configuration changes without approval;
  • consequence of a use rises;
  • a dispute or inquiry requires evidence; or
  • a provider changes components not versioned by the organization.

6.8 Provider and capability change monitoring

Purpose

Prevent provider changes in terms, prices, data use, safeguards, features, interfaces, locations, or availability from taking effect unnoticed after selection.

Intended outcome

Provider terms, prices, data use, safeguards, features, interfaces, locations, and availability are reviewed for continued fitness.

Decisions enabled

  • whether a change is acceptable;
  • what mitigation or negotiation is needed;
  • whether to continue, restrict, or exit; and
  • who must be informed or consulted.

Applicability

Workloads: All
Delivery modes: All
Condition: Modes B, C, and D directly; Mode A when it depends on a provider
Typical state: Shared

Key activities

  • identify the change channels for each provider, including notices, release notes, and status pages;
  • assign an owner who reads and records changes;
  • assess each change for effect on risk, cost, obligations, and outcomes;
  • test effects where behavior may have changed;
  • decide to accept, mitigate, or exit, and record the decision; and
  • feed findings to sourcing, portability, and risk decisions.

Required evidence

  • change notices received;
  • impact reviews;
  • acceptance, mitigation, or exit decisions;
  • results of tests after change; and
  • link to provider risk reviews.

Provider evidence

Ask the provider for:

  • advance notice of material changes;
  • release notes and documentation of behavior changes;
  • notice of terms, price, or location changes; and
  • availability and incident reports.

Common failure patterns

  • notices unread because they arrive by email to a shared inbox;
  • behavior changes found by users;
  • price changes accepted by default;
  • no owner; and
  • changes assessed without testing.

Capability levels

Level Observable condition
Emerging Provider changes are noticed by chance, and effects are discovered through use.
Defined Owners and channels for provider change are defined, and notable changes are recorded.
Managed Changes are assessed and tested, decisions are recorded, and findings reach related decisions.
Adaptive Monitoring adapts to provider behavior, and the organization anticipates changes and negotiates or exits in good time.

Review and exit triggers

  • a provider issues a material notice;
  • behavior changes without notice;
  • a change conflicts with an obligation or risk limit;
  • the provider's own status changes; or
  • no review has occurred by the due date.

Boundary

Watches what changes after selection, such as terms, prices, features, and locations. 1.5 chooses, 5.4 preserves the ability to switch, and 7.8 assesses provider risk at selection and renewal.

6.9 Rollback, replacement, and retirement

Full record: worked example 5 of 5 in Part 3, Capability model. The catalog entry in Part 4 gives the intended outcome, workloads, delivery modes, and condition.

6.10 Continuity and exit planning

Purpose

Prevent important work from stopping when an AI capability or provider becomes unavailable, unacceptable, uneconomic, or obsolete.

Intended outcome

Important work can continue when an AI capability or provider becomes unavailable, unacceptable, uneconomic, or obsolete.

Decisions enabled

  • which work must continue and for how long;
  • what alternatives are ready;
  • how much to invest in preparation; and
  • when to invoke the plan.

Applicability

Workloads: All
Delivery modes: All
Condition: Applied proportionately to consequence
Typical state: Required, with depth based on consequence

Key activities

  • identify work that depends on each AI capability and its tolerance for interruption;
  • define alternatives, such as manual procedures, other tools, or other providers;
  • set recovery objectives suited to consequence;
  • prepare data, access, and agreements that make alternatives usable;
  • exercise continuity and exit plans; and
  • update plans after changes and exercises.

Required evidence

  • continuity plan and dependency map;
  • alternatives and their readiness;
  • recovery objectives;
  • exercise results; and
  • update records.

Common failure patterns

  • manual fallback assumed but nobody remembers how;
  • backups of data with no way to use it elsewhere;
  • plans that cover provider outage but not provider exit;
  • exercises skipped; and
  • dependence discovered during the failure.

Capability levels

Level Observable condition
Emerging Continuity relies on improvisation, and dependence on AI is not mapped.
Defined Continuity plans and alternatives exist for the uses where consequence calls for them.
Managed Plans are exercised, alternatives are ready, and lessons update the plans.
Adaptive Continuity design adapts to dependence, provider change, and exercise findings, and the organization shifts work between alternatives with little disruption.

Review and exit triggers

  • an exercise fails to meet recovery objectives;
  • a provider fails, changes, or announces withdrawal;
  • dependence on the capability grows;
  • alternatives become unavailable; or
  • consequence of interruption rises.

Boundary

Keeps standing continuity and alternatives ready so work survives unavailability. 6.9 carries out the rollback, replacement, or retirement.