Capability records, perspective 5: Platform, engineering, and integration
Part of the SEI AI Adoption Framework (AIAF).
5.1 Capacity and service planning
Purpose
Prevent AI services from failing or becoming unaffordable under real demand because compute, throughput, latency, support, and skills were sized for the demonstration.
Intended outcome
Compute, storage, throughput, latency, availability, support, and skills can meet expected demand and consequence.
Decisions enabled
- how much capacity to reserve or buy;
- what service objectives to commit to;
- when to scale, throttle, or queue; and
- whether demand justifies the cost.
Applicability
Workloads: All
Delivery modes: A, D
Typical state: Required
Key activities
- forecast demand by use, including peaks and growth from adoption;
- set service objectives suited to consequence, such as availability and latency;
- size compute, storage, support, and people to the objectives;
- plan how the service degrades or scales under overload;
- monitor demand and capacity against forecast; and
- revise plans and objectives from observation.
Required evidence
- demand forecast and assumptions;
- service objectives;
- capacity and scaling plan;
- monitoring against forecast; and
- review decisions.
Common failure patterns
- sizing for average rather than peak load;
- ignoring provider quotas and rate limits;
- support staff not scaled with users;
- objectives not tied to consequence; and
- capacity decided after launch.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Capacity is addressed when users complain or limits are reached. |
| Defined | Forecasts and service objectives exist for in-scope services with owners. |
| Managed | Capacity and scaling plans are tested and monitored, and decisions are revised from observed demand. |
| Adaptive | Planning adapts to usage patterns and cost, and the organization shifts capacity across uses and providers as demand changes. |
Review and exit triggers
- demand exceeds forecast or objectives are missed;
- a provider limit or price changes;
- adoption expands to new groups;
- consequence of failure rises; or
- support load exceeds capacity.
5.2 AI runtime and service delivery
Purpose
Prevent runtime and service patterns from failing the availability, isolation, performance, location, and recovery needs of the work they support.
Intended outcome
Runtime patterns meet availability, isolation, performance, location, recoverability, and operating requirements.
Decisions enabled
- which runtime pattern to use;
- how workloads are isolated from each other;
- what recovery objectives apply; and
- whether the pattern can be operated by the organization.
Applicability
Workloads: All
Delivery modes: A, D
Typical state: Required
Key activities
- derive requirements from consequence, workload, data, and location rules;
- choose runtime patterns such as hosted, dedicated, or self-operated;
- design isolation between tenants, uses, environments, and sensitivity levels;
- set recovery objectives and prepare failover;
- test performance, isolation, and recovery; and
- record operating responsibilities.
Required evidence
- architecture and pattern decision;
- service objectives;
- isolation design;
- recovery and failover tests; and
- operating responsibilities.
Common failure patterns
- shared runtime with no isolation of sensitive uses;
- recovery never tested;
- pattern chosen for novelty;
- location requirements discovered late; and
- responsibilities unclear between provider and organization.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Runtime is whatever the project team could stand up, and recovery is untested. |
| Defined | Patterns, objectives, and responsibilities are defined for in-scope services. |
| Managed | Isolation and recovery are tested and results shape changes. |
| Adaptive | Runtime design adapts to measured behavior, cost, and risk, and the organization can change pattern with limited disruption. |
Review and exit triggers
- a test misses an objective;
- data sensitivity, consequence, or location rules change;
- the provider changes the service;
- an outage reveals a gap; or
- a new pattern becomes viable.
5.3 AI engineering, evaluation, and release tooling
Purpose
Prevent AI changes from being built, tested, and released in ways that cannot be repeated, explained, or reversed.
Intended outcome
Teams can build, configure, test, approve, release, observe, and reproduce changes consistently.
Decisions enabled
- whether a change is ready to release;
- which tools and registries are approved;
- what evaluation a release needs; and
- who may approve and deploy.
Applicability
Workloads: All
Delivery modes: A, D
Typical state: Required
Key activities
- define the toolchain for building, configuring, evaluating, and releasing;
- keep models, prompts, configurations, datasets, and evaluations under version control or registry;
- automate tests and evaluations in the pipeline and set pass criteria;
- separate duties for authoring and approving releases;
- record what was released, with what evidence; and
- reproduce earlier releases when needed.
Required evidence
- toolchain description;
- registries and version records;
- pipeline definitions and results;
- release approvals; and
- reproduction tests.
Common failure patterns
- prompts edited in production;
- evaluation run by hand and not repeated;
- releases that cannot be reproduced;
- no separation between author and approver; and
- multiple toolchains with different rules.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Changes are made directly by builders, and records live in individual notes. |
| Defined | A defined toolchain with versioned artifacts and release records exists for in-scope work. |
| Managed | Evaluation and approval are built into pipelines, releases are reproducible, and duties are separated. |
| Adaptive | Tooling adapts to defects and release outcomes, and the organization shortens delivery without weakening evidence. |
Review and exit triggers
- a release cannot be reproduced;
- a change reaches production without evaluation;
- a new model or tool class is adopted;
- pipeline failures or escapes increase; or
- ownership of the toolchain changes.
Boundary
Governs building and releasing an AI solution or shared substrate. 5.6 governs software development that uses AI assistance. Both can apply when a team uses AI assistance to build its shared AI platform.
5.4 Provider choice and portability
Purpose
Prevent dependence on a single provider or pattern from becoming a lock-in the organization cannot afford to leave.
Intended outcome
Architecture and commercial decisions preserve enough switching ability for the business consequence and expected lifespan.
Decisions enabled
- how much switching ability the consequence and lifespan justify;
- where to introduce abstraction and where to accept dependence;
- which contract terms to require; and
- whether a switch is feasible when needed.
Applicability
Workloads: All
Delivery modes: All
Typical state: Shared
Key activities
- map dependencies on models, services, data formats, and interfaces;
- decide how much portability is worth its cost;
- design interfaces, data formats, and prompts to limit provider-specific coupling where it matters;
- negotiate terms for data export, notice, and transition support;
- test switching or migration for important uses; and
- record accepted dependencies and revisit them.
Required evidence
- dependency map;
- abstraction and portability decisions;
- switching and migration tests;
- contract terms for export and transition; and
- accepted-dependence record.
Provider evidence
Ask the provider for:
- provider documentation of export formats and interfaces;
- contract commitments on data return, notice, and transition support; and
- evidence that exports work in practice.
Common failure patterns
- assuming a standard interface makes switching easy;
- portability designed for all uses at uniform cost;
- data that can be exported but not used elsewhere;
- tests never run; and
- dependency accepted without recording it.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Dependence on providers is recognized in discussion, but nothing is designed or tested. |
| Defined | Dependencies are mapped and portability decisions recorded for in-scope uses. |
| Managed | Switching or migration is tested where consequence justifies it, and contract terms support exit. |
| Adaptive | Portability decisions adapt to market and provider change, and the organization can move work when conditions require. |
Review and exit triggers
- a provider changes price, terms, or availability;
- a migration test fails;
- a use becomes more consequential;
- a credible alternative appears; or
- the dependency list changes.
Boundary
Designs architecture and contracts so the organization can switch. 1.5 makes the sourcing choice, 6.8 monitors provider change, and 7.8 assesses provider risk.
5.5 Cost engineering
Purpose
Prevent AI costs from growing unseen across licenses, usage, people, and operations until value cannot justify them.
Intended outcome
Costs across licenses, usage, people, and operations are forecast, attributed, monitored, and optimized, then reviewed alongside observed outcomes, validated value assumptions, and uncertainty.
Decisions enabled
- whether a use is affordable at its expected scale;
- where to optimize, cap, or stop;
- how costs are attributed and funded; and
- whether cost remains justified by the evidence of value, including where that evidence is incomplete.
Applicability
Workloads: All
Delivery modes: All
Typical state: Required
Key activities
- forecast cost across licenses, usage, infrastructure, people, support, and change;
- attribute cost to uses, teams, and outcomes;
- set budgets, alerts, and unit-cost measures;
- review variances and optimize, for example by selecting a cheaper model or reducing waste;
- review cost beside observed outcomes, validated value assumptions, and stated uncertainty, without presenting an unproven financial return as established; and
- account for provider price changes.
Required evidence
- cost model and assumptions;
- budgets and alerts;
- unit economics;
- variance and optimization reviews; and
- cost review that keeps observed outcomes, value assumptions, and uncertainty distinct.
Common failure patterns
- only license price counted;
- usage-based costs exceed forecast without notice;
- no attribution, so no one owns the cost;
- optimization that reduces quality unnoticed; and
- cost reviewed apart from outcomes, or an unproven financial return reported as established.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Costs are visible in invoices, and nobody forecasts or attributes them by use. |
| Defined | A cost model, budget, and owner exist for in-scope uses. |
| Managed | Costs are attributed, monitored against unit measures, and reviewed beside observed outcomes and stated value assumptions, with uncertainty recorded. |
| Adaptive | Cost practice adapts to price and usage change, and the organization adjusts models, providers, and designs to keep cost proportionate to the evidence of value. |
Review and exit triggers
- cost exceeds budget or forecast;
- unit cost rises or the evidence of value weakens;
- a provider changes pricing;
- usage grows beyond plan; or
- attribution shows an unowned cost.
5.6 AI-assisted development lifecycle
Purpose
Prevent AI-assisted software development from weakening the source, dependencies, credentials, environments, review, and release integrity it relies on.
Intended outcome
Development automation protects source, dependencies, credentials, environments, review quality, traceability, and release integrity.
Decisions enabled
- which AI development tools and uses are permitted;
- what review applies to AI-assisted changes;
- what access development agents may have; and
- how provenance and licensing risks are handled.
Applicability
Workloads: W2
Delivery modes: All
Typical state: Conditional, when AI assists software development
Key activities
- define approved tools and uses and what code and data may be shared with them;
- limit repository, credential, and environment access for assistants and agents;
- keep human review of AI-assisted changes meaningful, and mark such changes where useful;
- scan dependencies, secrets, and licenses on changes that include generated code;
- protect pipelines and release steps from automated changes without approval; and
- monitor defects and security findings linked to AI-assisted work.
Required evidence
- engineering policy for AI-assisted development;
- repository and pipeline controls;
- review and traceability evidence;
- scanning results; and
- defect and finding trends.
Common failure patterns
- credentials exposed to assistants;
- reviewers approving large generated changes without reading them;
- generated code with unknown license or dependency risk;
- agents with permission to modify pipelines; and
- proprietary code sent to unapproved services.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Developers use assistants individually, and rules on access and review are unwritten. |
| Defined | Approved tools, permitted uses, and access limits are defined for in-scope teams. |
| Managed | Review, scanning, and access controls are applied and checked, and findings are tracked. |
| Adaptive | Controls adapt to new tools and observed defects, and the organization measures effects on quality and security. |
Review and exit triggers
- a secret or proprietary code is exposed;
- defect or vulnerability rates change;
- a tool gains agent authority;
- license or provenance concerns arise; or
- review quality declines.
Boundary
Governs software development that uses AI assistance in any mode. 5.3 governs building and releasing an AI solution or shared substrate.
5.7 Integration architecture
Purpose
Prevent AI from becoming an unmanaged connection between systems, where data and actions flow without defined interfaces, identities, failure behavior, or ownership.
Intended outcome
AI exchanges data and actions through defined interfaces, contracts, identities, failure behavior, and ownership boundaries.
Decisions enabled
- which systems AI may connect to and by what means;
- what data and actions pass across each connection;
- how failures are handled; and
- who owns each side of the boundary.
Applicability
Workloads: All
Delivery modes: All
Condition: Where AI is integrated with other systems
Typical state: Conditional
Key activities
- describe integration patterns and approved interfaces;
- define data contracts, including what is sent, returned, and retained;
- assign identities and permissions to each connection;
- specify failure behavior, such as timeouts, fallbacks, and safe states;
- assign ownership to both sides of each boundary; and
- test failures and exits, including provider or system withdrawal.
Required evidence
- integration patterns;
- data contracts;
- identity and permission records;
- failure and exit tests; and
- ownership records.
Common failure patterns
- point-to-point connections created by scripts and plugins;
- no defined behavior when a downstream system fails;
- broad credentials shared across integrations;
- undocumented data flows; and
- ownership assumed by each side to sit with the other.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Connections are made by individual teams as needed, and flows are known to the builders. |
| Defined | Patterns, data contracts, and owners are defined for in-scope integrations. |
| Managed | Identities are scoped, failure and exit behavior are tested, and changes are controlled. |
| Adaptive | Integration design adapts to incidents and change, and the organization can reroute or withdraw connections safely. |
Review and exit triggers
- a connected system changes its interface or data;
- a failure causes unsafe behavior;
- new data or action types pass the boundary;
- ownership changes; or
- a connection is found that was not recorded.
5.8 Agent platform and tool gateway
Purpose
Prevent agents from reaching models, tools, data, and other agents through paths nobody can observe or enforce.
Intended outcome
Agents use approved models, tools, services, data, and other agents through observable and enforceable control points.
Decisions enabled
- which models, tools, and data an agent may use;
- which control points apply to each;
- how agent relationships are recorded and reviewed; and
- when access is changed or withdrawn.
Applicability
Workloads: All
Delivery modes: A, D
Typical state: Required
Key activities
- provide a registry of approved agents, models, tools, and services;
- route calls through a gateway or equivalent control points that enforce policy;
- maintain a tool catalog with descriptions, owners, and permission scopes;
- record relationships among agents, tools, and data;
- log calls for monitoring and investigation; and
- test that controls hold, including against attempts to bypass them.
Required evidence
- registry of agents and tools;
- gateway policy and enforcement records;
- tool catalog;
- relationship map; and
- test results.
Common failure patterns
- agents calling tools directly with no control point;
- tool catalog out of date;
- approval at creation with no later review;
- relationships between agents invisible; and
- gateway present but easy to bypass.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Agents reach tools through code written by their builders, and control is by convention. |
| Defined | A registry, a catalog, and defined control points exist for in-scope agents. |
| Managed | Policy is enforced at the gateway, relationships are recorded, and bypass tests are run. |
| Adaptive | Controls adapt to new agent patterns and observed behavior, and the organization can reshape agent access quickly when risk changes. |
Review and exit triggers
- an agent reaches a tool outside the catalog;
- a bypass test succeeds;
- new agent or tool types are introduced;
- a tool's behavior or ownership changes; or
- relationship maps diverge from reality.
5.9 AI observability and telemetry
Purpose
Prevent operators from being unable to see what AI is doing, what it costs, how users experience it, or what it has caused, which delays detection of failure and harm.
Intended outcome
Operators can connect technical behavior, cost, user experience, actions, outcomes, and incidents without collecting unnecessary sensitive data.
Decisions enabled
- what to observe and at what detail;
- which thresholds trigger an alert or action;
- who sees what, given privacy limits; and
- whether the AI continues, is limited, or is stopped.
Applicability
Workloads: All
Delivery modes: All
Typical state: Required
Key activities
- design telemetry that links technical behavior, cost, user experience, actions, and outcomes;
- set dashboards and alerts tied to decisions and owners;
- limit sensitive data in telemetry and control access to it;
- include provider-reported and organization-measured signals;
- test that alerts fire and that responders can act; and
- review telemetry and retire signals that no one uses.
Required evidence
- telemetry design;
- dashboards and alert definitions;
- privacy controls on telemetry;
- alert tests and responses; and
- review records.
Common failure patterns
- technical metrics with no view of outcomes;
- alerts nobody acts on;
- telemetry that stores sensitive content without need;
- reliance on provider dashboards alone; and
- blind spots at the points where agents act.
Capability levels
| Level | Observable condition |
|---|---|
| Emerging | Observation is by sampling and complaint, and visibility depends on the provider's tools. |
| Defined | Telemetry and alerts exist for in-scope services with named owners. |
| Managed | Signals link behavior, cost, experience, and outcomes. Alerts are tested and acted on, and privacy limits are applied. |
| Adaptive | Observability adapts to incidents and new behavior, and the organization detects emerging issues before users report them. |
Review and exit triggers
- an incident was not detected by telemetry;
- alerts are ignored or too frequent;
- new capabilities lack coverage;
- privacy concerns arise about telemetry; or
- a provider changes available signals.
