Capability records, perspective 5: Platform, engineering, and integration

Part of the SEI AI Adoption Framework (AIAF).

5.1 Capacity and service planning

Purpose

Prevent AI services from failing or becoming unaffordable under real demand because compute, throughput, latency, support, and skills were sized for the demonstration.

Intended outcome

Compute, storage, throughput, latency, availability, support, and skills can meet expected demand and consequence.

Decisions enabled

  • how much capacity to reserve or buy;
  • what service objectives to commit to;
  • when to scale, throttle, or queue; and
  • whether demand justifies the cost.

Applicability

Workloads: All
Delivery modes: A, D
Typical state: Required

Key activities

  • forecast demand by use, including peaks and growth from adoption;
  • set service objectives suited to consequence, such as availability and latency;
  • size compute, storage, support, and people to the objectives;
  • plan how the service degrades or scales under overload;
  • monitor demand and capacity against forecast; and
  • revise plans and objectives from observation.

Required evidence

  • demand forecast and assumptions;
  • service objectives;
  • capacity and scaling plan;
  • monitoring against forecast; and
  • review decisions.

Common failure patterns

  • sizing for average rather than peak load;
  • ignoring provider quotas and rate limits;
  • support staff not scaled with users;
  • objectives not tied to consequence; and
  • capacity decided after launch.

Capability levels

Level Observable condition
Emerging Capacity is addressed when users complain or limits are reached.
Defined Forecasts and service objectives exist for in-scope services with owners.
Managed Capacity and scaling plans are tested and monitored, and decisions are revised from observed demand.
Adaptive Planning adapts to usage patterns and cost, and the organization shifts capacity across uses and providers as demand changes.

Review and exit triggers

  • demand exceeds forecast or objectives are missed;
  • a provider limit or price changes;
  • adoption expands to new groups;
  • consequence of failure rises; or
  • support load exceeds capacity.

5.2 AI runtime and service delivery

Purpose

Prevent runtime and service patterns from failing the availability, isolation, performance, location, and recovery needs of the work they support.

Intended outcome

Runtime patterns meet availability, isolation, performance, location, recoverability, and operating requirements.

Decisions enabled

  • which runtime pattern to use;
  • how workloads are isolated from each other;
  • what recovery objectives apply; and
  • whether the pattern can be operated by the organization.

Applicability

Workloads: All
Delivery modes: A, D
Typical state: Required

Key activities

  • derive requirements from consequence, workload, data, and location rules;
  • choose runtime patterns such as hosted, dedicated, or self-operated;
  • design isolation between tenants, uses, environments, and sensitivity levels;
  • set recovery objectives and prepare failover;
  • test performance, isolation, and recovery; and
  • record operating responsibilities.

Required evidence

  • architecture and pattern decision;
  • service objectives;
  • isolation design;
  • recovery and failover tests; and
  • operating responsibilities.

Common failure patterns

  • shared runtime with no isolation of sensitive uses;
  • recovery never tested;
  • pattern chosen for novelty;
  • location requirements discovered late; and
  • responsibilities unclear between provider and organization.

Capability levels

Level Observable condition
Emerging Runtime is whatever the project team could stand up, and recovery is untested.
Defined Patterns, objectives, and responsibilities are defined for in-scope services.
Managed Isolation and recovery are tested and results shape changes.
Adaptive Runtime design adapts to measured behavior, cost, and risk, and the organization can change pattern with limited disruption.

Review and exit triggers

  • a test misses an objective;
  • data sensitivity, consequence, or location rules change;
  • the provider changes the service;
  • an outage reveals a gap; or
  • a new pattern becomes viable.

5.3 AI engineering, evaluation, and release tooling

Purpose

Prevent AI changes from being built, tested, and released in ways that cannot be repeated, explained, or reversed.

Intended outcome

Teams can build, configure, test, approve, release, observe, and reproduce changes consistently.

Decisions enabled

  • whether a change is ready to release;
  • which tools and registries are approved;
  • what evaluation a release needs; and
  • who may approve and deploy.

Applicability

Workloads: All
Delivery modes: A, D
Typical state: Required

Key activities

  • define the toolchain for building, configuring, evaluating, and releasing;
  • keep models, prompts, configurations, datasets, and evaluations under version control or registry;
  • automate tests and evaluations in the pipeline and set pass criteria;
  • separate duties for authoring and approving releases;
  • record what was released, with what evidence; and
  • reproduce earlier releases when needed.

Required evidence

  • toolchain description;
  • registries and version records;
  • pipeline definitions and results;
  • release approvals; and
  • reproduction tests.

Common failure patterns

  • prompts edited in production;
  • evaluation run by hand and not repeated;
  • releases that cannot be reproduced;
  • no separation between author and approver; and
  • multiple toolchains with different rules.

Capability levels

Level Observable condition
Emerging Changes are made directly by builders, and records live in individual notes.
Defined A defined toolchain with versioned artifacts and release records exists for in-scope work.
Managed Evaluation and approval are built into pipelines, releases are reproducible, and duties are separated.
Adaptive Tooling adapts to defects and release outcomes, and the organization shortens delivery without weakening evidence.

Review and exit triggers

  • a release cannot be reproduced;
  • a change reaches production without evaluation;
  • a new model or tool class is adopted;
  • pipeline failures or escapes increase; or
  • ownership of the toolchain changes.

Boundary

Governs building and releasing an AI solution or shared substrate. 5.6 governs software development that uses AI assistance. Both can apply when a team uses AI assistance to build its shared AI platform.

5.4 Provider choice and portability

Purpose

Prevent dependence on a single provider or pattern from becoming a lock-in the organization cannot afford to leave.

Intended outcome

Architecture and commercial decisions preserve enough switching ability for the business consequence and expected lifespan.

Decisions enabled

  • how much switching ability the consequence and lifespan justify;
  • where to introduce abstraction and where to accept dependence;
  • which contract terms to require; and
  • whether a switch is feasible when needed.

Applicability

Workloads: All
Delivery modes: All
Typical state: Shared

Key activities

  • map dependencies on models, services, data formats, and interfaces;
  • decide how much portability is worth its cost;
  • design interfaces, data formats, and prompts to limit provider-specific coupling where it matters;
  • negotiate terms for data export, notice, and transition support;
  • test switching or migration for important uses; and
  • record accepted dependencies and revisit them.

Required evidence

  • dependency map;
  • abstraction and portability decisions;
  • switching and migration tests;
  • contract terms for export and transition; and
  • accepted-dependence record.

Provider evidence

Ask the provider for:

  • provider documentation of export formats and interfaces;
  • contract commitments on data return, notice, and transition support; and
  • evidence that exports work in practice.

Common failure patterns

  • assuming a standard interface makes switching easy;
  • portability designed for all uses at uniform cost;
  • data that can be exported but not used elsewhere;
  • tests never run; and
  • dependency accepted without recording it.

Capability levels

Level Observable condition
Emerging Dependence on providers is recognized in discussion, but nothing is designed or tested.
Defined Dependencies are mapped and portability decisions recorded for in-scope uses.
Managed Switching or migration is tested where consequence justifies it, and contract terms support exit.
Adaptive Portability decisions adapt to market and provider change, and the organization can move work when conditions require.

Review and exit triggers

  • a provider changes price, terms, or availability;
  • a migration test fails;
  • a use becomes more consequential;
  • a credible alternative appears; or
  • the dependency list changes.

Boundary

Designs architecture and contracts so the organization can switch. 1.5 makes the sourcing choice, 6.8 monitors provider change, and 7.8 assesses provider risk.

5.5 Cost engineering

Purpose

Prevent AI costs from growing unseen across licenses, usage, people, and operations until value cannot justify them.

Intended outcome

Costs across licenses, usage, people, and operations are forecast, attributed, monitored, and optimized, then reviewed alongside observed outcomes, validated value assumptions, and uncertainty.

Decisions enabled

  • whether a use is affordable at its expected scale;
  • where to optimize, cap, or stop;
  • how costs are attributed and funded; and
  • whether cost remains justified by the evidence of value, including where that evidence is incomplete.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • forecast cost across licenses, usage, infrastructure, people, support, and change;
  • attribute cost to uses, teams, and outcomes;
  • set budgets, alerts, and unit-cost measures;
  • review variances and optimize, for example by selecting a cheaper model or reducing waste;
  • review cost beside observed outcomes, validated value assumptions, and stated uncertainty, without presenting an unproven financial return as established; and
  • account for provider price changes.

Required evidence

  • cost model and assumptions;
  • budgets and alerts;
  • unit economics;
  • variance and optimization reviews; and
  • cost review that keeps observed outcomes, value assumptions, and uncertainty distinct.

Common failure patterns

  • only license price counted;
  • usage-based costs exceed forecast without notice;
  • no attribution, so no one owns the cost;
  • optimization that reduces quality unnoticed; and
  • cost reviewed apart from outcomes, or an unproven financial return reported as established.

Capability levels

Level Observable condition
Emerging Costs are visible in invoices, and nobody forecasts or attributes them by use.
Defined A cost model, budget, and owner exist for in-scope uses.
Managed Costs are attributed, monitored against unit measures, and reviewed beside observed outcomes and stated value assumptions, with uncertainty recorded.
Adaptive Cost practice adapts to price and usage change, and the organization adjusts models, providers, and designs to keep cost proportionate to the evidence of value.

Review and exit triggers

  • cost exceeds budget or forecast;
  • unit cost rises or the evidence of value weakens;
  • a provider changes pricing;
  • usage grows beyond plan; or
  • attribution shows an unowned cost.

5.6 AI-assisted development lifecycle

Purpose

Prevent AI-assisted software development from weakening the source, dependencies, credentials, environments, review, and release integrity it relies on.

Intended outcome

Development automation protects source, dependencies, credentials, environments, review quality, traceability, and release integrity.

Decisions enabled

  • which AI development tools and uses are permitted;
  • what review applies to AI-assisted changes;
  • what access development agents may have; and
  • how provenance and licensing risks are handled.

Applicability

Workloads: W2
Delivery modes: All
Typical state: Conditional, when AI assists software development

Key activities

  • define approved tools and uses and what code and data may be shared with them;
  • limit repository, credential, and environment access for assistants and agents;
  • keep human review of AI-assisted changes meaningful, and mark such changes where useful;
  • scan dependencies, secrets, and licenses on changes that include generated code;
  • protect pipelines and release steps from automated changes without approval; and
  • monitor defects and security findings linked to AI-assisted work.

Required evidence

  • engineering policy for AI-assisted development;
  • repository and pipeline controls;
  • review and traceability evidence;
  • scanning results; and
  • defect and finding trends.

Common failure patterns

  • credentials exposed to assistants;
  • reviewers approving large generated changes without reading them;
  • generated code with unknown license or dependency risk;
  • agents with permission to modify pipelines; and
  • proprietary code sent to unapproved services.

Capability levels

Level Observable condition
Emerging Developers use assistants individually, and rules on access and review are unwritten.
Defined Approved tools, permitted uses, and access limits are defined for in-scope teams.
Managed Review, scanning, and access controls are applied and checked, and findings are tracked.
Adaptive Controls adapt to new tools and observed defects, and the organization measures effects on quality and security.

Review and exit triggers

  • a secret or proprietary code is exposed;
  • defect or vulnerability rates change;
  • a tool gains agent authority;
  • license or provenance concerns arise; or
  • review quality declines.

Boundary

Governs software development that uses AI assistance in any mode. 5.3 governs building and releasing an AI solution or shared substrate.

5.7 Integration architecture

Purpose

Prevent AI from becoming an unmanaged connection between systems, where data and actions flow without defined interfaces, identities, failure behavior, or ownership.

Intended outcome

AI exchanges data and actions through defined interfaces, contracts, identities, failure behavior, and ownership boundaries.

Decisions enabled

  • which systems AI may connect to and by what means;
  • what data and actions pass across each connection;
  • how failures are handled; and
  • who owns each side of the boundary.

Applicability

Workloads: All
Delivery modes: All
Condition: Where AI is integrated with other systems
Typical state: Conditional

Key activities

  • describe integration patterns and approved interfaces;
  • define data contracts, including what is sent, returned, and retained;
  • assign identities and permissions to each connection;
  • specify failure behavior, such as timeouts, fallbacks, and safe states;
  • assign ownership to both sides of each boundary; and
  • test failures and exits, including provider or system withdrawal.

Required evidence

  • integration patterns;
  • data contracts;
  • identity and permission records;
  • failure and exit tests; and
  • ownership records.

Common failure patterns

  • point-to-point connections created by scripts and plugins;
  • no defined behavior when a downstream system fails;
  • broad credentials shared across integrations;
  • undocumented data flows; and
  • ownership assumed by each side to sit with the other.

Capability levels

Level Observable condition
Emerging Connections are made by individual teams as needed, and flows are known to the builders.
Defined Patterns, data contracts, and owners are defined for in-scope integrations.
Managed Identities are scoped, failure and exit behavior are tested, and changes are controlled.
Adaptive Integration design adapts to incidents and change, and the organization can reroute or withdraw connections safely.

Review and exit triggers

  • a connected system changes its interface or data;
  • a failure causes unsafe behavior;
  • new data or action types pass the boundary;
  • ownership changes; or
  • a connection is found that was not recorded.

5.8 Agent platform and tool gateway

Purpose

Prevent agents from reaching models, tools, data, and other agents through paths nobody can observe or enforce.

Intended outcome

Agents use approved models, tools, services, data, and other agents through observable and enforceable control points.

Decisions enabled

  • which models, tools, and data an agent may use;
  • which control points apply to each;
  • how agent relationships are recorded and reviewed; and
  • when access is changed or withdrawn.

Applicability

Workloads: All
Delivery modes: A, D
Typical state: Required

Key activities

  • provide a registry of approved agents, models, tools, and services;
  • route calls through a gateway or equivalent control points that enforce policy;
  • maintain a tool catalog with descriptions, owners, and permission scopes;
  • record relationships among agents, tools, and data;
  • log calls for monitoring and investigation; and
  • test that controls hold, including against attempts to bypass them.

Required evidence

  • registry of agents and tools;
  • gateway policy and enforcement records;
  • tool catalog;
  • relationship map; and
  • test results.

Common failure patterns

  • agents calling tools directly with no control point;
  • tool catalog out of date;
  • approval at creation with no later review;
  • relationships between agents invisible; and
  • gateway present but easy to bypass.

Capability levels

Level Observable condition
Emerging Agents reach tools through code written by their builders, and control is by convention.
Defined A registry, a catalog, and defined control points exist for in-scope agents.
Managed Policy is enforced at the gateway, relationships are recorded, and bypass tests are run.
Adaptive Controls adapt to new agent patterns and observed behavior, and the organization can reshape agent access quickly when risk changes.

Review and exit triggers

  • an agent reaches a tool outside the catalog;
  • a bypass test succeeds;
  • new agent or tool types are introduced;
  • a tool's behavior or ownership changes; or
  • relationship maps diverge from reality.

5.9 AI observability and telemetry

Purpose

Prevent operators from being unable to see what AI is doing, what it costs, how users experience it, or what it has caused, which delays detection of failure and harm.

Intended outcome

Operators can connect technical behavior, cost, user experience, actions, outcomes, and incidents without collecting unnecessary sensitive data.

Decisions enabled

  • what to observe and at what detail;
  • which thresholds trigger an alert or action;
  • who sees what, given privacy limits; and
  • whether the AI continues, is limited, or is stopped.

Applicability

Workloads: All
Delivery modes: All
Typical state: Required

Key activities

  • design telemetry that links technical behavior, cost, user experience, actions, and outcomes;
  • set dashboards and alerts tied to decisions and owners;
  • limit sensitive data in telemetry and control access to it;
  • include provider-reported and organization-measured signals;
  • test that alerts fire and that responders can act; and
  • review telemetry and retire signals that no one uses.

Required evidence

  • telemetry design;
  • dashboards and alert definitions;
  • privacy controls on telemetry;
  • alert tests and responses; and
  • review records.

Common failure patterns

  • technical metrics with no view of outcomes;
  • alerts nobody acts on;
  • telemetry that stores sensitive content without need;
  • reliance on provider dashboards alone; and
  • blind spots at the points where agents act.

Capability levels

Level Observable condition
Emerging Observation is by sampling and complaint, and visibility depends on the provider's tools.
Defined Telemetry and alerts exist for in-scope services with named owners.
Managed Signals link behavior, cost, experience, and outcomes. Alerts are tested and acted on, and privacy limits are applied.
Adaptive Observability adapts to incidents and new behavior, and the organization detects emerging issues before users report them.

Review and exit triggers

  • an incident was not detected by telemetry;
  • alerts are ignored or too frequent;
  • new capabilities lack coverage;
  • privacy concerns arise about telemetry; or
  • a provider changes available signals.