LLM gateway migration runbook: provider cutover and rollback

Expected outcome

This runbook moves one or more LLM workloads from a direct provider integration, or from one gateway-routed provider, to a target provider through a gateway that the organization has already selected. The outcome is not “all providers are interchangeable.” It is a documented supported contract, a measured comparison, a staged cutover, an exercised recovery path, and an evidence packet that lets an operator explain what changed.

Start upstream with the enterprise AI gateway selection guide. That guide decides whether a gateway and its data, identity, resilience, and ownership tradeoffs are appropriate. This article begins after that decision: the gateway exists, a platform team owns it, and a specific provider, model group, or endpoint must change. The method is compatible with gateway products but does not assume any vendor’s configuration is a universal standard.

The central discipline is to preserve the application’s business contract while admitting provider differences. An OpenAI-compatible endpoint can make request shapes familiar, but it does not equalize model capability, tool-call behavior, response schemas, safety policy, context size, regional availability, quotas, billing meters, or failure semantics. A migration that ignores those differences simply discovers them in production.

Prerequisites

Identify the operational owner of the gateway, each application owner, the target-provider owner, security or privacy reviewer where sensitive inputs are involved, and an incident commander for the cutover window. Freeze the initial scope: named applications, endpoints, model aliases, environments, regions, users, data classes, and rollback owner. Do not use a broad “all traffic” migration as the first scope.

Collect a representative, versioned evaluation set before changing routing. Include normal inputs, long contexts, structured outputs, tool calls, refusals, malformed requests, timeouts, rate-limit responses, and a request that must not reach an external provider. The prompt evaluation playbook can help distinguish a useful regression set from a collection of easy prompts. Capture expected properties, not only preferred prose: schema validity, citation presence where required, tool-call allow/deny behavior, policy outcome, and human-reviewed task result.

Create dedicated gateway credentials per environment and protect provider credentials in the approved secret system. Instrument the gateway and client with a correlation ID that survives retries and failover. OpenTelemetry HTTP semantic conventions provide a portable baseline for recording request method, route, status, duration, and errors; add an application-safe model, deployment alias, provider, region, request class, token or usage measure, policy result, and retry/fallback indicator. Avoid storing prompts, completions, API keys, or customer identifiers unless the documented data policy permits it.

Workflow

1. Build an inventory before introducing a new route

Inventory every call path, including background jobs, SDK wrappers, batch processing, browser agents, evaluation jobs, internal tools, and emergency scripts. A direct call that bypasses the gateway can turn a clean dashboard into misleading evidence. For each path, record its caller identity, endpoint, model alias, provider-specific option, data classification, streaming mode, tool or structured-output requirement, regional constraint, latency and spend owner, retry behavior, and user-visible consequence of failure.

Do not count a model alias as an inventory. One alias can conceal multiple actual deployments, fallback destinations, or SDK features. Compare deployed configuration with source search, secret references, runtime telemetry, and gateway access logs. Resolve unknown traffic before cutover rather than allowing it to silently inherit a default route.

2. Write the compatibility matrix and the portable contract

Declare the supported subset at the gateway boundary: request type, messages or input parts, system instructions, tools, JSON or structured output, streaming, attachments, token controls, safety options, response fields, usage fields, and idempotency behavior. Then identify fields that must remain provider-specific or unsupported. The matrix should say “not supported” rather than transforming an uncertain feature into a plausible-looking success.

Capability	Current provider behavior	Target/provider gateway behavior	Migration decision	Test evidence
Text generation	Model alias and request format	Target alias and accepted fields	Translate only the documented common subset	Contract fixture and response schema
Tool calling	Tool schema, selection, arguments	Supported tool format and limits	Require explicit allowlist and human-review boundary	Allowed, denied, and malformed-tool tests
Structured output	Schema mode and validation result	Target schema mechanism or application validator	Fail closed on invalid required structure	Valid and invalid schema fixtures
Streaming	Event order, completion marker, cancellation	Target stream behavior	Preserve cancellation and incomplete-response handling	Disconnect and resume test
Usage and cost	Provider meters and units	Gateway-normalized fields plus raw provider record	Keep raw values for reconciliation	Billing and usage sample
Safety and refusal	Provider response and local policy	Target response and local policy	Do not use failover to evade a policy refusal	Refusal classification test

Treat the matrix as a release artifact. It also prevents the migration from becoming a new gateway-selection exercise. The LiteLLM review is a product-fit reference for one gateway implementation; it is not evidence that a particular workload’s model contract is portable.

3. Normalize errors without erasing diagnostics

Normalize errors into a small operational taxonomy, but retain the provider status, code, request ID, retry-after signal, raw-classification reference, gateway route, and correlation ID in protected logs. A useful taxonomy separates: invalid_request, authentication_or_authorization, policy_refusal, quota_or_rate_limit, context_or_payload_limit, transient_provider, gateway_or_network, timeout_or_cancelled, and unknown. Map each class to an action: reject and fix input, alert access owner, show a policy outcome, retry with bounded backoff, shrink or reroute only when approved, fail over only to a compatible route, or escalate.

Provider documentation is why the raw fields matter. OpenAI documents error and rate-limit behavior; Anthropic documents distinct API error responses. A 429 may reflect a quota or rate constraint, while a content-policy response should not be silently retried through a weaker route. LiteLLM’s reliability documentation also distinguishes regular, context-window, and content-policy fallback concepts. Make policy refusal and authorization failure terminal unless an approved policy explicitly says otherwise.

4. Shadow safely and compare outcomes

Shadow traffic means the production path returns the current approved response while an eligible copy is evaluated against the target route. It is not an invitation to duplicate every customer prompt. First check authorization, provider terms, data residency, retention, contract, and privacy constraints. Where live shadowing is not allowed, replay sanitized production fixtures, consented data, or synthetic workloads instead.

Log the same correlation ID, test-set version, model alias, gateway route, request-class label, output schema result, task score, latency, usage, error class, and fallback flag on both paths. Do not send shadow output to an end user or permit it to execute tools. Compare structured correctness and policy behavior before stylistic preference. A human reviewer should inspect meaningful disagreements, especially tool calls, refusals, citations, financial or operational decisions, and cases where one path returns a valid-looking but incomplete answer.

5. Apply budget and observability guardrails

Set stage-specific caps for token usage, provider spend, retry count, concurrent shadow requests, and fallback volume. Every numeric value in this runbook is an example, not an industry standard: a team might begin with a small named cohort, use a two-attempt retry cap for a particular safe read-only request class, or stop a stage after an example daily spend ceiling. Select actual limits from the workload’s cost model, provider quota, data sensitivity, user impact, and incident capacity.

The observability check should answer: did the expected route receive traffic, did the correct identity call it, did errors normalize as intended, did fallback occur only for eligible cases, did usage reconcile with provider records, and did any request lose tracing? Monitor completion rate, schema-valid rate, task-quality signal, policy outcome, p50/p95 latency, timeouts, queue depth, cost per verified outcome, retry volume, and unknown-error count. The agent observability guide helps place this evidence in the broader trace and human-override model.

6. Cut over in reversible stages

Start with an internal, read-only workload and a named cohort. Hold a go/no-go review after each stage, attaching the matrix, shadow comparison, metrics, incident log, and owner sign-off. Move only when the current stage meets its pre-agreed safety, contract, cost, and service criteria. Keep the old provider route configured and tested until the target route proves stable over the agreed observation period.

Do not combine a model migration with a prompt rewrite, tool expansion, identity change, or retention-policy change unless the evaluation plan can isolate each factor. One change per observable step makes rollback meaningful. If a feature cannot be carried through the portable contract, keep it explicitly provider-specific, defer it, or decline the cutover for that workload.

7. Exercise failover and rollback

Define eligible failover pairs by capability, data boundary, region, and policy. A fallback model must be compatible enough for the request class; it must not broaden data transfer, bypass a refusal, alter a required schema without detection, or repeat a side effect. LiteLLM’s own reliability guidance recommends testing fallbacks by triggering a relevant provider error in a non-production environment. Use a safe test deployment to prove error classification, ordered fallback, headers or trace fields showing the selected route, and user-visible behavior.

Set a rollback trigger before the cutover: a confirmed authorization or data-boundary breach; a material policy-control failure; an unrecoverable loss of traceability; a sustained quality or schema-validity regression beyond the locally approved tolerance; unexpected cost growth above the stage budget; or unavailable target service with no compatible safe fallback. These are categories, not universal thresholds. The trigger record must name the decision-maker, measurement window, evidence source, and route change that returns traffic to the last known-good configuration.

Reusable template

Use this compact packet for each workload and attach detailed records rather than copying secrets or customer content.

Item	Required record
Inventory	Application, owner, environment, identity, endpoint, alias, provider, data class, tools, streaming, and user consequence
Compatibility matrix	Supported fields, provider differences, unsupported features, transformation rules, and contract fixtures
Error taxonomy	Normalized class, raw status/code reference, retry/failover rule, escalation owner, and user-safe message
Shadow plan	Eligible input source, consent/data boundary, sample selection, response isolation, comparison metrics, and reviewer
Observability plan	Correlation ID, required attributes, protected logs, dashboards, alert owner, and reconciliation method
Budget guardrails	Stage-specific example caps, alert path, spend owner, and stop condition; never present examples as standards
Cutover stages	Named cohort, start/end, go/no-go evidence, old route retained, and rollback owner
Failover plan	Compatible pairs, prohibited classes, test method, trace proof, and recovery communication
Evidence packet	Approved scope, fixtures, results, metrics, incidents, raw provider references, decisions, and final configuration version

Failure modes

• Compatibility assumed from an endpoint name: a gateway accepts the call but tools, streaming, or structured outputs behave differently. Require the matrix and contract fixtures.
• Shadow traffic leaks data: prompts are copied to a target provider without an approved data boundary. Stop the shadow path and use sanitized or synthetic replay.
• Retries duplicate side effects: a timeout hides a completed tool call, then a retry repeats it. Keep side-effecting operations outside automatic retry or use verified idempotency controls.
• Normalized error hides a policy signal: a refusal becomes a generic server error and falls back to another model. Preserve the raw classification and block policy-evasion fallback.
• Metrics are not comparable: traffic differs by cohort, prompt version, region, or route. Label the run and compare like with like.
• Cost is observed too late: a broad stage runs before budgets and alerts exist. Set a locally approved example cap, owner, and stop condition before sending traffic.
• Rollback has no last known-good route: the old provider was removed during cutover. Retain and exercise the return route until the change is accepted.

Acceptance criteria

1. Every in-scope call path is inventoried, and gateway telemetry confirms no unmanaged production route remains for the stage.
2. The compatibility matrix documents supported, transformed, and unsupported behavior with passing and failing fixtures.
3. Error logs retain correlation ID and raw provider evidence while the application receives a stable, safe normalized class.
4. Shadow traffic obeys the approved data boundary, cannot reach users or tools, and produces a comparable evidence set.
5. The stage dashboard reports route, identity, latency, usage, cost, quality, policy result, retry, and fallback evidence without exposing secrets.
6. Budget guardrails and cutover gates identify their values as local examples or approved workload limits, not universal thresholds.
7. A safe failover and rollback exercise proves that the target route can be removed and the prior approved route restored with recorded evidence.

Next step

Run the packet for one low-risk, read-only workload and keep the scope fixed until the evidence is reviewed. Then apply the same template to higher-impact workloads, with tighter data and tool boundaries rather than a larger blind traffic percentage. If the migration uncovers a gateway ownership, credential, or resilience gap, return to the gateway-selection decision record before expanding the rollout.