Multi-API Orchestration: When Every Service Must Succeed or All Must Roll Back

A single B2B order touches four independent services — pricing, tax calculation, payment processing, and shipping. Each API is owned by a different vendor with different SLAs, different authentication schemes, different rate limits, and different ideas about what constitutes a valid error response.

The naive approach — calling each API sequentially in one endpoint — creates a fragile chain where any single failure breaks the entire order. Worse, a failure at the last step after all previous steps succeeded means you have an authorized payment charge with no corresponding order record — money trapped in limbo.

We needed an architecture that isolates failures, manages rollback, and gives the caller full visibility into exactly what succeeded and what didn’t. (What getting this right bought the business: 70% faster order processing.)

Compensation patterns are well-described in distributed systems literature. Implementing them reliably with real vendor APIs that each fail differently — that’s where production engineering matters.

Every external service sits behind an abstract interface. The orchestration layer works with normalized domain objects — Customer, LineItem, TaxResult, ShipmentLabel — and never sees vendor-specific types. Concrete implementations handle vendor-specific authentication, request formatting, response parsing, and error mapping.

This isn’t theoretical future-proofing. We swapped payment providers during the project. The change required one new implementation file and zero modifications to the orchestration logic. The interface boundary earned its cost on the first provider change.

For rollback, we implemented a compensation pattern: each successful step registers an undo operation. If any subsequent step fails, all registered compensations execute in reverse order. The payment void is the most critical — an authorized charge without an order record is the worst possible state. The orchestrator returns a structured result showing the status of every step, so a partially completed order is never silently swallowed.

The uncomfortable question compensation patterns usually skip: the undo can fail too. A payment void times out; an inventory release hits the same degraded API that broke the order in the first place. Pretending otherwise is how 'automatic rollback' quietly becomes a new source of stuck state.

Three rules keep failed rollbacks survivable. Compensations execute in risk order — the payment void runs first, because an orphaned charge is the state you can least afford to hold. Every step and every compensation is idempotent, keyed to the order attempt, so retrying an undo can never double-release inventory or double-void a charge. And a compensation that still fails after retries doesn’t disappear into a log — it lands in a dead-letter queue with the full step-by-step result attached, so a human resolves it from complete context instead of reconstructing the order across four dashboards.

That last path is deliberately boring. The goal isn’t a system where rollbacks never fail — it’s a system where a failed rollback is a ticket with everything attached, not a 2 AM archaeology project.

Each API has different rate limits and different consequences for exceeding them. We implemented per-provider rate management that respects each service’s constraints independently. When a provider approaches its limit, requests are spread across the available window instead of hard-rejected — maintaining throughput without triggering rate limit errors. The same bounded-resource discipline governs our serverless security middleware.

Caching reduces unnecessary network calls. Customer records, tax rates, and shipping quotes all have natural staleness windows where cached results are perfectly valid. Payment data is never cached — every authorization hits the network. The asymmetry is intentional: cache what’s stable, always fetch what’s transactional.

When a non-critical service degrades — shipping is slow, tax calculation times out — the system can degrade gracefully: complete the order with available data and queue the missing step for retry. When a critical service fails — payment authorization is rejected — the transaction stops and rolls back cleanly. The distinction between critical and non-critical is explicit in the orchestration configuration, not left to developer intuition during an incident.