INCIDENT // PRD-921

Root Cause: Order Fulfillment Failed

A post-mortem investigation into a cascading failure across the payment gateway and inventory service. Scroll to trace the diagnostic process.

T+00:00 // Discovery

Application Logs

Alert fired at 02:14 UTC. Initial signals indicated a spike in 500 errors originating from the `OrderService`. PagerDuty alerted the on-call rotation.

syslog

[02:14:12] ERROR OrderService: FAILED_TO_PROCESS_ORDER
[02:14:12] TRACE order_id=ord_9982x status=500
[02:14:12] FATAL Exception: Unhandled promise rejection
[02:14:13] WARN Retry attempt 1 failed

T+04:12 // Investigation

API Trace

Tracing the specific `order_id` through Datadog APM revealed the request was dying at the boundary between our gateway and the inventory service.

net_req

POST /api/v1/checkout200ms

POST /internal/inventory/reserve150ms

GET /v3/stock/verify5002ms (TIMEOUT)

T+12:30 // Deep Dive

Database Contention

Checked the RDS performance insights. The timeout was caused by a deadlocked query on the `inventory_ledgers` table. A massive sequential scan was locking rows needed by the checkout flow.

psql

EXPLAIN ANALYZE
SELECT * FROM inventory_ledgers
WHERE sku_id IN (
  SELECT id FROM skus WHERE category = 'holiday'
) FOR UPDATE;

-> Seq Scan on inventory_ledgers (cost=0.00..89234.00 rows=45000)
-> Lock waiting (pid: 8932)

T+25:00 // Resolution

Index & Optimization

Killed the offending ad-hoc reporting query. Pushed a hotfix to add a composite index on `(sku_id, status)` and refactored the reservation logic to use optimistic locking instead of `FOR UPDATE`.

FIX_DEPLOYED

PR #1042: Optimistic locking for inventoryMerged by @shamim • 2 files changed, +14 -4