Scaling a Payment Gateway to 10M+ Daily Transactions
Re-architected a legacy monolithic payment engine into a high-throughput microservices cluster using Go, Redis Streams, and Kubernetes, achieving 99.999% uptime with 38ms P99 latency.
Key Takeaways
- Decoupling transaction ingestion from async settlement via Redis Streams dropped P99 authorization latency from 800ms to 38ms.
- Sharding state across stateless Go services on Amazon EKS eliminated deployment restarts and unlocked 15,000 TPS burst capacity.
- Isolating raw cardholder data behind an ephemeral tokenization boundary reduced PCI DSS compliance audit scope by 70%.
The Challenge
A mid-market digital payments provider processing 2M daily transactions reached a critical performance ceiling. Their monolithic Java application required full JVM restarts for deployments, creating 4 to 8 minutes of recurring service disruption. Nightly reconciliation batch jobs failed under holiday transaction peaks, demanding manual intervention from finance engineers.
Peak-hour authorization latency spiked to 800ms at P99, triggering cascading timeouts across merchant checkouts. Additionally, storing cardholder data across shared database tables placed the entire monolith under strict PCI DSS audit scope.
Architecture & Technical Approach
We decomposed the monolithic architecture into five independent microservices built with Go 1.23:
- Ingestion Gateway: Terminates TLS 1.3 and validates merchant HMAC signatures.
- Tokenization Vault: Isolates raw PAN data behind an encrypted HSM boundary.
- Risk Scoring Engine: Evaluates velocity rules against an in-memory Redis cluster in sub-5ms.
- Payment Router: Multiplexes transactions to banking acquirers with automated failover.
- Async Settlement Worker: Micro-batches ledger state from partitioned Redis Streams.
Core Engineering Specifications
- Runtime & Concurrency: Go services running in distroless Docker containers on Amazon EKS.
- Message Bus: Redis Streams partitioned by merchant ID to guarantee FIFO ordering per account.
- Database Architecture: PostgreSQL 16 with Citus horizontal sharding for transaction ledgers.
- Observability: OpenTelemetry tracing propagated through B3 headers to Jaeger and Prometheus.
Quantitative Benchmarks & Results
| Metric | Monolith Baseline | Microservices Target | Production Result |
|---|---|---|---|
| Daily Transaction Throughput | 2.0M | 10.0M | 11.4M Transactions/Day |
| P99 Authorization Latency | 800ms | < 50ms | 38ms |
| Deployment Interruption | 4 to 8 min | Zero | Zero Downtime (Rolling) |
| Settlement Window | 24 Hours (Nightly Batch) | < 30 min | 12 min (Continuous Micro-Batch) |
| PCI DSS Code Audit Scope | 100% of Application | < 35% | 28% of Total Codebase |
| Production Availability | 99.95% | 99.999% | 99.9994% |
Production Reliability & Lessons Learned
Separating the synchronous authorization path (which must complete under 50ms) from asynchronous ledger settlement prevented database lock contention. Automated circuit breakers instantly isolate degraded banking gateways within 200 milliseconds.