The State Machine That Saved a National Payment System (and What I Learned Building It)

When building the settlement engine for a national Instant Payment System, we hit the limits of Spring State Machine's built-in persistence — snapshots lost transition history, sub-machine restore was unreliable, and there was no concurrency protection. We built a custom layer that serialized the full StateMachineContext to Postgres via Kryo after every transition using a StateMachineInterceptor, stored an append-only transition log for audit, and enabled standby replicas to restore any in-flight settlement and resume processing. Concurrency was solved with optimistic locking via a Postgres UNIQUE constraint. The system survived node failures, rolling deployments, and traffic spikes in production without losing a single transaction.