Post Mortem

(also posted to improvmx.com/blog/2025-09-23-post-mortem)

Summary

Spike in connections overloaded our SQL server and triggered a latent performance degradation in SQL lookups, causing delayed delivery until performance mitigation.

Timeline

11:30AM EST: Incoming mail connections spike to 30x steady state, causing increased load on our servers
11:49AM [Downtime Begins]: Emails begin being delayed/backed up.
11:58AM [*Fir...