PostMortem: OVH DNS Outage

Issue Summary:
DNS resolvers on our OVH data center provider went down. Our mail sending infrastructure was unable to pull new mail tasks from the queue, and unable to deliver them due to inability to lookup MX records.

Timeline:
7/14/2025
17:54 [Downtime Begins] OVH DNS Outage, Cargo cannot pull messages off SQS queue, the few messages it has cannot be delivered because it cannot lookup MX records
18:01 [First Alert] BetterStack pings Matthew for "Gmail Deliverability"
18:01 [Acknowledge Alert] Matthew acknowledges alert, begins investigating
Matthew at first thinks it's the Gmail Deliverabilities Alert being too sensitive.
18:09 Matthew realizes it's a system wide outage, and posts to #ops in slack
18:11 [Public Status Posted] Matthew posts to status page: https://status.improvmx.com/incident/619603?mp=true
18:21 [Recovery] Matthew sees messages come in again, posts to #ops
18:23 [Public Status Resolution] Matthew posts resolution to status page.

Root Cause:
We don't have backup DNS servers on our OVH infrastructure. It would have failed over automatically if we had Google or Cloudflare set as backups.

Resolution and Recovery:
OVH's DNS servers recovered on their own, and the mail queue immediately began completing.

Corrective and Preventative Measures:
- install dnsutils on all servers, to ensure DNS query tools like dig are immediately available
- setup backup DNS resolvers on OVH Servers

Deliverability issues have been mitigated, we are continuing to investigate, and will post a postmortem within 24 hours.

We are examining reports of deliverability issues across email providers.