FeedMail Blog

FeedMail was offline for 26 minutes. During this period the website was unavailable and feed updates were not sent. This outage was caused by our CoreDNS resolver failing. While FeedMail continued operating normally for a while as most operations such as feed fetching and mail sending don't rely on the Kubernetes DNS server FeedMail does use the Kubernetes DNS server for a few operations such as connecting to it's own database. When database connections needed to be refreshed the DNS resolution failure caused FeedMail to become unhealthy and it was unable to continue operation. Timeline All times are in UTC . 13:28 Start FeedMail goes down. Website is offline and feeds are not being checked. 13:32 Detection Automated monitoring reported that the FeedMail website was unavailable. 13:38 Automated monitoring reported that feeds were not being fetched. 13:42 Kubernetes cluster update was started. 13:53 Mitigated FeedMail was restored to operation. The website was ag...