Ghost would not log in

I had Ghost deployed using k8s for quite some time, the helm chart was running pretty smoothly and I could post my infrequent posts perfectly fine. Then one day... I couldn't. It was very confusing. In hindsight I could have checked the changelogs, alternatively it should be handled better in the code. But overall, the issue was very vague and difficult to diagnose.

TL;DR

It was the mail service blocking login. I use Hetzner and Amazon SES to send emails; I never had a reason to test that the emails actually sent and so I did not see that my send port was 465. Hetzner blocks this port by default, same for port 25, which makes absolute sense. Use port 2465 instead.

A recent addition to Ghost was to implement 2fa to the admin screen, which meant that when you try and log into the admin screen it tries to send out an email to confirm your identity. This request is getting blocked by Hetzner's firewall and resulting in a timeout.
Ideally there should be a short timeout to the SMTP server, allowing the webpage to respond saying "Your SMTP server did not respond within the expected timeframe." This would be a lot clearer than browser timeouts and no info.

I figured it was this when delving into the source code and found the session create endpoint. This eventually led me to finding that it would try and send an auth code email, which I knew port 25 was blocked but I did not know about 465. The timeout all makes sense now, sending an email and it would timeout would likely be some sort of firewall thing. Let me check its doing what I expected, since I already updated the config to use 2465 now. So then I delved into my configs and found that the SES config wasn't even being applied, it was falling back to using the "Direct" method. It was all basically down to the env vars not applying the use of SES, and Direct has a bug that lets it timeout when trying to send with no email config.

I swapped from using env vars to using a static config.production.json file. Once that was all sorted I was able to log in. Unfortunately for me I had already moved the blog to a good old fashioned Docker Compose deployment and killed off my k8s stack. I did half-think it was the database communication going all wonky, between Ghost and MySQL. All that effort and it wasn't even k8s at all 😅.