I’ve been working on a new project recently, which I’m excited to launch imminently. In fact, you can even see it in action at the bottom of this post 😊 However, as I sit here on this Saturday morning in the interminable queue for Oasis tickets I suddenly noticed that the SSL certificate for the site had expired a couple of days ago. Firstly, I need some monitoring on that, but also it had me scratching my head for a while as to how to resolve.

I’ve blogged previously about how I use Coolify for self-hosting and I absolutely love it, however the downside to using any abstraction is that when something goes wrong it’s not always easy to get to the root of the problem.

TL;DR: I ended up connecting to the server directly and restarting the Traefik container (named coolify-proxy) which forced the certificate to regenerate.

So, I started with what I could see in front of me in Coolify. Everything looked right, and it definitely hadn’t changed recently, so I fired off a re-deploy (several re-deploys) to see if that cleared the issue. Of course, it didn’t.

Next, I started reconfiguring to see if I could figure out what was causing it. Comparing it to other projects I have in Coolify I noticed that this more recent project had Caddy available as an alternative reverse proxy, so I started removing and editing tags and firing off further redeploys in case that was the issue. No luck.

Eventually I went beyond Coolify, connected to the server and got into the underlying Docker instance. When listing the containers (docker ps) I noticed there is a Traefik container named coolify-proxy so I pulled up the logs for that (docker logs coolify-proxy) and noticed the following.

Error renewing certificate from LE… Timeout during connect (likely firewall problem)

I noticed within the detail of that error that it was trying to connect to http rather than https, so I opened up port 80 on the sever - still to no avail. I was still on a bit of a wild goose chase though, so I jumped back into Coolify and started changing configuration settings such as switching off “force https”, explicitly including the http:// domain - multiple redeploys later and still no closer to a resolution.

I next observed some other errors in the logs:

Router defined multiple times with different configurations

This sent me down another rabbit hole. Especially, once I found this GitHub issue which has since been patched, but I started wondering whether my site had been created on the erroneous version. So, I deleted the site, re-created it - same issue with it using the expired certificate!

I was close to reaching out to the Coolify creator directly, when I tried one more thing. I found a post on the Traefik blog which gave me the instructions to force-update Let’s Encrypt certificates. As it happens it was as simple as:

docker restart coolify-proxy

New SSL certificate generated and we were back in business! I’m still not 100% sure what caused the initial issue, so that is going to need further investigation, but at least now I have until 29th November (next expiry date) to find out. Unless one of my other sites runs into the same issue first… 😬