I recently wrote about how I improved subdomain routing in Cushion. Everything went smoothly while developing the new routing and there were no issues at all when deploying it to the review app, but strangely, something went wrong when deploying it to staging. Cushion’s app routes rendered correctly, but its assets all 404’d.

At first, I thought that maybe the assets didn’t build properly, so I SSH’d into the staging server and checked the directory, but all the files were there. I then double-checked the routing code, thinking that maybe something might be off there, but that wouldn’t make any sense because the review app—the identical production environment—worked perfectly well. Then what could it be?

After spinning for a bit, I finally thought to check the HTTP_HOST environment variable—the one I was checking for the subdomain conditional. Both development and the review app returned the expected hostname, but the staging server returned the Heroku hostname—not the staging hostname. Interesting!

It didn’t take much longer (despite the hours I already lost) for me to realize the issue. The one difference between the environments is that staging uses AWS Cloudfront to cache the assets with a CDN—to mimic production. In the Cloudfront distribution behaviors, there’s an option, “Cache Based on Selected Request Headers”, that’s set to “none” by default, which is said to improve caching. By choosing this option, however, Cloudfront will essentially pass the origin’s headers through to the server rather than sending the front-facing host’s headers. That’s why HTTP_HOST returned the Heroku hostname instead of the staging hostname.

To fix the issue, I simply needed to select the “Whitelist” option and whitelist the HOST header. As soon as the Cloudfront distribution finished propagating, everything worked as expected. I was then able to deploy the new subdomain routing to production where everything continued to work. With a staging environment that’s 1:1 with production, I can catch issues like this ahead of time rather than scrambling to fix them. While that’s nice and all, this experience reiterates that even two decades into coding, I still find myself spinning my wheels for a few hours on an issue that takes two clicks to fix. That’s the developer’s curse.