Based on one of my new guiding principles for Cushion—“test code that should be tested”—I started looking into adding front-end tests, using Jest, to Cushion’s Heroku CI setup. Currently, CI takes roughly 6:30 to finish, which includes spinning up the server, installing Ruby and the Ruby gems (often cached), migrating the test database, then running all ~6,000 backend tests. Compared to how slow it used to be, this feels pretty good.

That said, adding front-end tests also means I need to set up the Node environment, which includes installing Node.js, Yarn, and Cushion’s Node.js dependencies, then running the Jest tests. Despite only having one initial test to start, this adds an extra two minutes to Cushion’s CI run, resulting in a total time of roughly 8:30. This is essentially 30% longer and will only get worse as I write more tests. I knew this when I started, but I hoped it wouldn’t be this bad for simply adding the Node environment.

Heroku CI uses the same “dynos”, or servers, that they offer to run Cushion’s web servers. By default, CI runs on their “performance-m” dyno, which is one of their more powerful (and expensive) dynos. While it is expensive to run 24/7, month-over-month, it literally costs pennies to run CI. Because of this, I wondered how much I could improve Cushion’s CI time by simply upgrading the dyno.

In Cushion’s app.json file, which is used to configure CI, I added a formation configuration to specify the new dyno:

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "size": "performance-l",
          "quantity": 1
        }
      }
    }
  }
}

On paper, this provides over 4x more compute speed and 5.5x more memory at only 2x the cost, but if it also speeds up the CI run, it would actually cost less than 2x—not that it really matters since we’re again only talking about pennies. If I could take a few minutes off CI time by spending a few extra pennies, I’d do it in a heartbeat—and I am! With the new dyno, I was able to shave off 3 entire minutes, bringing Cushion’s CI time down to 5:30. While this isn’t nearly as nice as it would’ve been pre-Node.js, it’s way better than 8:30.

Since I’m saving so much time, I was actually curious to find out exactly how much extra it costs. Using Heroku’s pricing example, it turns out that I’m only spending an extra penny per run:

// Before (performance-m at 8:30)
$250/month * 8.5 minutes / 43200 minutes/month = $0.05

// After (performance-l at 5:30)
$500/month * 5.5 minutes / 43200 minutes/month = $0.06

This is amazing, but I will say, it’s laughable how expensive and underpowered these dynos are compared to what I could get elsewhere. I’m sure I could run Cushion for a fraction of the cost at DigitalOcean, but with Heroku, I get to offload the devops responsibilities to their devops crew, which I gladly pay for instead of managing it myself, as a self-taught front-end dev with a graphic design degree. While the peace of mind is certainly worth it, I still wish their dynos’ specs were more up-to-date than the ones they announced in 2015.

Going forward, I’m satisfied with the ~5-minute CI time, but I’m also eager to explore Heroku CI’s ability to parallelize test suites. In the formation config above, you’ll notice the quantity field, which is used to spin up more dynos per CI run. This would do nothing for my single front-end test, but I’m really curious how much it could affect my ~6,000 backend tests. I’ll save that for another time.