Continuous Deployments

For years, deploying twice a week served us well. It gave us predictable release windows and enough coordination that production rarely felt risky.

That changed fast. AI-assisted development multiplied the number of pull requests we were opening, our new product needed constant hotfixes, and release branches got bigger every week. The process itself hadn’t changed; it was just suddenly the slowest part of shipping.

Peter Evans’ excellent create-pull-request action had been generating our production deployment PRs, and for a long time that was enough. But as releases grew, release-diff issues went from a weekly annoyance to an almost daily one.

To spread the pain around, I rotated “Ship Captains” every two weeks, chosen by a gloriously low-budget wheel spin that looked like it came straight from Temu. Sometimes the unlucky winner would lose most of a day untangling release issues before we could ship. The low point was a deploy PR titled “Thursday’s Production Deploy” that spent fourteen days in flight over the holidays and piled up 85 commits. Its entire description read “This PR merges the latest changes from main into production,” and Greptile skipped reviewing it automatically because “production deploy” was on its excluded-keywords list, so our biggest deploys got the least scrutiny.

Twice a week wasn’t going to survive any of this. We needed continuous deployment.

Earning the right to ship continuously

Before pitching it, I wanted to know where engineers were actually losing time. Our smoke suite had grown to eighteen minutes, so engineers regularly waited half an hour to discover a one-line type error, then another half hour to verify the fix. Pull requests lingered and merge conflicts piled up while everyone waited for the next window.

Around the same time, autonomous coding agents started contributing real code, and they could open pull requests much faster than we could reasonably ship them. Shipping more often was never really the goal, though. The point was shrinking the gap between writing code and finding out whether it actually worked.

Leadership gave me two focused weeks to build the first version, and I spent most of them reducing risk rather than automating. I cut the smoke suite from eighteen minutes to four by moving broader regression coverage into scheduled suites; the old suite’s breaking point had been a single flaky test that held nine pull requests out of production for over an hour. That meant accepting that some regressions would surface after deploy instead of before, so I added post-deploy health checks: a deploy that fails its check is automatically rolled back to the previous Vercel deployment, and a slipped regression means minutes of exposure rather than a blocked pipeline. I also set up builds so only the apps affected by a change get built and deployed.

It’s a custom pipeline mostly because the off-the-shelf routes didn’t fit. Vercel’s built-in promote reuses the staging artifact without rebuilding, and our environment config gets baked into the client bundle at build time, so promoting staging would have pointed production traffic at the staging database. Each environment gets a fresh build instead; the few extra minutes per deploy are the price of correctness. GitHub’s merge queue gates pull requests before they merge, but the failures that actually hurt tend to show up after merge, against a real deployment, so that’s where the gate went.

Once deployments got small, shipping stopped being an event. Nobody had to push the team to release more often; it was just the path of least resistance.

Designing for the engineers sitting twenty feet away

The automation lives in GitHub Actions, but Slack is the front door; a dashboard would have been one more tab nobody opened. Every pull request shows up in a dedicated channel under its author’s Slack identity, and the same message updates in place as it moves through the pipeline.

Draft. Opened. Awaiting reviews. Approved. Deploying to staging. Deploying to production. Shipped.

Once engineers started using it, I noticed how people decided whether a pull request was worth reviewing. Before clicking into GitHub, they checked two signals: Greptile’s confidence score and whether CI had passed. Those two usually answered the question before anyone read a line of code, so I surfaced both directly in the Slack message and kept them updated in real time.

Over time the feed turned into the team’s default view of engineering activity.

Measuring operational health

Each week the pipeline posts a deployment health report: deployment frequency, change failure rate, rollback rate, and median time to recovery (the DORA metrics, more or less). The trends matter more than the values; a metric moving the wrong way usually flags a problem before anyone feels it. Still, the raw numbers were hard to argue with: median merge-to-production time fell from about 36 hours to 10 minutes, and the change failure rate settled below 2% even as we went from two deploys a week to twenty a day. In the release-train era, about one release in three had needed an emergency hotfix behind it.

The problems we didn’t anticipate

The first thing that bit us was a failed rollback. One night a deploy failed its health check, the automatic rollback tried to force-push the production branch back, and branch protection rejected the push, killing the recovery halfway through. Production quietly kept serving the bad commit, because the pipeline only inspects its own state while a run is in progress; between runs, nothing was watching. Rollbacks now land as regular commits instead of force-pushes, and a drift monitor compares what each production app is actually serving against the production branch every fifteen minutes, pinging the channel when anything diverges.

Smoke-test failures were the other big one. We had notifications, but engineers still had to dig up context before they could do anything useful with them. The alert now includes the pull request, the author, the failing tests, links to CircleCI, and our troubleshooting guide: enough to decide at a glance whether to investigate now or let the next deployment heal it. More often than not, the answer comes from a single prompt to an LLM with the logs attached.

Knowing when not to ship

Continuous deployment doesn’t mean production should always be changing. Customer demos or infrastructure maintenance sometimes want a stable environment, and I didn’t want the answer to be release branches again. Promotion freezes pause production deploys while engineers keep merging into main; lift the freeze and production catches back up on its own.

Freezes cover the pipeline, but not every feature should go live the moment it merges. Some span several pull requests; some are finished but waiting for the right moment to be released into the wild. With no release branches there’s nowhere for that work to sit, so it ships dark behind Vercel Flags: the code deploys with everything else, and turning it on for users is a flag flip rather than a deploy.

Looking back

Today the team averages roughly twenty production deployments a day, and our deployment costs are about what they were at two a week: each deploy now builds only the affected apps and runs a four-minute suite instead of an eighteen-minute one.

Building confidence in the system took much longer than building the automation. Almost every improvement after the first version started as a passing comment in Slack or a habit I noticed over someone’s shoulder, and that turned out to be my favorite part of the project.

These days the system has mostly faded into the background; engineers merge and deployments happen without anyone watching. The Ship Captain wheel is retired.