Release Web Software to Production in the Morning
Until recently, most development shops I’ve been a part of have performed software releases to production after hours, typically late at night. The conventional thinking is that releasing off hours minimizes the risk of disrupting users if there are bugs in the release that were not found in QA or if the web site must go down during the release process.
Over the past few months, we’ve tried something different. We release once every 3 weeks or so. The day before the release, the dev team works long and hard to ensure that QA is completed, all bugs are fixed and verified in QA environments. Then we all go home, get a good nights sleep, then start the release first thing in the morning when everyone is back at work. Our releases usually take about 45 minutes, but some can last several hours if we are pushing a lot of configuration changes. We rarely need to bring production completely down for a release, as we can shift load and take clusters out of service using a load balancer.
Reasons why I’ve come to really like this approach:
- If you release at the end of the day, often times not all your development staff is around. I’ve seen this many times where developers don’t stick around because they don’t have any code going in the release. But, that doesn’t mean you might not need that particular developer if something goes wrong, especially if you have to rollback.
- First thing in the morning, your development staff is fresh and ready to go. If you release at then end of a long day, and something bad happens in production, it can be brutal to try to figure it out. It’s like working 12 hours, then someone handing you a calculus problem. Good luck. The developer is much more likely to fix the issue quickly and correctly early in the day.
- Releasing in the morning also makes it easier to observe the production environment for a time after the release. A production issue might not be apparant right after the release, it may take an hour or two (ahem, a queue backing up). Releasing in the morning allows development to keep an eye on production during the day and make sure everything is running smoothly.
Releasing in the morning might will not work for everyone. If your production environment has so much load that it can’t afford part of the cluster going down, then this isn’t possible for you. But if you can technically do it, I suggest you give it a try. I’d love to hear back from others about their experiences with this.
About
Thanks for visiting! My name is Keith Forsythe and this is my blog about product development, product marketing, mobile devices and general business practices at an Internet Startup.
I am currently co-founder and Director of Product Development at RideCharge, Inc (Taxi Magic), a startup mobile and web taxi booking service. Prior to RideCharge, I was an IT project manager as CVS/Caremark, and a software engineer at Baxter Healthcare. I have a MS in E-Commerce Technolgy from DePaul University and a BS in Chemistry from the University of Illinois at Champaign Urbana.
