Release Web Software to Production in the Morning
Until recently, most development shops I’ve been a part of have performed software releases to production after hours, typically late at night. The conventional thinking is that releasing off hours minimizes the risk of disrupting users if there are bugs in the release that were not found in QA or if the web site must go down during the release process.
Over the past few months, we’ve tried something different. We release once every 3 weeks or so. The day before the release, the dev team works long and hard to ensure that QA is completed, all bugs are fixed and verified in QA environments. Then we all go home, get a good nights sleep, then start the release first thing in the morning when everyone is back at work. Our releases usually take about 45 minutes, but some can last several hours if we are pushing a lot of configuration changes. We rarely need to bring production completely down for a release, as we can shift load and take clusters out of service using a load balancer.
Reasons why I’ve come to really like this approach:
- If you release at the end of the day, often times not all your development staff is around. I’ve seen this many times where developers don’t stick around because they don’t have any code going in the release. But, that doesn’t mean you might not need that particular developer if something goes wrong, especially if you have to rollback.
- First thing in the morning, your development staff is fresh and ready to go. If you release at then end of a long day, and something bad happens in production, it can be brutal to try to figure it out. It’s like working 12 hours, then someone handing you a calculus problem. Good luck. The developer is much more likely to fix the issue quickly and correctly early in the day.
- Releasing in the morning also makes it easier to observe the production environment for a time after the release. A production issue might not be apparant right after the release, it may take an hour or two (ahem, a queue backing up). Releasing in the morning allows development to keep an eye on production during the day and make sure everything is running smoothly.
Releasing in the morning might will not work for everyone. If your production environment has so much load that it can’t afford part of the cluster going down, then this isn’t possible for you. But if you can technically do it, I suggest you give it a try. I’d love to hear back from others about their experiences with this.
4 Comments »
Leave a Reply
About
Thanks for visiting! My name is Keith Forsythe and this is my blog about product development, product marketing, mobile devices and general business practices at an Internet Startup.
I am currently co-founder and Director of Product Development at RideCharge, Inc (Taxi Magic), a startup mobile and web taxi booking service. Prior to RideCharge, I was an IT project manager as CVS/Caremark, and a software engineer at Baxter Healthcare. I have a MS in E-Commerce Technolgy from DePaul University and a BS in Chemistry from the University of Illinois at Champaign Urbana.
[...] There are more good reasons too, read the full article on SkepTek. [...]
Pingback by Simpltry » Blog Archive » Deploying Web Software Well Rested | February 27, 2009 |
Good post. It’s never fun to release something at 2am and then wake up to a 7am call that the site is going haywire and there are already 20 support emails about it.
Great points, especially the third: there’s nothing worse than getting paged when the site falls over under peak load after a late-night Herculean release!
Late night releases are especially annoying for east-coast shops that have to wait for their west-coast customers to sign off before starting the release. Waking up a little early to do an early morning release is much nicer than staying up all night waiting for west-coast peak to tail off.
Day-time releases are not without risk, though. As you’re bringing nodes down for the release, you’re running with diminished capacity, so you’ve got to make sure the remaining nodes can handle peak load. Sometimes the release itself can trigger additional load (such as when external javascript version numbers are bumped, invalidating browser caches, or when image sizes change, invalidating CDN caches). The onslaught of traffic as clients refresh their caches can be enough to overwhelm the remaining nodes and send the site into a tailspin.
im new to this