Until recently, most development shops I’ve been a part of have performed software releases to production after hours, typically late at night. The conventional thinking is that releasing off hours minimizes the risk of disrupting users if there are bugs in the release that were not found in QA or if the web site must go down during the release process.
Over the past few months, we’ve tried something different. We release once every 3 weeks or so. The day before the release, the dev team works long and hard to ensure that QA is completed, all bugs are fixed and verified in QA environments. Then we all go home, get a good nights sleep, then start the release first thing in the morning when everyone is back at work. Our releases usually take about 45 minutes, but some can last several hours if we are pushing a lot of configuration changes. We rarely need to bring production completely down for a release, as we can shift load and take clusters out of service using a load balancer.
Reasons why I’ve come to really like this approach:
- If you release at the end of the day, often times not all your development staff is around. I’ve seen this many times where developers don’t stick around because they don’t have any code going in the release. But, that doesn’t mean you might not need that particular developer if something goes wrong, especially if you have to rollback.
- First thing in the morning, your development staff is fresh and ready to go. If you release at then end of a long day, and something bad happens in production, it can be brutal to try to figure it out. It’s like working 12 hours, then someone handing you a calculus problem. Good luck. The developer is much more likely to fix the issue quickly and correctly early in the day.
- Releasing in the morning also makes it easier to observe the production environment for a time after the release. A production issue might not be apparant right after the release, it may take an hour or two (ahem, a queue backing up). Releasing in the morning allows development to keep an eye on production during the day and make sure everything is running smoothly.
Releasing in the morning might will not work for everyone. If your production environment has so much load that it can’t afford part of the cluster going down, then this isn’t possible for you. But if you can technically do it, I suggest you give it a try. I’d love to hear back from others about their experiences with this.
The agile project management technique of planning poker is a popular estimation process for software development. My favorite Agile author Mike Cohn presents a good description of the practice on his blog here: http://blog.mountaingoatsoftware.com/?p=14.
Let me start off by saying that I really like planning poker. It’s a very clever way to estimate user stories. However, from a tech startup standpoint, where time is a critical resource, planning poker becomes a non-essential part of the agile process. Here’s why:
- The cost is material. My typical 2 to 3 week iteration has approximately 40 stories. I find that it takes between 2 to 5 minutes per story. Using 3 minutes as average, that brings us to 120 minutes. So my team of 10 people, spend 2 hours in planning poker. Multiply that by $75/hour and you get a cost of $1,500, not to mention the opportunity cost of not coding. At a startup, where time is so valuable, this is a tough pill to swallow.
- After several iterations and planning poker sessions, I began to realize that I could estimate just as good as the developers. I would compare actual hours reported by my developers spent on user stories to the story points estimated and calculated that my estimates had the same standard deviation as the whole team did the estimation. Even if you continue to practice planning poker, this is an excellent exercise to refine your estimating skills.
- Take shortcuts. The average bug tracked in our bug tracking software is about a 1 story point. Skip estimating every bug and just apply this average.
- If deadlines are less critical to your particular situation, then why care so much about your estimates? My web application dev team often has a release plan that changes every couple weeks. We are not tracking a burn down to know when the release is complete, because the story make-up of the release changes so frequently. The burn down chart is not actionable for us.
On particularly large stories or Epic’s, I ask the developer who will probably be working on the story to give me their estimate. But for the majority of user stories, I take care of the estimation. Maybe you can too?
This is one of the first books that I have read on the topic of Web Analytics. This book is not a deep dive of a particular tool like Google Analytics or Omniture, but rather a comprehensive description of a web analytics framework (Trinity) and a 6 month plan to implement. The book is targeted more towards medium to large companies. A small startup like my own couldn’t begin to implement all of the suggested actions. In some chapters, Avinash clearly notes which strategies (such as the testing chapter) are best for companies with limited resources, but overall he doesn’t always do this. I would have like to see him make minimum suggestions or short cuts for companies that don’t have even 1 FTE dedicated to web analytics.
A few key points from the book that I really found beneficial:
- Heuristic Evaluations: An easy way to quickly determine if a web page is designed well by going over a checklist of about 20 basic design rules of thumb
- 10/90 Rule: Spend 10% of your web analytics budget on tools and 90% on people.
- Web Analytics function should align with the business, not IT.
- 10% variability in web tools is not uncommon.
- 2% is standard conversion rate
- In Google, type in “site: <www.yourdomain.com>”. This will show you how well Google has indexed your site. If Google shows 50 pages, and you know you have 200, then something is stopping its spider from indexing your entire site.
- http://snipurl.com/poplink. Great tool to quickly view your website ranking across multiple search engines for a search phrase.
- 80% of the time, company employees are wrong about what customer’s want.
- About 20 search phrases represent the majority of search engine traffic to your website.
I definitely recommend this book, but be ready to pick apart what you can realistically do in the amount of time you can afford to spend on web analytics.
To quote Luke… “You ask the impossible”. Although making your boss at work and your family at home happy may not be as difficult as lifting an x-wing fighter out of a murky Degobah swamp with nothing more than your thoughts, it is still quite the challenge.
Everyone wants more. Your family wants more of you. Your board of directories want more. Your employees and co-workes want more. No one is ever happy with the amount of time you are able to devote to them.
A manager at a startup should expect to work 50 to 60 hours a week. Find an amount within that range that is sustainable, and try to stick to this. I find that I can do 10 hours a day during the week, and 5 hours over the weekend. So to get your hours in and be the most productive you can, try out these tips.
- Try to do 8 hours in the office, and 2 at home during the week. Sitting next to your spouse on the sofa in the evening allows you to multitask: spending time with the family while working.
- Be home for bedtime. If you have kids, make sure you are home at least 3 times a week for bedtime. I find that I can leave work around 6pm, get my kids bathed and put to bed, eat, then get back to work. Life is short, and you shouldn’t miss this special time of the day.
- I like to be home at 5 on Fridays, so I will work one late night a week so that I can cut out early on Friday.
- Save email for later. I find that addressing my email at night, when I’m a little tired, is a good use of my brain in a tired state. Doing this regularly gets people used to expecting 24 hour turn around on emails, rather than immediate response. Plus, doing email all at one time at the end of the day allows you to not get interrupted during the day.
- Use your time efficiently. Eat lunch at your desk. Get a smartphone with email. Your drive to work is a great time to check mail.
For those times that your spouse looses their patience with you, I suggest reminding them that if building a successful business was easy, everyone would do it.
The one position I continue to have trouble hiring for is QA. I manage a development team of 8 others. 5 Ruby on Rails developers, 2 Java and C# developers, and a system administrator. My team is quite sharp. Finding a QA person that can fit in to a Internet Startup Agile Development team has been quite a challenge. And if you are a small company, like mine, you only get one person for this role. The person you hire needs to be a rockstar. No budget to have multiple people.
So what makes a great QA person? Well I believe you should look for the following qualifications:
- QA Vocabulary. Nothing is more annoying than a QA candidate who can’t tell me the difference between white box and black box testing. There is a good set of software QA terms here http://geekswithblogs.net/srkprasad/archive/2004/06/02/5795.aspx and here http://www.aptest.com/glossary.html
- QA testing skills. I usually take a story that is ready for testing and have the candidate take a shot during the interview. I find it useful to leave the candidate alone while they do this.
- Resumes that have grammer/spelling errors… trash ‘em.
- Extension of the product manager. I don’t get to review every story that is developed and release, but the QA engineer does. So your QA person is an extension of your product people/person. It is critical the QA person get a feeling for what your product manager likes and dislikes.
- Hard Ass. You need to make sure that your QA candidate can stand up to manipulative developers that try to convince the QA person to “it’s not a bug, it’s a feature”. The QA person needs to not roll over everytime they are steamrolled.
- Flexible schedule. That’s right, the QA candidate needs to be comfortable with working long days and nights, especially just prior to a release.
- Relevant experience. Hiring a QA person for web testing that only has tested windows clients is not a good idea. You are a startup, you need results fast. Your not in training mode.
And of course, finding this person sure isn’t easy… so good luck!