To Release or Not To Release
Deciding when to release software is a tough decision. There are drawbacks to both shipping too early and too late, and unless you have a strategy for determining when software is ready to be released, you could be damaging your company's bottom line and reputation.
Shipping software that still contains major defects will not only harm your relationship with existing customers but also significantly impact your operating costs. Studies suggest that fixing a bug in the field costs as much as 10 times more than fixing it during the development phase. It’s therefore critical that your release is of high quality before it goes out to customers.
Then again, if you're a start up, your main priority is not future operational costs. It’s being able to survive another quarter. And the only way to achieve that is by getting the product out the door and in the hands of customers – now!
So the question remains: when should you release your product?
Understand What You're Shipping
While it’s not necessary to reach the zero-defect mark before releasing your software, it is necessary to understand how many defects it contains. The only way to get a feel for the quality of the product is to test it appropriately.
People often think of Quality Assurance as a phase where testers verify the functionality of the product. While this is true, functionality is only a subset of what needs to be tested. What about performance, scalability, usability, reliability, installation, security, and failure paths? Determine ahead of time which of these areas you want to test, document them in your test plan, and do not release the software until you've successfully executed the required test cases.
Another way to ensure sufficient test coverage is to plant software defects, a method known as defect seeding. This practice requires you to purposely introduce software bugs in various areas of the product without informing the QA team. When QA has detected and reported all of the seeded defects, you can be confident that they have properly tested all areas of the software.
Gate All Software Defects
Once all areas of the software have been tested and defects have been entered in a tracking system, you need to determine which defects need to be addressed prior to releasing the application, and which ones can wait until a maintenance release or a patch. Many companies use metrics to determine how many defects should be fixed. For example, a management team might have a policy to fix 95% of all serious defects, and 80% of all minor defects, or they may set the objective to only allow 20 serious and 50 cosmetic defects in the final release.
While those methods are somewhat effective, I instead recommend that you put a team together that includes representatives from Product Management, Product Development and Quality Assurance. This team should meet on a regular basis to review the list of defects, assess them, and gate them against a specific release and/or patch. While this method is tedious, it gives you a more realistic view of the quality of the product.
Generally speaking, there are areas of the software that are more important to your customers than others. Why should you waste time hardening a section of your software that your lead customer will not use? Why not instead document the defects in a release note, ship the release to this lead customer, and address the remaining bugs in a maintenance release? Keep in mind that, from a business perspective, time-to-market is more important than quality, and while you do not want to ship a product loaded with critical defects, you should never aim for the zero-defect mark. Just remember to manage your customer’s expectations, and be honest about the quality of your release.
What about Industry Statistics?
Some companies prefer to use industry statistics to determine when to release their software. Two frequently used metrics include the number of bugs found per thousand lines of code (KLOC), and the number of bugs fixed per KLOC.
While such statistics give you a sense of control, they do not provide a realistic view of the quality of the product. Statistics are only valid if you can control all the independent variables, which is usually not the case. Perhaps this release had more junior developers than the previous one. Perhaps you're using a different programming language that is a lot less permissive. Perhaps you QA entrance criteria were too simple. Take my advice. Do not strictly rely on statistics to determine when your software is ready to be shipped. If you do, there is a strong possibility that you're going to ship too early, or too late!
Create a Release Checklist
No matter which strategy you choose to guarantee the appropriate test coverage and prioritize and gate defects, plan ahead of time, and track your progress with a checklist. Before releasing your product, make sure each criterion on the checklist has been met, without any compromises. After all, if you don't follow your plan, why bother planning in the first place?
This article was originally published on www.gantthead.com.