Over the last month I have presented or participated in several user groups and conferences. There was Southern Fried Agile in Charlotte, Agile 2010 in Orlando, a PMI Atlanta professional growth event, meetings of Agile Atlanta and the Atlanta Scrum Meet-up, and Product Camp Atlanta (not surprisingly in Atlanta). These conferences give me an opportunity to meet with people doing, or attempting to do agile in the enterprise. There are a lot of common patterns that I am seeing. One of the most common patterns is that we are doing QA all wrong in most companies.
Imagine if someone approached you up front and said “I have an idea. Let’s design our product development system like this. First, development is going to take in less then precise requirements. Then development is going to build something from the requirements. Then we are going to deliver this to a separate group that will then figure out how to test it for accuracy. Then that team will test it. When the product doesn’t pass the test we just defined we will return it to development for them to fix it.” Is this the way that your test organization operates with your development organization? Does the relationship between QA and development seem reasonable? Why are challenges between QA and development so prevalent in our organizations?
We Don’t Understand Quality Assurance
Quality Assurance and Quality Control are different things. We have Quality Assurance organizations – but most often they are involved in Quality Control control activities.
Quality Control: Find defects. These activities are focused on validating the deliverable. They are performed after the product (or some aspect of the product) has been developed. It also enforces fit to specification. This protects production and our customers from getting defective deliverables that do not meet the specification. But when done on a completed product it isn’t doesn’t enable flow and defects found downstream of the deliverable result in rework in development.
Quality Assurance: Prevent defects. These activities are focused on the process used to create the deliverable. They are performed as the product is being developed. So, Quality Assurance should be helping us improve our processes so we don’t create defects in the first place. A key point to this is that creating a common understanding of the specification and how we will validate it up front is really important to eliminating defects.
If your Quality Assurance activities are primarily focused on finding defects and ensuring fit to requirements after the fact then your QA group is primarily doing Quality Control. Getting QA involved in improving the specification communication process through is a good place to start. A specific technique might be exploring communicating acceptance criteria to development with the same examples QA will use to perform acceptance tests.
We Don’t Understand the Cost of Rework
Finding defects after the product has been delivered ensures that you will generate rework. Rework from failed QA tests are extremely expensive to flow. Sending work back into development interrupts work in process. This has a dramatic effect on cycle time across the system – which results in increased cost of every item in the system.
After the fact testing is also typically done in batches when a release or iteration is delivered to QA from development. Defects uncovered in these batches create bottlenecks at development. Just like when traffic backs up on the highway, bottlenecks in product development result in a long term cost to the overall system.
We can dramatically reduce the cost and adaptability of the system by reducing this rework. One way to address this defect bottleneck is to perform quality control frequently – not in batches. Another way is to prevent defects through effective quality assurance in specification understanding and in development. This will reduce the number of defects produced that escape to Quality Control.
We Have Designed the Product Development System Wrong
We used to teach that there needed to be a separation of testing and development. Having them both report to the same person was like having the person that wrote the checks in accounting also reconcile the check book. It was too easy to cheat. We created QA teams that were completely separate from development – and that reported up through different management structures. I was talking to the director of development at a company where the QA and development chains of command meet at the CEO. When you design the organization to QA and development in different silo’s you will make it very difficult to have effective Quality Assurance involvement. You will ensure Quality Control behavior.
This separation made sense at one point (I think). It doesn’t make as much sense today – especially as the resulting silos result in slower delivery times and increased costs. What has changed in the last 10 years is:
- the cost of setting up testing environments is much lower
- our ability to rapidly and frequently produce testable deliverables for these environments is much greater, and
- our ability to perform acceptance, integration, regression, performance, load, and stress testing frequently is enhanced.
Quality Assurance needs to be part of the development team and process. We need to change our thinking around this.
So we need to integrate Quality Assurance very early in the process. We need to ensure early arrival of acceptance test examples to development. But, we still need independent test teams when we are doing large or complex projects that require integration of many team’s outputs. Independent test teams can be running parallel testing or test-immediately-after delivery. When they are getting high quality components, they focus on user experience, performance, load, and stress testing. Move everything you can into the development teams (enhanced with quality assurance) so you can produce low defect escape rates. Then eliminate the defect bottleneck and increases in cycle time by treating most of the defects at the independent test team as new specifications to development.
We have the Wrong Expectations
We Expect Perfection – Not Improved Value
Our QA organizations tend to be focused on defect identification and perfection in delivery. The focus of QA, and everyone in the organization, needs to be on producing value-add and delighting the customer – not on defect-free code. This doesn’t mean that defects are okay – but every application has some defect or limitation. Unless you are building aircraft navigation systems or other critical systems you are going to deliver some defects. Make sure the defects don’t block business value – but there is a point where it is more important to the customers and business to deliver value.
We Expect Local Optimization – Not Flow
Rather than perfecting Quality Control, QA can add a lot of value helping the organization enhance the flow of value through the entire product development system. Delivering a unit of value requires analysis, development and testing. There is no value delivered internally. The local optimization problem of improving one area at the expense of another is not useful. Crossing the boundaries between analysis, development, and testing is needed to optimize flow of value. The various teams need to figure out how to do it together – not just improve their area.
Expect our Current Constraints to be Permanent
Organizations labor under the belief that their constraints are fixed. Find a way to break down the barriers that lead to your constraints. Don’t accept the current constraints as permanent. Increase thinking at the system level to find ways to improve the product development process.
We Don’t Use the Tools Available to Us
There is an impressive amount of automation and related techniques available at very low cost to help build the QA environment that allows for fast feedback, building on progressively tested levels, and improving opportunities for frequent testing. Developers should be using tools that support automated unit testing and only checking in code that passes all their unit tests. This provides a basis for validation and refactoring. Test driven development or test just after development should be ubiquitous – but it is not. Continuous Integration environments that ensure that each check-in results in a valid and testable platform help teams perform integration and build validation. This helps everyone work on a stable build and ensures that no one gets way off track. There are automated tools for acceptance testing and some levels of UI testing. When the tools are combined you can build a very powerful regression testing environment that can also rapidly perform performance, load, and stress testing.
The lack of general adoption of these tools probably has multiple roots. Sometimes it can be difficult to leverage all of them with legacy systems. They require new skill sets and generate new artifacts that have to be maintained. On the surface, they automate tests that are currently performed by QA groups and so their is fear and organizational threat. But some subset of these tools should be broadly adopted by each team. QA teams can focus their efforts on valued added items like Exploratory Usability Testing and Release Validation.
Summary
Most organizations are doing QA all wrong. It is time to start doing it right, the cost of low quality in dollars, time, customer satisfaction, and the engagement of employees depends on it.
- Switch their primary focus to Quality Assurance
- Help the organization reduce re-work
- Integrate QA and QC with analysis and the development organization and work to eliminate defects – not just identify themF
- Enable value, flow, and ongoing improvement
- Leverage proven tools and techniques across the Product Development Organization to increase the value delivered
The status quo is not sufficient to meet the needs of most organizations. While not all of these approaches will be right for every situation, the rate of adoption is too low to be justified. There is far too much latent value being tied up in organizations because we are doing QA all wrong.. What obstacles do you see in your organizations?