Effective System Testing

With Agile, we all recognize the value of unit tests. System testing steps down quite a bit, but still important to us. We want to find the issues and fix them with as fast as possible, and as accurate as possible.

For tests, we all know this theory:

• If you change one thing that cause a problem, then we are quite sure the suspect. 
• If you change two or more things at the same time that cause a problem, we just confused.

So, it is a good thing change one thing at a time. In some teams which are funded well, they usually have one or more test system that just run a system forever, and call this kind of test system as longevity system.

However, let's do some simple math.

Assuming there are 7 defects in your system but in fact you have no ideas whether or not they exist. As a team, you need to find them and fix them in a given period, say a week and make sure you have a high quality product before you release to the customers. How can you achieve your goal?

If you can only handle one thing at a time, and you are lucky enough to encounter all of those 7 defects one by one, and fix them one by one, than you can do them one in a day by average.

However, we seldom could be so lucky. Especially if you are "careful", you can't find defects until them emerge at your customers' environment, which will hit you back hard.

And, remember, the one thing at a time solution may work well with sustaining project when you are facing a system has been in the market for 20 years. But with new products? It will never work. Slowing down the verification of fixing process will help you push a lot of unsolved defects into the market, much more than you could imagine.

So, it is not we don't know the one-at-a-time thing, but we simply can't. To make a good product, to effectively test your system, you have to do these:

• Make parallel testing possible. Decoupling is the key to this. Always think about how you are going to test your system, and how you can make sure you know this part of the system is working or not.
• Expose the problems, including the unknown as much as possible. Fail fast is the way to make things work. When with doubt, throw out the problem. This will help you find out the unknown as soon as possible. When the unknown becomes known, then you can provide the solution.
• Be sensitive. Whenever you are working on something, you need to have your theory and know what would happen. When something goes different, either you theory is wrong or the system is not implemented in the way you think. Be sensitive to your calculation and the system you build. You can both improve your theory and the system, both are very important. Many people think I had a very good troubleshooting skill, but actually I have better calculation skill, and a very good sense for what is right and wrong.
• Be confident and determined. Since the system has been well decoupled, you can make sure one thing will not interfere the others; since the all unknown are exposed as much as possible, your system are built based on things you know and you know well, or you can catch it right away; and since you have your theory and calculate it well, the system should work in the way you expect. Now you can be confident and determined. This is very important, and even more important than you would expect, because nowaday, we have to build system upon many open sources components, or commercial third party ones. We expect them provide us good quality, work exactly like their document. Usually they do, but sometimes, they don't. If you are not confident enough with you system, you have to spend a lot of time tracing problems that really not your own fault.

System testing is not easy. But with things I said above, you can make it much easier and more effective. 

My colleague, John M once told me, never slow down the development, but speed up the verification. His word makes a lot of sense to me.