30.10.13
29.10.13
23.10.13
22.10.13
19.10.13
18.10.13
7.10.13
6.10.13
Effective System Testing
With Agile, we all recognize the value of unit tests. System testing steps down quite a bit, but still important to us. We want to find the issues and fix them with as fast as possible, and as accurate as possible.
For tests, we all know this theory:
For tests, we all know this theory:
- If you change one thing that cause a problem, then we are quite sure the suspect.
- If you change two or more things at the same time that cause a problem, we just confused.
So, it is a good thing change one thing at a time. In some teams which are funded well, they usually have one or more test system that just run a system forever, and call this kind of test system as longevity system.
However, let's do some simple math.
Assuming there are 7 defects in your system but in fact you have no ideas whether or not they exist. As a team, you need to find them and fix them in a given period, say a week and make sure you have a high quality product before you release to the customers. How can you achieve your goal?
If you can only handle one thing at a time, and you are lucky enough to encounter all of those 7 defects one by one, and fix them one by one, than you can do them one in a day by average.
However, we seldom could be so lucky. Especially if you are "careful", you can't find defects until them emerge at your customers' environment, which will hit you back hard.
And, remember, the one thing at a time solution may work well with sustaining project when you are facing a system has been in the market for 20 years. But with new products? It will never work. Slowing down the verification of fixing process will help you push a lot of unsolved defects into the market, much more than you could imagine.
So, it is not we don't know the one-at-a-time thing, but we simply can't. To make a good product, to effectively test your system, you have to do these:
- Make parallel testing possible. Decoupling is the key to this. Always think about how you are going to test your system, and how you can make sure you know this part of the system is working or not.
- Expose the problems, including the unknown as much as possible. Fail fast is the way to make things work. When with doubt, throw out the problem. This will help you find out the unknown as soon as possible. When the unknown becomes known, then you can provide the solution.
- Be sensitive. Whenever you are working on something, you need to have your theory and know what would happen. When something goes different, either you theory is wrong or the system is not implemented in the way you think. Be sensitive to your calculation and the system you build. You can both improve your theory and the system, both are very important. Many people think I had a very good troubleshooting skill, but actually I have better calculation skill, and a very good sense for what is right and wrong.
- Be confident and determined. Since the system has been well decoupled, you can make sure one thing will not interfere the others; since the all unknown are exposed as much as possible, your system are built based on things you know and you know well, or you can catch it right away; and since you have your theory and calculate it well, the system should work in the way you expect. Now you can be confident and determined. This is very important, and even more important than you would expect, because nowaday, we have to build system upon many open sources components, or commercial third party ones. We expect them provide us good quality, work exactly like their document. Usually they do, but sometimes, they don't. If you are not confident enough with you system, you have to spend a lot of time tracing problems that really not your own fault.
System testing is not easy. But with things I said above, you can make it much easier and more effective.
My colleague, John M once told me, never slow down the development, but speed up the verification. His word makes a lot of sense to me.
5.10.13
3.10.13
Preconditions for Agile
I had lunch with two good friends, and we talked about Agile. With my experience, I truly believe that Agile is the way we can make things right. But one of my friends is skeptical.
After I heard his concerns, I understood what he concerned about.
We talked about communication. He said communications also meant disruptions. Shouldn't we avoid too much disruptions in the team? I said yes. There is a precondition for interaction in Agile, which is that we should avoid unnecessary communications.
Then I talked about evolutionary design, which is also my friends' concern. He prefers good design upfront and I said yes, you are not wrong, because evolutionary design was once proved failed. In early days of software development, some 1980's, software pioneers found that there was a change cost for software development, which meant that if we found something need to change at the earlier phase, it would cost much less than we change something at the later phase. Say, for example, if we find out a problem when doing requirement analysis, it could cost us one dollar. If we don't find this issue but leave it to test phase, it could cost us 1000 dollars. With this thought, we have the waterfall SDLC.
However, along with the software development experience, people found out waterfall was not the way we can successfully develop software. Waterfall was based on one assumption:
After I heard his concerns, I understood what he concerned about.
We talked about communication. He said communications also meant disruptions. Shouldn't we avoid too much disruptions in the team? I said yes. There is a precondition for interaction in Agile, which is that we should avoid unnecessary communications.
Then I talked about evolutionary design, which is also my friends' concern. He prefers good design upfront and I said yes, you are not wrong, because evolutionary design was once proved failed. In early days of software development, some 1980's, software pioneers found that there was a change cost for software development, which meant that if we found something need to change at the earlier phase, it would cost much less than we change something at the later phase. Say, for example, if we find out a problem when doing requirement analysis, it could cost us one dollar. If we don't find this issue but leave it to test phase, it could cost us 1000 dollars. With this thought, we have the waterfall SDLC.
However, along with the software development experience, people found out waterfall was not the way we can successfully develop software. Waterfall was based on one assumption:
- We could find the right answers upfront.
But actually we couldn't. Not for requirements, not for development tools. We even have problem with human resources, because people come and go more often than we would have expect.
That why we have to step back and think about evolutionary design and iterations.
The key issue we need to handle in evolutionary design is about the change. If we can flatten the change cost curve, then we can use evolutionary design again. In another word:
- If we can change something easily later, it is less important for us to have it right upfront.
So, this is another precondition for Agile: you need to make sure you can change the current design, decision, or implementation easily before you apply Agile. Otherwise, you are going to prove something has been proved in 1970's.
The answer for this was from Kent Beck. He found out two things to solve the change curve issue:
- Unit tests
- Refactorings
But my friend asked me again: sometimes unit test doesn't help, why?
Here comes that precondition for unit test: you have to have a system that unit test friendly.
Unit test means that you can verify the system with a very small unit and make sure it work as expect. Two things need to address here:
- Your system quality could be assured mainly by unit tests.
- Unit tests needed to be simple and small.
Say, for example, if quite a portion of your system are running on some logic that your unit test could not reach, then how can unit tests help the quality?
On the other hand, if your unit tests are monsters, very difficult to understand or very difficult to trace problem, then how can those unit tests help improve the velocity?
That's why we have to advocate decoupling and cohesion. That's why we need Agile architecture for Agile development. That's why we need SOLID.
That are all preconditions for Agile development. If you miss them, you are not doing Agile at all, and you are possibly going to fail, and fail very badly, even worse than those days you are following waterfall.
2.10.13
Subscribe to:
Posts (Atom)