Why? Simply because this is the way software is built. To be more specific, there are two reasons for that:
Firstly, software is built by people mentally, instead of physically. There are many other things are built mentally, such as paintings, poems, and music. However, look at all those other things, how many are built by group of people? And how many of them need to be "useful"? For software, it need to build by group of people, and it need to be useful.
Secondary, software system are based on a rapidly changing industry, which is the silicon industry. The Moore's Law is the destiny of software engineering, and where the miserable comes from. One day a friend of mime argued with me why software engineering couldn't work as civil engineering. I asked him to imagine if the material used for constructions became 50% lighter in 18 months, 50% thinner in 18 months, what would happen in this industry. Then he got my point. And yes, this is where the problems come from. The whole industry of software engineering essentially dynamic. I really don't think that Product managers know what software we should be, and I would doubt any software architect know how to build a piece of software. Basically, software engineering doesn't work top-down. With less experience for lower level of engineers, it can't be bottom-up either. So how to build our software? To make things even worse, I don't think our customers know what software they need.
So basically, we are at an industry that no one knows how to build the products, and no way to build the product right. We are screwed from beginning, like it or not.
The only way to build a software product, in my opinion, is to just-build-it. Really? Let me talk about it a little bit more.
I would like to define some kind of distances for software development. For example, the distance of test cycle, the distance of feature development cycle. These two are the most significant ones of them. So Unit Test are so important, because we reduce the distance of test cycle into some sub-second ones. But for feature development cycle, let's talk about some "normal" way.
In some teams, if it is not most teams, we have Product Managers to "collect" a list of features for the development teams. Whether or not being Agile, the features will be pushed down to the development team, and the team try their best to develop them or feedback to the Product managers, or the POs. Most of time there are some back-and-forth processes until we settle down, and there come our features.
In this way, I saw some teams who develop some simple feature in half a year, while those features could be implemented some other ways in several days. The communication between engineers and Product managers are so expensive, not to mention there are other communication costs between engineers.
The first Agile manifesto is: individuals and iterations over processes and tools.
But what does it mean? Different people have different view though. My experience tells me that we need to encourage our engineers to help finding the right features we really want to build, not having them just simply top-down. For the POs, they simply need to understand what those features are, and how they could be useful to the customers, and prepare to explain to the customers that why it doesn't work in the way the customers are thinking about.
This is how Google is doing I guess, according to the book "How Google Tests Software". And I truly believed this because I found out many good features in my products were from those short distance of development.
There are many other tricks in software development. The rule of thumb is being Agile, which I truly believe in. But Agility is not just a word, we need to make it real.
