Innovate or Die - Feedback

Tighten the feedback loop during CTP and beta testing
The originally public 10.5 beta called for two builds with 2½ months (10 weeks) after each for comment. However, beta newsgroup traffic would imply that there was only significant input for 3 weeks after each build release. That means that after each build 7 out of the 10 weeks between builds was largely “dead” time which allowed user interest to wane. User input could be focused better if there were more interim releases with fewer new features between builds. That allows the users to focus more on each new feature in turn. One of the key concepts of Extreme Programming (Release Often) is based on this concept:

The development team needs to release iterative versions of the system to the customers often. The release planning meeting is used to discover small units of functionality that make good business sense and can be released into the customer's environment early in the project. This is critical to getting valuable feedback in time to have an impact on the system's development. The longer you wait to introduce an important feature to the system's users the less time you will have to fix it.

It should also help maintain interest in the beta. Finally, the developers should be able to use the interim builds as an opportunity to respond to user feedback and introduce fixes to new functionality during the beta. This is similar to the Extreme Programming concept of Iterative Development:

Iterative Development adds agility to the development process. Divide your development schedule into about a dozen iterations of 1 to 3 weeks in length. Keep the iteration length constant through out the project. This is the heart beat of your project. It is this constant that makes measuring progress and planning simple and reliable in XP...Concentrate your effort on completing the most important tasks as chosen by your customer, instead of having several unfinished tasks chosen by the developers.

Doing so will help ensure that the user feedback is properly addressed. The point of the beta should not to be to ensure that there are no bugs in what the engineers designed and implemented, but to determine if the new features actually address end users needs and to allow tweaking of them to better address those needs. That is, it should be conducted more as prototyping than QA.

That's something that Microsoft has apparently discovered that it needs to address. That's why the introduced the CTP concept, and there is word that they may eventually stop doing traditional beta entirely.

Scott Ambler also notes that the tighter the feedback loop, the less expensive it is to do development:

Fundamentally....the reason why the agile cost of change curve has flattened is because we follow techniques which reduce the feedback cycle. Agile techniques, shown in green, have short feedback cycles and therefore are at the flat end of the curve. Traditional techniques, shown in red, have longer feedback cycles and therefore are at the higher-cost end of the curve.