Test styles

Link

I've revisited Rusty Miller's post about various test styles. Cem Kaner's paper is a nice refresher that got me thinking about how we could do so much more with our software testing processes.

The subtle point I take away from Dr. Kaner's paper is that it takes excellent people to drive quality into software. Quality is not a bi-product of a collection of test cases per se.

I think primarily because of a very limited budget, we have relied pretty heavily on dogfooding our products to users through Beta programs. But even with all the money in the world, it's been invaluable getting real users using the product as soon as possible. This is especially true with business process software that is usually highly customized to a company's internal processes. There is no way a bunch of isolated developers is going to get it right. Get something in front of users as soon as possible.

It's certainly a fine line though. Customers can lose faith immediately if they encounter a blocker bug or an excessive amount of significant bugs. I think that's where exceptional expectation setting (hand holding, hugging, donuts, etc.) by your project manager is absolutely critical. It's also a challenge to keep users using the product and providing feedback through constant bug fix cycles. This is especially true if they have alternatives to the software or their daily job functions don't require the use of your product, or simply because they're too busy. Having a formal communication channel like public bug tracking, user groups, or even simply a daily conference call helps this. What you don't want is to ask the question, "who's using our software right now" and not know. The value of a Beta program is diminished nearly completely when this situation becomes reality.

Also, the most common reaction to Beta software is, "it would be really cool if it could do 'x'." Suddenly the project you thought was going to wrap up in a month or two has doubled with remaining work. I think this is where the contract process or statement of work process is so critical. It's nice to fall back on an agreement for the original scope of the project. Obviously you can't think of everything up front so implementing processes that produce shorter code cycles after a release are critical. Its always easier to break it to a client that a feature didn't make it in this release but will be in the next one only four weeks out.