Beta test projects

[beno]

Hi NeoBookers,

First, thanks a lot to Dave for this new space to exchange ideas and learn new things related to NeoBook.

As this thread title suggest, I have to learn how to organize, manage and keep track of a succesful beta test process for my products.

It may seem obvious that I feel I´m not doing it right. But would like to learn and listen (read?) from the experienced NeoFriends we are lucky to have in this forum.

So, thanks in advance for your contributions and ideas.

beno

[Wrangler]

All I can say about beta testing is that you can't do enough of it. Even after extensive testing, there always seems to be a bug or two that gets by. The number of bugs depends, of course, on the complexity of the software. The more bells and whistles, the more chance of a bug.

I have tried shareware titles in the past where the author thought that those who downloaded his title would be the beta testers. This is a big mistake. They will delete it and move on if there are bugs that prevent him/her from trying all the features offered.

It is important that you, at a minimum, get rid of any big bugs that prevent the software from running like it should. Access violations, blue screens of death, program hanging, etc. should not happen. The only bugs left at release time should be minor ones, if any at all.

Whenever possible, I try to use the software myself for a period of time before making it available for further beta testing. At the very least, you must sit down and spend time poking around in every nook and cranny trying to break it. And don't just check it by running it like it should be run. Do things to it that most users wouldn't, and see what happens. I also suggest doing this with the compiled version, as opposed to running it in the IDE. During development, I make it a habit to extensively test each component, or feature, while coding it. This should get rid of most of the big ones. When you feel it is ready, go through it again. Don't assume that because you told neobook to do it, it will do it the way it was intended.

Once you feel you have found any errors yourself, it is time to test on as many different platforms as possible. Neobook apps are very friendly. The interaction with Windows is very minimal, but depending on the features offered, could behave differently in different versions of Windows. Since most of us don't have a number of different machines with different operating systems laying around, it makes sense to let others try it on theirs. So, how do you do that?

Well, there are a couple of ways to do that. You can go to those who currently use your software, if the title is a version upgrade, much like Dave does with Neobook. Since Neobook is a programming environment, Dave is luckier than most. Programmers understand what a bug is, and the input will be of higher quality. But the "average person" understands little if anything about it. These folks are excited about new features you have implemented, and may be willing to endure the possible errors to get a chance to try it out for you. However, they must be instructed as to the proper way to do the testing:

1. They should know that paying attention while running the program is important. They should report to you what they did to cause the error in detail, and also what the error said. (this is where it is important to anticipate and trap errors with verbose dialogs in your program). They should be provided with an email address that they can report to.

2. Know that this IS a beta, and the end results may be unpredictable. It should be installed into a seperate folder, and not over top of their existing installation.

3. They should include in their report what their system and hardware configuration is, what operating system, how much ram, processor speed, and video card type. Windows provides this information, and you could also provide a reporting system in your app, allowing them to email from within the program, where it gathers this information for them, and sends it to you in the email. There are commandline utilities and nb plugins that will help with this. This feature can be removed before final release. If they can't provide this information, they probably don't qualify to perform the testing. At least SOME knowledge of how a computer operates is necessary to get reports that you can do something with.

Your testers could also include family, and friends. I like to go this route, and I entice them with a free copy of the final release for their efforts. These folks would show more dedication to the effort, because they know and respect you. If your software is targeted toward kids, get kids to play with it. A parent could help with the reporting. If the handicapped will be using it, get them to test. The handicapped will be more sensitive to what they are doing, and actually make better testers than most.

You could advertise the testing program on your web site. It would be important here to offer something in return. A free copy of the final release, a discount on the final release, or maybe a t-shirt, or credit on the About page. More than likely, this method will require weeding out those that want to sign up just for the prize. Might be wise to provide an online application form, asking personal information, as well as their system configuration. Not that this information could be verified, but it will provide more of a commitment.

No matter who does the testing, keep in mind that they will tell others about your creation. Word of mouth is the best advertising. Giving away free copies to them will justify the expense (which, to you, is no more than labor, in most cases). Of course, this will vary according to what it is being tested. For example, it wouldn't behoove Dave to use this method for neobook. On the other extreme, it WOULD make sense when dealing with kids. What they like, generally their friends will like. They will tell their friends at school, and they will pressure Mom and Dad to purchase a copy for them.

On your part, it is important that you respond to error reports, and let them know you have received it, and added it to the list. A mailing list could be established to keep track of bugs that have already been reported. This could also be used to notify testers of an upgraded version to try which has corrected bugs already reported.

DO's:

- Communicate with your testers. Keep them apprised of what is going on.

- Try to make the testing experience as easy as possible.

- Try to provide them with the tools to generate quality reports.

- Be patient. Beta testing can be difficult, and frustrating. Be prepared for further interrogation to get to the bottom of an error.

- Try to fix the errors right away. This strengthens your commitment to the project.

- Keep notes on each error reported, and who reported them. The resulting list of testers will provide higher quality testers for the next release.

DON'TS:

- Don't release beta, and then go on vacation. Your lack of interest will be their lack of interest.

- Don't promise them things, and then not deliver. If you offered free copies, send to them before public release. If not, maybe give them a certificate, or credit within the program itself. They will be more likely to do it for you again. Everyone likes a pat on the back, or see their names in lights. It may seem silly to some, but satisfying to others.

Well, that is all that comes to my mind at the moment. Anyone else?

[beno]

Hi Wrangler,.

Wow!, thanks for your sharing your experience and ideas.

You a are a kind person.

Saludos,

beno