"As a practical matter, the level of effort devoted to testing should reflect the importance of the software to the enterprise that maintains or uses it."

--Howard Rubin and Brian Robbins 

Manage Testing as a Continuous Process

The test program is one of the most valuable resources available to a program manager. Like a diamond mine or an oil field, the test program can be the source of vast riches for the program manager trying to gauge the success of his/her program. Unfortunately, it is often a resource that goes untapped. Few program managers exploit the information available to them through their test program. Rather, they view testing as a necessary evil - something that must be "endured" with as little pain as possible. This attitude often clouds their horizon and effectively shrouds the valuable information available via the test program.
 
Used effectively the test program can provide a rapid and effective feedback mechanism for the development process, identifying failures in the software construction process and allowing for early correction and process improvement. These benefits only accrue, however, if the test program is managed like a "program", with the discipline, rigor, and structure expected of a program. This includes establishing objectives, devising an implementation strategy, assigning resources and monitoring results.
 
Used in this context, testing becomes a pervasive, systematic, objective confirmation that the other parts of the development process are achieving the desired results. The program manager that exploits the information available through testing recognizes that the test program, not only evaluates the product, it also evaluates the process, by which the product was created. If a defect is found in testing, other checks and balances installed further upstream failed. This information can then be used for real-time improvement of the process.

This is not to say that testing is a panacea for all program problems. For software systems of any significance, it is a practical impossibility to test every possible sequence of inputs and situations that might be encountered in an operational environment. The objective of a test program is to instill confidence in users and developers that the software product will operate as expected when put into service. Because it is impossible to check all possible situations, testing is, by its very nature, a sampling discipline. A typically small number of test cases will be executed and, from the results, the performance of the system will be forecast. Testing is by its very nature a focused snapshot of a system. It is also inherently inefficient. Like the oil field mentioned earlier, for all the logic and analysis done beforehand, many probes come up "dry", with the system operating exactly as expected.
 
This is not to slight the value of testing. A test that is tailored to and consistent with development methodologies provides a traceable and structured approach to verifying requirements and quantifiable performance.
 
Two types of testing are performed at each level (software subsystem and system).

1. Integration testing (based on design)
2. Acceptance testing (based on requirements)

In organizing a test program, four key principles should be followed.

• Every CSCI requirement for every CSCI should trace into at least one test case.
   
• Test cases should include white-box tests with path-coverage levels appropriate for the software being tested.
   
• Test cases for integration testing should include black-box functional, interface, error recovery, out-of-bounds input, and stress tests.
   
• Test cases for the fully integrated system should include tests against operationally realistic scenarios.

Preparation of test cases is something that should start very early in a program. The best place to start them is during requirements definition. During the infancy stages of a program, focus on testing provides significant advantages. Not only does it ensure that the requirements are testable, but the process of constructing test cases forces the developer to clarify the requirement, providing for better communication and understanding. As test cases are designed, they should be documented along with their assumptions. These test products should become contract deliverables and should form the basis for regression tests needed during the maintenance phase of the software life cycle. However, test cases are not immune to errors. As such they, too, should be inspected to eliminate defects.
 
Effective testers are often characterized as a different breed. They tend to look a problem from a different perspective than a developer. The latter group look at problems to see how they work. He effective tester looks at it the see how it won't work. Both perspectives are necessary if defects are going to be rooted out of complex software systems

top

---