In the last post, I introduced the testing situation—when is testing complete, how are tests performed, and what are the results of the tests? In this post, I discuss the test cases themselves: important use cases, architecturally significant requirements, and view-specific use cases.
Important use cases
Some use cases are more important than others. These are the use cases that characterize the functionality of the system in broad terms. If these use cases are satisfied, the other functionality should be added easily. These are not the architecturally significant requirements, which I will discuss next; these are use cases that expose functionality.
One question frequently raised about use cases is how many are enough? A strength of use cases is that they are very specific and provide an excellent method for documenting specific requirements and exploring details of the design. A drawback of use cases is that they are very specific. It takes a great many use cases to characterize all of the requirements of a system. Again, it is a cost-benefit analysis that determines how many use cases to analyze during the test phase. How long it takes to analyze a use case will depend, but a half hour has been our experience, on average. At that rate, analyzing 100 use cases will take 50 hours. Is that worth it? It depends on the context of the development. Usually, fewer than a dozen will enable capturing the major functionality.
Architecturally significant requirements
Architecturally significant requirements are essential test cases. These are the requirements that drive the architecture design. If they are not satisfied, the design is not adequate.
Architecturally significant requirements can be expressed as constraints—“the system shall have a database”—or as quality attribute scenarios. Constraints as test cases are easy to check—either they are satisfied or they are not. If architecturally significant requirements are quality attribute requirements, then quality attribute requirements are tested according to one or more of the techniques enumerated above.
View-specific use cases
Examining the current design hypothesis from the perspective of different views also is a source of tests. We can generate test cases using the three collections of views: modules, component and connector, and deployment. In general the response to these view-specific questions has two portions: (1) what policy should be chosen? and (2) what mechanisms are to be used to implement the chosen policy? In some cases, the choice of a policy is a business decision and should be discussed with the business representative for the project. Each question leads to one or more view-specific use cases that have the form “Does the current design hypothesis support the policy decision that has been made?”
Representing to make testing easier
The questions we propose in this section depend on having multiple views of the current design hypothesis. If there is not a module, a component and connector, and a deployment view of the current design hypothesis, then they should be created.
The set of module views provides the basis for examining several different categories of questions. The common features of the module views are responsibilities and modules into which those responsibilities are placed. All of the questions with respect to the module view deal with detecting responsibilities not yet in the design hypothesis or detecting allocations of those responsibilities to modules that lead to difficulties when making changes.
- Growth cases. What happens when usage of the system grows? Are there additional responsibilities that will be needed to manage an order of magnitude more users than currently expected? Should allowance be made to make sure expansion is easier?
- Modifications. What are the expected set of modifications, and what allowance should be made?
- Exceptions. What responsibilities are necessary to manage exception handling? Should a common method of handling exceptions be defined as a pattern and utilitized throughout the system?
- Version upgrade. What should happen when a new version of an existing system is deployed? Have formats changed, and should the new version support the old formats? What about changes to the user interface? Should multiple user interfaces be supported?
- Deferred binding time. Which functions will be bound at runtime? Each of these functions will require mechanisms to support the late binding.
- Self identification. To what extent will the system be able to report its version number, patch level, date of installation, and other identifying information?
- License management. How is the right of the user to use the system to be validated?
Component and Connector View
Component and connector views provide the basis for examining questions concerning parallelism and resource management. Component and connector views are all concerned with threads, processes, and resources. In general, the method used to answer each question is to trace the series of events on a thread that results from the stimulus framed in the question. While performing the trace, look for points of resource contention that will indicate the necessity for synchronization among threads. Also look for opportunities for parallelism that indicate that a new thread could be created. In certain circumstances, a process or thread should be killed. Look for such circumstances. Finally, look how user state is managed. Which elements are statefull and which are stateless will have important implications for scalability and reliability. The questions are
- Multiple users. What happens when two users perform similar tasks simultaneously? Which resources might be used by two users concurrently? These resources will need synchronization mechanisms.
- One user performing multiple activities simultaneously. What happens when a user wishes to initiate a second concurrent activity? Is the first one suspended, terminated, or allowed to continue? Where is the state for the first activity maintained? Will the state for the two activities be kept separate?
- Start up. What activities must occur during start up to support steady-state operation? Must threads be created that manage user input? How are connected devices detected and initialized? A related question is, When are connected devices detected? How are the characteristics of these devices determined? Start up can be a time-consuming operation, so determining opportunities for parallelism is important during consideration of start up. For those start up activities that could happen in parallel, additional threads should be spawned.
- Shutdown. What functions should be performed during shut down? What state must be saved? How are resources freed during shutdown? Resources such as memory, disk space, locks, and so forth must be released. Not all portions of the system may be able to clean up after themselves. For example, one portion may be waiting for an unavailable network. In this case, that portion must be killed and the resources it was using must be freed. This implies that the resources being used by a process must be known to the system outside of the process.
The deployment view introduces the concept of processors and networks to the component and connector views. In a component and connector view, a stimulus activated a thread within a process and the process was assumed to execute on a single processor. In the deployment view, the thread may travel over networks and multiple processors. So many of the questions asked during consideration of concurrency can also be asked again with respect to network activities. As with the other two sets of views we have discussed, the questions imply a policy that in turn implies mechanisms. The following are questions that should be considered in addition to those considered during examination of the concurrency view:
- Installation. How is the system to get to its target processors? Is the system to be distributed through some distinct media or over a network? If the system is distributed over a network, how are running versions of the system to be handled? What kind of suitability checks are to be performed before installation?
- Initialization. What are the network and processor considerations with respect to initialization? Will each processor have its own initialization or will the initialization be centralized? Is the knowledge of connected devices limited to the processor to which they are connected or is this knowledge shared? What kind of registration scheme is to be implemented to support late binding? When considering the module view, late binding was relevant to function. In the deployment case, late binding is relevant to the mapping of processes to processors or the addition or removal of processors to support scalability.
- Processing across processors. How are processes to be assigned to processors? Is the assignment static on deployment or is it dynamic? If dynamic, what algorithm is used to make the assignment? How are queries assigned to various processes?
- Messaging over the network. What load will the deployment decisions place on the network? How many messages of what size are being sent between two processors? Should delivery be guaranteed or is it acceptable to lose some messages?
- Disconnected operation. What should happen when the system is disconnected and when it reconnects? Mobile systems frequently are in situations where they have no network connectivity. In such cases, there must be a policy that covers actions when the mobile system is disconnected and when it is reconnected. The architecture must support mechanisms to implement the policy.
- Failure of an element. Elements of a computer system fail. What is desired when an element fails? Different elements will have different policies. Scalable systems may have stateless components to support retrying and redirecting queries when a processor fails. Network failure may be treated as a disconnect or alternative routings may be tried. In any case, there should be policies that control desired behavior of the system when a failure occurs and mechanisms to implement that behavior.
If there are other views that are specific to the system being designed—for example, a service-oriented architecture view—then they can also be used to generate questions to be used in the test phase.
– Len Bass, SEI