In this last post, I discussed where the initial hypothesis comes from in the generate-and-test cycle. In this post, I will begin discussing the test phase.
The test phase of generate and test subjects the current design hypothesis to a variety of tests and uses the results of these tests to inform the generation of the next design hypothesis. Before we discuss the tests that are applied in this phase, we will discuss three other questions: when are we done, how are the tests applied, and what are the outputs of the tests?
When is the testing phase done?
Ideally, the testing phase ends when all of the tests are passed by the current design hypothesis. If this should happen, then there is no question about terminating the architectural design process.
More likely, however, is that the architectural design process is terminated when either the time and budget allocated to the architectural design is completed or when the current design is “good enough” to proceed. This assumes that the project-planning process has allocated time for the architectural design. This time should be between 5% and 10% of the total project time depending on the size and complexity of the project.
Once the initial design has been accomplished, further design proceeds in parallel with development. As development proceeds, the initial design will have to be refined.
How is testing performed?
There are several different techniques that can be used to test either a design or particular design decisions.
- Thought experiments. This is the most common method to test a design. A variety of use cases or quality attributes scenarios are used to examine the design, and the architect or other examiner traces the use case or scenario through the design and determines whether it will be satisfied and if not, why not.
- Modeling. For some quality attributes, most notably performance, analytic models exist. A model of the current design can be created, and this model can be used to predict the relevant quality attribute properties and see if they meet the scenarios.
- Simulation. A simulation can be constructed of the system to test its quality attribute properties. The simulation can be executed with a workload or other input that represents how the system will be used. The quality attribute response predicted by the simulation is compared to the required response to see if the current design satisfies the situation being tested.
- Prototypes. A prototype can be constructed to test particular portions of the design in particular situations. The prototype should be executed with inputs that represent the situations being tested. The responses of the prototype can be used to determine whether the current design hypothesis is suitable at least in the aspect that the prototype represents.
All of these techniques take time to perform. Thought experiments typically take the least time, the other techniques more. The time the other techniques take will depend on the fidelity of the model, the simulation, or the prototype. Since these techniques take time, they should be used only on important aspects of the current design hypothesis. The choice of which aspects is a cost/benefit decision. The cost of performing the test is compared to the benefit of knowing information with more fidelity than that available from thought experiments. In other terms, how much risk would be reduced by knowing the predictions of the model, a simulation, or a prototype?
What are the results of the tests?
Performing the tests will yield one of four results.
- The current design hypothesis is satisfactory with respect to the test.
- The test yields responsibilities that must be added to the next design hypothesis.
- The test identifies quality attribute scenarios that are not satisfied by the current design hypothesis and the portion of the design hypothesis that causes the scenario to be unsatisfied.
- The test identifies constraints that will be carried forward to the generation of the next design hypothesis.
This is the information that will be carried forward to the generate-next-design-hypothesis phase.
– Len Bass, SEI