A few weeks ago I posted a blog entry about our new effort in Architecture-Based Testing. The project’s goal is to help find answers to the two questions shown below:
Since then, we have listened to stakeholder feedback and changed our name to Architecture Support for Testing. Many feel that this more accurately describes what we are trying to provide.
More substantively, though, we have taken some steps toward answering both of these questions. To answer the first question, we have written a catalog of architectural design approaches that lead to systems that are more testable. These approaches include patterns, styles, and tactics for testability. We have also uncovered a rich set of metrics for object-oriented systems that can help an O-O architect chart a course to a testable system. The catalog will be published soon as an SEI technical note.
To answer the second question, we have begun building fault models for architecture design approaches. A fault model is a list of faults to which the design approach is susceptible. For example, a pipe-and-filter architecture is susceptible to these faults:
- Filter produces wrong output.
- Pipes talk to the wrong filters, or filters talk to the wrong pipes; the system is “mis-wired.”
- Two pipes that both supply a filter lead to out-of-order reading/writing of data, causing a filter error.
- End-to-end latency is too large.
A service-oriented architecture is subject to these faults (just to name a few):
- A service violates its contract.
- It accepts/delivers values outside the bounds defined for the service.
- It fails to meet performance expectations.
- A component containing the service fails, as observed by termination of a process.
- A node containing the component fails as observed by inability to reach the node.
And so forth. In addition to faults that can occur–that is, faults you potentially “buy into” when you choose the architectural design approach–there are also faults that cannot occur. These are the faults you “buy out of” when you choose the approach. For instance, if you choose a state-machine architecture with which to embed the control logic of a system, and your system doesn’t shortcut that logic anywhere else, then you can be assured that system-wide control logic errors can occur only in the state machine.
These fault models can inform testers how to make use of the architecture to construct a more effective testing regimen. They can concentrate more on faults that can occur and less on faults that cannot. In extreme cases, analysis that shows with 100% certainty that a particular fault cannot occur might take the place of corresponding tests. In a development project in which testing dollars are tight (and show me a project where they aren’t!), we hope to use architecture to effect better testing outcomes: better results for less cost.
The sketch below illustrates. Architectures come with fault models (unfortunately, usually not written down, but we’re trying to do something about that). Fault that can occur under that architecture are listed. Analysis, if available, can be used to reduce the testing necessary for those faults. Otherwise, the fault model makes those faults clear to the tester, who can construct tests accordingly. Tests that cannot occur can help the tester reduce emphasis on testing for those faults.
We continue to solicit your feedback and interest in this project. In my next post, I’ll describe a workshop we held recently, and describe two upcoming ones.
– Paul Clements, SEI