A new project at the SEI
The SEI’s Architecture-Centric Engineering (ACE) Initiative has launched a new research project called “Architecture Based Testing.” I thought I would use this opportunity to tell everyone about it, and to ask for your feedback.
By architecture-based testing, we mean using a system’s architecture to inform and guide the system’s testing activities. While there has been substantial work devoted to this topic in the research community, not much of that research seems to have filtered into communities of practitioners. Hence, the promise of architecture-based testing–to use architecture to reduce the time and expense of testing and to increase its effectiveness–remains unfulfilled.
Charting the territory
Work in architecture-based testing can be broadly categorized as follows:
- Foundations: General treatment of architecture-based testing, laying out the vocabulary, key concepts, principles for reasoning, theoretical bases, and so forth. It includes what it means for a system to be more testable than another, and how to measure testability.
- Architecture Analysis: Architecture-based predictive analysis of systems (without code present). This includes static analysis of architectures, model checking of architectural descriptions, simulation of architectures, etc.
- Testing: Using architecture to define test assets to test code for functional/quality attribute (QA) correctness. We can use architecture to develop test plans, test paths, test cases for code, or regression tests when architectures or code changes. We can also use architecture to help us define and set up a project’s integration and test infrastructure.
- Conformance: Using architecture to define tests for code compliance/conformance to architecture.
- Design: Designing an architecture to make systems more testable. This work includes finding and cataloging patterns and tactics for testability, designing testing as a service, and capturing anti-patterns: what not do to in the architecture if you’d like a testable system.
Categories 2, 3, and 4 can be considered in a combined fashion, under a broader category we might call “Analysis and testing, given an architecture,” as shown in the following table:
|Testing whether system will meet its requirements||Testing for conformance of code to architecture|
|Code present||3. Testing||4. Conformance|
|Code not present||2. Analysis||N/A|
Overlaid on each of these categories is a value proposition: What benefit do activities in a category bring to a software organization? And how are the activities best carried out? Hence, process and measurement issues play a large role.
Where is our project concentrating its focus?
The overall goal of our project is to help answer this question:
How can we (better) use architecture to help test our systems?
In broad terms, we are trying to draw connections between architecture and testing, in both directions.
In one direction (from testing to architecture), we are trying to understand how, given an architecture, testers can use it to make testing more effective. In the other direction (from architecture to testing), what if we aren’t just “given” an architecture at all? What if we can design it up front to make testing more effective? Those are the questions we’re trying to answer.
Since testing is such a dreadfully expensive part of the software development, we think that even a small improvement might make a big difference.
Although projects tend to evolve in response to exigencies that arise over time, here is our current plan for addressing each part of the roadmap given earlier:
|Roadmap Area||SEI Contribution|
|1. Foundations||Publish papers to raise awareness of architecture-based testing, catalog previous work, and make architects and testers aware of the possibility that architecture-based testing can significantly lower the cost and raise the effectiveness of testing. For example, we have started an architecture-based testing “community of interest” on sprucecommunity.org, the site that pairs real-world driving problems with research solutions, and we are compiling a comprehensive bibliography on architecture-based testing.|
|2. Analysis||Learn how architecture analysis can lead to improved testing outcomes. This might include obviating the need for certain tests, pointing out the need for other tests, or reducing the cost of still others.|
|3. Testing||Run an architecture-based testing workshop for testing practitioners. We will capture what the best practices are (or might one day be) for using architecture during the testing phase of software system development in order to reduce testing’s cost or increase its efficacy. The output of the workshop will be a set of important model problems in testing. These are problems that, if solved, would result in a significant decrease in project resources devoted to testing and/or a significant increase in system quality given an expenditure level.|
|4. Conformance||No work currently planned in this area.|
|5. Design||Compile a catalog of architectural design approaches (e.g., styles, patterns, tactics) and anti-approaches that lead to systems that are more testable.|
We are reaching out to the community to help us in two ways.
First, you can help us capture architectural approaches for designing testable systems. Tell us what you do during design to help testers do their jobs. We’ll record everyone’s responses, and put the best ones up (with attribution, unless you tell us you’d rather remain anonymous) in a future blog post.
Second, what would you consider a “model problem” in architecture-based testing? That is, what problem (if solved) would let the use of architecture in testing make a substantial improvement in test cost or effectiveness?
Finally, we’d also be delighted to hear your opinion about our project, and/or suggestions about future work in this area.
Thanks in advance.
– Paul Clements, SEI