The SEI Architecture Based Testing Project: Feedback Requested

A new project at the SEI

The SEI’s Architecture-Centric Engineering (ACE) Initiative has launched a new research project called “Architecture Based Testing.”  I thought I would use this opportunity to tell everyone about it, and to ask for your feedback.

By architecture-based testing, we mean using a system’s architecture to inform and guide the system’s testing activities. While there has been substantial work devoted to this topic in the research community, not much of that research seems to have filtered into communities of practitioners. Hence, the promise of architecture-based testing–to use architecture to reduce the time and expense of testing and to increase its effectiveness–remains unfulfilled.

Charting the territory

Work in architecture-based testing can be broadly categorized as follows:

  1. Foundations:  General treatment of architecture-based testing, laying out the vocabulary, key concepts, principles for reasoning, theoretical bases, and so forth. It includes what it means for a system to be more testable than another, and how to measure testability.
  2. Architecture Analysis: Architecture-based predictive analysis of systems (without code present). This includes static analysis of architectures, model checking of architectural descriptions, simulation of architectures, etc.
  3. Testing: Using architecture to define test assets to test code for functional/quality attribute (QA) correctness. We can use architecture to develop test plans, test paths, test cases for code, or regression tests when architectures or code changes. We can also use architecture to help us define and set up a project’s integration and test infrastructure.
  4. Conformance: Using architecture to define tests for code compliance/conformance to architecture.
  5. Design: Designing an architecture to make systems more testable. This work includes finding and cataloging patterns and tactics for testability, designing testing as a service, and capturing anti-patterns:  what not do to in the architecture if you’d like a testable system.

Categories 2, 3, and 4 can be considered in a combined fashion, under a broader category we might call “Analysis and testing, given an architecture,” as shown in the following table:

Testing whether system will meet its requirements Testing for conformance of code to architecture
Code present 3. Testing 4. Conformance
Code not present 2. Analysis N/A

Overlaid on each of these categories is a value proposition: What benefit do activities in a category bring to a software organization? And how are the activities best carried out? Hence, process and measurement issues play a large role.

Where is our project concentrating its focus?

The overall goal of our project is to help answer this question:

How can we (better) use architecture to help test our systems?

In broad terms, we are trying to draw connections between architecture and testing, in both directions.

In one direction (from testing to architecture), we are trying to understand how, given an architecture, testers can use it to make testing more effective. In the other direction (from architecture to testing), what if we aren’t just “given” an architecture at all?  What if we can design it up front to make testing more effective? Those are the questions we’re trying to answer.

Since testing is such a dreadfully expensive part of the software development, we think that even a small improvement might make a big difference.

Although projects tend to evolve in response to exigencies that arise over time, here is our current plan for addressing each part of the roadmap given earlier:

Roadmap Area SEI Contribution
1. Foundations Publish papers to raise awareness of architecture-based testing, catalog previous work, and make architects and testers aware of the possibility that architecture-based testing can significantly lower the cost and raise the effectiveness of testing. For example, we have started an architecture-based testing “community of interest” on, the site that pairs real-world driving problems with research solutions, and we are compiling a comprehensive bibliography on architecture-based testing.
2. Analysis Learn how architecture analysis can lead to improved testing outcomes. This might include obviating the need for certain tests, pointing out the need for other tests, or reducing the cost of still others.
3. Testing Run an architecture-based testing workshop for testing practitioners. We will capture what the best practices are (or might one day be) for using architecture during the testing phase of software system development in order to reduce testing’s cost or increase its efficacy. The output of the workshop will be a set of important model problems in testing. These are problems that, if solved, would result in a significant decrease in project resources devoted to testing and/or a significant increase in system quality given an expenditure level.
4. Conformance No work currently planned in this area.
5. Design Compile a catalog of architectural design approaches (e.g., styles, patterns, tactics) and anti-approaches that lead to systems that are more testable.

Help wanted!

We are reaching out to the community to help us in two ways.

First, you can help us capture architectural approaches for designing testable systems. Tell us what you do during design to help testers do their jobs. We’ll record everyone’s responses, and put the best ones up (with attribution, unless you tell us you’d rather remain anonymous) in a future blog post.

Second, what would you consider a “model problem” in architecture-based testing? That is, what problem (if solved) would let the use of architecture in testing make a substantial improvement in test cost or effectiveness?

Finally, we’d also be delighted to hear your opinion about our project, and/or suggestions about future work in this area.

Thanks in advance.

Paul Clements, SEI


3 responses to “The SEI Architecture Based Testing Project: Feedback Requested

  1. Pingback: Tweets that mention The SEI Architecture Based Testing Project: Feedback Requested | SATURN Network Blog --

  2. This project makes sense to me. It reminds me of Test-Driven Design, and Behaviour-Driven Development. And just like those, I am very fond of the aspect of your project where testability helps to drive the architecture. If testability were in the conversation at the earliest stages, it might influence buy or build decisions in a positive way. Even when you buy a system, it is worth asking how it can/will be tested.

  3. Ms. Sangeeta Johari

    Testing in computer era, has become heart of software system. as in a system there are many modules, so after completion of one module, that should be rigorously tested
    1. Each instruction of module, how much time it takes to execute, the looping is hanged out.
    2.When modules began combing then flow graph or matrix table is made so that error can be detected easily.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s