Model Problems in Architecture Support for Testing: The Votes Are In

Recently in this space I’ve described the SEI’s recent research in architecture support for testing (AST), with which we are striving to understand how architecture can be used to lead to better testing outcomes. I also described a testing practitioner’s workshop we held at the SEI in February, which resulted in a set of 29 model problems in AST. These are problems that, if solved, would make a substantial difference in the testing of software. I invited readers of this blog to add their votes to those of our workshop participants to produce a ranking of the model problems in terms of importance, using the following scale:

  • VH (Very High) = 5 (meaning that the respondent places a very high value on this capability)
  • H (High) = 4
  • M (Medium) = 3
  • L (Low) = 2
  • VL (Very Low) = 1 (meaning that this is a capability that is not at all valuable to the respondent)

As promised, here are the results.

The top-ranked model problem was REQ1, which scored 4.18. You thought it was a bit less important than our workshop participants did–they (collectively) gave it a score of 4.30.

Scenario name REQ1
Source Tester
Stimulus A tester chooses a test set to test the system for requirements satisfaction.
Environment The architecture is complete. System test has not yet begun.
Response The tester uses an architecture analysis tool that identifies the smallest number of tests to run to provide coverage of 98% of the requirements. Redundant tests are eliminated.
Response measure Performing the analysis is much less costly and time-consuming to run than the tests it replaces.
Original notes from workshop Can we use the architecture to tell us that (out of all of the huge number of tests possible) if we run a small number of tests, we will have covered 98% of the requirements?

The second highest-ranked problem, scoring 4.09, came from the set of model problems having to do with integration testing.

Scenario name INT15
Source Tester
Stimulus The tester needs to develop integration tests.
Environment The architecture of the system is complete and detailed.
Response Automated tools use the architecture to generate test cases and test subs for the components within the system.
Response measure The tester asserts that 90% of the generated tests should be run.
Original notes from workshop How can architecture be used to drive integration testing?Generation of test stubs from high-level descriptions (Common Information Model CIM)

Generation of test cases from more detailed descriptions (Multispeak)

Keyword-based test automation–use of domain-specific language. Can we use the domain-specific language in the architecture models so that we can test sequences of the keywords… basis for auto-generation of tests

After that came a four-way tie for third place. These scenarios all scored an average importance rating of 3.91:

Scenario name REQ2
Source Tester
Stimulus A tester chooses a test set.
Environment The architecture is complete.
Response The tester uses an architecture analysis tool that identifies the smallest number of tests to run to provide coverage of the highest-risk areas of the system. Redundant tests are eliminated.
Response measure Performing the analysis is much less costly and time-consuming to run than the tests it replaces.
Original notes from workshop Can I use architecture to use tests to cover high-risk or high-probability-of-error areas in the system?

  • Design for testability… of the architecture to meet those requirements (encapsulation, isolation…)
  • Predictive analysis… to analyze for satisfaction of those reqs.
  • Generation of test artifacts… to test for those requirements
  • Conformance… is necessary for the other three to work.
Scenario name SPL1
Source A stakeholder who wants a change to product B.
Stimulus A change request, which will result in core assets being modified, is initiated for product B, which is a certified product.
Environment The products in the product line must be certified to a specified standard. Core assets have been used as part of the product implementation. Several products have used these core assets. There is a policy that if any core asset changes, all products that depend on that core asset are rebuilt.
Response The changes to the core assets are made and the products, A and B, dependent on those assets, are rebuilt using approved tactics. An architecture-based analysis proves that no recertification of A, which only has new implementations of the same core assets, is needed. Evidence accumulated during the changes is used to expedite the recertification of B.
Response measure The effort to recertify B is significantly less than the effort to certify B originally while maintaining the level of confidence.
Original notes from workshop Testing Certified Product Line Assets : Can AST help to reduce certification costs for products by identifying and managing certifications levels of core assets?

  • Change core or variant assets for next Product B without affecting certification of Product A
    • Cost is an issue
    • Risk to product delivery
    • Takes too much time to perform testing for delivery
    • What does it take to certify a parent product having capability and then apply certification to other products?
    • Product A has completed the certification and a problem is found in a separate product. How can we prevent certification process if Product A needs to pick up change?
    • Does early or late binding help in reducing certification costs?
Scenario name INT4
Source Tester
Stimulus The tester seeks to uncover performance bottlenecks in the integrated system as quickly as possible.
Environment The architecture and system integration are complete.
Response An analysis of the system architecture identifies the components and threads most likely to cause the system to fail in terms of its performance thresholds. The analysis could also be used to predict the components that will cause performance issues when integrated into the system.
Response measure Performance problems are discovered early and not late in the testing cycle.
Original notes from workshop
  • Can the architecture be used to predict and define the bottlenecks within the system and then be used as a basis for integration testing?
  • Utilization and resource availability can be bottleneck predictors.
  • Capturing sufficient performance information to determine high transaction components
  • Predicting integration risk
Scenario name INT7
Source Test designer
Stimulus The test designer must create a set of tests for system integration.
Environment The architecture is complete.
Response The designer uses an automated tool that analyzes the architectural descriptions of the components and generates lists of observable and controllable entities.
Response measure The tool generates 100% of the observable entities provided by the components within the system.
Original notes from workshop How can a module definition describe what can be observed?

Now, of course, the real fun begins: finding solutions to these problems.

Thanks for participating.

– Paul Clements, SEI

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s