Model Problems in Architecture Support for Testing: Cast Your Vote!

In a recent post, I mentioned a workshop in Architecture Support for Testing that was held at the SEI in February. The output of that workshop was a set of 30 model problems. These are problems that, if solved, would result in a significant decrease in project resources devoted to testing and/or a significant increase in system quality given an expenditure level. Since we are investigating the relationship between architecture and testing, each of the model problems has a flavor of architecture to it as well as a focus on testing.

Our workshop participants are, at this writing, casting their votes for the most important of these problems, but while they are doing that, I wanted to give this readership the same opportunity. The most important of the model problems (as determined by voting) will be taken to the Researchers’ Workshop on Architecture-Based Testing in Pisa, Italy, in late March.  There, they will be put before some of the leading researchers to solve, or try to solve, or begin to solve, or begin to think about solving.

Here are the model problems, grouped into four broad areas of subject matter.  Each model problem is phrased as a scenario describing a capability we wish we had.  (The model problem is providing that capability.)  Instructions for voting appear at the bottom.

1.    Architecture, Testing, and Requirements

Scenario name REQ1
Source Tester
Stimulus A tester chooses a test set to test the system for requirements satisfaction.
Environment The architecture is complete. System test has not yet begun.
Response The tester uses an architecture analysis tool that identifies the smallest number of tests to run to provide coverage of 98% of the requirements. Redundant tests are eliminated.
Response measure Performing the analysis is much less costly and time consuming to run than the tests it replaces.
Scenario name REQ2
Source Tester
Stimulus A tester chooses a test set.
Environment The architecture is complete.
Response The tester uses an architecture analysis tool that identifies the smallest number of tests to run to provide coverage of the highest-risk areas of the system. Redundant tests are eliminated.
Response measure Performing the analysis is much less costly and time consuming to run than the tests it replaces.
Scenario name REQ3
Source Unit testers
Stimulus Unit testing of a module begins
Environment Modules are coded, not yet integrated. Each module’s responsibilities with respect to requirements are documented.
Response Analysis is run on individual modules to find errors with respect to the requirements.
Response measure Cost of performing the analysis is very low, and the results save significant project cost and rework.
Scenario name REQ4
Source Stakeholder who provides new requirements
Stimulus New requirements have been proposed for a fielded system.
Environment Requirements analysis phase of a new version of a system
Response Analysis of the requirements yields a report on the architectural changes the new requirements will necessitate, as well as a report on the cost of making those changes and making the corresponding changes to the implementation.
Response measure The cost of performing the analysis is very low.
Scenario name REQ5
Source Tester
Stimulus System testing is about to begin.
Environment The system, when fielded, will consist of a very large number of nodes (for example, every soldier in a division may carry a node, as well as all of the division’s vehicles), a prohibitively large number for a test and integration lab to handle.
Response The system is tested in the test and integration lab using the nodes that are there. Analysis is performed to give high confidence that this testing is sufficient, and the system will perform correctly in the field.
Response measure The cost of the analysis is low; the confidence measure it provides is very high.

2.    Architecture, Testing, and Software Product Lines

Scenario name SPL1
Source A stakeholder who wants a change to product B.
Stimulus A change request, which will result in core assets being modified, is initiated for product B, which is a certified product.
Environment The products in the product line must be certified to a specified standard. Core assets have been used as part of the product implementation. Several products have used these core assets. There is a policy that if any core asset changes, all products that depend on that core asset are rebuilt.
Response The changes to the core assets are made and the products, A and B, dependent on those assets, are rebuilt using approved tactics. An architecture-based analysis proves that no recertification of A, which only has new implementations of the same core assets, is needed. Evidence accumulated during the changes is used to expedite the recertification of B.
Response measure The effort to recertify B is significantly less than the effort to certify B originally while maintaining the level of confidence.
Scenario name SPL2
Source The testing process
Stimulus Integration test suites must be selected for products in the software product line.
Environment Products in the product line are defined in terms of features. All core assets are thoroughly unit tested.
Response An architecture-based product test suite construction algorithm analyzes the features, combinations of features, and dependencies among features that define each product. The algorithm uses optimization techniques to select the smallest number of products to cover all of the features.
Response measure The percentage of products that have to be tested to achieve complete feature coverage is much less than an exhaustive test suite but how much less will vary based on the degree of similarity among the products.
Scenario name SPL3
Source Stakeholder who initiates new capability
Stimulus A new capability, which spans multiple subsystems, is added to the product line.
Environment An operating software product line organization has a validated architecture and functioning products. The new capability impacts multiple core assets and requires the addition of new assets.
Response An architecture-based unit test suite construction algorithm analyzes the core asset interfaces and their dependencies in the architecture to specify a sufficient set of unit tests to achieve an acceptable level of coverage. 

An architecture-based integration test suite construction algorithm identifies the smallest set of existing core assets that are needed to combine with each new core asset to provide acceptable integration test coverage.

Response measure Both algorithms produce test suites that are significantly smaller than suites selected using non-architecture-based methods while providing the same level of coverage.
Scenario name SPL4
Source Stakeholder who initiates new product
Stimulus A new product that was not originally envisioned is added to the product line.
Environment An operating software product line organization has a validated architecture and functioning products. The new product may require additional core assets.
Response If new core assets are created to contribute to the new product, an architecture-based core asset testing algorithm is used to construct a test suite from existing and new tests. Then an architecture-based product test suite construction algorithm analyzes the core assets that make up the new product. It selects existing test cases that apply to the product in a first pass and then specifies additional new test cases, which will have to be constructed, in a second pass until a minimal set is achieved.
Response measure The percentage of the total tests that must be newly constructed is very low (but will vary depending on the percentage of core assets that are new to the product line).
Scenario name SPL5
Source Architect
Stimulus A change is needed to achieve a different level of a non-functional requirement.
Environment An operating software product line organization has a validated architecture and functioning products.
Response An architecture-based product test suite construction algorithm identifies and analyzes the changes made to the architecture. The algorithm analyzes the existing test suite and identifies changes that must be made to the test suite to achieve suitable coverage.
Response measure The percentage of the test suite that must be changed and executed is very low.
Scenario name SPL6
Source Test manager
Stimulus A need to understand the effectiveness of the testing process
Environment An operating software product line organization has a validated architecture, functioning, tested products, and specified, documented test assets.
Response An algorithm queries the test-reporting and defect-reporting databases. The query retrieves test results and coordinates these with defect reports to determine how many tests are required to locate a defect. The defect yield, a measure of effectiveness, is reported.
Response measure The results of the defect yield analysis are sufficiently detailed so that differences in test effectiveness between two testing methods can be discerned.
Scenario name SPL7
Source Architect
Stimulus A design decision involving decomposition is being studied.
Environment An operating software product line organization has a validated architecture and functioning products.
Response An architecture-based analysis of the as-is architecture and the will-be architecture compares the testability of the two architectures and gives guidance on what decision to make.
Response measure The analysis is sufficiently sensitive that it gives a clear indication of which design is preferred a high percentage of the time.

3.    Scope of Architecture and Architecture Support for Testing

Scenario name SCO1:  Requirements with cross-cutting impacts
Source Users
Stimulus New requirements are precipitated that increase system load by adding more users and/or increasing transactions.
Environment System is under normal (runtime) operation.  Large distributed database with many users (e.g. 10<x<5000), each user has a local workstation.  Database is common to all users 

OR

System is in the development phase (Where can “testing” of artifacts effectively be done? On what artifacts? At what degree of confidence?)

Response AST techniques are used to analyze the new requirements for their impact on the architecture and its ability to support the new load while still maintaining required quality attributes.
Response measure The proper feedback occurs within an acceptable time.
Scenario name SCO2:  Changed requirements with cross-cutting impacts
Source Requirements team or user community
Stimulus Make the system multiuser (with up to 1000 users), necessitating a change to the architecture.
Environment System is in the design phase
Response AST techniques are used to plan regression testing.
Response measure The proper feedback occurs within a specified acceptable time.

4.    Architecture, Testing, and Integration

Scenario name INT1
Source Test Designer
Stimulus A test designer is creating tests for the system
Environment The architecture is complete.  The system has multiple modes of operation.
Response The test designer uses a tool to generate a set of tests that adequately exercise the different system modes.
Response measure Adding a test to the generated set doesn’t uncover any new faults, but with confidence that the generated test set covers all modes.
Scenario name INT2
Source Architect
Stimulus An architect is structuring the system’s mission threads to support incremental integration.
Environment The architecture is complete. The system is being integrated incrementally.
Response The system’s mission threads are tagged with the components necessary to support the thread.
Response measure A tester can use a definition of what has been integrated to determine the subset of mission threads that can be tested.
Scenario name INT3
Source Test Designer
Stimulus The test designer must determine the order in which to test the system’s mission threads.
Environment The architecture and system integration are complete.
Response The mission threads are prioritized according to criticality.
Response measure Testers can use the prioritized list to ensure that all critical threads are tested even if integration test time is cut short.
Scenario name INT4
Source Tester
Stimulus The tester seeks to uncover performance bottlenecks in the integrated system as quickly as possible.
Environment The architecture and system integration are complete.
Response An analysis of the system architecture identifies the components and threads most likely to cause the system to fail in terms of its performance thresholds. The analysis could also be used to predict the components that will cause performance issues when integrated into the system.
Response measure Performance problems are discovered early and not late in the testing cycle.
Scenario name INT5
Source Tester
Stimulus The tester is determining whether the system meets performance requirements.
Environment The architecture and system integration are complete. The architect has defined a complex relationship between individual performance parameters.
Response The tester is able to analyze the relationship determining appropriate loadings of the separate performance parameters in order to provide confidence that overall system performance will meet the defined relationship.
Response measure Testing is accomplished with a small number of test sets ensuring, with confidence, that the performance parameters can be achieved.
Scenario name INT6
Source Test designer
Stimulus The designer is creating cases to provide confidence in the system integration.
Environment The architecture is complete.
Response The designer uses some form of architectural analysis to develop a definition of observable (and controllable) entities within the system.
Response measure The analysis takes less time than developing tests (providing similar confidence in the system integration) manually.
Scenario name INT7
Source Test designer
Stimulus The test designer must create a set of tests for system integration.
Environment The architecture is complete.
Response The designer uses an automated tool that analyzes the architectural descriptions of the components and generates lists of observable and controllable entities.
Response measure The tool generates 100% of the observable entities provided by the components within the system.
Scenario name INT8
Source Test designer
Stimulus The designer determines what observation points to add to the system components.
Environment The architecture is complete but development has not begun or is only partially complete.
Response An automated (preferably) analysis of the mission threads determines observation points in specific components that will provide most benefit to the testers. The resultant test points are then included in system development.
Response measure Testers get the data necessary to ensure that the system components are appropriately integrated with fewer test points than expected without mission thread analysis.
Scenario name INT9
Source Tester
Stimulus The testers need to ensure that the integrated system isn’t violating privacy policies.
Environment The architecture is complete and contains details with respect to data privacy.
Response An analysis of the architecture determines a set of tests with respect to data privacy; such tests would define roles and their access to sensitive data.
Response measure Tests generated from the architecture lead to greater confidence that the system will not inadvertently disclose private data.
Scenario name INT10
Source Test designer
Stimulus The test designer wants to test that certain behaviors cannot occur within the system.
Environment The architecture is complete and contains information on threads and behaviors that the system must not be able to perform.
Response Analysis of the architecture derives tests for the known-to-be-forbidden behaviors.
Response measure The test suite tests for all behaviors known to be forbidden as well as those the system should exhibit.
Scenario name INT11
Source Tester
Stimulus The tester needs to test a variety of similar interfaces.
Environment The architecture and system integration are complete.
Response The tester is able to automatically generate tests for a variety of instances of a given interface using a single test set and generated test stubs.
Response measure The test generator can generate 90% of the tests needed to test the different interfaces.
Scenario name INT12
Source Tester
Stimulus The tester must test the integrated system in a variety of different environments.
Environment The architecture is complete and the system is integrated; the architecture describes known variations in the operational environment.
Response An analysis of the architecture enables the tester to simulate the environmental variations (differences in communicating systems, platforms, …) and test the system behavior in a sufficient number of environments in order to gain confidence in correct execution. Ideally, the test set will be optimized to provide sufficient coverage without testing all environmental variations.
Response measure Confidence that the integrated system will operate correctly is obtained without testing all environmental variations.
Scenario name INT13
Source Budget team
Stimulus The budget team needs to prepare estimates for the cost of integration testing.
Environment The architecture is complete.
Response Automated analysis of the architecture predicts cost and schedule associated with integration testing.
Response measure Predicted costs are within 10% of the actual costs of integration testing.
Scenario name INT15
Source Tester
Stimulus The tester needs to develop integration tests.
Environment The architecture of the system is complete and detailed.
Response Automated tools use the architecture to generate test cases and test subs for the components within the system.
Response measure The tester asserts that 90% of the generated tests should be run.
Scenario name INT16
Source Testers
Stimulus A tester wants to perform integration testing for a component into a running system.
Environment The architecture of the runtime environment (system of systems) is complete and up to date. The operational  system must continue to operate.
Response The tester adds the component to the running system and performs necessary tests on it.
Response measure Production instances of component respond to the tests without polluting live data with the test data from the component in question.

Voting

The scenarios are listed below, by name. For each scenario, please add a RANK using one of the following values, according to the value you place on the capability described by the scenario. Ranking a scenario VH (Very High) means that you place very high value on that capability, whereas a VL (Very Low) scenario represents a capability that is not at all valuable to you.

  • VH (Very High)
  • H (High)
  • M (Medium)
  • L  (Low)
  • VL (Very Low)

Any scenario you don’t rank will be counted as “VL (Very Low).”

Copy and paste the list below into a document, add your rankings, and e-mail it to me at clements@sei.cmu.edu by March 31.  We’ll post the results here.

<snip>

REQ1

REQ2

REQ3

REQ4

REQ5

SPL1

SPL2

SPL3

SPL4

SPL5

SPL6

SPL7

SCO1

SCO2

INT1

INT2

INT3

INT4

INT5

INT6

INT7

INT8

INT9

INT10

INT11

INT12

INT13

INT14 N/A

INT15

INT16

(end)

– Paul Clements, SEI

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s