REQUIREMENTS BASED TESTING

IT project failure has become a very common factor during the past 10 years; statistics show that the root causes of many unsuccessful projects are due to poor requirements. James Martin shows in his study that 56% percent of bugs identified in projects are errors introduced in the requirements stage (Borland Software Corporation, 2005); therefore, getting the requirement right is essential. This can be achieved by requirements based testing approach, this essay is a discussion on what is requirement based testing, why it is important, how can it be applied to reduce bugs in software products, and what benefits can be gained.

 “Requirements Based Testing is the practice of aligning requirements at the beginning of a software project with quality expectations for testing –to ensure final deliverables consistently meet business needs.” (Alho, 2007). The above definition implies that using requirement based testing practices help ensure that the final outcome of the software consistently meets the business needs. Requirement based testing address two major concepts, first it focuses on validating whether the requirments are correct and makes sure that the requirements are up to the standards (ie complete,unambigous and logically consistent ), secondly it designs a set of test cases from the requirements. The design and the code should fully meet those requirements. This can be achived using black box testing. Therefore RBT can be defined as a methodology which defines set of rules to test the requirements and maximize the software quality by developing the right product.

However one can come to the conclution that the RBT process is tredious aditional work when developing a software product. But RBT is important because requirements are a critical factor when developing a software and you must get it right from the beginning of the development life cycle. Research has shown that most software failures are due to the following reasons

• Requirements and specifications are incomplete.
• Requirements and specifications change too often.
• There is a lack of user input (to requirements). (Mogyorodi, 2003)

All the above mention reasons are circled around requirements, therefore it is a critical need to verify and validate the requirements from the requirement stage itself.

Early bug detection reduces the cost of bug fixing; as mentioned above most root cause of bugs are due to inadequate requirements. According to the above definition of RBT, it improves the requirement and reduces software bugs from the requirement stage itself. This clarifies rather than following the traditional approach of testing which is performed after the code is delivered; therefore, it is important to follow the RBT process to eliminate software bugs and increase software quality.

To understand the RBT process it is important to know the RBT methodology. RBT methodology can be defined in 12 steps (Mogyorodi, 2003). Out of the 12 steps, five of the most important steps are taken in to consideration in-order to understand the effectiveness of the RBT process.

1.      Validation of requirements against business objectives – this ensures that the requirements satisfies the objectives of the project and meets the business need of the project.

2.      Review of requirements by stakeholders- refines the requirements by getting feedbacks from the stakeholder before going to the next phase.

3.      Development of use-cases - Map requirements against a task-oriented or interaction-oriented view of the system to make sure the requirements are complete.

4.      Application of language analysis techniques- Identify ambiguous, unclear or inconsistent phrases in the requirements and fix them to guarantee the consistency and clarity of the requirements.

5.      Logical consistency checks performed and test cases designed. - Design test cases that are 100% equivalent to the functionality in the requirements.

(Mogyorodi, 2003)

This gives a clear understanding on how the RBT process can improve the software quality and identify errors from the very first stage itself, due to validation of requirements against business objectives. This also indicates that the RBT process does not only detect bugs but it also address bug prevention by integrating testing throughout the development life cycle. At the same time RBT practices eliminates the most common risk of incomplete and changing requirement specifications issue because of its application of language analysis techniques and traceability functionalities. 

 

Figure 1: Requirements Based Testing process flow  (Aharonovitz, 2008)

Figure 1 show the actives which are undertaken by the RBT process. This shows the goals of RBT using separate boxes. Arrow heads direct how they can be achieved.

The first goal of the RBT process is ensuring the quality of requirement. It can be achieved by ensuring that business needs are met, maximizing user/domain expert’s involvement, and validating whether the requirements are correct and none conflicting. The following methods must be applied to ensure the quality of the requirements: Review of requirements against 

business objectives, by domain experts/business users, and map the requirements against use cases ambiguity analysis of requirements.  (Aharonovitz, 2008)

Designing logical test cases. This can be achieved by expressing the requirements using cause-effect cards, generation of test cases, and the use of truth tables or unique path detention can derive the equivalent test cases for the requirements to optimize the number of test cases.  (Aharonovitz, 2008)

The next two goals, ensuring the quality of the test cases, and the quality of design and code can be achieved by reviewing the test cases by requirement authors, domain experts, end users and also by reviewing test cases by designers and developers in the code review process.

Last goal of the RBT process is completing and executing tests which ensure that the actual behavior of the system is the expected behavior and meets the business needs. (Borland Software Corporation., 2006)

Moty Aharonovitz, senior director of Product Strategy at Borland Software, states that traceability plays a critical role in the RBT process; he claims that maintaining traceability information between requirements, test-cases and tests is crucial. This determines which test cases or tests should be changed when a specific requirement changes, and the impact of change. (Aharonovitz, 2008)

In conclusion RBT process and activities describe that RBT depends on two major functionalities 1) Quality assured requirements and 2) test cases. Using the following functionalities throughout the development life cycle development teams can guarantee that they are building the right product with the lowest defects.

To ease the RBT process and use it effectively organizations can use tools which automate the RBT process. For example Borland software cooperation has taken several approaches to automate the RBT process and has developed a set of tools, namely Borland Caliber DefineIT - designed to ensure that software requirements are defined completely and accurately from the beginning, Borland CaliberRM - designed to facilitate collaboration, impact analysis, and communication in the definition and management of changing requirements, and  SilkCentral Test Manager - delivers visibility and control of project quality aligned with business requirements and is integrated with CaliberRM to maintain bi-directional traceability and change impact analysis between business requirements and test activities (Borland Software

Corporation., 2006). By using an automated RBT process it would ease the software development cycle, and the final outcome will avoid downstream defects and project delays.

From the above discussion it is a fact that when a company follows RBT methodology, they will be rewarded with better products and profit. Considering the facts discussed above requirement based testing can be dived in to three forms from the business point of view; 1) Business benefits, 2) Operational benefits and 3) financial benefits.

According to the RBT methodology as discussed above each of the categorized benefits are analyzed using the RBT functionalities. Beginning with the business benefits which can be achieved by clear, complete and correct requirements which lead to developing what the customer really wants; thus resulting in higher customer satisfaction and improved market agility. Operational benefits focus on benefits while developing the software. As discussed above RBT has a well structured methodology to improve tractability and visibility of the requirements which eases the development cycle. This will optimize the QA activities ensuring the quality of the product. Furthermore it enhances the software quality by introducing requirements test coverage which will ensure all requirements are satisfied by the end product. Furthermore financial benefits can be looked at from the revenue prospect of an organization. By following the RBT process and activities development teams can reduce compliance risks and improve impact analysis. Moreover the most important aspect of financial benefits are related to profit;  by identifying the defects early and fixing them in a very early stages it reduces re-work, helping save time and money which will leads to gaining more profit.

Although there are many advantages of using the RBT process in the development life cycle Robin F. Goldsmith, JD claims that there can be draw backs associated with RBT process as well. He states that there are limitations imposed by inadequately defined requirements, assessing the extent to which the requirements are complete, the appropriate level of test case details, and inclusion of requirements based unit tests by developers. However he also states the above pitfalls can be overcome by identifying the strengths and often unrecognized weakness of requirement base tests, and understanding the importance of testing based on business as well as system requirements and discovering how to identify more of the necessary but often overlooked test cases (Goldsmith, 2008).

In conclusion the requirements based testing approach is a very important process to be followed due to the fact that many IT project failures are associated with requirements and requirement management. The requirements based testing process introduces a well formed procedure from the requirement gathering stage to the deploying stage of the product. It has an effective methodology to gather requirements without flaws, and defines how to test the requirements properly in-order to ensure that the right requirements are captured. At the same time it has methods to keep track of changing requirements, and ways to define requirement based test cases including maximum coverage with a minimum number of test cases. It is a fact that by following the above mentioned process most of the software bugs can be identified early, requirements specifications can be produced clearly, and requirements changes can be managed effectively. Even though there can be pit falls in the RBT process, if it is followed properly one can avoid many draw backs and gain a lot of benefits from it.

Would use of design by contract have prevented the Ariane 5 disaster?

 

Ariane 5 was a European expendable launch system designed for delivering satellites to space, exploded 40 seconds after the lift –off which caused around 500 million damage to the world.  When developing the Ariane 5 software, some codes were reused from the Ariane 4 system which was used successfully for Ariane 4 but had  different flight characteristics than Ariane 5. Soon after the explosion an inquiry board was called to investigate the disaster and they found that explosion was due to a software error in the inertial reference system. Specifically a 64 bit floating point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16 bit signed integer. The number was larger than 32,767, the largest integer storable in a 16 bit signed integer, and thus the conversion failed. (Ladkin, 1998)

The most simplest answer is Yes but if i elaborate more ..

It is known Ariane 5 had failed due to a reuse specification error (Ladkin, 1998). If software engineers and designers had followed the design by contract methodology this disaster could have been avoided.

Design by contract follows set of methods which provide some tools for formulate the contracts, to increase the quality of the software with correctness and robustness. Which would also enhance the performance of  the reusability of software components, which could have avoided and prevented its destruction. Author would like to discuss the following key concepts of DBC which would have prevented Ariane 5 disaster.

• Explicitly handle exceptions from the design stage.

Design by contract force all the fundamental constraints in each code module must be explicitly handled and mentioned from the design state itself. Design by contract propose several approaches to be taken to satisfy the above conditions. Such as preconditions and post conditions which validates fundamentals inside the functions, and invariant which specify the legitimate values of the class attributes(validate the fundamental contracts inside a class). In Ariane 4 if the  software crew had mentioned the constraints, from designing the module itself and put  the needed preconditions post conditions and invariants, this error would have detected from the reusing state  and change the values accordingly in Ariane 5 module. And assertions can be verified during the testing stage to verify the boundaries.

• Provide systematic documentation for the software components.

 “The requirement that the horizontal bias should fit on 16 bits was in fact stated in an obscure part of a document. But in the code itself it was nowhere to be found!” (Jézéquel, 1997)

            If all the critical assertions and exceptions were documented and all the constraints were mentioned in the design specification document, QA team and review team would have given the proper attention to the validations and done the unit tests accordingly. One cannot blame QA team and the review team that they have not done their job right because you cannot test all software components in a large system to a given restricted timeline (you can test a lot but not all). Therefore if the documents are systematically produced reusing stage itself one can identify the defect and fix it. And QA team would give a critical testing on the boundaries and exceptions which had fundamental constraints mentioned in the documentation. Design by contract make the documentation easy and it explicitly mention the pre conditions post conditions and invariant to make the documents more clear.

Design by contract make the code self documenting it clearly classify the exceptions and the constraint which must be handled inside each functions and classes. And if the proper assertions are mention from the code itself review team would identify the critical points of the code, but in Ariane 5 code had not mentioned about the fundamental constraints which leads review team to abound the above error.