Written by Member
Software Testing Techniques
Introduction
Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure.
This method of test design is applicable to all levels of software testing: unit, integration, functional testing, system and acceptance. The higher the level, and hence the bigger and more complex the box, the more one is forced to use black box testing to simplify. While this method can uncover unimplemented parts of the specification, one cannot be sure that all existent paths are tested.
Test design techniques
* Equivalence partitioning
* Boundary value analysis
* Decision table testing
* Pairwise testing
* State transition tables
* Use case testing
* Cross-functional testing
Testing Strategies/Techniques
* black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function
* data outside of the specified input range should be tested to check the robustness of the program
* boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output
* the number zero should be tested when numerical data is to be input
* stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems
* crash testing should be performed to see what it takes to bring the system down
* test monitoring tools should be used whenever possible to track which tests have already been performed and the outputs of these tests to avoid repetition and to aid in the software maintenance
* other functional testing techniques include: transaction testing, syntax testing, domain testing, logic testing, and state testing.
* finite state machine models can be used as a guide to design functional tests
* According to Beizer the following is a general order by which tests should be designed:
1.Clean tests against requirements.
2. Additional structural tests for branch coverage, as needed.
3. Additional tests for data-flow coverage as needed.
4. Domain tests not covered by the above.
5. Special techniques as appropriate--syntax, loop, state, etc.
6. Any dirty tests not covered by the above.
Black Box Testing Strategy
Black Box Testing is not a type of testing; it instead is a testing strategy, which does not need any knowledge of internal design or code etc. As the name "black box" suggests, no knowledge of internal logic or code structure is required. The types of testing under this strategy are totally based/focused on the testing for requirements and functionality of the work product/software application. Black box testing is sometimes also called as "Opaque Testing", "Functional/Behavioral Testing" and "Closed Box Testing".
The base of the Black box testing strategy lies in the selection of appropriate data as per functionality and testing it against the functional specifications in order to check for normal and abnormal behavior of the system. Now a days, it is becoming common to route the Testing work to a third party as the developer of the system knows too much of the internal logic and coding of the system, which makes it unfit to test the application by the developer.
In order to implement Black Box Testing Strategy, the tester is needed to be thorough with the requirement specifications of the system and as a user, should know, how the system should behave in response to the particular action.
Various testing types that fall under the Black Box Testing strategy are: functional testing, stress testing, recovery testing, volume testing, User Acceptance Testing (also known as UAT), system testing, Sanity or Smoke testing, load testing, Usability testing, Exploratory testing, ad-hoc testing, alpha testing, beta testing etc.
These testing types are again divided in two groups: a) Testing in which user plays a role of tester and b) User is not required.
Testing method where user is not required:
Functional Testing:
In this type of testing, the software is tested for the functional requirements. The tests are written in order to check if the application behaves as expected.
Stress Testing:
The application is tested against heavy load such as complex numerical values, large number of inputs, large number of queries etc. which checks for the stress/load the applications can withstand.
Load Testing:
The application is tested against heavy loads or inputs such as testing of web sites in order to find out at what point the web-site/application fails or at what point its performance degrades.
Ad-hoc Testing:
This type of testing is done without any formal Test Plan or Test Case creation. Ad-hoc testing helps in deciding the scope and duration of the various other testing and it also helps testers in learning the application prior starting with any other testing.
Exploratory Testing:
This testing is similar to the ad-hoc testing and is done in order to learn/explore the application.
Usability Testing:
This testing is also called as ‘Testing for User-Friendliness’. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user.
Smoke Testing:
This type of testing is also called sanity testing and is done in order to check if the application is ready for further major testing and is working properly without failing up to least expected level.
Recovery Testing:
Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications.
Volume Testing:
Volume testing is done against the efficiency of the application. Huge amount of data is processed through the application (which is being tested) in order to check the extreme limitations of the system.
Testing where user plays a role/user is required:
User Acceptance Testing:
In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to.
Alpha Testing:
In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers.
Beta Testing:
In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers.
Black Box Testing Example
In this technique, we do not use the code to determine a test suite; rather, knowing the problem that we're trying to solve, we come up with four types of test data:
1. Easy-to-compute data
2. Typical data
3. Boundary / extreme data
4. Bogus data
For example, suppose we are testing a function that uses the quadratic formula to determine the two roots of a second-degree polynomial ax2+bx+c. For simplicity, assume that we are going to work only with real numbers, and print an error message if it turns out that the two roots are complex numbers (numbers involving the square root of a negative number).
We can come up with test data for each of the four cases, based on values of the polynomial's discriminant (b2-4ac):
Easy data (discriminant is a perfect square):
a b c Roots
1 2 1 -1, -1
1 3 2 -1, -2
Typical data (discriminant is positive):
a b c Roots
1 4 1 -3.73205, -0.267949
2 4 1 -1.70711, -0.292893
Boundary / extreme data (discriminant is zero):
a b c Roots
2 -4 2 1, 1
2 -8 8 2, 2
Bogus data (discriminant is negative, or a is zero):
a b c Roots
1 1 1 square root of negative number
0 1 1 division by zero
As with glass-box testing, you should test your code with each set of test data. If the answers match, then your code passes the black-box test.
Advantages of Black Box Testing
* Efficient when used on Larger systems
* As the tester and developer are independent of each other, test is balanced and unprejudiced
* Tester can be non-technical.
* There is no need of having detailed functional knowledge of system to the tester.
* Tests will be done from a end user's point of view. Because end user should accept the system. (This is reason, sometimes this testing technique is also called as Acceptance testing)
* Testing helps to identify the vagueness and contradiction in functional specifications.
* Test cases can be designed as soon as the functional specifications are complete
Disadvantages of Black Box Testing
* Test cases are tough and challenging to design, without having clear functional specifications
* It is difficult to identify tricky inputs, if the test cases are not developed based on specifications.
* It is difficult to identify all possible inputs in limited testing time. So writing test cases is slow and difficult
* Chances of having unidentified paths during this testing
* Chances of having repetition of tests that are already done by programmer.
Dimensions for Examining Testing
There are five dimensions that can be used for examining testing:
1. People (who does the testing)
2. Coverage (what gets tested)
3. Risks (why you are testing)
4. Activities (how you are testing)
5. Evaluation (how you know you’ve found a bug)
Characteristics
Let’s use this system to understand and clarify the characteristics of black box and white box testing.
People: Who does the testing?
Some people know how software works (developers) and others just use it (users). Accordingly, any testing by users or other nondevelopers is sometimes called “black box” testing. Developer testing is called “white box” testing. The distinction here is based on what the person knows or can understand.
Coverage: What is tested?
If we draw the box around the system as a whole, “black box” testing becomes another name for system testing. And testing the units inside the box becomes white box testing. This is one way to think about coverage.
Another is to contrast testing that aims to cover all the requirements with testing that aims to cover all the code. These are the two most commonly used coverage criteria. Both are supported by extensive literature and commercial tools. Requirements-based testing could be called “black box” because it makes sure that all the customer requirements have been verified. Code-based testing is often called “white box” because it makes sure that all the code (the statements, paths, or decisions) is exercised.
Risks: Why are you testing?
Sometimes testing is targeted at particular risks. Boundary testing and other attack-based techniques are targeted at common coding errors. Effective security testing also requires a detailed understanding of the code and the system architecture. Thus, these techniques might be classified as “white box.”
Another set of risks concerns whether the software will actually provide value to users. Usability testing focuses on this risk, and could be termed “black box.”
Activities: How do you test?
A common distinction is made between behavioral test design, which defines tests based on functional requirements, and structural test design, which defines tests based on the code itself. These are two design approaches. Since behavioral testing is based on external functional definition, it is often called “black box,” while structural testing—based on the code internals—is called “white box.” Indeed, this is probably the most commonly cited definition for black box and white box testing.
Another activity-based distinction contrasts dynamic test execution with formal code inspection. In this case, the metaphor maps test execution (dynamic testing) with black box testing, and maps code inspection (static testing) with white box testing.
We could also focus on the tools used. Some tool vendors refer to code-coverage tools as white box tools, and tools that facilitate applying inputs and capturing inputs—most notably GUI capture replay tools—as black box tools. Testing is then categorized based on the types of tools used.
Evaluation: How do you know if you’ve found a bug?
There are certain kinds of software faults that don’t always lead to obvious failures. They may be masked by fault tolerance or simply luck. Memory leaks and wild pointers are examples. Certain test techniques seek to make these kinds of problems more visible. Related techniques capture code history and stack information when faults occur, helping with diagnosis. Assertions are another technique for helping to make problems more visible. All of these techniques could be considered white box test techniques, since they use code instrumentation to make the internal workings of the software more visible. These contrast with black box techniques that simply look at the official outputs of a program.
To summarize, black box testing can sometimes describe user-based testing (people); system or requirements-based testing (coverage); usability testing (risk); or behavioral testing or capture replay automation (activities). White box testing, on the other hand, can sometimes describe developer-based testing (people); unit or code-coverage testing (coverage); boundary or security testing (risks); structural testing, inspection or code-coverage automation (activities); or testing based on probes, assertions, and logs (evaluation).
