Testing: Testing is operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component. –- source: IEEE
When testing, we execute a set of test cases. A test case specifies how to perform a test. At a minimum, it specifies the input to the software under test (SUT) and the expected behavior.
📦 Example: A minimal test case for testing a browser:
- Input – Start the browser using a blank page (vertical scrollbar disabled). Then, load
longfile.htmllocated in the
- Expected behavior – The scrollbar should be automatically enabled upon loading
Test cases can be determined based on the specification, reviewing similar existing systems, or comparing to the past behavior of the SUT.
A more elaborate test case can have other details such as those given below.
- A unique identifier : e.g. TC0034-a
- A descriptive name: e.g. vertical scrollbar activation for long web pages
- Objectives: e.g. to check whether the vertical scrollbar is correctly activated when a long web page is loaded to the browser
- Classification information: e.g. priority - medium, category - UI features
- Cleanup, if any: e.g. empty the browser cache.
For each test case we do the following:
- Feed the input to the SUT
- Observe the actual output
- Compare actual output with the expected output
A test case failure is a mismatch between the expected behavior and the actual behavior. A failure is caused by a defect (or a bug).
📦 Example: In the browser example above, a test case failure is implied if the scrollbar remains disabled after loading ‘longfile.html’. The defect/bug causing that failure could be an uninitialized variable.
Here is another definition of testing:
Software testing consists of the dynamic verification that a program provides expected behaviors on a finite set of test cases, suitably selected from the usually infinite execution domain. -– source: Software Engineering Book of Knowledge V3
Some things to note (indicated by keywords in the above definition):
- Dynamic: Testing involves executing the software. It is not by examining the code statically.
- Finite: In most non-trivial cases there are potentially infinite test scenarios but resource constraints dictate that we can test only a finite number of scenarios.
- Selected: In most cases it is not possible to test all scenarios. That means we need to select what scenarios to test.
- Expected: Testing requires some knowledge of how the software is expected to behave.
Explain how the concepts of testing, test case, test failure, and defect are related to each other.