Index of /~billard/se/cs4320/ex3
Name Last modified Size Description
![[DIR]](/icons/back.gif)
Parent Directory 20-Apr-2008 08:49 -
![[ ]](/icons/layout.gif)
flow.pdf 19-May-2009 09:03 26k
![[ ]](/icons/unknown.gif)
print.bat 19-May-2009 09:03 1k
printdoc.bat 19-May-2009 09:03 1k
CS 4320 Software Testing and QA: Ex3 CODE COVERAGE
==================================================
NOTE: Draw your FLOWGRAPHS for Exercises 3.1 and 3.3 as plain text or
make a Word doc.
Platform: Paper
Given: source code
Goal: set up white box test cases using statement, decision, condition coverage.
In white box test, the engineer has access to the sources and uses the sources
to design test cases. The idea behind code coverage is that, if more lines of code
are actually executed given various test cases, then we are more confident that the code
is good (assuming the results are accurate). We will never be able to prove the absence
of bugs though, no matter how many test cases we run. And we don't just want to run
arbitrary number of cases. There's an intelligent way to design the necessary number
of cases called path coverage using cyclometric complexity. That will be done in Ex4
but this current exercise is in preparation for Ex4.
Coverage Strategies for White Box Test:
Statement Coverage (all-nodes): execute each statement at least once
Branch Coverage (all-edges) : execute each IF/WHILE with TRUE and FALSE
(also called decision coverage)
Path Coverage (all-paths) : execute each predicate of IF/WHILE with TRUE and FALSE
(also called condition coverage)
Before given the details, here is the bottom line:
1. Statement = Branch = Path
if (cond1) {
stmt1;
} else {
stmt2;
}
All three techniques will generate 2 test cases, one with cond1=TRUE, one with cond1=FALSE.
2. Statement <> Branch = Path
if (cond1) {
stmt1;
}
Branch and Path as before but Statement will do only 1 test case: cond1=TRUE.
3. Statement = Branch <> Path
if (cond1 && cond2) {
stmt1;
} else {
stmt2;
}
Statement and Branch will generate 2 test cases, say cond1=TRUE and cond1=FALSE.
This will be sufficient to "touch" both stmt1 and stmt2.
Path will generate 4 test cases:
cond1 cond2
-----------
T T
T F
F T
F F
However, since C compilers usually do "lazy" evaluation, the third case is irrelevant (or throw out
the fourth case).
EXAMPLE 1:
In condition coverage, we are concerned with each of the predicates.
Consider this segment of code:
if (a && b)
x;
else
y;
z;
Some compilers (C) perform lazy evaluation. If a is false, then don't
bother to test b. Other compilers (Pascal) would go on to test b.
Here is a problem:
if (i > 0) && (i++ < 10)
A lazy evaluation with the first condition false, would never increment i.
Or perhaps the second condition calls a function which performs some
important operations.
An expanded form of the multiple condition if-statement:
LAZY EVALUATION:
if (a)
if (b)
x;
else
y;
else
y;
z;
The corresponding FLOWGRAPH:
F T
+----a----+
| |
v F v
y<--------b
| | T
| v
| x
| |
+--->z<---+
We now add an "artificial" edge from the bottom (z) to the top (a):
..............
. .
F . T .
+----a----+ .
R4 | R1 | R3 .
v F v .
y<--------b .
| R2 | T .
| v .
| x .
| | .
+--->z<---+ .
. .
..............
(See flow.pdf for a nicer version.)
The artificial edge is just a mathematical enhancement used by graph
theorists (so that every edge is "reachable" from every other edge).
Note that the code does NOT execute back to the top.
The figure also has labeled the 4 separate regions: R1, R2, R3, R4.
and 2 predicates (T/F branches): a, b
and 5 nodes: a, b, x, y, z
and 7 edges: a-y, a-b, b-y, y-z, b-x, x-z, z-a (artificial)
Now, we are ready to make an important definition that will help us
determine the exact number of test cases we need for White Box Test.
The number is computed by the CYCLOMETRIC COMPLEXITY (CC):
Predicates + 1 = Edges - Nodes + 1 = Regions - 1
or
P + 1 = E - N + 1 = R - 1
That is, there are 3 different ways to compute CC, and they all yield
the same result.
In the above example, P = 2, E = 7, N = 5, R = 4, hence
CC = 2 + 1 = 7 - 5 + 1 = 4 - 1 = 3
This implies that there are exactly 3 test cases that will "cover" the code:
a b
------------
True True
True False
False *
where the wildcard (*) means the value of b could be either True or False;
The easiest way to remember CC is just: Predicates + 1.
Look at all the IF and WHILE statements and count the number of predicates:
if ((i<=1) and (j>10))
while ((k==0) || (k > 5)) {
// do something
}
How many predicates? (Answer: 4, hence CC=5)
CYCLOMETRIC COMPLEXITY will used extensively in Ex 4. For now, we'll continue
to examine other introductory material.
NON-LAZY EVALUATION:
if (a)
if (b)
x;
else
y;
else
if (b)
y;
else
y;
z;
F T
+----a----+
| |
F v T |
+--b--+ |
| | |
| | F |
+->y<-+<-----b
| | T
| v
| x
| |
+--->z<---+
With an artificial edge from z to a, this has 5 regions and 3 branch
nodes (predicates), hence a cyclometric complexity of 4. It also has
6 nodes, not 5, because b appears in TWO DIFFERENT nodes. With 9 edges
(including artificial), then E - N + 1 also has the cyclometric complexity
of 4.
The 4 test
cases would be:
a b
------------
True True
True False
False True
False False
EXERCISE 3.1:
Consider this segment of code:
if (a || b)
x;
else
y;
z;
Expand the code into individual predicates for both LAZY and NON-LAZY
evaluation and draw the corresponding flowgraphs.
In each case, what is the cyclometric complexity?
EXAMPLE 2:
Consider this function for absolute value:
int abs(int x) {
if (x >= 0) x = 0 - x;
return x;
}
This function is incorrect. The predicate should be: x < 0
A white box statement coverage test requires that every statement be
executed at least once.
Statement Coverage x Result Status
-------------------------------------------
1. IF-True 0 0 Pass
TEST RESULT: Pass
TOTAL TEST CASES: 1
Statement coverage would say that the code is correct when, in fact, it is
not correct.
Decision (Branch) coverage requires that each IF (and WHILE) evaluate to
TRUE and FALSE:
Decision Coverage x Result Status
-------------------------------------------
1. IF-True 0 0 Pass
2. IF-False -5 -5 Fail
TEST RESULT: Fail
TOTAL TEST CASES: 2
Decision coverage arrived at the correct result: the code has a fault.
Note that condition coverage would be the same as decision coverage in
this case because the IF statement has only one predicate.
EXAMPLE 3:
Consider this function with two predicates in the IF statement:
int check(int x) {
int flag;
if ((x <= 0) && (abs(x) <= 10))
flag = 1;
else
flag = 0;
return flag;
}
This code is incorrect because of the use of the incorrect abs() function.
Statement Coverage x Result Status
-------------------------------------------
1. IF-True 0 1 Pass
2. IF-False 5 0 Pass
TEST RESULT: Pass
TOTAL TEST CASES: 2
Decision coverage is the same as statement coverage here and gives the
same wrong answer.
Condition Coverage x Result Status
-------------------------------------------
1. IF-T/T 0 1 Pass
2. IF-T/F -15 1 Fail
3. IF-F/T 5 0 Pass
4. IF-F/F 15 0 Pass
TEST RESULT: Fail
TOTAL TEST CASES: 4
The above is for NON-LAZY evaluation but even LAZY evaluation would
still have test case #2 which indicates the fault.
EXAMPLE 4:
Consider this function that computes insurance premiums:
int liability(int age, char sex, int married) {
int premium;
premium = 500;
if ((age < 25) && (sex = 'M') && (!married))
premium += 1500;
else
{
if ((sex = 'F') || (married))
premium -= 200;
if ((age > 45) && (age <65))
premium -= 100;
}
return premium;
}
Statement Coverage Age Sex Married Test Case
---------------------------------------------------------
1. IF1-True <25 Male False (1) 23, M, 0
2. IF2-True * Female * (2) 50, F, 0
3. IF3-True >45,<65 * * (2)
TOTAL TEST CASES: 2
Decision Coverage Age Sex Married Test Case
---------------------------------------------------------
1. IF1-True <25 Male False (1) 23, M, 0
2. IF1-False * Female * (2) 23, F, 0
3. IF2-True * Female * (2)
4. IF2-False >=25 Male False (3) 50, M, 0
5. IF3-True >45,<65 * * (3)
6. IF3-False <=45 Female * (2)
TOTAL TEST CASES: 3
EXERCISE 3.2:
Fill-in the table for condition coverage. Try to re-use test cases
whenever possible. You might have to change an earlier test case so
that it works for the earlier case and a later case. If (1) can
be used again, then just list (1) by itself - similar to the
above tables.
Condition Coverage Age Sex Married Test Case
---------------------------------------------------------
1. IF1-T/T/T <25 Male False (1) 23, M, 0
2. IF1-T/T/F
3. IF1-T/F/T
4. IF1-T/F/F
5. IF1-F/T/T
6. IF1-F/T/F
7. IF1-F/F/T
8. IF1-F/F/F
9. IF2-T/T
10.IF2-T/F
11.IF2-F/T
12.IF2-F/F
13.IF3-T/T
14.IF3-T/F
15.IF3-F/T
16.IF3-F/F
TOTAL TEST CASES:
EXERCISE 3.3:
Given this C code:
while (a)
x;
if (b)
y;
z;
1. Draw a FLOWGRAPH of the code.
2. # of predicates (P) =
3. # of regions (R) =
4. # of nodes (N) =
5. # of edges (E) including artificial =
6. cyclometric complexity = P + 1 = R - 1 = E - N + 1 =
7. List the paths necessary to achieve statement coverage.
Make as few paths as possible but touch every statement.
Example: a,x,a,b,z touches statement x and z.
8. List the paths necessary to achieve branch coverage.
Make as few paths as possible but go TRUE/FALSE on each IF/WHILE condition.
You do not need to go TRUE/FALSE on each predicate within a compound condition.
9. List the paths necessary to achieve all-paths coverage.
Make EXACTLY the `cyclometric complexity' number of paths (your answer to 6 above).
You DO need to consider the individual predicates within a compound condition.
But note that we are doing LAZY evaluation.
The best way to solve this: try to add just one new edge for each path.
(Of course the first paths may touch more than one edge).
Always be cautious, not greedy. The best approach is to bounce off predicates with
FALSE first, and only later go for the inside part. Once you touch a new edge
in a path, try not to touch any more if you can help it.
To help with this, it's obvious that the complexity is 3 because there are
2 predicates. So make 3 paths but DON'T make any paths longer than 5 nodes.
Fill them into the left part of the table, but be sure each path touches a new edge
as you list them:
a-x x-a a-b b-z b-y y-z
Path 1: *
Path 2: *
Path 3: *
--------------------------------------------
Path 4: a,x,a,b,y,z 1 1 1 0 1 1
* fill-in the path here, along with the 0's and 1's.
Note that a Path 4 has been listed, which is 6 nodes long. All 6 possible edges
(excluding artificial) are listed across the top of the table.
What are those 1's and 0's?
TRUE or FALSE: Is this particular edge part of the listed path?
As for Path 4, ALL edges are in the path EXCEPT b-z.
a) Make 3 paths, no longer than 5 nodes, touch a new edge each time.
b) Fill-in 1's and 0's.
c) Do you see a relationship between the paths?
Is Path 4 equal to the ADDITION of two other paths MINUS the remaining path?
How to do this math? Just use the 1's and 0's on a per column basis.
But it has to work across the board.
Here's an example:
a-b c-d e-f
+ Path 1: e f 0 0 1
- Path 2: a b e f 1 0 1
+ Path 3: a b c d 1 1 0
---------------------------
= Path 4: c d 0 1 0
Therefore, Path 4 = Path 1 - Path 2 + Path 3.
Note that Path 4 serves as the goal, and you can tell which paths to add and
which paths to substract.
Now, if you can do this with YOUR table you've just proven that Path 4 is DEPENDENT
upon Paths 1, 2, 3. And Paths 1, 2, 3 are LINEARLY INDEPENDENT (an important topic in Ex4).
There's no way to add or subtract some of them to yield another.
But since Path 4 is the addition and subtraction somehow of Paths 1, 2, 3
then this could just be rewritten as Path 2 (say) is the combination of
1, 3, 4 somehow. Doesn't matter. Any THREE of them can be considered
INDEPENDENT and the last one DEPENDENT on the other three.
(In the above example, Path 2 = Path 1 + Path 3 - Path 4.)
How to avoid all this math and be SURE you have CC # of paths that are
independent? ALWAYS TOUCH A NEW EDGE! There will ALWAYS be CC # of INDEPENDENT
paths, and if you do it right, you CAN touch a new edge. And the math
implies that if you touch a new edge (put a 1 in a column), it can't be
dependent on the others because there are only 0's above it.
If you DO NOT touch a new edge each time, the paths MAY still be independent
but you would not be sure - without exhaustively trying all possible additions
and subtractions.
Be cautious, don't be greedy.
Now you're ready for Ex4.
Project Notebook:
README
Exercise 1
Exercise 2
Exercise 3