AP Computer Science Principles · Unit 1 lecture · Part 9 of 11 · 8 min

1.4 Testing with defined inputsChoosing tests that actually find bugs

Mr. Thompkins
A logic error only shows up when you feed the program the right input. So the real skill isn't running tests, it's choosing them. Here's how professionals choose.

In the development process, testing uses defined inputs to make sure an algorithm or program produces the expected outcomes. Defined means chosen on purpose, ahead of time, along with the result you expect. Then you compare: expected versus actual. If they differ, you have found a bug, and you use the results of the test to revise the program.

Which inputs? The course is specific: defined inputs should demonstrate the different expected outcomes, and include values at or just beyond the extremes, the minimum and maximum, of the input data. In practice that means: one test for each kind of result the program can give, plus tests right at the edges, where a cutoff or a limit lives.

?Wait, what's the difference between > and ≥?
x > 50 means "greater than 50", so 50 itself is false. x ≥ 50 means "greater than or equal to 50", so 50 is true. The two agree for every value except 50 itself, which is exactly why you must test at the cutoff.

Watch it work. A store offers free shipping on orders of 50 dollars or more. Here's the procedure someone wrote.

1PROCEDURE shipping(total)2{3  IF (total > 50)14  {5    RETURN(0)6  }7  RETURN(7)28}
  1. line 3 Meant to be free shipping for 50 or more
  2. line 7 Every other order pays 7
Free shipping for 50 or more. Read line 3 carefully.

Test it with "normal" orders, 30 and 80, and everything looks perfect: 30 pays 7, 80 ships free. Those tests are useful, because they show both expected outcomes, but they can't catch this bug. Only an order of exactly 50 does: the requirement says it ships free, and the code charges 7, because 50 > 50 is false.

test inputexpectedactualresult
3077pass
8000pass
4977pass
5007FAIL
5100pass
A test table: input, expected, actual. Only the boundary value 50 reveals the error.
47484950515253just belowat the cutoffjust abovepays 7ships free> and ≥ disagree only at 50 itself
Boundary values for a cutoff at 50: just below, at, and just above.
?Wait, how do you know the expected outcome?
From the program requirements. You can't pick good tests, or know the right answers, without knowing what the program is supposed to do. That's why the course says requirements are needed to identify appropriate defined inputs.

Two more kinds of tests catch a lot: unusual inputs, like an empty list, a zero, or a negative number, and extremes, like the smallest and largest values allowed. If a timer accepts 5 to 60 minutes, test 4, 5, 60 and 61. If a procedure takes a list, test an empty list and a list of one element.

Pause and think
A procedure is supposed to return true for ages 13 to 19, inclusive. Which ages would you test, and why those?
I thought about it. Show me
Test both sides of both cutoffs: 12 and 13 at the bottom, 19 and 20 at the top, plus one value in the middle like 16 and one far outside like 5. A bug at either edge, like writing > 13 or < 19, shows up at 13 or 19.
Find the bug with a test
This loop is supposed to display 5, 3, 1 and then stop. Choose a way to test it, run the test by tracing, describe what goes wrong, and fix it.
1n ← 52REPEAT UNTIL (n = 0)13{4  DISPLAY(n)5  n ← n - 26}
  1. line 2 n goes 5, 3, 1, -1, ... and never equals 0
output › 5 3 1 -1 -3 ... forever
A countdown by 2s.
  1. Test by tracing: write a table with one row per check of the condition, tracking n and whether n = 0 is true.
    Why?
    REPEAT UNTIL checks its condition before every pass and stops the first time it's true.
  2. Check 1: n is 5, not 0, so display 5 and n becomes 3. Check 2: display 3, n becomes 1. Check 3: display 1, n becomes -1.
  3. Check 4: n is -1. -1 = 0 is false, so it keeps going: -1, -3, -5, ... n skipped right over 0. The loop never ends.
    Why?
    Counting down by 2 from an odd number never lands on 0 exactly. This is a logic error that shows up as an infinite loop.
  4. Fix: change the condition to REPEAT UNTIL (n ≤ 0). Now at n = -1 the condition is true, and the loop stops after displaying 5 3 1.
    Why?
    Using ≤ instead of = makes the stop condition catch any value at or past the goal, not just one exact value.

Answer: It displays 5 3 1 -1 -3 and never stops, because n jumps from 1 to -1 and never equals 0. Changing the condition to n ≤ 0 makes it display 5 3 1 and stop.

checknn = 0?displayed
15false5
23false3
31false1
4-1false-1
5-3false-3 (and on, forever)
The trace shows the problem at check 4: n has passed 0 without ever equaling it.
Stuck? Get a hint
  1. Write the value of n at each check: 5, 3, 1, ...
  2. What comes after 1 when you subtract 2?
  3. Does n ever equal exactly 0? What condition would still be true once n has gone past 0?
Check yourself
A procedure should return "pass" for scores of 65 or higher and "fail" for anything lower. Scores are whole numbers from 0 to 100. Which set of tests is most likely to reveal an error in the comparison?
  1. A20, 50, 90
  2. B64, 65, 66
  3. C0, 100
  4. D70, 80, 90
Show the answer

B. 64, 65 and 66 test just below, at, and just above the cutoff. A mistake like score > 65 would wrongly fail 65. The other sets never touch 65, so both versions of the comparison give the same results on them.

Check yourself
Which statement about testing is true?
  1. AIf a program passes five tests, it has no logic errors.
  2. BGood tests use defined inputs with known expected outcomes, including values at and just beyond the extremes of the input.
  3. CTests are only needed after the program is finished.
  4. DTesting with random inputs is always better than choosing inputs on purpose.
Show the answer

B. Testing compares actual outcomes with expected outcomes for chosen inputs, and boundaries are where bugs hide. Passing a few tests never proves there are no errors, which is the trap in the first choice, and testing happens throughout development.

Exam tip from Mr. T

When a question asks for the "best set of test inputs", look for the set that sits on both sides of every cutoff. Big, normal-looking numbers almost never win.

  • Testing uses defined inputs with expected outcomes. Compare expected and actual, then revise.
  • Choose inputs that show every expected outcome, plus values at and just beyond the minimum and maximum.
  • Test at each cutoff (just below, at, just above) and with unusual inputs like an empty list or zero.
  • Program requirements tell you which inputs to test and what to expect.