I have seen (and been involved in) this debate a lot. These are the opposing arguments:
1. You should never use white box testing as then you are testing the implementation not the interface of the module. The tests should not have to change when the implementation changes.
2. You should always use white box testing since you can't possibly test every combination of inputs. Making use of knowledge of how the module is implemented allows you to test inputs that are likely to give incorrect results, such as boundary conditions.
there are different
interpretations...
of the opposite of
black box testing
interpretations...
of the opposite of
black box testing
Paradoxically, both of these arguments are valid. The problem comes down to what is meant by white box testing. Unfortunately, the meaning has never been clear since it is obviously just the opposite of black box testing and everyone knows what black box testing means. The problem is semantic -- there are different interpretations of what is meant by the opposite of black box testing.
Here is my attempt to describe two different meanings of white box testing:
Meaning 1: Test that a module works according to its (current) design by checking how it behaves internally by reading private data, or intercepting private messages.
Meaning 2: Test a module using knowledge of how it is implemented internally, but only ever testing through the public interface.
Like cholesterol, there are both bad (Meaning 1) and good (Meaning 2) forms of white-box testing.
Bad White Box Testing
Recently a colleague came and asked me how a Unit Test should access private data of the module under test. This question completely threw me as I had never needed, or even thought, to do this. I mumbled something about not really understanding why he wanted to.
Later, when I thought about it I realised that Unit Tests should not be accessing private data or methods of the modules they are testing. Doing this would make the unit test dependent on the internal details of the module. Unit Tests should only test the external behaviour of a module never how it is actually implemented.
As I said in my previous post (Unit Tests - What's so good about them?), one of their main advantages is that they allow you to easily modify and refactor code and then simply run the Unit Tests to ensure that nothing has been broken. This advantage would be lost if the test depends on the internal implementation of the module. Changing the implementation (without changing the external behaviour) could break the test. This is not what you want.
Unit Tests should only test the interface of the module and never access private parts.
Good White Box Testing
The whole point of Unit Tests is that they are written by the same person(s) who wrote the code. The best tests are those that test areas that are likely to cause problems (eg, internal boundary conditions) and hence can only be written by someone with an intimate knowledge of the internal workings of the module.
For example, say you are writing a C++ class for infinite precision unsigned integers and you want to create some Unit Tests for the "add" operation (implemented using operator+). Some obvious tests would be:
assert(BigInt(0) + BigInt(0) == BigInt(0));
assert(BigInt(0) + BigInt(1) == BigInt(1));
assert(BigInt(1) + BigInt(1) == BigInt(2));
// etc
Of course, the critical part of a BigInt class is that carries are performed correctly when an internal storage unit overflows into the next significant unit. Doing simple black box testing you can only guess at how the class is implemented. (8-bit integers, 32-bit integers, BCD or even just ASCII characters could be used to store the numbers.) However, if one wrote the code one could know that, for example, internally 32-bit integers are used, in which case this is a good test:
// Check that (2^32-1) + 1 == 2^32
assert(BigInt::FromString("4294967295") + BigInt(1) ==
BigInt::FromString("4294967296"));
This test will immediately tell you if the code does not carry from the lowest 32-bit integer properly. Obviously, you need other tests too, but this tests a critical boundary condition.
To find the sort of defects that the above test checks for, using black box testing, would require a loop that would probably take days to run. (I will talk next month about why Unit Tests should not be slow.)
Which is The Accepted Definition?
The whole confusion about White Box Testing is that the distinction above is not clear. Most definitions imply the "good" definition, but then say something that contradicts this. For example, the Wikipedia page on White Box testing (as at November 2013) does not make the distinction but seems to imply the "good" definition, but then suggests it is talking about the "bad" form - for example it says it is like like in-circuit testing which is the hardware equivalent of accessing private data.
My general conclusion, talking to colleagues is that the "good" definition is what most people think White Box Testing is or should be, but there are some people that use the "bad" definition.
I don't really care which definition of white box testing you think is correct as long as you distinguish between them, and as long as you specifically use "good" white box testing with your unit tests.
Disadvantages of White Box Testing
At this point I will note a few problems.
First, with some code the distinction between interface and implementation is blurred. This is a bad thing for many reasons, one of which is you don't know if you are doing "bad" white box testing. See
Best Practice for Modules in C/C++ on how to separate interface from implementation. Also see the sections on Information Hiding and Decoupling at Software Design.
Also with white box testing it is easy to forget to add new tests when the implementation changes. For example, using the BigInt example above, imagine that the code was enhanced to use 64-bit integers (eg, because most users were moving to 64-bit processors where there would be a large performance boost). If, after changing the implementation, the test was not modified (or a new test not added) to check for overflow of the 64-bit unit then the Unit Tests would not be complete.
Conclusion
There are two points to this post.
1. Make sure you understand what someone means by white box testing. When they say Unit Tests should not use white box testing then they are probably talking about "bad" white box testing which accesses the internals of the module under test.
It is pointless
to write Unit Tests
that only do
black box testing
to write Unit Tests
that only do
black box testing
2. When writing Unit Tests you should always use "good" white box testing to test for different combinations of parameters, boundary conditions and other likely problem areas.
It is pointless to write Unit Tests that simply do black box testing. The whole point of Unit Tests is that they are written by whomever wrote the code, in order to test likely problem areas.
No comments:
Post a Comment