Practical Testing

One of the common complaints about TDD and unit testing that I see is that the examples used aren’t real. People often say that the examples used are toys and that writing such tests adds little or no value. To be honest, I often find myself agreeing with them.

One of the problems of adding unit tests to an existing code-base or driving a new project with TDD is deciding exactly where to spend your testing efforts. This is more of an issue when adding tests to existing code as I personally find that the safety of TDD on new code becomes slightly addictive…

Anyway, in an attempt to show how adding unit tests to existing code can be worthwhile I’ve decided to write a series of blog entries on Practical Testing of real world code…

The code I’ll be using for first examples is a class that is used by my high performance, I/O completion port based networking code, The Server Framework; The CCallbackTimer. class provides a very light-weight timer manager that runs on its own thread and is reasonably challenging to test. I have a subtle bug that I need to fix in it and reproducing the conditions that cause the bug are quite difficult; first run your PC for 49.7 days…

The class lives in one of our tools libraries and is coupled to several other classes and one other library. First I’ll present the code and the fragment of the library that’s required to use it. Then I’ll add a test harness for the library and write a simple test for the class. Then I’ll finally start working on the bug fix and some tests that prove it’s fixed and make sure it stays fixed.

I’ll use this entry as an index that I’ll update as I post the future entries.

• 1 - Introduction - where we describe the code that we’ll test and explain the problems we’re hoping to fix.
• 2 - The first test - where we create a test harness and a unit test.
• 3 - Test 2, Enter the mocks - where we write our first real test and struggle to write a reliable tests when time is involved…
• 4 - Taking control of time - where we tell the object under test where to source its time data from, rather than just allowing it to decide for itself.
• 5 - Testing shouldn’t be this hard - where we make the time source more complicated to enable us to control the multi-threaded nature of the class under test.
• 6 - Tests refactored - where we fix the bug that we discovered in part 5, clean up the tests and finally write the test that shows up the problems that happen when GetTickCount() rolls over to 0.
• 7 - Fixing the tick count wrap bug - where we finally put in a fix for the bug that shows up when GetTickCount() rolls over to 0.
• 8 - Once more, with tests first - where we see what the code might look like if we had developed test first rather than by following the HITIW methodology.
• 9 - More tests, more development, notice the order? - where we write a couple more tests for the new version of the timer queue and do just enough development to make one of them pass.
• 10 - Fixing the tick count wrap bug, again - where we fix the tick count bug again, only better.
• 11 - Moving away from the simplest thing - where we add back some real world functionality so that the timer queue can support more than one timer.
• 12 - Threading is orthogonal - where we realise that the threaded aspect was orthogonal to the real work and add it back in an optional way.
• 13 - Missing functionality - integrating the new code into a client of the old code exposes some missing functionality.
• 14 - Bitten by the handle reuse problem - where we use the tests we’ve built to support a redesign to remove the handle reuse problem.
• 15 - Testing payback - where we use the tests we’ve built to support a redesign to improve the performance of the code.
• 16 - Fixing a timeout bug - where we use the tests we’ve built to support a bug fix.
• 17 - A whole new approach - where, once again, we use the tests we’ve built to support a major redesign of the code.
• 18 - Removing the potential to deadlock - where, yet again, we use the tests we’ve built to support a major redesign of the code.
• 19 - Removing the duplicate code - we finally remove the duplicate code that was introduced in part 17.
• 20 - Mind the gap - Some of the new tests that we added in part 18 were not actually testing as much as we thought they were. This time we add more tests and then fix a race condition.
• 21 - Looking at Performance and finding a leak - Adding some new functionality and tests to measure performance before setting out to improve it unearths an unexpected memory leak.
• 22 - Performance: Some you win… - Making some performance related changes that seem to indicate that more substantial changes are necessary; perhaps there’s another way.
• 23 - Another new approach: timer wheels - Where I trade memory for speed and switch to using a timer wheel algorithm.
• 24 - Removing test duplication - A quick time-out to clean up the mess of duplicate tests and shift into full on TDD mode by using all of our existing tests on our new implementation.
• 25 - Nothing is free - Implementing more of the timer wheel shows that our timer dispatch is more costly than it might be if we were running on embedded hardware; but still not too bad.
• 26 - More functionality, more refactoring and a new bug - Implementing one shot timers in the timer wheel exposes a bug in how we set all timer wheel timers.
• 27 - Fixing things… - This time around we fix the bug we exposed last time and locate and fix a couple of memory leaks…
• 28 - Finishing the timer wheel - Where I finish the timer wheel implementation and compare the performance to the timer queues.
• 29 - Fixing the timer wheel - Some bugs emerge during integration testing.
• 30 - Reducing contention - Where the tests support the addition of custom STL allocators and a private heap to reduce contention and improve performance of the timer queue.
• 31 - A bug in DestroyTimer - Where I fix a new bug, test first.
• 32 - Intrusive containers - Where I replace the STL set used in the timer wheel with a custom intrusive set.
• 33 - Intrusive multi-map - Where I replace the STL map used in the timer queue with a custom intrusive container.
• 34 - Potential reentrant locking deadlock - Where the change to using non-reentrant locks in 32 needs to be rolled back, but not before we write a test to prove that it’s broken.
• 35 - Heap Corruption - Where I expose a heap corruption bug by writing a test and then fix the bug.
• 36 - Timeout handle wrap - Where I expose a potential bug, fix it and then refactor the problem code away.
• 37 - Bringing the code up to date - Where I update the code to the latest version used in The Server Framework.
• 38 - Bringing the code up to date, again… - Where I update the code to the latest version used in The Server Framework and support VS2019 and VS2017.
• 39 - 19 years’ unit testing the same code - A precis of where we are and how we got here and an minimal update to support the latest VS2017, 2019 and 2022 compilers.
• 40 - Code updates and new functionality - Code updates and new functionality.
• 41 - GoogleTest - Moving to GoogleTest.

TickShifter v0.1 - A tool that allows you to control the value returned by GetTickCount() and some sample programs built with the first and last versions of the timer queue code that demonstrate the code failing due to the original rollover bug and working once it is fixed.