I’ve just spent a day tracking down a bug in a pending release of The Server Framework. It was an interesting, and actually quite enjoyable, journey but one that I shouldn’t have had to make. The bug was due to a Windows API call being inserted between the previous API call and the call to GetLastError() to retrieve the error code on failure. The new API call overwrote the previous error value and this confused the error handling code for the previous API call.
Previously on “Practical Testing”… I’m in the process of replacing STL containers with custom intrusive containers in the timer system that I have been developing in this series of articles. The idea is that the intrusive containers do not require memory operations for insertion or deletion as the book-keeping data required to store the data in the container has been added to the data explicitly. This reduces the potential contention between threads in the application and, hopefully, improves overall performance.
Back in 2004, I wrote a series of articles called “Practical Testing” where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests.
I have a tendency to write unit tests that are a little more invasive than they need to be; these tests make sure that not only are the results as expected but also that as many of the side-effects and interactions with other objects are as expected as well. So, for example, in my current WebSockets development for The Server Framework I have some tests which test that the correct data is delivered to the client of the API that I’m developing and also test that the API interacts with its buffer allocator correctly and doesn’t leak memory.
I’m nearing the end of my WebSockets implementation for The Server Framework and have been dealing with various protocol compliance issues. Whilst I have decent unit test coverage I haven’t, yet, sat down and produced compliance specific unit tests which walk through the various parts of the (ever changing) draft RFC and test each aspect. Looking back, I probably should have taken this approach even though the RFC was fluid. Anyway.
Dhanji over at Rethrick Construction has written an interesting piece on the value of unit testing.
I agree with his conclusion; “So the next time someone comes to you saying let’s write the tests first, or that we should aim for 80% code coverage, take it with a healthy dose of scepticism.” But then I tend to take everything with a dose of scepticism…
I also agree with the fact that sometimes the tests get in the way of refactorings that you’d like to do and sometimes the tests give you more code that needs to be maintained and that they often appear to slow down your development time.
Back in 2004, I wrote a series of articles called “Practical Testing” where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests.
My tangential testing that began with my problems with commands run via WinRs during some distributed load testing are slowly unravelling back to the start. I now have a better build and test system for the server examples that ship as part of The Server Framework. I have a test runner that runs the examples with memory limits to help spot memory leak bugs and a test runner that checks for lock inversions.
As I mentioned, I’ve been adjusting my build system and have finally got to the point where my lock inversion detector is suitable to run on all of my example servers during their test phase on the build machines. I’m working my way through the various example server’s test scripts and adjusting them so that they use the lock inversion detector, can be easily configured to run the full blown deadlock detector and also can run the servers under the memory profiling test runner that I put together earlier in the week.
My theorising about the strange memory related failures that I was experiencing with my distributed testing using WinRS have led me to putting together a test runner that can limit the amount of memory available to a process and terminate it if it exceeds the expected amount. With this in place during my server test runs I can spot the kind of memory leak that slipped through the cracks of my testing and made it into release 6.
