Testing

I’m nearing the end of my WebSockets implementation for The Server Framework and have been dealing with various protocol compliance issues. Whilst I have decent unit test coverage I haven’t, yet, sat down and produced compliance specific unit tests which walk through the various parts of the (ever changing) draft RFC and test each aspect. Looking back, I probably should have taken this approach even though the RFC was fluid. Anyway.

Dhanji over at Rethrick Construction has written an interesting piece on the value of unit testing. I agree with his conclusion; “So the next time someone comes to you saying let’s write the tests first, or that we should aim for 80% code coverage, take it with a healthy dose of skepticism.” But then I tend to take everything with a dose of skepticism… I also agree with the fact that sometimes the tests get in the way of refactorings that you’d like to do and sometimes the tests give you more code that needs to be maintained and that they often appear to slow down your development time.

Back in 2004, I wrote a series of articles called “Practical Testing” where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests.

My tangential testing that began with my problems with commands run via WinRs during some distributed load testing are slowly unravelling back to the start. I now have a better build and test system for the server examples that ship as part of The Server Framework. I have a test runner that runs the examples with memory limits to help spot memory leak bugs and a test runner that checks for lock inversions.

As I mentioned, I’ve been adjusting my build system and have finally got to the point where my lock inversion detector is suitable to run on all of my example servers during their test phase on the build machines. I’m working my way through the various example server’s test scripts and adjusting them so that they use the lock inversion detector, can be easily configured to run the full blown deadlock detector and also can run the servers under the memory profiling test runner that I put together earlier in the week.

My theorising about the strange memory related failures that I was experiencing with my distributed testing using WinRS have led me to putting together a test runner that can limit the amount of memory available to a process and terminate it if it exceeds the expected amount. With this in place during my server test runs I can spot the kind of memory leak that slipped through the cracks of my testing and made it into release 6.

Previously on “Practical Testing”… I’ve been looking at the performance of the timer system that I developed and have built a more specialised and higher performance timer system which is more suitable for some high performance reliable UDP work that I’m doing. Whilst developing the new timer wheel I began to consider the thread contention issues that the timer system faced and came up with a notation for talking about contention (see here).

Previously on “Practical Testing”… I’m writing a timer wheel which matches the interface used by my timer queue. This new implementation is designed for a particular usage scenario with the intention of trading space for speed and improving performance of some reliable UDP code. The last entry completed the development of the timer wheel. This time we fix a couple of the bugs that I’ve discovered since I started to integrate the code with the system that it was developed for.

Previously on “Practical Testing”… I’m writing a timer wheel which matches the interface used by my timer queue. This new implementation is designed for a particular usage scenario with the intention of trading space for speed and improving performance of some reliable UDP code. Over the last four entries I’ve implemented various parts of the timer wheel and adjusted the test code so that I could reuse the tests that I already had for the other implementations with my timer wheel.

Previously on “Practical Testing”… To deal with some specific usage scenarios of a piece of general purpose code I’m in the process of implementing a timer wheel that matches the interface to the timer queue that I previously developed in this series of articles. Last time I left myself with a failing test. The problem is that setting a new timer on the timer wheel sets a timer that’s relative to the time that timer wheel thinks is ’now’ and the timer wheel’s view of the current time could be slightly behind reality; see the previous entry for a diagram that explains the problem.