When your mocks are executable

The size of the “units” that I test with my unit tests varies quite considerably. Usually I tend to work at the single class level; the test will provide mocks for the services that the class requires and the tests will ensure that the class behaves correctly and interacts as expected with the its service providers. Sometimes though the units consist of a whole subsystem, still often a single class, but one which wires up and collects many other classes into a single component; again the service providers that it uses may be mocked but they’re often only right on the edge of the functionality that is being tested. These chunky unit tests ensure that the code that makes up the component integrates together and often allows the testing of large sections of an application in a controlled, unit test style.

This week I’ve been unit testing a class that pulls together around 90% of my multiple process debugger code. The tests are harder to write than tests for single class units but they’re extremely useful.

The debug tools that I’m working on at present can run as console mode applications or GUI apps. Eventually there will be a single tool that either runs as a console app or with a GUI, but right now there are two. In fact, at present, the GUI application is just a collection of prototype display code that no longer integrates with the debug engine… The debug engine is a class that gathers together all of the code required and presents a relatively simple interface to the client code, you can register a “monitor” interface with it and then get told of various events that the kind of debug tools that I’m writing might be interested in. The usage pattern is pretty much that you create the debug engine and pass it a “program settings” class that tells it how to configure itself and you plug in a monitor if you’re interested in any of the events and then you simply call LaunchSession() to start the target process and that’s it.

Generally the GUI program is interested in displaying debug events from the monitor and the console program isn’t. The GUI relies on the debug events to drive the display of windows for the debug session and for each of the processes being debugged and to display data that’s returned by the debugger from the target processes. The console program simply launches the session and then calls WaitForSessionToComplete() to sit and wait for everything to finish. The GUI allows the user to adjust the settings that affect the target processes as they run, the console requires that you set your requirements at the start and then just leave it to it.

The interface to the debug engine is simple but there’s a lot of stuff going on under the covers and that makes unit testing the class quite complex. The debug engine is pretty much the “top” of the pile of objects that are plugged together to form the body of the tool; this is where the dependency injection stops and, apart from the monitor interface, there is very little that can be mocked. In the past most of the “chunky” objects that I’ve unit tested have had a “top” and a “bottom” that could be mocked up; the trading system that I put together for a client last year had tests that mocked the data access layer at the bottom and the monitoring layer at the top and drove 90% of the system from fake data. This time things were slightly more complex, the mocking that was needed at the “bottom” was done in terms of creating several small executables that could be controlled and monitored by the unit test during the test and that the debug engine could use as target processes. This is something that, like testing multi-threaded code, at first can look impossible to test but gets easier once you start.

Since the tests that I’ve written recently are for code that is built on top of a stack of several libraries that each add a little more value and refine the focus a little more I’ve actually been writing these “mock exe” tests for a little while now. This means that although our “chunky” unit tests are testing a lot of functionality I have many finer grained unit tests that I can switch to when I get a failure in the “chunky”, high-level tests. For example, the high-level test might find that the debug engine fails to work with a CLR exe due to strangeness in how CreateProcess() deals with managed executables that you’d like to create in a suspended state; this may result in rework in a lower level library but that rework would be supported by that library’s tests and not the higher level library’s tests. Once a problem is found using the high-level test a test for it can be written at the appropriate level in the library/class hierarchy.

The simplest mock test executable probably looks something like this:

int WINAPI _tWinMain(HINSTANCE, HINSTANCE, PTSTR, int) 
{
   CManualResetEvent running(CLocalName(_T("MyLib::TestExeRunningEvent")));
   
   running.Set();
   
   CManualResetEvent shutdown(CLocalName(_T("MyLib::TestExeShutdownEvent")));
   
   shutdown.Wait();
   
   return 0;
}

When the debug engine launches this target exe the unit test can wait for the target to start up by waiting on events from the debug engine and waiting for the named manual reset event that the target process sets when it has started. The target exe will then sit there, waiting until the unit test tells it to shut down by signalling the named shutdown event. Once the unit test decides that it’s time to shut down it can inform the target process and wait for the debug events from the debug engine. The lower level multi-process debugger unit tests can make sure that the debugger is informed of all the expected things that happen whilst the process becomes ready to run and the higher level tests can simply check that the debug engine class is responding correctly to the multi-process debugger class’s events. By the time the high-level test has confirmed that the target has been launched and the “running” event has been set the lower level code has taken care of suspending the target in a “safe” state during start up, injecting code and configuring it once injected and then resuming the target. This high-level unit test is retesting functionality that has already been tested in the debug tools library where we would have tested the multi-process debugger’s ability to launch a target and control its startup and the libraries above that where we would have tested the code that can inject code into the target and communicate with it once injected. The high-level test is testing the integration and the fine tuning of the kind of events that it needs to report to the code that will be built above it.

A slightly more complex mock target exe might be one that launches a child process.

int WINAPI _tWinMain(HINSTANCE, HINSTANCE, PTSTR, int) 
{
   CManualResetEvent running(CLocalName(_T("MyLib::ParentTestExeRunningEvent")));
   
   running.Set();
   
   CManualResetEvent launch(CLocalName(_T("MyLib::ParentTestExeLaunchEvent")));
   
   launch.Wait();
   
   CStartupInfo startupInfo;
   
   CProcessInformation processInfo;
         
   const _tstring processName = GetModulePathName() + _T("\\TestExe.exe");
   
   const _tstring commandLine = _T("\"") + processName + _T("\"");
   
   if (0 == ::CreateProcess(
      processName.c_str(),
      const_cast<tchar*>(commandLine.c_str()),
      0,                   // process attributes
      0,                   // thread attributes
      FALSE,               // inherit handles
      CREATE_NEW_CONSOLE,  // creation flags 
      0,                   // environment
      0,
      &startupInfo,
      &processInfo))
   {
      return ::GetLastError();
   }
   else
   {
      CManualResetEvent launched(CLocalName(_T("MyLib::ParentTestExeLaunchedEvent")));
   
      launched.Set();
   }
   
   CManualResetEvent shutdown(CLocalName(_T("MyLib::ParentTestExeShutdownEvent")));
   
   shutdown.Wait();
   
   return 0;
}

This mock target process allows the debug engine to be tested to ensure that it deals correctly with multiple process debugging, etc.

More complicated unit tests might require more from the mock processes but since each unit test will be focusing on testing a single aspect of the debug engine we can usually continue to rely on the relatively simple communication afforded by named events.

Testing “chunky” units often requires that you think a little bit more about exactly how you can mock up the services that you require, it’s worth it though!