Debugging network protocols with journaling

One of my long-term clients has hundreds of cloud machines running instances of their server, each server maintains thousands of reliable UDP connections using a custom protocol that we’ve developed over the years. When things go wrong it’s often hard to work out why. Even though we have reasonable unit test coverage of the code that runs the UDP protocol, it’s hard to build tests that cover every possible scenario.

Testing, discipline and detail

The manual process around updating broken links is due to be replaced by a simple link checker that I’ve been writing in Rust. It’s not quite ready yet but it’s nearly there… I was updating a few broken links today and came across this from 2004; “Software development is about discipline and detail; code quality starts to decay as soon as developers forget this. All code decays, but tests can help to make this decay obvious earlier.

Wayback

This blog has been around a long time and the internet tends to rot. This means that quite a lot of the links on old posts are broken. I’m slowly fixing these broken links to use “The Wayback Machine” but it’s complicated to automate as the resulting URLs need to include a timestamp of a valid snapshot and can’t just include a ‘rough idea of the date’. So I’m fixing the broken links manually by watching the posts that are accessed the most and manually checking the links and fixing them up.

20 years of blogging...

On the 3rd of May 2003 I posted the first entry on this blog. I then proceeded to “back fill” the blog with various things that had either been posted before in other places or had been laying around waiting for me to have somewhere to put them. This is why although the blog began in 2003 the archives go back to 1992. What I said on the 10th anniversary of this blog is still apt:

Adventures with \Device\Afd - test driven understanding

I’ve been investigating the ‘sparsely documented’ \Device\Afd interface that lies below the Winsock2 layer. Today I use a test driven method for understanding and documenting the API. TDU - Test Driven Understanding When trying to understand a new API I always like to end up with executable documentation in the form of tests that show the behaviour of the API. I write these tests in the same way that I write any tests; writing a test that fails and then adjusting so that it passes.

Quick and dirty analysis of memory allocations in Visual Studio code

Yesterday I was bemoaning encapsulation and how it was hiding what was going on inside my objects (and quite right too, what good would it be otherwise?). The issue is that the object I was interested in, and each of the objects that formed it, were allocating more memory that expected. It wasn’t so much that the object was bigger than expected, just that there were more allocations than I expected and that for some reason destroying lots of these objects is taking longer than I would expect.

The cost of encapsulation

I’m debugging performance issues with a C++ server that has been stalling and then failing to recover. I’ve reached a point where we can generate the problem using a network interruption that causes multiple connections to disconnect at the same time. The fixed sized thread pool that services these connections becomes overloaded with work that requires it to clean up connection objects for the disconnected connections and all of the threads in the pool spend far too long fighting over the lock to the heap as they try and return memory to it to clean up the connection objects.

Building OpenSSL 3.x for x86 and x64 on Windows for side by side deployment

Back in August 2012 I shared my scripts for building OpenSSL on Windows. These have changed a little since the ones I had for the 1.0.x and 0.9.x releases of OpenSSL. The main idea is the same, the scripts build the OpenSSL code as both static libs and DLLs for both x86 and x64 and allow you to have all of the files next to each other in the same directory by adding various ‘warts’ to the file names.

Practical Testing: 41 - GoogleTest

I’ve been writing a series of blog posts, called “Practical Testing”, about testing real-world, multi-threaded code. Up until now I’ve used my own, home grown, unit testing framework. When I started out with this series back in 2004 there wasn’t much in the way of mature testing frameworks for C++ and, over the years I haven’t really found a need to switch from my own stuff that I understand and that works pretty well.

Practical Testing: 40 - Code updates and new functionality

Nineteen years ago I began a series of blog posts, called “Practical Testing”, about testing real-world, multi-threaded code. As with most code that works well, and is used by lots of people, we’re still changing it and improving it and using it. I’ve just done a precis of how we got here and now it’s time to continue the journey. As I hinted at the end of the last episode, there were some outstanding issues to deal with and some new functionality to add and test.