Bug in overlapped UDP port unreachable errors?

I’m currently fixing a bug in the UDP side of The Server Framework. The problem is that I’m failing to handle how Winsock reacts when an overlapped SendTo() results in an ICMP Port Unreachable response from the remote host. This error gets reported by failing a pending WSARecvFrom() on the same socket by returning 0 bytes from the WSARecvFrom() with an error code of ERROR_PORT_UNREACHABLE.

This would be fine if the framework didn’t assume that all errors from an overlapped WSARecvFrom should mean that we don’t post another read… You may recall from when I was doing some work on making the framework survive low resource situations that we can’t just issue another read when the previous one failed due to a low resource situation as we would just continue the problem, so we back off and recover later. The problem I have now is two-fold; a) we can post another read in this situation as this one failed due to non local resources related issues and b) the “later” when we recover from the lost read is too late for this particular client’s usage pattern.

Our UDP server operates as follows; the server listens on a port and issues a number of overlapped reads (WSARecvFrom()’s) on that port, this number is the “listen backlog”. As each read completes the server issues another new read and processes the data that has arrived. If there’s an error reported from the read then the server doesn’t issue another read now, but, instead, waits until a socket is closed (and released) and then compares the number of reads it currently has outstanding with the number that it should have outstanding and tops up the pending reads if necessary.

Our client’s ‘reliable UDP’ code holds on to sockets for each ‘connection’ until the connection is either closed cleanly or allowed to time out. If the remote end of the connection closes its socket (and doesn’t disconnect cleanly) then the server will attempt to send data (a protocol level ping, or any reliable data that the remote end hasn’t send an ACK for) until a connection timeout condition occurs. For each ping that the server sends to a client that has closed its socket the server will get an ERROR_PORT_UNREACHABLE error returned to one of its pending reads and it wont issue a new read. Each of these failures reduces the number of pending reads that the server has. If the server has enough clients in ‘dirty disconnect’ mode then it will be continually reducing its read queue and if sockets are not being closed quickly enough then it will end up in a situation where there are no outstanding reads and incoming datagrams from connections that are still active will be lost (and the remote end of the reliable UDP connection will eventually resend this lost data).

So, the bug fix to the framwork is to deal with ERROR_PORT_UNREACHABLE errors as non local resource related read errors and immediately post a new read to replace the one that has just failed.

But there’s more…

It seemed to me that since Winsock was bothering to report the remote ‘disconnect’ to me I might as well make use of it. In the client’s reliable UDP system we can waste a lot of energy trying to resend data to clients that have disconnected badly. This is especially annoying since we’re getting an error from Winsock after each send to the client that is no longer there and we’re ignoring it. We could be using the ERROR_PORT_UNREACHABLE error to enable us to shut down the server’s end of the connection as soon as the client closes their socket rather than waiting for the server end of the connection to time out… Well, we could if Winsock was actually giving us all the information that we need…

When you issue an overlapped WSARecvFrom() you provide a space for Winsock to store the remote address that has sent a datagram to you. You can then use this remote address to send data back. Unfortunately when the WSARecvFrom() fails with ERROR_PORT_UNREACHABLE (on Vista x64 at least) then Winsock only fills in the address portion of the remote address, the port is set to the listening local port and not the sending remote port… This makes the resulting address all but useless to me and is a pity as the ICMP port unreachable response that the networking layer is receiving and which is responsible for causing the WSARecvFrom() to fail has full details of the remote address and port that caused the failure and so Winsock has all the information that it needs to be able to report this useful information to me…