Just yesterday, I had a discussion with the CTO of a current client. He’s a 20 year industry veteran and he was wondering why Microsoft hadn’t stopped in their tracks and underwent some massive code reviews to get rid of the problem for once and for all.ChristophDotNet
Because it’s a very hard problem? How many lines of old code have you and the CTO personally reviewed and decided were completely bug free?
So you have 50million lines of code. Some of it is from waaaay back. You probably don’t have lots of tests for all the code. Due to its age it’s probably “evolved” and isn’t the neatest, nicest code in the world. It’s an OS, some of it’s probably fairly complex stuff ;) You need to review it and find security flaws, or any bugs that in a given circumstance can cause security flaws. Once you find them you need to remain backwards compatible and fix the flaws without introducing any new flaws. Remember that you probably don’t have an automated test suite with 100% coverage. Do you still feel confident that you can do the job? Can you give me a date when it will be done? An idea of how much it will cost?
And how are we measuring the success of this endevor? We’re not looking at the total number of security bugs that you fixed during the blitz, we’re looking for no bugs to remain. Still confident? If even one bug remains and even if you find and fix it the day after the blitz officially ends then the whole thing will be classed as a failure.
“Look, there’s a bug they didn’t find. Bad Microsoft. Why oh why didn’t they work harder. How can I convince my clients that their software is any good at all”.
Surely we should be looking at the shape of the curve (bugs over time) and comparing it to, oh, they don’t have a curve for the OS that they didn’t spend ages fixing… What’s more, if you throw loads of very bright people at a problem like this I’d personally expect that they’d continue to have “AHA” moments long after they’ve been told to stop. They’ll keep fixing things even if they’re not supposed to…
Perhaps they should own up and publish all of the faults that they found during the blitz. I’m sure everyone would take that really well ;) I’m not much of a marketing bod but even I know that it would be hard to give that idea wings.
So, even though Microsoft could and, by all accounts, did, pull all of its best people onto the task it’s still a decidedly non trivial task and there are no meaningful measures to say if it was a success or not… I think we’re lucky they bothered to try at all.