OddThinking

Many years ago, I was testing and debugging some software for a set of machines on a rack. The system was designed to have high availability – the four nodes were in constant communication across eight cables, mirroring their databases and transparently recovering if there was a failure in the system. During my testing and debugging, each component had low reliability – some part of the system crashed or otherwise required rebooting frequently – the MTBF for each component was about 2 hours.

I was pleasantly surprised one day to notice that the system had been up for a whole week! No single node had lasted more than a half-day, but the system had been robust in the face of that. Woohoo!

I was reminded of this experience this week – without the pleasant surprise – as my PC repeatedly froze up at random intervals during extended, intensive data recovery operations. It froze up about 15 times – once it survived 8 hours before locking up, once it lasted 33 seconds.

The difference between this experience and the last one was that there was no synchronisation with other machines and no recovery points. Every time it froze, I lost everything since my last (manual) save. As a result, I couldn’t just walk away and ignore the computer while the data recovery software running – I had to come back every half-hour and interrupt it to save the progress that had been made.

Sigh.