In 2006 or 2007, a software upgrade of the disk controllers on the principal systems went wrong. For a week, they were out of operation. It was one of the biggest crises in the history of the government office, and perhaps the biggest of all. At the time, Kees and I were working on the systems renewal project at another location. The other database administrators dealt with the issue, as did many others. I knew there was a serious problem as we received regular updates by email, but I didn’t realise how serious it was. After a week, the telephone rang at home. It was 9 PM. My wife, Ingrid, took up the phone. It was the IT director. He said there was an emergency and asked me to come to the office. His voice reflected fear. ‘As if the Titanic had hit the iceberg,’ Ingrid later noted.

I hurried to the office and arrived by 9:30 PM. Many people were still in. It was a massive crisis. There was an atmosphere of fear. The database administrator on duty, Dirk-Jan, brought me up to speed. I searched the database log files, found the error messages, and typed them into the Google search bar. In this way, I found a document on the Internet with the remedy. I then repaired the failures and brought the systems online one by one. Board members and senior managers were standing around me, watching me type. Solving the issue wasn’t complicated, but few people used Google to find the answer at the time.

I learned that the last backup was over a week old, and the mirror copy was offline. You may know what backups are and why you might need them, but you may not know what a mirror copy is. A mirror copy is a safety measure. If you own a computer or a mobile phone, it contains data. That data is on a device. In the early 2000s, it was usually a hard disk. If that disk fails, your data may be gone forever. If you lose some photographs of your late cat, you might feel sad about it, but after a few years, you get over it, perhaps after consulting your psychiatrist and taking a lot of pills.

Corporations can’t afford to lose their data. That would bankrupt them. Their business is their data. Without it, they are out of business. If you have a backup, only the data from after the latest backup may be lost, but that can still kill you, most notably if you haven’t made a backup for a week. We were a government agency, so it wouldn’t have bankrupted us, but it would have been a national political scandal.

Corporate computers have multiple data storage groups in different locations. If one group catches fire or stops operating because of a failed software upgrade, the other groups still have the data. These groups are called mirror copies. We had two groups: the original and the mirror. You can imagine my bewilderment. We had no backup, and the copy wasn’t available. So much had gone wrong that it was a miracle that I succeeded in recovering all the data. But having no mirror and no backup meant we were still on the brink.

An even greater surprise was yet to come. The managers and the board wanted to return to business as usual and run the backlog of batch jobs. Then I said, ‘This is perhaps the most important advice I will ever give in my entire career. Don’t start the batch jobs yet. We are on the proverbial edge of the precipice. Running the jobs might just push us over. Everything went wrong for a week and there is no guarantee whatsoever that it will be all right now. We should bring the mirror copy back online and make a backup first.’

They planned to ignore my advice. Bringing the mirror copy back online and taking a backup would take eighteen hours of precious time. It was a lot of data to back up, as it was everything we had. I was a low-ranking official while the IT director had claimed there was nothing to worry about. But he had left the building. I kept stressing that making a backup was the right thing to do. ‘If something goes wrong that could finish us,’ I told them. It was the worst crisis ever. And so, I pressed for an extensive check-up to see if everything was in order. On that, they could agree.

During the check-up, I found another failure that everyone had overlooked. That scared the managers and the board, prompting them to start another meeting. And then they followed my advice. The IT director was no longer there, and they faced a determined saviour who told them in no uncertain terms that they were about to do something stupid. The operators brought the mirror copy online and made a backup before we resumed normal operations. In this way, rational decision-making prevailed. Nothing went wrong anymore, but no one could have known that beforehand.

If it had gone wrong, the agency probably would have survived. Operations would likely have had to stop for several weeks—that had already happened for a week—and it may have been impossible to recover all the data. That would have made the headlines. But it never came to that. When the journalists of the local newspaper smelled a rat, the board could tell these journalists that the situation was under control and that the data was safe. My wife’s comparison of this situation with the Titanic having hit the iceberg was not entirely apt. Saving the Titanic once it had hit the iceberg was technically impossible. It would have required a miracle. What I did may have appeared to be a miracle, but it was technically possible.

The audit department later evaluated the crisis. The auditors noted that after a week of failures, all the problems suddenly vanished, which they found already hard to believe. What they found even more difficult to fathom, and they stressed the inconceivability of it during a meeting, was that after a week of irrational decision-making, sanity suddenly took hold as we had brought the mirror copy back online and made a backup. They couldn’t figure out why that happened. Our management had kept them entirely in the dark. I didn’t enlighten them either, as it would make our management and board appear incompetent.

My manager, Geert, complimented me for handling the situation. He stressed that my colleagues had been content with me. ‘I was a pleasant colleague,’ he added. Strangely enough, Geert didn’t say something like, ‘Your contribution was critical in saving us from a disaster.’ It reveals something about Geert’s thinking. To him, it was teamwork. Geert wasn’t present that evening, so he may not have learned the details of what transpired. And so, it didn’t help my career. A few years later, the other senior database administrators received a higher salary grade, but I did not. Geert was involved in that decision.

When the office was on the brink, I knew what to do and was determined not to let the ship sink. Our management and the board were clueless and had lost it. Fear gripped them, making them listen to reason. Now it seems that my dealing with the crisis could be a harbinger of things to come.

Latest revision: 4 August 2025

Featured image: Der Untergang der Titanic. Willy Stöwer (1912). Wikimedia Commons. Public Domain.