Grapevine snail. Jürgen Schoner.

Learning Opportunities

In 2005, the government agency I worked for embarked upon an ambitious systems renewal project. The IT director hoped to make a mark by replacing our existing IT systems with something new. As the story goes, meaning I was not a first-hand witness, but several colleagues said more or less the same, so that it could be the truth, the IT director visited Oracle headquarters near San Francisco. He could get a steep discount on the Oracle ERP solution. ERP can administrate an entire business as a total-for-everything solution, but only if you model your business the way ERP prescribes. In other words, every department had to change how it conducts its affairs. That would upset our entire organisation. That makes ERP awkward.

Instead of admitting he had bought the ERP on a whim and writing off the license fee, which would have been a minor loss, the purchase had to be justified. Possibly, the IT director wanted to save face. He then hired business consultants to assess the quality of our IT systems and propose a solution. Not surprisingly, their report stated that our systems were obsolete and beyond repair, even though they were doing fine even fifteen years later, and that we should switch to the Oracle ERP as that would save a lot of money. There is no such thing as coincidence here.

There was a strategic information planning phase before the project started. Cap Gemini business consultants organised brown paper sessions where employees could help identify the business requirements. We had to list our components. And the consultants would call them services. I wrote the word database on brown paper. I stuck it to the wall, and the Cap Gemini business consultant called it a database service. That did not seem like a proper strategic business analysis to me. Anyone could paste the word service behind the name of a component. It was the moment when it dawned upon me that it might not end well. There were political troubles early on. A Cap Gemini project leader left because she didn’t share the vision of the IT director. It appeared he surrounded himself with yes-men.

As a student, I learned in the Information Planning course that you must first define the business requirements and then select a solution based on these requirements. It can save you a lot of trouble. The business consultants did the opposite as they tried to justify a choice already made. Over time, it grew into an all-encompassing megalomanic plan. Most departments didn’t want the IT department to determine how they operate. And so, they planned to supplement the Oracle ERP with several special-purpose modules. Making special-purpose software would make ERP pointless, as not having to make special-purpose software was the whole point of using ERP.

Once the project had started, Oracle began to determine its direction. Oracle consultants soon found that we needed more solutions from Oracle and, of course, the latest technology, sometimes not yet proven to work. Our decision-makers agreed. So when the special-purpose software failed, they tried new ideas like using a business process model language on top of ERP. In this way, the business model could be tailor-made while using ERP. No one had done that before for good reasons. However, the consultants were there to make money for their employers by making the customers happy, so they tried the idea. The pointlessness demoralised me. Goals were constantly changing as ideas were failing. Employees became stretched to their limits for years in a row. I was an Oracle database administrator and was especially hard hit as our job role expanded to everything related to Oracle. The stress began to take its toll, and I began to experience repetitive strain injury.

One of the database administrators, Kees, was a tech genius who enthusiastically jumped on every crazy plan and did a lot of overtime. In this way, he became the manager’s darling. I lagged behind him, but the other database administrators couldn’t keep up. The situation soon went out of control. It was like having a trojan horse inside our department. Kees didn’t look after the department’s interests but the project’s. The things he invented and built had to be kept up and running by us. He didn’t make a lot of notes but was always willing to help us if we didn’t understand It.

Our managers had no clue what they were doing and believed that more advanced technology like Oracle RAC clusters would fix things, but that only contributed to our troubles. RAC was pointless, required additional knowledge, and was prone to failure. I would then sarcastically note, ‘The most stable RAC cluster has one machine.’ That was the same as not having RAC at all. Pesky problems piled up on my desk, and I constantly had to learn new skills while project leaders pressured me to work on plans that were bound to fail. That couldn’t go on. I want to help people, but I shouldn’t fix other people’s problems or work on ideas that will never succeed. There was always too much work, while most of it was pointless, and there was only so much I could do.

I made a few radical changes and worked only on the issues that mattered most while trying to avoid working on things that were bound to fail. Whenever multiple project leaders pressured me, I referred them to my manager, saying, ‘He should set priorities.’ Whenever a colleague tried to bequeath his pesky problem to me, I explained how he could fix it himself. ‘It begins with using Google,’ I said time after time. In 2006, not many colleagues used search engines. Google helped me to solve more issues at a faster pace. Often, someone else has had the same problem and posted the solution on the Internet. It was also a learning opportunity. If you solve problems, others will relay their problems to you, and you end up drowning in problems. When people can find someone else to fix their problems, they won’t do it themselves. And perhaps it is better to let them fail if they can’t.

As things were spiralling out of control, our management decided we needed more qualified database administrators. Our salaries were too low to attract the right people, so they gave the new hires and Kees a higher salary grade. They also increased the number of temporary positions. The headcount went from four to fourteen database administrators. And still, we couldn’t handle the workload.

The year 2008 neared its end, and then it appeared that A* was God and that She had a plan with me. Long-lasting stress can cause psychosis. My perspective on what mattered changed profoundly. The job at the office became a sideshow. I had to step up on rational thought to deal with the craziness of it all. I cut my working hours and began to work on a plan for the future of humanity. I nevertheless succeeded in giving a good impression. My manager at the time, who had been a database administrator previously, held me in high regard because I tried to manage the workload by focusing on what mattered. After all, even fourteen database administrators couldn’t handle the work, so something was seriously wrong. It was like being the only rational person in an insane environment. I didn’t know how to end the madness. I only tried to deal with it within the limits of my possibilities.

As we needed so many database administrators, temporary hires came and went. One of them had roots in the former Dutch colony, Suriname. He once made a surprisingly frank statement. He worked for his uncle, he told me, who also came from Suriname. But apart from him, his uncle only hired Dutch people. ‘That is because the Dutch are more reliable,’ he added, and, ‘They do the job as agreed. You don’t have to check on them.’ I supposed it was true. Otherwise, he wouldn’t say that. People from Suriname don’t take things as seriously as the Dutch.

After several years and spending over 100 million euros, our corporate delusion finally went into operation. It did nothing but accept incoming messages and acknowledge their reception. An experienced programmer could build that in a day. It was a bloated set of software and machines with the latest technology but capable of nothing. The IT director organised a party in which he proudly announced that we finally had a system under architecture. That made me think our architects were incompetent, and it would take years to erase that impression.

Again, as the story goes, our board then made a politically brilliant move by selling the project as a success to the government in The Hague, arguing that after spending more than 100 million euros, they needed a few million more to fix the remaining bugs. The board used that money to hire a software company to rebuild everything in Java. The ERP only did the financial administration. In this way, it appeared that the new software was just an add-on. The budget was tight. There was no money for testing and fixing bugs. As you can see, image is everything. That is why corporations invest in brands and marketing.

The new Java systems crashed nearly every day. They consumed a lot of resources and produced a lot of data, for instance, millions of messages no one dared to throw away. You could use them to find out what went wrong. And so much went wrong. Working with these systems was dreadful, not only for IT workers but also for the end users. I had grown cynical about everything management was doing. Only one project succeeded, building an old-fashioned Designer/2000 system. Had we made the other systems using Designer/2000 or its successor tool APEX, from the start, it might have cost less than 5% of what we had spent, and it would have worked. Architecture came with a price tag and had nothing to show for it. If you see an ugly building, an architect likely designed it and made it far more expensive than it needed to be. The core problem, however, was complexity, or trying to implement one solution for everything in one go.

I estimated that the new Java systems used 100 times as much memory and disk space as the Designer/2000 systems for doing the same job. That was an understatement because making a fair comparison was difficult. The outcome of the calculation was 3,000 to 16,000 times as much. The new systems started with low numbers of transactions and would scale up, and there is a margin of error. Excessive resource consumption didn’t seem like a problem at first. The price of memory and disks went down over time. Cheap resources make people wasteful, and you can see that everywhere around you. But we were too far ahead of our time in terms of wasting resources, that is.

The data storage became overburdened. The IT director fired the manager responsible for the data storage, citing that corporations like Google and Facebook could scale up fast. But we weren’t Google, but a government agency with a few hundred IT employees. I was on leave when it happened. The following Monday, I learned about it. And I thought, ‘The IT director is responsible for the mess. He is the one who should be fired.’ A few hours later, a soccer club, VVV from Venlo, fired its trainer, who had the same first and last name as our IT director,1 a remarkable coincidence. After five years of maintenance, costing millions more, the Java systems gradually began to work properly.

Featured image: Grapevine snail. Jürgen Schoner (2005). Wikimedia Commons. Public Domain.

1. VVV-Venlo ontslaat trainer Van Dijk. Nu.nl (20-12-2010). [link]

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.