Jokers on Files.

Joking jokers

In 2002, I started to work as an Oracle database administrator at a government agency near home. Most people in the Netherlands know about the agency because it processes traffic fines. For that reason, it isn’t popular with the general public, just like the Internal Revenue Service. So if someone asked who my employer was, I kept it vague and said the government or the Department of Justice. It didn’t take long before something went seriously wrong. On my second day on the job, one of the production systems crashed after running the batch jobs, leaving a corrupt database, and with the benefit of hindsight, that was a bit peculiar. After two days of searching, I still hadn’t found the exact cause. When I restored the backup of the previous evening, which was still valid, and ran the batch jobs, the database became corrupt again. It probably was a software bug, so I advised restoring the backup of the previous evening and upgrading the database software to the latest version and seeing if it would solve the issue. Instead, the IT director declared a crisis and set up a multi-disciplinary task force to deal with the situation.

The head of the task force was a corpulent project leader who decided we should find the cause, which I hadn’t uncovered. I just wanted to fix the problem. Every day at 10 AM, there was a meeting to discuss the state of affairs. Every day I proposed to upgrade the database software to see if it would help. And every day, my proposal was brushed aside. I would have done it myself, but I was new on the job, and they used VAX VMS, an operating system I wasn’t familiar with, so I couldn’t install software or restore backups on my own. Two weeks later, after our experts had all weighed in and also after hiring a database corruption expert from Oracle, the cause remained elusive, and managers were getting desperate. Finally, they were willing to consider my suggestion. And it solved the problem. It was a harbinger of things yet to come. During a review, they grilled me for not being interested in researching the cause. I said that solving a crisis was more important as it was a production system, and the users needed it to work. And by the way, the upgrade demonstrated that it was a software bug.

A few months later, my employer hired a security officer. Probably the audit department had advised it. He was a guy in a suit who soon began to make our work harder by implementing unnecessary procedures. For instance, we had to lock up our Oracle manuals in a secure location after work and bring the keys to the porter’s lodge. But our manuals were public information like Windows manuals. Today, you can find this information on the Internet. At the same time, Mulder, the system that processed the traffic fines, had a superuser named MULDER with the password MULDER. Everyone knew that and could mess with the traffic fines. I notified the security officer, but being a true bureaucrat, he had more important things to do, such as attending meetings, inventing procedures and making management reports. Other systems had this issue too. And so, I contacted a few senior programmers, and we fixed that problem.

There were other issues with access rights too. As they would say in the course Professional Skills, ‘There was room for improvement.’ If a new employee came in, the service desk made a ticket stating, ‘Create user account X as a copy of account Y,’ and sent it from one department to another. Usually, it took two weeks for the ticket to pass through all our departments, and system administrators made errors. Hence, account X was rarely exactly like account Y. If people switched departments or left, the defunct access rights usually weren’t deleted. Perhaps the audit department had figured this out, as our management soon initiated a project role-based access rights (RBAC).

RBAC works like so. You have a role in a department. In ordinary language, it is your job. For your job, you need access to an array of systems. Your job description determines which rights you need, for instance, reading specific data or changing it. As a rule, employees should not receive more access rights than required to perform their tasks. RBAC is about the rights an employee in a specific job role needs. Business consultants came in and defined job roles and access requirements. A programmer then built an administrative database. But the database wasn’t connected to our systems, so there was no guarantee that the access rights in our systems matched the administration. And if you know how things fare in practice, you know that the administration would soon become stale and pointless. People are lazy, make errors, and forget things. And that would change once the administration and our systems connected. If the administration connected but was wrong, people couldn’t do their jobs properly, so the administration had to be constantly updated.

In 2004, I secretly began building an account administration system named DBB using Designer/2000, leaving the bureaucrats out of the loop because they would probably stand in the way and make it harder for me. Only my manager and a few colleagues knew about it. DBB automated granting and revoking access rights in our systems the RBAC way. It took me nine months as I also had to do my regular work as a database administrator. But when I was ready to implement DBB on the production databases, the bureaucrats became aware of what was happening and tried to block it. In early 2005, I introduced it sneakily with the help of the people from the service desk who wanted to use it. They installed the DBB client programmes on their personal computers. And I was a database administrator, so I could install anything I wanted on any database.

The outcomes were spectacular. The service desk now created the accounts, so the tickets didn’t have to pass through so many departments. We created accounts in one day instead of two weeks. And the service desk could reset passwords on the spot instead of relaying the request to a department, bringing down the time to reset passwords from hours to seconds. And the access rights accurately reflected job roles. So, once DBB was operational, the opposition crumbled, and DBB became a regular application, even though not an official one, and we had RBAC forcefully implemented.

The logo of DBB was a drawing made by Ingrid. She had drawn it for another purpose. It features jokers grinning at a set of file folders. To me, these folders symbolised bureaucracy. DBB joked with the bureaucrats as the bureaucrats considered it a rogue system. Supposedly, I was one of those jokers, so I made one of them my avatar on the web. DBB was my love child, just like Fokker once was Jürgen Schrempp’s. And so, I ensured DBB could survive if I ever left the agency. I produced design documents and manuals and built DBB according to accepted Designer/2000 practices. We had a lot of Designer/2000 programmers, so they could easily have maintained DBB. But I hadn’t followed the proper procedures when building and implementing it, so it never became official. So, if something went wrong, it was not a mere incident, as would be the case with any other system, but a cause to replace DBB. And something went wrong once.

For over ten years, bureaucrats devised plans to replace DBB. Our management started two projects to replace it. The first effort stalled because they had underestimated the complexity of the matter. They might have thought, ‘If one guy can do it, how difficult can it be?’ In 2016, a new project team realised it was pointless to replace DBB as it was doing fine and replacing it was costly. The newer Java systems ran on Postgres databases and used web access, so they didn’t use DBB. And our management planned to decommission the old Designer/2000 systems so DBB could retire by then.

And so, I wondered how bureaucrats think and concluded that it is like so, ‘If I mess things up but stick to the rules and follow procedure, no one can blame me. If do the right thing but do not follow procedure, and something goes wrong, my job is on the balance.’ If something has gone wrong, the government hires consultants to investigate the issue and propose changes to the procedures to prevent it from happening the next time. Sadly, the next time, the situation may be different, and then it goes wrong again. You might think it is better to do away with procedures, but in a government administration, that might not be a good idea. The role of government is to provide and implement rules. Just imagine that every government employee does as he sees fit. Nevertheless, there could be room for improvement.

DBB not only joked with the bureaucrats. The joke was also on me and in a most peculiar fashion. In June 2010, I received a highly unusual request from a system administrator to drop a user account manually. That hadn’t happened for several years. DBB usually took care of that, but for some unknown reason, DBB failed to drop this particular account. The username was ELVELVEN. If you read that aloud, you say eleven elevens in Dutch, a reference to the 11:11 time-prompt phenomenon. Usernames consisted of the first one or two characters of the employee’s first name followed by the employee’s last name. In this case, the user’s last name was Velven. To me, 11:11 signals a combination of two related unlikely events. And indeed, the joke had a part two, and it was even more peculiar.

In 2014, I tested an improvement to DBB. My test signalled that an illegal account had sneaked into our systems. The username was AD******, the first character of the first name followed by the last name of A******* [the lady who might be God and appears to stalk me with coincidences]. Had she been employed with us, this would have been her username. And her name isn’t common, so this was unnerving, even more so because it was the only username that popped up. It couldn’t be her, or could it? It turned out that a guy with the same last name as hers had worked for us. His first name began with an A too. And the account wasn’t illegal. I had mixed data from two different dates in the test, which made it appear that this account had sneaked in illegally. Just imagine the odds of only this account popping up.

In 2005, my manager promised me a promotion. He told me that I had managed to introduce DBB. ‘You had a vision and you made it happen and you overcame all the opposition, and now we have RBAC,’ he said. He added that I was the best database administrator of the lot. I doubted that and said we had a tech genius in our department who was better than me. And then he said, ‘Having the right vision and making it happen are far more important.’ Only, he didn’t formalise the promotion, so I tried to make him put his promise into writing. I asked him several times to do that. And then, he took on a new job somewhere else, so I feared I would end up empty-handed. After all, I hadn’t many friends in high places.

Just before he left, I pressed him again to put his promise into writing. As the promotion had not yet come through, he wrote I could get a minor wage increase, and then he filed it for processing at the human resources department. A few weeks later, they summoned me to the human resources department. A bureaucrat had come up with a technicality. I couldn’t even keep the minor wage increase. That was a breach of contract, plain and simple, but to bureaucrats, only rules and procedures count. My previous manager had already left, so they blamed it on him, and his temporary replacement didn’t care as he also was on his way out. As I had put a lot of effort into having it in writing, and my manager had already fobbed me with a minor wage increase, I walked out of the meeting angrily.

When I arrived home, Ingrid told me that a freelance agency had offered me a job. It was the first offer of this kind since I started working for my employer. And so, I made a rash decision and resigned. With the benefit of hindsight, it was a remarkable coincidence that the freelance agency called me on this particular day. It didn’t take long before I started to have second thoughts. Out of the blue, a strong feeling emerged that it was a wrong decision. I can rationalise it by saying there weren’t many jobs for database administrators near home. And the issues with my son didn’t allow me to work far away from home while my physical condition didn’t allow for long travels. That may all be true, but these considerations were not the real reason. The feeling became so strong that I had no other choice but to reverse course and try to undo my resignation.

There was a new manager, and he accepted my change of mind. He pledged to do his best to restore my confidence in my employer. Due to a bureaucratic error, I missed the promotion again a year later. I began to distrust him and feared he might not make good on his promise. That didn’t happen at the time, but he soon gave the tech genius a higher pay grade and left me out. And several years later, after he had risen in rank, in another remarkable coincidence, he tried to take away the pay grade that came with the promotion when I switched to Java programming. Nevertheless, he was a very competent manager who later played a leading role in improving the IT department. After some years of bureaucratic wrangling, the promotion finally came through.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.