ITIL Processes Taken to Extremes
Posted: May 8, 2008 Filed under: xfiles 13 CommentsA DBA from another team went on vacation, and asked me to do few urgent tasks for him while he is gone.
One of the tasks was to refresh test schema from production data. In his email he commented that in his team the process includes opening a change in the Change Management system for this task “in case we drop the production schema instead of the test schema by mistake”.
I was very surprised by this. In my team, the ticket asking for the refresh was considered enough documentation, and there was no need to open a change. Also, if you regularly drop the wrong schema, opening changes will not help you.
I asked around for explanation for the different procedure. Turned out that few month ago someone from the other team dropped the wrong schema by mistake. It was recovered from backup with no issues, but it still caused an hour or two of downtime for the users. Which means that we need to open an Incident, and of course, every Incident has to contain action items for preventing same issue from reoccurring. In this case, the manager who reviewed the incident noticed that there was no Change open for the schema refresh, which means that the DBA did not follow the right procedure! The natural action item for the change was “Instruct DBAs on Change Management procedures”. DBAs were properly instructed (at least some of them) and are now opening a change before dropping the wrong schema.
Which just goes to show how procedures can’t replace common sense.
Just a simple Hadoop DBA 
ONe method I’ve used to get around this is to set putty or whatever terminal emulator I’m using to have a different background colour depending on the type of server. typically red for production green for everything else. Helps identifiy which is which when the hostnames are similar and very usefull when performing migrations at 4am when everything starts to get a bit confusing..
Great blog entry. It demonstrates a prime example where you can train somebody to follow a procedure or elements of it, alternatively if one educates the person or team as to why the procedure and ultimately the process exists then an understanding can be realized as to the roles the various parties undertake in the procedures/process. This does’nt just apply to ITIL though…..it is life as a whole.
Regards. steve@itilnews.com
To be fair to ITIL (not something I say very often) this sort of thing is not the fault of ITIL per se. ITIL just says there should be procedures in place and that there should be a mechanism for reviewing and improving those procedures.
Now reviewing and improving procedures _could_ mean shortening or even removing procedures. It just never gets interpreted that way. Inevitably procedures bloat and engender further procedures. Which is why we end up with 10 Step Guides to Replacing a Light Bulb : http://www.telegraph.co.uk/news/uknews/1572905/MPs%27-guide-to-clearing-up-broken-light-bulbs.html
Cheers, APC
What Change Management system do you use?
Do you have a Configuration Management system as well?
My company is looking at new systems and I would be very grateful for any suggestions.
chris_c – good idea! I’ll definitely try that!
APC,
Are you sure you are not confusing ITIL with ISO? ITIL is quite specific about the procedures that should be followed. ISO standards seem to be more about “procedures for managing procedures”.
In any case, we started following ITIL two years ago, and so far I’m very happy with it. Most of the new processes we have definitely improved the quality of our work, if only by forcing me to slow down and think about what I’m doing 🙂
But sometimes good processes get abused. Having a process that forces people to learn from incidents is a good idea, but only as long as people really want to improve.
We also have a “rule” that says “Don’t do risky maintenance on Friday”. Great idea, but some people interpret it as “Do nothing on Friday” or even “Do nothing on Friday, Thursday and Wednesday”…
Andrew:
We are currently using HP’s PPM for change and incident management. Its not perfect, but it worked well and we managed to integrate it with HP’s CMDB product and HP Discovery and Mapping solutions quite well.
However, since this was a hack and PPM is not intended for this purpose, we are in the process of evaluating the move to a more “official” ITIL solution. This will happen in the next year.
Of course once you actually configured a CMDB, you will only use products that integrate with the CMDB you have (because there is no chance in hell you’ll be going through the configuration process ever again).
So we will stay with HP’s UCMDB and Discover, add HP’s Service Center, maybe add HP’s Change Control solution.
Interesting times 🙂
We have all kinds of this stuff…it seems crazy. But after working ridiculous hours, one more set of eyes is a good thing.
I don’t like it, but I think it helps me most of the time.
I do like the idea of a different background color for the production environments…
[…] ITIL processes can’t make up for common sense, says Chen Shapira. She gives an example of another DBA asking her to take over a few tasks, which involved testing in production data and opening a change in the change management system. She also cites a previous incident which cost users 1-2 hours of downtime. […]
Regarding different background colors when using putty: Here’s a bat file to automate setting that up. Helpful if you have many hosts already, or if you want to enter a bunch in a hurry. It adjusts registry keys needed for the default foreground and background colors. Place the three lines below in a bat file, say putty_color.bat. Then supply the target hostname when calling it. The lines below will make the foreground yellow.
c:\putty_color.bat yourhost
reg add HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions\%1 /v Hostname /d %1 /f
reg add HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions\%1 /v Colour0 /d 0,0,0 /f
reg add HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions\%1 /v Colour2 /d 255,255,128 /f
Want to see a list of sessions that you’ve already got in the registry for putty? This will help you make a series of calls to the bat file above.
reg query HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions
I normally use Reflection and paint my foreground as Red for production databases, Green for DRC’s, Blue for Test databases and default for others which do not fall in the above categories…………
So, when issuing a DROP USER/TABLE/TABLESPACE I just need to make sure on which color I am working on……
Looks like you need to have high Priority for this Change process. And your management or Change approver would have also planned for Outage time as this can have impact on the users 🙂
Frankly ITIL processes Taken to Extremes..
[…] mattina vengono fuori i problemi. Non pensavo che in aziende grandi e strutturate, dove si usa ITIL e dove ci sono “team” di DBA ecc ecc. succedessero cose […]