We have an application which manages projects. Every once in a while the users make a huge mistake, or they miss the data from 5 days ago, or they want another copy of a project to play with, so they call the DBA. The DBA grumbles a bit, asks few questions, and within an hour or two or five they have a copy of the project.
Few month ago the users discovered that the DBAs automated the process and they just run a script to copy the schema that contains the project. At that point they thought – “Hey! we can run scripts too!”, why should we put up with the DBA grumbles and questions when this is so simple?
Users petitioned to management, and over the DBAs horrified protests it was decided to build UI front end to the script and let the users run it whenever they need to instead of going through the DBA team.
The tool was deployed on Monday.
On Tuesday the storage manager wondered why the diskspace taken by the application grew by 500G overnight.
On Wednesday users complained about awful performance on the application. It is possibly related to 100% CPU and long IO queues caused by 4 simultaneous import processes. Storage manager announced that he has no more space to allocate to this application.
Today, Thursday, the users opened a request to DBAs “We tried to import schema X several times with the new tool. It doesn’t work. Please restore the schema ASAP. We already breached SLA.”. Turned out that the tool does not handle a user with tables on a non-default tablespace. Also turned out that the schema takes over 100G, that most of the data was imported on each attempt and that the tool does not clean up the schemas after failure.
Thankfully, everyone agreed to stop using the tool and review the processes and procedures around schema restores.
Maybe one day tools with cute UI will replace DBAs, but they need to be very good tools.
