Yes, Virginia, You Really Need A DBA

We have an application which manages projects. Every once in a while the users make a huge mistake, or they miss the data from 5 days ago, or they want another copy of a project to play with, so they call the DBA. The DBA grumbles a bit, asks few questions, and within an hour or two or five they have a copy of the project.

Few month ago the users discovered that the DBAs automated the process and they just run a script to copy the schema that contains the project. At that point they thought – “Hey! we can run scripts too!”, why should we put up with the DBA grumbles and questions when this is so simple?

Users petitioned to management, and over the DBAs horrified protests it was decided to build UI front end to the script and let the users run it whenever they need to instead of going through the DBA team.

The tool was deployed on Monday.

On Tuesday the storage manager wondered why the diskspace taken by the application grew by 500G overnight.

On Wednesday users complained about awful performance on the application. It is possibly related to 100% CPU and long IO queues caused by 4 simultaneous import processes.  Storage manager announced that he has no more space to allocate to this application.

Today, Thursday, the users opened a request to DBAs “We tried to import schema X several times with the new tool. It doesn’t work. Please restore the schema ASAP. We already breached SLA.”. Turned out that the tool does not handle a user with tables on a non-default tablespace. Also turned out that the schema takes over 100G, that most of the data was imported on each attempt and that the tool does not clean up the schemas after failure.

Thankfully, everyone agreed to stop using the tool and review the processes and procedures around schema restores.

Maybe one day tools with cute UI will replace DBAs, but they need to be very good tools.

Advertisements

10 Comments on “Yes, Virginia, You Really Need A DBA”

  1. Noons says:

    or maybe one day IT managers will develop a brain and stop pandering to the idiotic “we don’t need no dbas” brigade?

  2. prodlife says:

    I think IT managers *are* the “we don’t need dbas” brigade.

  3. hey ddl,
    in my former project euler sponsor, we started writing an apex application that allow duplicating until time of production to a test environment. I think it was a good idea to relieve the dba from boring tasks.

    If your development team is unable to make an usable self-service application from your dba script, do not blame the manager!

    Whale, maybe you enjoy doing point in time recovery yourself btw 🙂

  4. Freek says:

    Laurent,

    I think that for application like described here, you will always need some kind of “gate keeper” to stop end users of doing silly things.
    Sure, the application could be made more fool proof, but without a gate keeper you will end up with buying a lot of disk space, memory and cpu power to sustain all the copies that will be generated.

    regards

  5. the disk and cpu should be ordered by the application owner, not the dba, and the application owner should be notified about disk usage

    any operation should of course check for sufficient disk usage

  6. Chen Shapira says:

    Hi Laurent,

    Yes, it is clear that the application was buggy. I don’t blame my manager, I blame IT management that allows development to produce DB tools without passing specification and tests through the DBA team.

    I also believe that tools should be used by people who understand how they work, understand their limitations and are responsible for the results of using the tools.

    Everything is just “running a script”, and you can build a cute UI around everything. But things will still go wrong, and you still need people who understand what to do when things go wrong.

    I suspect that in each organization there is someone who is responsible for capacity and resource management. It could be the application owner, but in our case it is the DBA team. In any case, whoever is responsible for these resources should be able to control the use of tools that use up these resources in an unplanned way.

  7. Hi Chen,
    Wise words, and yes I have understood that not only the developers wanted to use that application, but also they wanted to use it without your blessing, which is a huge mistake of them 😦

    To prevent this the dba should never gives unlimited tablespace to those teams !

    you still need people who understand what to do
    I love you so

    In any case, whoever is responsible for these resources should be able to control the use of tools that use up these resources in an unplanned way.

    yes, as I said, use quotas, profiles, restrict login, lock table owner schemas in prod, and change your system password to something else than MANAGER 😀

  8. Sadik says:

    🙂 Very interesting, maybe one day good UI tools will replace DBAs…! I doubt if the day will ever come. Consider this in a typical manufacturing organisation (the types I work for) there are hundreds of users who would bring the system down if it was even for a week’s absense of the dba.

  9. prodlife says:

    Hi Sadik,

    I’m on vacation now. I hope the DBs are doing fine 🙂

  10. Ludovico says:

    ROTFL! I implemented several similar tasks through RMAN duplicate, consistent copies, exp/imp.
    I think it’s a DBA job to implement and develop the procedures that do the dirty things.
    In my environments the whole procedures are often started by “flag files” touched by application GUIs. “Flag files” and crontabs. This is how duplicate procedures and GUIs interact.

    1 */2 * * * [ -f /GO_duplicate ] && [ -x /do_duplicate ] && /do_duplicate


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s