First Encounter with Big Corporate

My friends were teasing me for working in a very large company, and how we have all those “all hands meetings” with hundreds of people participating from all over the world. This reminded me of an old story that I want to share.

This happened over five years ago. I was still an application developer at the time. The customer was a huge telephony company, from an exotic country. The project was to move about 250G of data from our database to their database, and then connect it to an application over in the customer’s data center. I was on the project as the application expert. I would be working with special tools developed for this migration, and with the DBAs on the customer side. The main challenge for this project was that we were working under very tight deadlines.

Obviously the customer was warned about the amounts of data involved. I also explained that we will not be doing direct loading, but bulk inserts. These generate redo, and they must prepare for the amounts of redo generated. I was very proud that although I was not (yet) a DBA, I could already explain about redo 🙂

Anyway, we started the project by creating the user, and then ran some scripts to create the empty segments. Later we will load the data into them. Our 250G of data was not uniformly distributed between the 200 tables in the schema. We actually had less than 10 very large tables, and large amount of smaller tables and a significant amount of empty tables. Half way through the object creation our tablespace ran out of space. Anticipating large amounts of data, someone configured HUGE extents. OK, these things happen, lets fix that and try again. We need to get going fast, loading the data was expected to take about 24 hours.

At around 12pm in my timezone, which was around 2am for the customer, their database crashed. Out of space of archive logs. I called their DBAs one after another, leaving messages, but didn’t manage to catch (=wake up) anyone. This was surprising, the DBAs I worked with in my company used to wake up at any hour.
6 hours later, one of the DBAs called me back. I told him that his database is down, was done for the last 6 hours, and this is holding up our urgent data loading project.

His response: “Oh, this is really bad. I must call a meeting to discuss this”.

Two days later the DB was up and running and I could continue the data migration. I love the whooshing sounds deadlines make as they pass by.

At that time I was horrified at their response, becuase I knew that when our servers crash, our DBAs leaped to save it, they did not call any meetings. Now I have some perspective from the DBA side of things and I still don’t get it. We have procedures on how to deal with a DB that ran out of redo space (it is relatively common disaster), and these procedures do not include any meetings.

—————————-

Kevin Closson likes my blog! This makes me very happy, not just because it drives hoards of readers my way (hey everyone! hope you like it here!), but also because Kevin’s blog is the very first Oracle blog I started reading regularly. Kevin is one of the people I most admire in this field. His knowledge of OS, network, storage and hardware is unbelievable. I’ve always been more interested in how stuff actually works than in how to do things and Kevin’s blog is just full of this kind of knowledge.

Advertisements

5 Comments on “First Encounter with Big Corporate”

  1. Surachart says:

    Hi Chen Shapira,

    That’s a good story.
    Anyway, DBA at my Office wake up any time… any hour. 😉

  2. Robert Klemme says:

    Chen,

    you wrote “I’ve always been more interested in how stuff actually works than in how to do things […].” – I feel similarly. But, ultimately the knowledge of how things work internally is interesting because we can better _use_ them, don’t we?

    Btw, you also gave a nice example for cultural differences which make it sometimes very hard to work with people in other regions of the world. I’ve experienced this myself…

    Cheers

    robert

  3. prodlife says:

    Robert,

    I think the ability to use knowledge is just a convenient excuse.
    I noticed that when you go to “backup and recovery” class, some people are fascinated by explanations about how redo works, and others just want the RMAN scripts.

    Of course if you understand redo you will end up doing better recoveries, but redo is also fascinating for its own sake.

  4. Robert Klemme says:

    Hm… I’m not sure I am convinced, yet. 🙂

    There are certainly different approaches to learning among humans. For some it is sufficient to just get the “basic facts” while others need to understand inner workings of something (I’m certainly in the latter camp). I am not sure whether I would find redo interesting if I never had a chance of using that knowledge – even if indirectly (e.g. by considering the volume of an update when investigating slowness).

    I guess there are a lot of interesting aspects of C# programming language but I don’t feel inclined to investigate them since there is zero chance at the moment that I would be using any of that knowledge. OTOH, I find other machinery fascinating although I will never work with it. I guess it all depends on the subject and my relation to it.

    You certainly can’t avoid using your knowledge of redo considering your occupation, so you never know what makes you find it fascinating… 🙂

    Thanks for sharing!

    robert

  5. prodlife says:

    I’m not sure either 🙂

    On one hand, I agree that I do not learn partitions simply because we don’t use them

    On the other hand, I learned quite a lot about OS scheduling algorithms, simply because its an interesting topic. I never found any use for that knowledge (but I admit I tried).


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s