First Encounter with Big CorporatePosted: January 16, 2009
My friends were teasing me for working in a very large company, and how we have all those “all hands meetings” with hundreds of people participating from all over the world. This reminded me of an old story that I want to share.
This happened over five years ago. I was still an application developer at the time. The customer was a huge telephony company, from an exotic country. The project was to move about 250G of data from our database to their database, and then connect it to an application over in the customer’s data center. I was on the project as the application expert. I would be working with special tools developed for this migration, and with the DBAs on the customer side. The main challenge for this project was that we were working under very tight deadlines.
Obviously the customer was warned about the amounts of data involved. I also explained that we will not be doing direct loading, but bulk inserts. These generate redo, and they must prepare for the amounts of redo generated. I was very proud that although I was not (yet) a DBA, I could already explain about redo 🙂
Anyway, we started the project by creating the user, and then ran some scripts to create the empty segments. Later we will load the data into them. Our 250G of data was not uniformly distributed between the 200 tables in the schema. We actually had less than 10 very large tables, and large amount of smaller tables and a significant amount of empty tables. Half way through the object creation our tablespace ran out of space. Anticipating large amounts of data, someone configured HUGE extents. OK, these things happen, lets fix that and try again. We need to get going fast, loading the data was expected to take about 24 hours.
At around 12pm in my timezone, which was around 2am for the customer, their database crashed. Out of space of archive logs. I called their DBAs one after another, leaving messages, but didn’t manage to catch (=wake up) anyone. This was surprising, the DBAs I worked with in my company used to wake up at any hour.
6 hours later, one of the DBAs called me back. I told him that his database is down, was done for the last 6 hours, and this is holding up our urgent data loading project.
His response: “Oh, this is really bad. I must call a meeting to discuss this”.
Two days later the DB was up and running and I could continue the data migration. I love the whooshing sounds deadlines make as they pass by.
At that time I was horrified at their response, becuase I knew that when our servers crash, our DBAs leaped to save it, they did not call any meetings. Now I have some perspective from the DBA side of things and I still don’t get it. We have procedures on how to deal with a DB that ran out of redo space (it is relatively common disaster), and these procedures do not include any meetings.
Kevin Closson likes my blog! This makes me very happy, not just because it drives hoards of readers my way (hey everyone! hope you like it here!), but also because Kevin’s blog is the very first Oracle blog I started reading regularly. Kevin is one of the people I most admire in this field. His knowledge of OS, network, storage and hardware is unbelievable. I’ve always been more interested in how stuff actually works than in how to do things and Kevin’s blog is just full of this kind of knowledge.