RAC tricks – rolling patch with a shared home

We had to apply yet another opatch, but we are only allowed 4 hours of downtime per system per month and we already used our monthly budget, so we need a way to apply an opatch without any downtime.

On some of our systems, it is not an issue. They are RAC systems where each node has its own $ORACLE_HOME on its own server. We take one node down, apply the patch, start the node, stop the other node, apply the patch, start other node. Patch installed on both nodes, no downtime for our customers. Win-Win.

But what do we do about our other systems? The ones which share a single $ORACLE_HOME on a filer? Where we need to take both nodes down for applying the patch?

A co-worker came up with a brilliant idea:

Stop one node. Use the filer power to duplicate $ORACLE_HOME. Connect node to new home, just make the change in /etc/fstab, the database will never notice the difference.
Apply patch in new home. Start database in new home. Now stop the second node and connect it to the new home as well. Start the node in the new home. We have a patched DB with no downtime in a shared home system! We even have a built in rollback – connect one node after the other back to the old home, where we didn’t apply the patch. In my experience rollback of opatches don’t always work, so having a sure rollback plan is a great bonus.

We tested it today in a staging environment and it seems to work well. Now we just need to convince management that we should do it in production. It looks like a great solution, but in my experience management hates approving any plan that does not appear in Oracle manuals. For all their talk of innovation and “thinking outside the box” they are a very conservative bunch. I can understand the extreme risk aversion of IT management, but if you never do anything new, you can never improve, and thats also risky.

Advertisements

9 Comments on “RAC tricks – rolling patch with a shared home”

  1. John says:

    Hi – the first approach you used, in a RAC system, is fine, except that it doesn’t work for patches that change the database; only the software. Or am I missing something huge, here?

  2. prodlife says:

    Hi John,

    You know, I can’t recall a patch that actually changed the database. Those are usually upgrades (from 10.1 to 10.2 for instance), or maybe a big patchset. Indeed, these will require a real downtime even in a RAC environment.

  3. Paresh says:

    Hi,

    I liked your idea to apply patch in shared $ORACLE_HOME environment without impacting business. Could you please tell us the exact steps for filer power to duplicate $ORACLE_HOME?

    Thanks for sharing knowledge with us.

    Paresh.

  4. prodlife says:

    The exact steps depend on your filer and should be checked with filer vendor. I don’t do this part, I ask my storage admin.

  5. Ken says:

    How did you handle registering the new $OH in CRS?

  6. prodlife says:

    Hi Ken,

    Since the only change I make is in the mount (I mount a new volume with the same path as the old one), the value of $OH never changes, and I’ve no need to register a new one in OCR.

    • sriram says:

      Hi,

      Thanks for sharing this quick win tip , but are you sure that we dont need to run a catpatch.sql for these interim patchsets? some times it is mandatory to update the dictionary?

      • prodlife says:

        @Sriram
        The trick is for OPatches, which normally don’t require dictionary rebuild. Patchsets normally require a complete shutdown, even with local home, for exactly this reason.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s