By Jim Czuprynski | Gary Gordhamer, Editor

As I celebrate my 35th year in information technology and my 15th year as an Oracle DBA, I’ve finally come to the conclusion that the role of a DBA often combines the roles of an airline pilot and diagnostic physician.

For starters, no one really notices us when the plane is flying trim and level; it’s only when the plane is screaming toward the ground at 10,000 feet a minute that our worth within the organization is really acknowledged.

When application performance suddenly degrades, it often seems that it’s the Oracle DBA who is expected to shed her pilot’s stripes and immediately transform herself into a simulacrum of that brilliant yet cantankerous television doctor, Dr. Gregory House. Suddenly, she is responsible to perform a differential diagnosis process to solve the problem, regardless of whether the problem is database-related or not.

One place I believe Oracle DBAs could really use their diagnostic prowess and thus add value to their IT organization is to actively and accurately diagnose depreciatory behavior patterns and less-than-progressive ways of thinking. To assist you in your role as diagnostician, I’ve grouped these rather common afflictions into five maladies that I’ve named Uniquitis, HypoRMANosis, Tengenertia, Toolichondria, and Idiomyopia.

Uniquitis

Symptoms

If you’re an Oracle DBA and you constantly hear statements like these from application developers, QA technicians, IT managers, and even your CIO:

• “Our application workload demands are like none other.”
• “Only our team/developers/DBAs/CIO really understands what we’re trying to accomplish here.”
• “We don’t need to attend conferences. Our team knows what it’s doing!”

… then your team is likely suffering from what I like to call Uniquitis.

Uniquitis (u-neek-i-tis): The belief that your IT organization’s problems are unique and that no other organization is facing or has ever faced precisely these same problems.

Uniquitis may also manifest itself as an extreme reluctance to adapt to new releases of Oracle (cf. Tengenertia) or to adjust a team’s development style to take advantage of new industry standards like Agile or DevOps.

Prognosis

Unchecked, this leads to ever-narrowing organizational vision. Independent thought and creative problem solving may be discouraged in favor of organizational inertia, but DBAs are in the best position to rise to the challenge.

Treatment

One of the best ways to overcome this malady is to actively collaborate with other DBAs in your industry to find out how they’re solving similar problems within their organizations. Also, communicate with other IT professionals, especially by attending technical conferences to get new perspectives. And most importantly, don’t be afraid to say: “Surely, someone else must have already encountered these challenges!”

HypoRMANosis

Symptoms

Oracle offered Recovery Manager (RMAN) as early as Oracle 8i, but there are still at least a few IT organizations that have yet to adopt it fully. During my 10 years as an Oracle University instructor and now in my role as a strategic solutions consultant, I’ve heard some interesting comments from Oracle DBAs regarding its capabilities. Here’s a brief sampling:

• “We’ve always used [non-RMAN third-party tool set] and have never had a problem.”
• “RMAN can’t handle our required RPO or RTO SLAs.”
• “Our system/storage administrators take full responsibility for our backup (and disaster recovery) strategies for all databases.”

If you’ve ever overheard anyone on your DBA teams say something similar, it’s likely that your IT organization is suffering from HypoRMANosis.

HypoRMANosis (hi-po-ar-man-O-sis): An extreme reluctance to leverage all of the features of Oracle Recovery Manager to back up, restore and recover Oracle databases.

Another disturbing trend I’ve often seen is severe underutilization of key RMAN features, such as hardly (or never!) leveraging the Fast Recovery Area (FRA); rarely testing restoration and recovery strategies; and, most especially, the failure to fully leverage Oracle Database flashback features, especially Flashback Versions Query, Flashback Query or Flashback Data Archive (renamed to Temporal History in Oracle 12c). If your team has already made the transition to Oracle 12c, by the way, be sure to check out its related temporal validity features and the ability to create bi-temporal queries.

Prognosis

Whenever I encounter these symptoms, I like to ask the following question: If your storage administrator won’t be standing next to you when it’s time to restore and recover your databases at some ungodly hour in the middle of the night, then why is he empowered to tell you how best to back them up?

As Oracle DBAs, we are fully responsible for guaranteeing the complete recoverability of each and every database within its specified RPO and RTO. With rare exception, RMAN is simply the best tool to achieve that goal, especially with Oracle 12c’s new capability to create image copy backups in a multi-piece format using parallel processing. If your organization is still using some other backup and recovery strategy, it’s time to re-evaluate RMAN’s latest capabilities, because the lifeblood of your company, its production data, is ultimately dependent upon its recoverability.

Treatment

Here’s a short list of possible prescriptions:

A robust RMAN strategy for database backup, restoration and recovery does require you to understand the latest capabilities of RMAN, so be sure to test multiple restoration and recovery scenarios that correspond to your organization’s needs as well as the current release of your Oracle Database.

Take the time to get comfortable with advanced RMAN techniques like Tablespace Point-In-Time Recovery (TSPITR) and Flashback Database; in extreme recovery scenarios, they usually offer dramatically shorter recovery times than a complete restore of the entire database.

Don’t be afraid to valiantly challenge RPO assumptions from your user community and leverage flashback tools and features (especially Flashback Versions Query) instead of restoring ancient backup files to alternate servers just to run a report on the state of data at some prior point in time.

Tengenertia

Symptoms

If you find that you and your fellow Oracle DBAs seem to be constantly ignoring the very features designed to make an Oracle DBA’s life easier, then, unfortunately, you may be suffering from Tengenertia.

Tengenertia (10-gen-UR-shia): The tendency to operate an Oracle 11g or 12c database like it’s still an Oracle 10g database; also, exhibiting extreme resistance to explore and adopt latest features of current database release.

Indications of an acute case of Tengenertia may include:

• Avoiding use of Enterprise Manager 11g Grid Control or 12c Cloud Control
• Clinging desperately to STATSPACK reports instead of what I like to call “The Three As”: Active Session History (ASH), Automatic Workload Repository (AWR) and Automatic Database Diagnostic Monitor (ADDM)
• Ignoring beneficial features of the Oracle Optimizer like SQL Plan Management and Adaptive Cursor Sharing and instead attempting to tune complex queries only via optimizer hints

Prognosis

While it may be comforting to cling to the same way of doing things over and over, it’s important to remember that without leveraging the full features of the latest version of your Oracle Database, it may be possible that database application performance may actually be several orders of magnitude worse than optimal. Also, many of the features of 11g and 12c were designed to alleviate DBA stress and actually increase the number of databases that a single DBA can manage at one time, so by refusing to adapt to new ways, a modern Oracle DBA will simply become increasingly over-burdened and eventually burn out.

Treatment

Much like avoiding adult-onset diabetes, a lifestyle change is required. This includes an active embrace of the latest release’s newest features by reading through the appropriate new features guide and then making those new features part of your Oracle DBA tool belt going forward. And when the next release is imminent, I strongly suggest getting your hands on that guide as soon as possible so that you’re aware of what changes are on the horizon.

Toolichondria

Symptoms

Have you recently overheard comments or complaints from your colleagues, especially from your CIO, CTO or DBA managers?

• “These [Oracle/third-party] tools are so expensive!”
• “We want to avoid vendor lock-in.”
• “We’ve always used [insert obscure tool here] because it’s [free/open source/not Oracle].”

If you have, your IT organization is most likely suffering from a syndrome that I’ve labeled Toolichondria.

Toolichondria (tul-e-KON-dri-a): Exhibition of extreme resistance to leveraging built-in tools or other licensed features of the current Oracle Database release.

As an Oracle DBA, you may encounter Toolichondria just after joining a new IT organization. You realize that the current DBA team is still relying on the same SQL scripts for application workload performance tuning when they deployed Oracle 9i, even though all of the organization’s databases were updated to Oracle 12cR1 six months ago. Even worse, the team hasn’t implemented any kind of central enterprise management or monitoring tool sets to keep an eye on their Oracle Databases’ performance; instead, they are relying on terminal window upon terminal window filled with database alert logs being tailed, or some ancient shell scripts scraping information from alert logs every few minutes. In my opinion, this approach to database management is like posting Facebook updates from a flip phone.

Prognosis

As our CIOs and CFOs often say, you can’t manage what you can’t measure. The big advantage that I see with Oracle Database’s diagnostic and tuning tools is that they provide accurate and reliable metrics for determining a database’s performance independently of their application workloads. Without accurate metrics, a performance issue can rapidly degrade into a massive finger-pointing exercise among storage administrators, network administrators and Oracle DBAs that achieves nothing but frustration.

It’s also important to be able to measure database performance even when there’s no performance issue apparent so that a good baseline of acceptable performance is established. That’s something AWR handles quite admirably. Otherwise, an Oracle DBA who’s only responding to performance anomalies is like an apprentice carpenter that forgets the old adage of “measure twice, cut once.”

Finally, while I certainly understand that avoiding vendor lock-in is an admirable goal, I’d rather have only one throat to choke in the unlikely case that a metric is slightly incorrect or misleading.

Treatment

One of the best ways to overcome Toolichondria is to follow in the footsteps of the giants of IT that have come before us. If you haven’t heard of the acronym RTFM, you probably haven’t been involved with information technology for very long, so it’s time to learn that it stands for Read The Fine Manuals. Regardless of the tools chosen, it’s your responsibility as an Oracle DBA to understand their power as well as their limits.

Once you’ve RTFM’d sufficiently, I heartily suggest embracing your chosen tools as force multipliers; they enable your transition to what I envision as a DBA “Army of One” — someone who can manage hundreds of different databases simultaneously and is always fully aware of the state of every database in her organization.

Idiomyopia

My role as an Oracle University instructor in core database technology prepared me for one of my favorite job responsibilities: interviewing potential consultants on their technical skill sets. Over the last five years, I’ve noticed an accelerating trend: candidates who appear quite comfortable with exaggerating the depth of their Oracle DBA expertise. Here’s a few recent disturbing examples:

One candidate’s resume stated she had significant experience with Oracle RAC; however, she couldn’t explain what would happen to a database instance when I accidentally terminated the LMON background process on the first node of the cluster. (She insisted that I’d have to restart the instance on that node manually.)

Another candidate couldn’t explain why ASM was crucial for the efficient operation of storage cells on an Exadata Database Machine even though his resume listed “certified Exadata installer” as one of his core competencies.

Another candidate listed “a deep understanding of Oracle Data Guard technology” on his CV. When I probed further, he was unable to list reasons why Data Guard Broker would be helpful when managing a switchover operation between primary and physical standby databases.

Symptoms

If you’ve recently had the misfortune of having to perform damage control or clean up after a severely under qualified Oracle DBA, that incessant muttering under your breath is probably a mild symptom of a disease I call Idiomyopia.

Idiomyopia (i-di-o-my-O-pe-a): The unnecessary tolerance of incompetent Oracle DBAs.

This is a difficult disease to discuss because it’s essentially our industry’s dirty little secret. After all, who wants to admit that a totally underskilled resource has somehow slipped past their organization’s careful screening process and now inhabits a role for which he’s seriously under-qualified? Hopefully, my screening efforts helped prevent the accidental hiring of a less-than-qualified consultant.

However, your organization may not have been so lucky due to any number of reasons, including poor initial screening through your company’s human resources team, or hiring underqualified DBAs based only on what their CVs claim (also known as “purple squirrel” syndrome), or failing to perform technical screening under tightly controlled conditions.

The effort it takes to locate and onboard a new Oracle DBA is often exhausting. In highly secured environments, it can take weeks just to get security approval, user accounts and laptops provisioned; unfortunately, this can often lead to pressure to retain an incompetent resource just because it took so long to get them hired.

Prognosis

The most important aspect of Idiomyopia is to first recognize the painful reality and avoid denial of the affliction; after all, doing nothing means this cancer will metastasize to other parts of your IT organization. The damage that an unskilled or incompetent resource can inflict on a production Oracle Database is virtually unlimited — and we haven’t even raised the specter of malicious intent on the part of that resource.

Finally, it’s important to realize that other competent resources will definitely be affected. How will coworkers feel after they have spent their evenings and weekends preparing for OCA or OCP certification exams when they realize that the value of those certifications are rendered meaningless by the practice of hiring someone with unproven or — even worse — falsified credentials?

Treatment

Just like cancerous cells, the best approach is to perform immediate surgery to remove the tumor. In other words, suggest that the resource be asked to move on to a job that’s more fitting to his skill sets. Once removed, it’s equally important to triage to eliminate future vectors of infection by making sure that your IT organization has established better screening criteria for future interview efforts.

Finally, it’s important to monitor for remission and remain hopeful for an eventual cure. The good news is that the cure might actually be coming sooner than we all think: My IOUG colleagues and I have been discussing potential methods to award unquestionable proof of identity to those of us who have seriously endeavored to become true IT professionals through plain old hard work, careful attention to detail and valid Oracle certifications.

Oracle DBA: Job, Career or Profession?

Whether you identify your Oracle DBA role as airline pilot, diagnostic physician or a mixture of both, one thing is certain: Dealing with the myriad afflictions that we encounter within our IT organizations is no simple task. We’re often involved in crucial aspects of ongoing application development as well as “keeping the lights on” for existing and often complex systems. These responsibilities require us to leverage cross-discipline skills as we engage other IT professionals on different teams. Much like Dr. House, these unique relationships within our IT organizations truly enable us to diagnose and perhaps maybe even cure some of these issues. Isn’t it time that we band together in a campaign to uplift the designation of Oracle DBA to that of a true profession?