By Eric Mader | Edited by Michelle Malcher

It’s 2 a.m. on a Saturday morning. A seemingly-simple deployment of quarterly bundle patches is causing issues with a RAID controller. The production customer portal is down, and the phone is ringing every 15 minutes for a status update.

The patch was thoroughly tested in development, but the production servers were purchased two years later and don’t match. The application team is waiting to roll out a minor update to the portal once database patching is complete. But – even if the database comes back online, it’s rumored that the developers are not 100 percent sure that the Billing table structure exactly matches what was tested in the lower environments and the new application code may not work.

The deployment process, which only took 60 minutes in development when applied two weeks ago, is now into hour six and eating into the weekend with no end in sight.

We’ve all been there. When the rubber meets the road, it seems like the database, operations and development teams are never quite in-sync. No matter how thoroughly executed, tested, documented and validated, something in the production environment is never quite the same as the lower environments – be it a hardware inconsistency, a code mismatch, or even the dreaded typoed command. To add insult to injury, code always seems to change between the time it leaves a developer’s laptop and when it is deployed to a mission-critical system.

Solving these issues is the primary motivator behind DevOps. Often misinterpreted as a buzzword endorsing development teams taking over platform and infrastructure roles, DevOps is all about development and operational modernization. Rather than operating as siloed, asynchronous teams as has been the norm for decades, DevOps represents a fundamental, holistic, organization-wide shift not only in processes and tools, but in people and culture.

Shifting to Continuous Processes

One of the fundamental changes for a DBA in a modernized development world is the team on which they operate day-to-day. The role will be less a part of a dedicated team consisting of just DBAs, and more a part of a broader cross-functional team consisting of other roles such as application developers, testers, engineers, architects and various subject matter experts.

The reasoning is simple: Rather than communicating and synchronizing across siloed teams, all activities within the various layers of an architecture operate from the same pipeline. The pipeline consists of stages, which are mapped to the various environments within the development lifecycle (e.g. Development, Test, QA, Stage, Production). Each stage consists of automated scripts that deploy all code and artifacts to an environment without requiring manual activities or human intervention.

This pipeline is often referred to as the Continuous Integration/Continuous Delivery (CI/CD) pipeline. It ensures that all changes are made in one direction, and that changes are not made in mission-critical environments without first being made and vetted in lower environments. Since all deployment activities are automated, teams can significantly reduce risk by deploying more frequently and with fewer changes.

The challenge is integrating databases with this pipeline. Unlike most application deployments, databases typically contain immutable components. For instance, the critical business and transaction data within a database usually cannot be easily or quickly destroyed and rebuilt. Similarly, certain fields may need to be obfuscated or replaced with sanitized data in non-production environments. Database processes that seasoned DBAs are used to performing manually need to be automated, and components such as schema changes and database patches will be made in tandem with application and architecture changes.

When undertaking a DevOps modernization effort, it helps to view the database as three tiers. The Instance tier is often the easiest to integrate into a pipeline and includes the instance objects (tables, views, etc.). The Platform tier includes the Oracle database software, and varies in complexity depending on organizational investment in data center hardware and abilities to use elastic options such as public clouds. The Data tier is often the most difficult to integrate: it contains the business and transaction data within the database.

Quick Wins with Instances

A great place for organizations to begin their database development and operations modernization journey is in the Instance tier. Similar to application code, DBAs should begin defining schema objects within SQL scripts and storing in the same version control as the other application code. Platforms such as git create a “paper trail” – all schema changes are completely tracked, traceable, and can be quickly rolled back to previously-known working scripts. These platforms also allow for code branches with pull requests. Database developers can specify DBAs to perform code reviews before a branch is merged into a stage’s branch and an automatic deployment occurs.

When scripting object changes, a simple approach is to build a DDL containing CREATE and ALTER statements for all schema the objects, ordered by dependencies. Take care, however, with tables. For databases that contain smaller, mutable data sets, ensure that the data is reloaded as part of the script. For larger, immutable sets, ensure that the script preserves the data within the table.

Organizations should also consider refining permission models within the database so that the automation processes within the pipeline are only allowed to perform critical actions, such as DROP, CREATE and ALTER commands. This will prevent database users, developers and DBAs from circumventing the pipeline and performing untracked modifications.

What About the Platform?

The Platform tier includes the Oracle database software. To de-risk deployments at all architecture tiers, upgrades and patches should be included in the CI/CD pipeline. In general, to align with DevOps ways of working and help DBAs transition, Oracle announced a shift in their database versioning strategy at OpenWorld 2017.

Traditionally, Oracle released a major version containing many enhancements and features on an unpredictable timeline. For example, there was a six-year span between the initial releases of 11g and 12c, and upgrades generally require major consideration and effort. To mitigate, Oracle released patch sets consisting of incremental changes and fixes – however, the timeframe was again somewhat unpredictable, and the changes still numbered in the thousands and carried a notable amount of risk. Even quarterly Patch Set Updates (PSUs) and Security Patch Updates (SPUs) required careful planning and testing, and often leave instances vulnerable to critical security issues for months or even years at a time.

To align with organizations that are operating within CI/CD pipelines and releasing more frequently, Oracle shifted to major releases tied to the current calendar year. The first year-based version – 18c – was released in January 2018. It is expected that 19c will arrive in early 2019, with 20c following in 2020. The idea is de-risking upgrades: by providing a major release more often, upgrades will occur more often and with fewer changes. Organizations will no longer have to plan and execute major, painful upgrades every few years.

In addition, Oracle plans for Release Updates once a quarter which will contain minor enhancements and security patches. In Q4 of 2019, DBAs can roll out version 19.4 to the pipeline for development, testing, and eventual automated deployment to production. If necessary, Oracle will also provide Release Update Revisions.

Rolling out software updates and patches relies heavily on the actual platform. Depending on the organization and deployment type, major upgrades may include automated software deployments via kickstart scripts. Oracle publishes validated Docker images of Oracle Database instances on the Container Registry, which can be quickly started with arguments pointing to storage locations containing existing database data. OPatch is fairly easy to automate: DBAs can build shell scripts that can call the apply and rollback arguments and return error codes to the pipeline to indicate success or failure.

Tackling the Data

Often, the most challenging database tier to integrate into a CI/CD pipeline is the Data tier. In most databases, it is unreasonable to expect an entire database to be dropped then re-built on every deployment. Similarly, it doesn’t make much sense to drop an entire schema just for the sake of deploying an upgrade or patch. In some circumstances, the data set may be small or trivial enough to be dropped and re-created as part of the schema’s SQL scripts.

As mentioned, the SQL script defining the schema structure and checked into version control should preserve data within tables. In addition to empty schemas, scripts should be tested against existing schemas containing representative data sets to ensure preservation. Redo logs are a powerful tool to leverage if there are concerns about dropped or altered data during automated deployments.

Begin the Journey

Overall, remember that DevOps is a journey – it is unreasonable to expect full maturity overnight or even within the scope of a single modernization effort. Continuous Integration and Continuous Delivery are just two aspects of modern development. Organizations should also consider Continuous Deployment, which includes zero-downtime concepts such as automated Blue – Green and Rolling deployment strategies. Also included is Continuous Monitoring, which involves triggering automated recovery operations based upon real-time events.

By viewing database development and operations in tiers, organizations can begin identifying opportunities for better aligning with other teams and development efforts. DBAs need to identify areas for better aligning with overall development efforts, and areas which will require longer-term changes to reach modernization maturity.

Let the journey begin!