Security in a DevOps World


By Ben Golub, executive chairman and interim CEO, Storj Labs

The IT industry has no shortage of buzzwords and fads, and it may seem as if DevOps, Microservices and Containers are nothing but the latest examples. But underneath the hype and hyperbole, IT is undergoing substantial and profound changes. DevOps and associated trends both create a mandate for new approaches to security and provide lessons for how to approach security.

The world of IT has changed dramatically and fundamentally over the past 15 years.  They are summarized in the table below. But, suffice it to say that there have been dramatic increases in:

  • The number of applications and services
  • The speed with which those applications are updated
  • The number of “machines” on which applications are run
  • The number of environments in which those applications run
  • The number of individuals/entities interacting with those applications.

As a result, security models that are manual, bespoke and aimed at securing individual machines, applications and deployments need to evolve. The modern security environment, like the modern IT environment, needs to evolve to focus on automation, reliability, and the creation of secure, dependable and repeatable processes to ensure a secure and resilient environment.

ApplicationsMonolithicModular, composed of large numbers of shared services
Release cadenceInfrequent, waterfallFrequent, agile
InfrastructureA few large servers in a datacenterLarge numbers of smaller, virtualized or containerized servers
DatacentersOn premise traditionalPublic, private, hybrid cloud
"Machines"Small number tightly tied to appsLarge numbers, short lifecycles, supporting multiple apps
"Clients"Small number that are locked down and belong to the organizationLarge number, multi-platform, belonging to multiple organizations

As a security practitioner, you might find the DevOps movement to be too touchy-feely. Many of its evangelists talk about joining people together across silos and working together on common business goals. But for all of the kumbayah talk surrounding it, DevOps (and its sister terms), do represent a fundamental change in both scale and approach. That approach is rooted in concepts of resilience, repeatability, and automation.

In its most basic form, DevOps brings together development teams and operations teams to create and deliver code faster and more effectively, and aims to create robust, secure, and resilient approaches to building, shipping, testing, and running code.  The DevOps model also encourages continuous improvement that makes software more resilient. DevOps teams use automation and continuous monitoring to write code in small chunks, deploy it rapidly and get quick feedback, thereby reducing risk because developers don’t have to dig through endless lines of code to figure out the source of a bug or vulnerability.

By embracing automation, clean interfaces and repeatable processes, DevOps teams achieve a number of important objectives:

  • Increasing the speed and scale of operations
  • Limiting the variables associated with build/ship/release, so that quality can be improved and root causes of issues can be more quickly diagnosed
  • Allowing for easy roll back/recovery when problems arise.

In this environment, security must also evolve to both support the pace and scale, as well as enjoy the benefits that come from this new approach.

Why Airplanes Rarely Crash the Same Way Twice

One major complication with the old-school way of developing and deploying software of any kind is the difficulty it presents for finding the root cause when a problem occurs, and then ensuring the same type of problem doesn’t happen again.

Several years ago, Atul Gawande published a hugely influential book called The Checklist Manifesto. In it, Gawande, a medical doctor, compares outcomes in commercial aviation to those in intensive care units. Both environments are complicated and can be marked by catastrophic failures. The problems and solutions in these industries are similar to those being faced by security and software development teams.

In the healthcare industry, over 30 percent of all failures are a result of human error. For example, a patient gets the wrong drug dosage or develops a line infection—potentially fatal mistakes that have no relation to the underlying illness. These tragedies too often are preventable, but they are repeated over and over again.

In contrast, commercial aviation is dramatically error-free. While crashes do occasionally occur, they are rarely identical to a previous one. Whatever its faults, the airline industry understands the importance of quality control and has leveraged automation to ensure that mistakes aren’t repeated. Every plane has to pass the same checklist of elements before it can take off. If an accident does take place, the industry figures out the root cause and adds it to the preflight checklist so that a similar accident doesn’t happen again.

As a result, flying has become incredibly safe. By making these incremental adjustments and encoding them through automation and repeatable processes, errors are almost never repeated because they have been fixed in every plane before it is allowed on the runway.

Similarly, in computing, if you break tasks down to smaller chunks, you can more easily automate common mishaps and avoid having to worry about them again in the future. That doesn’t prevent new problems from occurring, but at least you can rule out a host of variables, speeding up your ability to find the root cause and roll back if necessary. Following this process will make your security foundation much more resilient.

Security in the DevOps World

In a DevOps world, code is continuously written, rewritten, built, tested and deployed. Generally speaking, the “code” is deployed as container images to a centralized registry. New containers are continuously deployed to a large number of “machines” located in multiple locations, and those machines are continuously being modified and changed as well.

In this new world, manual processes must become automated. Furthermore, security approaches that previously focused on the secure deployment of a monolithic piece of slow-moving code on a small number of machines must adapt as well. Creating secure, automated, repeatable and cryptographically enforced processes is a priority.

In this new world, there are three key areas for security to drive standardization, automation, control, and continuous improvement:

  1. Trusted, automated processes for building, testing, security scanning and signing images
  2. Trusted, automated processes for deploying signed images using policy and the intersection of signed images and machine identity
  3. Tested, automated processes for creating, maintaining and managing machine identities and authorizations.

DevOps Is for Security, Too

I’m guessing some of you are still skeptical about the benefits automation can bring to security processes, let alone doing so in a way that mirrors what DevOps teams do. You’re probably worried about ceding traditional security controls—such as segregation of duties, change approval processes and manual security reviews at the end of projects—for processes that seem to be chasing a constantly moving target.

Automation actually addresses these and many other security concerns. Instead of having to worry about whether a DevOps team is using proper security hygiene during the software development process, you can make sure that security controls are built-in organically. As you discover errors or vulnerabilities during that process, you can take the approach flight investigators take and make sure those mistakes are not repeated by building corrective actions into workflows. You can also make sure that DevOps teams do not shirk security controls by incorporating items like code-signing and machine identity management in a safe and trackable way.

Finally, because automation leads to consistent and reliable outcomes, you can participate closely with DevOps teams to more quickly pinpoint any security issues that occur during the development process. This improved communication repositions security as a collaborative partner aligned with DevOps team goals rather than a naysaying, distant voice that always seems to say “no.”


In short, security evolves in multiple critical ways that mirror those described above for IT, as the table below shows.

Application TestingBespoke testing of individual app/machine combos before going to stagingAutomating and creating secure code testing and review processes that result in cryptographically signed images
Application DeploymentFurther bespoke testing before going to productionAutomating and creating secure deployment policies based on the combination of signed code and machine identities
InfrastructurePerimeter security and a few locked down machinesAutomated and repeatable processes for creating, configuring, testing and approving “machines” which are automatically assigned a cryptographic identity
Overall GoalPrevent unknown bad things from happeningPrevent known bad things from happening, respond quickly when there are anomalies and build a resilient environment and organization

Perhaps most critically, security evolves from a focus on preventing issues to building resilient, reliable and sustainable processes and organizations.

Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms and Conditions and Privacy Policy.