It’s easy to believe that releasing software at a slower pace means the software gets released more securely. While it’s sometimes counter-intuitive, my experience has been the exact opposite: quick releases are beneficial for shipping secure products.
Back in 2014 when I was running security and privacy for social products at Google, OpenSSL was hit with a rather nasty security vulnerability. We needed to update the application code immediately. Our fixed Android apps were publicly available within a few hours while the iOS apps were in review with Apple.
Unfortunately, the rest of the Google team took weeks to remediate the risk and many companies took far longer. What allowed us to move quickly while the rest of the organization was behind? The social products at Google had the kind of CI/CD release setup for mobile applications that many organizations even today only dream of. Pair that with a truly excellent team of release engineers and we were able to get the job done. The organization as a whole, however, did not.
And there’s no single entity to blame. Excellent engineers who cared about their users and security worked on each of these teams, but the difference was the social teams already had processes in place to make, test, approve, and roll out releases quickly.
Patching software in production, especially low-level infrastructure like the kernel and operating system, is deeply challenging. Subtle changes can cause instability and performance problems, leaving teams reluctant to patch on a relatively frequent basis because every time they do they may end up fighting fires. But the problem compounds: when the team releases less frequently, it also needs to release more changes at once. And as a result, there are only more corresponding chances for something to go awry.
When a security team needs to roll out an urgent security patch, we see the logical and painful culmination of this cycle: many changes must get implemented at once and teams are not practiced or well-versed in these adjustments. All of this combines to make security patches slower and riskier than needed.
Compounding this issue: it’s not necessarily apparent that any particular patch is a security patch. The Linux kernel, for instance, has been notorious for not labeling security fixes, but the problem has become more widespread because a developer may legitimately not realize they are fixing a security issue. Instead, they might be removing or fixing an old feature or cleaning up old code. That’s why it’s best practice to keep up on patches in general.
As a security professional, it’s easy to believe and worry that faster releases are less safe. Here’s the reality: such concerns are valid unless the proper protections are in place. The pace of releases in a continuous deployment world can only stay reasonable with automated protections like code scanning, extensive testing, and production monitoring. Humans can’t operate at that speed and frankly can’t do as good of a job at fixing certain flaws, as even the best engineers haven’t managed to wire their brains into the systems. However, organizations should place humans in the mix to do the tasks we are best at, like talking through designs and reviewing particularly interesting bits of code.
The cloud disrupted tech with infrastructure that moves faster than anyone could have imagined 20 years ago. Now, it’s become a security feature and business imperative for our software releases to at least match that speed.
Lea Kissner, chief information security officer, Lacework
