Software companies succeed on their ability to ship valuable features quickly. Ideally, companies would give teams ample time to take security into consideration and think through all the implications of their code and configurations. But when the deadline pressure gets turned up, security becomes less of a priority.
While this may sound frightening, sometimes there are legitimate reasons to forego the tightest security constraints in favor of faster time-to-market. And there are steps companies can take to ensure that basic security hygiene gets practiced. Companies don’t need to maximize security at the cost of all else, but rather navigate the trade-off in a way that benefits users, employees, and the business. Let’s take a look at the tension between speed and security from an engineering standpoint, and identify how we can strike a balance that keeps us moving forward quickly and securely.
The incentive towards speed
Organizations have a natural bias towards speed at the expense of security. Peers and managers are bound to notice whether the team delivers on time or at a slower pace. Teams that ship fast are rewarded; but when they ship late, customer and other stakeholders may get upset.
On the other hand, it’s hard to prove the security level of the code to top managers. When security teams work well, security goes unnoticed. And even when security’s done poorly, it’s often months or years before flaws are discovered – or worse, breached. By then, the people responsible may have moved to other teams and companies and never experience the consequences of their mistakes.
Given the natural bias towards speed, here are four steps organizations can take to slow the process down and focus on security:
- Automate wherever possible. There’s no replacement for a thorough code review done by a seasoned engineer. But automated scanning tools can complement human reviewers. There are both open source and commercial tools out there for static code analysis, CVE scanning, configuration validation, and policy enforcement.
- Set reasonable policies. While development teams can find enforcing automation policies a challenge, it’s important to draw some hard lines around what teams can and can’t do without approval. For example, security teams might want to write policies to prevent the creation of world-readable S3 buckets, or disallow containers running as root in the company’s production Kubernetes cluster.
- Have regular security-focused learning sessions. The team needs to know all the ways security flaws crop up. While learning sessions can take away from valuable development time, they’ll help keep security top-of-mind for each of the development engineers. As a part of each session, ask the team to identify relevant parts of the company’s application that’s worth a second look given what they’ve learned.
- Give recognition to anyone who points out security concerns. If people don’t feel heard or appreciated when talking about security, they’ll stop speaking up. By acknowledging their concerns and praising their diligence in front of the group, managers will encourage others to adopt a security-focused mindset.
The more the team comes into contact with security-related ideas and information – whether through the findings of an automated scanner or through a group learning session – the more they’ll think about security as they go about their day-to-day work.
When to opt for speed
Speed tends to trump security at most organizations, so it’s important to consciously correct for that. But I often see engineers who simply invert the bias and insist on tackling every minute security detail no matter how much it holds up the business. Strike a balance between speed and security, so as not to maximize one at the expense of the other. When trying to navigate the trade-off between safety and speed, ask the following questions:
- What’s the type of threat? Not all security threats are equal. If there’s a chance an attacker could exfiltrate personal information about the company’s end users, hold the release. But if it’s a denial-of-service on a beta project, it makes sense to ship with a “ToDo” to tighten the code up later on.
- How likely is an exploit? This becomes a particularly important question when dealing with security findings from automated scanning tools. Perhaps Trivy has discovered a CVE related to Git in one of the company’s Docker images – if the application never uses Git, the chances of it being exploited are near zero.
- What upstream protection does the company have? Some security issues sit far downstream from the first line of defense. For example, if the company runs an application with known vulnerabilities, it probably doesn’t want to expose it to external traffic. But if it’s running behind a firewall and isolated from other parts of the infrastructure, the team may decide it’s a manageable risk; the firewall will keep out any external attackers, and the team can minimize the splash damage of an attack through isolation.
In the cases where the company does opt for speed over security, it’s critical that it documents those decisions, creates tickets and “ToDos” to fix them, and keeps the larger team aware of potential issues. As applications evolve, the answers to the questions above might change, so it’s important that all decisions affecting security are recorded and tracked.
Navigating the trade-off between speed and security takes some work. It requires overcoming deep institutional biases, cultural norms, and personal habits. And the natural bias towards speed means the team will have to constantly reevaluate its security posture and consider areas that might have gone overlooked in the rush to ship.
Some may still find the idea of an acceptable level of security risk a hard pill to swallow. But it’s not possible to eliminate risk – the best security and ops teams simply know how to manage risk effectively. By asking smart questions and establishing good habits, the team can ship at a competitive speed while maintaining a strong security posture.
Robert Brennan, director of open source, Fairwinds
