The Hidden Security Risk in Cloud-Native: Out-of-Date Software

The cloud-native world is highly focused on security, but a surprising number of organizations may be falling behind on one important task: Keeping their software up-to-date with the latest patches and security releases.

Buoyant, creators of the popular open source Linkerd service mesh, recently performed an analysis of Linkerd adoption in the wild. We examined the version information from tens of thousands of Linkerd clusters around the world that had been running for at least 30 days. Our analysis revealed exactly how prevalent out-of-date software is: Of these clusters, almost 50% were running versions at least six months out of date; 20% were running versions at least a year out of date and a full 10% were running versions that were out of date by 18 months or more. The full distribution of version age is summarized in the graph below.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

Linkerd is an interesting project for this analysis for two reasons. First, Linkerd is very widely adopted, and Buoyant has been publishing Linkerd releases for many years. The magnitude of this data allows us to perform a comprehensive analysis.

Second, and perhaps more interestingly, Linkerd is fundamentally a tool for security, used to add features like zero-trust authorization policy and drop-in mutual TLS to Kubernetes clusters. Because of this, we take security very seriously: We perform annual, public third-party security audits; we ship releases rapidly when CVEs are discovered in underlying dependencies and so on. If there’s anything you should keep up-to-date in your environment, it’s Linkerd.

If this distribution skew is what happens to Linkerd, one can only imagine what the rest of the ecosystem looks like!

The Challenge of Keeping Cloud-Native Software Up-to-Date

The risk of outdated software is not specific to cloud computing, of course. Nor is it a new category of security vulnerability. The Open Worldwide Application Security Project (OWASP), a nonprofit foundation that works to improve the security of software, lists “Vulnerable and Outdated Components” as sixth in its 2021 OWASP Top Ten list of common vulnerabilities, including in its definition of the failure to “fix or upgrade the underlying platform, frameworks, and dependencies in a risk-based, timely fashion.”

What is unique is the fast-paced nature of the cloud-native ecosystem. Kubernetes itself provides only one year of patch support for recently released “minor” versions and ships three such versions each year. At Buoyant, we release a new stable version approximately every eight weeks and produce new Linkerd “edge releases” almost weekly. The challenge of keeping software up-to-date in such an ecosystem is not insubstantial, especially when upgrading software is often considered a form of “toil,” the lowest class of work in the SRE ontology—something to be automated if at all possible.

A Light at the End of the Tunnel for Linkerd

For the open source Linkerd project, at least, we’re taking active steps to address this. In the upcoming Linkerd 2.13 release, Buoyant will be providing free, automated upgrade assistance, vulnerability reporting and health monitoring to all Linkerd adopters anywhere in the world via our Buoyant Cloud automation platform for Linkerd.

As the creators and stewards of the Linkerd project, we believe the days of the take-it-or-leave-it approach to open source are quickly waning. It’s not enough for us simply to make Linkerd available—we have a responsibility to ensure that every Linkerd cluster in the world is healthy, secure and up-to-date. That’s exactly what we plan to do.