Hard Truths of Kubernetes Secrets Management

The recent CircleCI breach highlights the risk of storing secrets in places like private code repositories (GitHub), scripts, configuration files, files encrypted at rest, CI/CD pipeline code or even Kubernetes Secrets where they cannot easily be rotated, audited, authenticated and secured. CircleCI recently advised customers to “[i]mmediately rotate any and all secrets stored in CircleCI” and to “review internal logs for their systems for any unauthorized access.” CircleCI recommended that their customers not store any “long-lived” credentials in CircleCI.

CircleCI isn’t alone. There are several high-profile breaches related to unsecured secrets in the news recently:

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

• During the same week as the CircleCI breach, Slack reported a “threat actor downloaded private code repositories” after a “limited number of Slack employee tokens were stolen and misused.” Thankfully, no “downloaded repositories contained customer data [or] means to access customer data.” Sadly, this isn’t the only high-profile leak of source code, but this proves that source code is a terrible place to keep secrets.
• The Uber breach reported on September 19, 2022, was spread throughout the enterprise because hard-coded secrets embedded in a PowerShell script allowed the attacker to gain high-level access and escalate privileges. This breach is yet another good example of why you shouldn’t put secrets in code.

Kubernetes Secrets Aren’t Secure

There are two kinds of secrets in Kubernetes: Built-in and customized.

• Built-in secrets are automatically created by Kubernetes service accounts and attached to containers along with API credentials. These can be disabled or overridden as needed.
• Customized secrets enable you to define your sensitive data and create a custom secret to store it.

While Kubernetes Secrets are safer and more flexible than direct deployment in the Pod or Docker image creation, there are several drawbacks.

Kubernetes Secrets stores usernames and passwords as base-64 encoded strings. Although obscured from casual browsing, text encoding isn’t secure. This is even called out in Kubernetes Secret’s official documentation (see figure below).

[Text reads: Caution: Kubernetes secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.]

Further, stored secrets are only visible within the cluster where the secrets are kept. This isn’t a great fit for many cloud deployment scenarios. Organizations commonly have applications running containerized in Kubernetes but also have other applications built using serverless functions like AWS Lambda and other cloud-native tools.

There’s also a common misconception that if you simply encrypt the secrets at rest, you’re secure. But that means that you will still have to manually rotate the secrets as needed, and, as seen in the CircleCI breach, that can quickly get overwhelming and become a critical issue if you need to urgently rotate secrets.

Best Practices for Secrets Management in Kubernetes

Fortunately, developers can take several key steps to bolster the security of their Kubernetes environment, including:

• Understanding what authentication and authorization secrets you need to control and their ideal location in your Kubernetes cluster.
• Knowing the most secure way secrets can be configured, stored and managed for your deployed pods.

Regardless of where you store secrets, carefully map out exactly which containers need access to each of your secrets. Don’t share secrets anywhere they aren’t absolutely needed.

Using a centralized secrets management solution makes audit, access control and secrets management more manageable by giving organizations a centralized view of their overall Kubernetes security landscape. This helps you understand and control how individual containers and services interact with each other.

To limit unauthorized access and prove compliance, regularly audit critical systems access. Central audit trails provide visibility into critical security events.

Implement role-based access control (RBAC) and multifactor authentication, such as certificates. RBAC authorizes secret access based on a security policy detailing a role-based, time-based and task-based repository of access management. However, this isn’t enough protection if the secrets provider gives the secret to an imposter. Thus, you need to authenticate containers requesting secrets with strong multifactor authentication using multiple attributes only available to trusted containers, such as certificates.

And a simple rule that’s often overlooked: Rotate and change secrets regularly.

The Case for Centralized Secrets Management

Under a constant barrage of cybersecurity threats, enterprises must focus on what they can control to mitigate risk and potential exposure. Because there is no well-defined IT perimeter, the threat landscape is now asymmetric, making a zero-trust security mindset essential. Protecting non-human/machine identities and the secrets that represent them is critical to defending against the next attack. Any static credential or secret need to be removed from all scripts and source code. Security teams need to holistically manage access across the entire enterprise without silos or blind spots, with the ability to automatically rotate credentials as needed. A centralized secrets management system is the best option for authenticating, authorizing and auditing non-human access because it allows organizations to fully understand who has access to what and to automatically rotate or revoke access as needed.