Overcoming Kubernetes Governance Challenges

Enterprises are quickly discovering new Kubernetes use cases to increase agility and application performance. While Kubernetes provides more freedom to run applications across a variety of infrastructures, broader adoption means a proliferation of new clusters and increased complexity in how containers are run, monitored and managed.

While managing one Kubernetes cluster is not trivial, trying to manage multiple Kubernetes clusters on hybrid or multiple clouds becomes exponentially more difficult.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

There are a number of common challenges organizations run into when attempting to efficiently rein in multiple clusters. According to research from D2iQ, as many as 94% of surveyed organizations using cloud-native technology noted that Kubernetes is a source of complexity for their organizations. Anticipating these challenges allows for quick resolution should they arise.

Developers should keep the following in mind when developing a strategy to overcome these three common governance challenges in Kubernetes deployments:

Lack of Visibility and Management

As the number of clusters grows and spreads, managing and tracking their activity and growth becomes increasingly difficult. It is also harder and more time-consuming to troubleshoot any problems that may arise; if different software is involved, a single solution cannot be applied to each version. Lack of centralized governance and visibility into what’s happening within the Kubernetes environment negatively impacts application availability and performance and, ultimately, the organization’s bottom line.

Problems also arise if a cluster goes down unexpectedly—troubleshooting problems requires time and resources, and if there are dozens of potential software versions in use, managing all of them across the organization is increasingly difficult. Unlike planned downtime, when teams are generally able to understand what is needed to mitigate impacts for customers and projects, unplanned downtime makes this preparation impossible. Operators must be able to consistently administer, manage and obtain insights about their infrastructure.

Operational Complexity and Overhead

Spreading numerous Kubernetes clusters across different business units creates challenges in user identity tracking—especially if users onboard, offboard or change teams. Operators lose the flexibility to define user roles, responsibilities and privileges to ensure the right people are performing the right tasks within the environment. They also face difficulties identifying role violations, assessing governance risks and performing compliance checks. When more time is spent chasing issues or putting out fires, there is less time for efficient operations.

Empowering Both Developers and Operators

As developers work to implement Kubernetes, there must be a balance between the developer’s independence and the operator’s ability to easily manage the policies and procedures necessary to maintain the overall system. While the autonomy of developers is crucial, this freedom shouldn’t come at the expense of the environment’s security. Multiple solutions across multiple clusters often lead to more opportunities for attacks to slip through the cracks unnoticed. The alternative approach, where there is more consistency for a few select solutions, allows standardization across organizational clusters. The challenge comes when addressing the fine line between a developer’s ability to innovate and an operator’s ability to maintain the necessary procedures and governance.

Faced with these challenges, enterprises need concrete solutions to more efficiently manage Kubernetes’ growth. Without an actionable strategy, enterprises will be unable to troubleshoot problems and lack a framework for consistently tracking and implementing necessary procedures. Additionally, there is no clear process for managing the relationship between developers and operators, leaving their respective roles without structure and, in some cases, at odds with one another. To address these potential pitfalls, there are a number of tried-and-true steps that enterprises can implement:

Multi-Cluster Visibility and Management: Operators need the ability to centrally view, manage and consolidate disparate clusters as they are discovered to better optimize resources in a cost-effective manner and troubleshoot issues without losing valuable time. To mitigate the effects of unplanned downtime, teams can:

1. Create a runbook of worst-case scenarios and assign team members responsibilities in the event of an emergency 
2. Make sure the patch release process is well-defined so solutions can be rolled out to customers quickly
3. Run a post-mortem once the unplanned downtime is resolved to reflect on what was learned and what could be improved next time

Configuration Management: Operators need a close-up view and understanding of potential vulnerabilities in the software, enabling them to troubleshoot problems more quickly and more efficiently based on available resources. Providing operators with this control enables organizations to meet compliance standards and simplify the provisioning of services.

Authentication and Access Management: Operators need to simplify individual logins and permissions to service the needs of a wide range of clusters with centralized policy-driven capabilities.

Building and Maintaining Line of Business Relationships: Operations doesn’t want to hinder ITs efforts, and should work to streamline management tactics without restricting the available technology. Cultivating balance between developers and operators is key to successfully managing the overall environment.

The increasing adoption of Kubernetes is giving developers the freedom to create their own environments, but it has placed additional responsibilities on operators and managers. To manage organizational and security requirements, IT professionals and operators must work in tandem to maximize the impact of Kubernetes deployments—starting with a proper Kubernetes governance plan is a must.

Including developer teams and key stakeholders from the beginning of the planning process is also critical, and is the surest way to achieve successful Day 2 operations in production environments.