Best of 2023: Unified Kubernetes Monitoring With OpenTelemetry

As we close out 2023, we at Cloud Native Now wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2023.

A year ago, I told you that OTel was at the core of Sumo Logic’s collection strategy and that we were betting our future on OpenTelemetry. Back then, I said that unified collection was a unicorn idea that the industry was trying to pursue. Today, I am happy to share that we have launched fully unified Kubernetes monitoring for all signals (logs, metrics, traces, events)! With the advent of our latest Helm chart release, we not only have fully-unified collection, but the project includes auto-instrumentation and more efficient collection than prior Helm chart releases.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

“But what is a Helm chart, and why does unified collection matter for Kubernetes monitoring?” you ask.

Understanding Helm Charts and Unified Collection

Think of Helm charts as the package manager of the Kubernetes universe. For those working with Linux-based systems, it’s akin to the “apt” or “yum” for Kubernetes. They are designed to encapsulate and simplify the deployment, installation and upgrade processes for Kubernetes applications.

Practitioners still need to monitor and manage their Helm chart versions while adhering to a semantic versioning convention. This method offers significant manageability, allowing operators and developers to precisely deploy specific application versions, ensuring both compatibility and stability. Furthermore, if a new chart version presents breaking changes or encounters issues, operators have the flexibility to effortlessly roll back to a prior stable version. Additionally, Helm charts can be interdependent on other charts, and this structured versioning ensures that dependent charts harmoniously align with their correct versions.

OpenTelemetry has now expanded beyond unifying receivers and making a unified collection of signals. Now, there is an increase in automation via auto-instrumentation and a relatively new working group for semantic conventions. Each of these working groups works with Helm charts quite closely to achieve the original goal of the OTel project (which is to unify collection).

The Advent of Unified Collection for Kubernetes Monitoring

The image on the left shows the chaos of disparate and separate collectors. You have a hodgepodge of vendor-provided collection, instrumentation and backends. This is what data collection looks like without using the OTel standard. Your telemetry pipeline is backed up with myriad agent-specific code, and you are likely still using manual instrumentation held together by various ad-hoc scripts.

On the right, you can see what the architecture for a fully unified OpenTelemetry pipeline looks like. This is what the latest Helm chart release is able to provide for Sumo Logic users. Sumo Logic’s commitment to open source projects further solidifies the potential of unified metrics, events, logs and traces (MELTs) collection processes in the foreseeable future. This enables correlated telemetry, auto-instrumentation and a single agent to allow for a smaller, more efficient data collection footprint.

Moreover, teams represented by the image to the right are likely using auto-instrumentation to help with their Kubernetes infrastructure monitoring. This saves time significantly over manual instrumentation. Something that used to take hours now takes minutes! Some auto-instrumentation SDKs Sumo Logic supports or has contributed to include: Go, Istio, Kubernetes Tracing, JavaScript, Java, .NET, Python and Ruby on Rails. Notably, we also recently added PHP auto-instrumentation upstream; it is explained in detail with code examples and installation steps in this opentelemetry.io blog post by Przemek Delewski.

The transition to a unified monitoring approach with Kubernetes marks a pivotal moment in how we approach and manage data collection. This cohesive framework, backed by auto-instrumentation and semantic conventions, will not only unify collection but begin to define the future of the OTel project.

The Future of OpenTelemetry

Now that unified collection is possible through a single agent, what’s next for the OTel project and community? Is OTel still relevant?

As we step into the future, OTel will have to grow beyond unifying receivers and collection agents. The new wave of work coming from the project seems to be more focused on ease of use and automation. It is already starting to focus on standards for agent collector management with Open Agent Management Protocol (OpAMP) compliance.

The OTel standard seems to be trending toward remote management of vast fleets of data collection agents that align with the OpAMP protocol. This ensures ease of use via efficient agent management by facilitating real-time agent status reporting and configuration management en masse. Additionally, OpAMP touts a variety of security features, including auto-updating capabilities that support both upgrades and downgrades, TLS certificate rotation and a robust connection credential management system. These automation features enable secure communication between agents and servers, thereby minimizing potential threats.

The most notable feature OpAMP unlocks for observability vendors is streamlined remote configuration. OpenTelemetry agents have intricate functionalities and demand precise configurations to capture data accurately. Unified collection on fewer agents is great, but more is needed for ease of use and configuration for agents themselves. Dynamic adjustment capabilities from OpAMP based on agent status reports and the ability for these agents to report to an OTLP-compatible backend means that users can keep track of agent-specific metrics dynamically (such as resource usage and data processing rates). This ensures that agents function within desired thresholds and fosters proactive management without practitioners experiencing burnout.

OTel agents often undergo updates or require new packages to enhance their data collection capabilities. OpAMP simplifies this process, making the distribution and installation of these packages straightforward and reduces the need for SRE acrobatics due to extensive manual intervention. Standardized protocols like OpAMP will become indispensable to unlocking ease of use for the future of simplified, secure, and efficient agent management.

If you would like to learn more about Kubernetes monitoring best practices, check out my e-book. By reading that, you can learn more about the history of Kubernetes, a primer on Kubernetes architecture, and how to leverage OTel for K8s infrastructure monitoring. I also include a section on future trends and where the cloud observability industry seems to be heading.

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2023 – November 6-9, 2023.