Could Nomad Reignite Orchestrator Wars?

Most technology pundits agree that Kubernetes won the container orchestration wars, triumphing over alternatives like Docker Swarm and Mesos. Google even showcases a featured snippet for the exact moment of victory—November 29, 2017—the day AWS announced their Elastic Container Service for Kubernetes (EKS).

With growing adoption and a strong community foothold, Kubernetes appears to be the clear industry-standard container orchestrator. Yet, Kubernetes is no cure-all. Users cite mounting complexity with new features, developer experience qualms and security misconfigurations as potential roadblocks to implementation. It’s not that these drawbacks can’t be overcome, but most stakeholders admit Kubernetes’s inherent complexity makes onboarding to the platform and securing it rather cumbersome.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

For these reasons, some still use alternative container orchestrators. I recently spoke with Chang Li from HashiCorp on why she foresees more organizations dropping Kubernetes in exchange for more usable options. Below, we’ll consider some of the drawbacks to Kubernetes and see how it stacks up against Nomad as an alternative.

The State of Kubernetes

Nearly 90% of container deployments are now orchestrated, research from Datadog finds. Half of container deployments are running Kubernetes, either self-hosted or in a managed service such as EKS, GKE, or AKS. “The orchestration market matured rapidly,” describes Li, with Kubernetes becoming the dominant player.

Yet, Kubernetes introduces much complexity, which is heightened as new features are introduced. And, managing multiple tools and workflows could be a burden for some teams. “The more feature-rich ones offer more fine-grained control, but at the expense of operational complexity,” writes Cindy Sridharan.

Some organizations are turning to a full-cycle approach, wherein a developer owns their code through to production. For this new paradigm to work, developer experience for DevOps tools is crucial. Yet, the operational overhead associated with Kubernetes is very significant, describes Li. “We see a trend of K8s getting more complex as it grows in more features,” Li says.

In the rush to adopt Kubernetes, some red flags were ignored. Issues could include a lack of security best practices for this new environment, as 47.9% of Kubernetes security violations were due to insecure defaults, a recent study finds. Some teams lack good transparency into container workload utilizations. Most Kubernetes instances are underutilized, causing potential overspending. Kubernetes is also primarily focused on containers, leaving out scheduling for alternative formats.

Multi-Orchestrator Pattern

As a result, Li now sees a renewed multi-orchestrator pattern emerging. With the ubiquity of container scheduler usage, Li believes there is room in the market for alternative orchestrators to cater to specific needs. Multiple orchestrators may even coexist within the same organization.

At organizations heavily using Kubernetes, Li notes the growing adoption of Nomad for new projects among specific enterprise business units. These situations may require less fine-grained and more simplistic orchestration needs. Small and medium sized organizations without the staff to support Kubernetes may also turn to Nomad. For example, Roblox, Q2 Software, Cloudflare and Bowery Farming all use Nomad in production.

Nomad also is more flexible than Kubernetes when it comes to technology support. Whereas Kubernetes only supports certain container types (Docker, containerd, CRI-O, etc.), Nomad also supports non-container and legacy formats, making it a more generalized solution.

Kubernetes vs. Nomad

“Kubernetes has become the de facto standard for container orchestration,” says Datadog. Though Kubernetes is mature and versatile, it requires many components to manage. Here’s a general, high-level comparison between Kubernetes and Nomad:

Kubernetes

Benefits

• Fine-grained control.
• Superior community and support.
• Kubernetes has become a source of truth for some organizations, who are using it for other capabilities like RBAC.
• Expanding tooling community.
• Google, AWS, Microsoft all offer managed K8s versions.

Drawbacks

• High complexity and difficulty to onboard.
• K8s itself has many interoperating components to deploy and manage.
• Requires new security forethought, especially for self-hosting.
• Potential for runaway costs.
• You must learn new networking models (node IP, pod IP, internal/external IP).
• May require additional tools to manage.

Nomad

Benefits

• Simpler and easier to get started with.
• Less configuration may be better for smaller teams who can’t handle the complexity.
• Helps deploy, manage different workloads: Nomad supports workloads outside of containers, like Java, Windows apps and even binary.
• No new overlaid networking model to learn.

Drawbacks

• Immature in comparison to Kubernetes.
• Lacks diverse, strong community support.
• Narrower tooling community.
• Vendor-biased: intimately tied with HashiCorp ecosystem.

Never One Tool Fits All

When comparing divergent technologies, it’s good to recognize the complementary strengths. “There is always a space for a second solution; an alternative solution,” says Li. Since Nomad is “exactly the opposite of K8s,” Li believes it could fill some gaps that K8s won’t be able to offer in the long run.

If your team is already using containers in production, Kubernetes is a mature and versatile option. However, if you’re a small group or independent business unit wanting to get something together fast, especially one that supports various workloads, Nomad could be an excellent generalist approach.

When Kubernetes and Nomad, think of it this way:

“Kubernetes is primarily a platform for distributed containerized applications that happens to have a scheduler built in, whereas Nomad is primarily a flexible scheduler that can be used to build a platform comprised of hybrid deployment artifacts,” writes Cindy Sridharan.

When it comes down to it, the end goal is supporting an excellent application running on top, not the orchestration means, Li describes. Therefore, she foresees orchestration as becoming further abstracted from engineers in the future. Managed services have already taken a step in this direction. “Managed services alleviates a lot of operational complexity,” says Li.

Yet, the unique flavors of managed services make multi-cloud challenging to support. It requires substantial reconfiguration to migrate a workload from EKS to GKE, Li says. With some layers over Kubernetes, when something goes wrong, it could require double the effort to figure out problems, she adds. Kubernetes managed services or other container orchestration tools that hide operational complexity mustn’t cause additional troubleshooting.

“Organizations tend to chose an orchestrator based on their product, budget and timeline,” said Li. “It’s never one tool fits all.”