Scheduled Maintenance? That’s Another Way of Saying Technical Debt

Recently, I wrote about maintenance windows and how they should not be used for modern, cloud-native applications. I talked about why maintenance windows are a bad idea and explained why your customers still consider maintenance windows as downtime. Customers want to access your web application on their schedule, not your schedule.

No truly modern, cloud-native application should require downtime to perform a product upgrade or routine maintenance. A modern, cloud-native application should have no trouble operating 24/7, with no need for scheduled maintenance windows, ever.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

Increasing Technical Debt

If your application requires maintenance windows due to some historical architectural issue, then you should address it for what it truly is: An architectural problem. This downtime requirement is actually technical debt imposed on your application by your application’s systems and processes. This technical debt is costing you customer satisfaction and it’s costing you money. After all, your customers don’t care why your application is down. They just care that it is down.

As your application grows and expands, it will be harder to justify having a regular downtime window. Just like any other technical debt, you’ll make your application easier to maintain, grow and scale by reducing the technical debt associated with the application—and this includes removing the need to use maintenance windows to make changes.

You must be able to expand your application without impacting your customers.

But my Change Really Does Need Downtime to Deploy!

That’s rarely the case. In most cases, the need to bring an application down to perform an upgrade isn’t a necessity. Rather, it’s a shortcut. It’s generally substantially easier to design migration strategies if your application is down during the upgrade than it is to design migration strategies that don’t require downtime. Put another way, it’s much easier to bring the application down than to try and do the same work while it is still operating.

Designing downtime-less migration strategies can be a challenge. They take more time to implement and may use more resources. Hence, they cost more money to implement. But, in the vast majority of cases, this additional effort and cost is worth it because:

1. More Thought and Attention. Requiring downtime-less migrations forces the team to spend time on the entire migration solution. Applying this additional attention may allow other potential issues or concerns to surface that were not previously considered during the simpler and quicker discussion using downtime.
2. Lost Customer Opportunity. There is a lost-opportunity cost caused by losing or inconveniencing customers when you bring your application down. The lost-opportunity cost is, for most successful companies’ applications, substantially greater than the development costs involved in using a better migration strategy.

The first point is worth emphasizing. We, developers, tend to get lazy. When we believe we can freely take down our application to perform an upgrade, we take advantage of that and build a simpler upgrade mechanism that assumes the application can be down. If, instead, we do not allow for the option of bringing down the application, then developers must focus on the upgrade process more than they otherwise normally would.

When developers are required to think about the operational impact of a change, they tend to produce fewer operational issues than when they simply “throw it into production” and do not consider such ramifications. When you depend on maintenance windows, overall quality and availability suffer.

Sometimes, You Don’t Have a Choice

But I recognize that, sometimes, you don’t have a choice. Unfortunately, many applications simply can’t be upgraded without impacting the availability of the application. Some applications, including relatively modern applications, simply must be brought down to perform some types of upgrades.

The majority of changes that require downtime involve data migrations. Changing the size, shape or structure of data is often difficult or nearly impossible to do without bringing the application—or parts of the application—down, even briefly.

Still, even in this case, don’t use predefined maintenance windows. Having a predefined, predetermined, regularly scheduled maintenance window is simply an invitation to allow teams to make use of the window. Even for upgrades that don’t involve data manipulation and could, in fact, be done without any downtime, the implemented migration plan typically will require downtime more often if your organization has predefined organizational maintenance windows than if you don’t have such windows defined.

The default answer should always be “No downtime to perform an upgrade.” Upgrades that must require downtime to perform should be treated as anomalies and go through a special, much more rigorous approval process. The bar to allow an upgrade that forces application downtime should be very high, and the process to overcome that bar should be highly scrutinized.

It should not be easier to “just bring the app down” than it is to think deeper about the problem trying to be solved.