Runbook Automation as a Baseline for Controllability and Observability

How Transparency and Knowledge-Sharing Optimize Production Environments

Some of the highest priorities for engineers – from NOC Engineers, DevOps & Site Reliability Engineers – are the automation and optimization of their production environments. Many companies today face tough challenges with their Network Operations Centers (NOCs) or production environments. These challenges fall into the hands of engineering teams.

Some of the best practices implemented by engineers, especially in production environments that are expected to run 24/7, include controllability and observability. The practice of these two is the key that enables team members to gain transparency over the procedures that are part of the production environment. Observability is also crucial for sharing knowledge about these processes. A powerful tool that introduces more observability to production environments is runbook automation (RBA), also known as playbook automation.

What Is Runbook Automation? 

Put simply, a runbook is a list of procedures or actions that need to be carried out for every alert or a combination of multiple alerts. It is also often seen as a knowledge base that is constantly updated. Since NOC engineers often face the same types of malfunctions that require similar, if not identical, actions, creating and managing a runbook can make the process shorter and more effective. Runbook automation refers to the automation of these actions and procedures, and must be balanced between human and automated actions for an effective “hybrid” that increases efficiency and reliability.

The concept of runbook automation simultaneously provides advantages and challenges for the production environment. This practice’s main inherent advantage is to minimize some of the operational costs related to manual assignments of business processes.

For instance, runbook automation allows us to offload some of our daily assignments, lower the risk related to human error, and perhaps even enhance the quality of service we offer. Overall, runbook automation comes with the following three advantages:

• It offers users the capability to be proactive, i.e., taking action before a problem occurs by predicting issues based on known signs or identifiers
• It offers easy access to the operations capabilities needed to help you complete your tasks
• It automates workflows that extend throughout your existing manual commands and automation

It should be emphasized, though, that runbook automation is not intended to take the place of your current scripts, tools, manual commands, or API calls. That being said, runbook automation is quickly becoming the key interface between humans and tools to improve operations procedures.

In short, runbook automation involves the establishment of a workflow that integrates all processes, procedures, or tools, which all make essential parts of the production environment. In doing so, runbook automation makes the daily operations of a production environment more transparent, and thus more observable (and known), to all parties involved.

Which Challenges Does Runbook Automation Help to Solve?

When it comes to operations, re-organizing bits is the simplest part. After all, one already has the scripts, tools, and manual commands that manipulate files, copy artifacts, and call APIs. The problem is the following: Only a few select people in your company possess the know-how that’s needed to call upon and leverage those scripts, tools, and manual commands.

The knowledge isn’t shared, it’s siloed. There is no transparency about who’s doing what. There is a lack of both up-to-date knowledge and sufficient authorized access, which prevent others from being able to directly take part in any operations activity. As a result, everything (provisioning, incident management, diagnostics, maintenance, reporting, and more) falls to a few already overloaded and bottlenecked subject matter experts.

Some of the following challenges may sound familiar to your team:

• Bottlenecks develop around subject matter experts.
• Incidents take more time than necessary due to the fact that only a limited number of people can pursue action.
• Escalations are prevalent, which causes additional interruptions and disruptions, which consequently prevents planned work to improve business. By establishing a workflow and spreading out all processes, procedures, and tools across one big spectrum of activities, runbook automation provides the ability for all engineers on the team to observe exactly what is going on in every stage of production. The result is improved productivity both from an operational and technical standpoint.

What Does Production Look Like with Runbook Automation?

Once runbook automation is implemented, some of its immediate benefits include:

• Higher customer satisfaction – Due to increased uptime and availability by restoring service in a shorter time, and preventing service disruptions and outages before they happen.
• Less time wasted on waiting – Encourage action by replacing “open a ticket and wait” with “here’s the button to do it yourself”.
• There are fewer interruptions – Minimize the repetitive and tedious requests that take up your team’s time and hold up other work.
• Briefer incidents – Allow workers closest to the problem to be more proactive by quickly and effectively taking action.
• Reduced number of escalations – Avert disruptive and costly escalations that interrupt your overworked subject matter authorities.

Most importantly, all elements of the production environment have literally become “observable” to every engineering member involved. Teams are no longer siloed and limited to knowing only what is happening within their respective area of work. Instead, the production environment becomes orchestrated by one big family of engineers working on different parts of the puzzle that are visible to one another. In a 2015 McKinsey report titled “The Four Fundamentals of Workplace Automation,” it was asserted that by specifically using automation tools that were commercially available, marketing executives on average could save 15% of their labor time.

In the years since this report was published, there have been dramatic changes in the industry, namely the fact that most, if not all, SaaS products are now natively integrated or connected by third-party providers into other platforms and systems. If McKinsey were to perform that 2015 study again today, it is most likely that the 15% statistic would be much higher.

While there exists today vast potential to automate the production environment, we also need to understand and differentiate between what should and should not be automated. However, the point remains that to increase controllability and observability within your team and optimize your production environment, you will have to start automating somewhere.

How Do I Get Runbook Automation for Production?

The first thing is to take a step back and check your current levels of observability within the organization and, more specifically, within your NOC or production environment. Here are some questions you can ask:

• What is the SLA?
• Who is the owner of the service (i.e., responsible for it)?
• Can or should this process be automated?
• Can issues from this service be detected prior to service disruption?

The more your team is doing the above, the better your levels of observability are within the production environment.

Most likely, you want to implement runbook automation to establish a workflow and connect all the dots of the production environment. Therefore, first things first: Take note of all the existing processes, procedures, or tools that are currently being used by your engineers. The next step is to decide which ones are helpful in optimizing production (as some may actually be wearing production down). Once you have decided which elements are critical for production, you are ready to start streamlining these elements into your runbook and can initiate its automation.

Whether you are running applications on-premise or in the cloud, or are a scale-up startup or a multinational corporation, MoovingON.ai Platform takes away the pain of managing the day-to-day operations of your NOC.

Read here to learn more about the moovingon.ai Platform and how it can help your operation.