Since 2010, Digital Enterprise Journal (DEJ) surveyed tens of thousands of organizations around different issues related to monitoring and improving the performance of digital services. Recently, DEJ conducted a new survey of more than 2,400 organizations centered around the current state of visibility into the performance of these services and findings of this research, combined with a trending analysis of historic data, will be included in DEJ’s upcoming study, “Managing Visibility Into Performance of Digital Services – The Case and Roadmap For a Fundamental Transformation”.

The study is leveraging DEJ’s Maturity Framework, which identifies a group of Top Performing Organizations (TPOs), a leading 20% of the research participants based on their performance. This DEJ’s Research Note summarizes some key takeaways from this upcoming study.

Where we are and how we got here

DEJ’s research identified a steady trend that shows a decline in the number of organizations that are satisfied with visibility into performance of digital services, which started in 2012. Over that period the volume of that decline has also been increasing, resulting in a 34% decrease over the last 3 years, in the number of organizations describing their visibility capabilities as effective. Simultaneously, the research shows a 79% increase in the business impact of issues with IT performance over the last 3 years.

Participating organizations also reported the number of challenges they are experiencing and the graphic below shows the issues that significantly increased the most over the last 3 years.

These challenges can be grouped into 3 key areas: 1) business drivers; 2) technology trends; 3) internal processes and strategies. Most of these challenges are not only causing a decline in organizations’ ability to manage the performance of digital services, but the research also shows that the magnitude of their business impact is constantly increasing.

In addition to these factors, a high number of organizations reported they are still struggling with addressing persistent challenges such as: 1) lack of actionable context of performance data (69% of organizations); 2) inability to prevent issues before users are impacted (65%); 3) identifying a root cause of performance issues (62%); 4) alert noise (61%); 5) identifying the impact of change (58%); 6) inability to correlate data from different sources (58%).

A trend lasting more than a decade, where organizations keep investing in monitoring tools while their performance stays on a declining trajectory, is driven by a variety of factors and the three most significant ones can be summarized as: 1) outcomes organizations are looking to drive; 2) a lack of enterprise-wide approach; 3) attributes of the solution selection approaches.

Outcomes that organizations are looking to drive

DEJ’s research shows that TPOs are more likely to focus their visibility efforts to drive outcomes, such as improving user experience, resilience and enabling innovation, while other organizations are primarily focused on availability of digital services and improving their competencies for troubleshooting performance issues. As a result, TPOs are 3.8 times more likely to generate new revenue from digital services, as compared to all others.

Putting more emphasis on outcomes that go beyond “firefighting” also has a stronger impact on an organization’s business goals. The research shows that organizations see optimizing operational cost and mitigating service outages as two primary areas of how visibility initiatives can contribute to the business. However, DEJ’s study reveals that improving user experience and reducing inefficiencies to support innovation have an even stronger business impact.

The role that visibility and monitoring solutions play in the enterprise

The study shows that lack of visibility into performance of digital services has an enormous impact on goals that are very high on business and IT executives’ agendas. The research shows that: 1) 38% of visibility capabilities that organizations purchased are not being used, resulting in $1.6 million of unnecessary spend, on average; 2) 72% of IT and business executives are not aware of all of the visibility tools that are being deployed in their organizations. However, only 16% of organizations are taking a unified, enterprise-wide approach for managing their visibility initiatives.

DEJ’s research shows a disconnect between business and technology executives and IT practitioners when it comes to goals and priorities around managing digital services. While business executives see the most value in creating unique user experiences, modernization, building more adaptive and customer-centric organizations and employee enablement, their organizations’ visibility efforts are mostly centered around service uptime and reducing cost.

This shows that visibility into the performance of digital services should be an enterprise-wide, strategic initiative that includes processes for bringing all key stakeholders on the same page and implementing capabilities that would help connect the dots between technology competencies and business goals.

Vendor selection processes

DEJ’s research shows that TPOs are 2.4 times more likely to be using “impact on business outcomes” as the key criteria for selecting visibility solutions, while other research participants are more focused on solving tactical issues they currently have on hand. As a result, these organizations reported deploying, on average, 3.6 different visibility solutions before they identified the one that is the best fit for their needs.

The research also shows that 34% of organizations replaced their visibility solutions over the last 24 months. 55% of these organizations reported “not a good fit for our use case” as the key reason for replacing these solutions.

Each organizations’ approach for evaluating visibility solutions should be based on two core areas:

• Rethinking outcomes they are looking to drive and using them as key selection criteria for evaluating deployments of visibility solutions
• Conducting situational analysis based on their specific use case(s) to identify the right mix of capabilities that are the best fit for them

Additionally, velocity and the amount of change they are experiencing, both internally and externally, is driving organizations to adopt a different approach for understanding the alignment of their visibility initiatives with broader technology and business trends. Instead of looking for a better “mouse trap” for addressing their persistent challenges, organizations need to take a harder look into new technologies that are entering the enterprise and understand requirements for managing them.

So, let’s look at where we are now. The research shows two major certainties: 1) the business impact due to a lack of visibility is constantly growing; 2) more than a decade worth of experience provides solid proof that playing catch up and approaching this issue a certain way is not working.

This calls for a fundamentally different approach and DEJ’s analysis of TPOs practices helped identify 8 key principles on winning strategies for improving visibility into performance of digital services.

Principles of a brand-new approach

Context

Seventy-one percent of organizations that participated in DEJ’ study, “17 Areas Shaping the IT Operations Market in 2018”, reported that their IT performance data is not actionable. The study also found that “more data can actually have a negative impact on the performance – unless this data is delivered in a context that is actionable and relevant.” Since the five years that this study was conducted, organizations now have a better understanding of this issue and the market has gotten to the point where the term “creating actionable insights” has found a place in most of the vendors’ marketing messaging.

Importance of the context and “actionable insights” is no longer news to organizations, but the question now becomes “what does that really mean for them?”. The research shows that 69% of organizations are still struggling with putting data into actionable context and the chart below shows some of the key capabilities that TPOs are more likely to be deploying to address this issue.

Visibility from the user’s perspective – Customer and Employee

TPOs in DEJ’s research reported that the quality of user experience (both for customers and employees) is the only true indicator of the success of their initiatives for visibility into performance of digital services. However, 73% of all other organizations reported that their visibility capabilities do not capture performance of digital services from the user’s perspective. This is one of the key reasons why visibility initiatives are failing. DEJ’s research reveals that organization are losing, on average, $17.39 million annually due to the lack of capabilities for a true user experience monitoring (also known as DEM – digital experience management), which represents an increase of 21% over the last three years, as shown in DEJ’s 2021 study, “The Total Business Impact of IT Performance” (this number includes only quantifiable “hard” cost and it doesn’t capture areas such as productivity, engagement, etc.). Some of the key areas that contribute to this number are shown in the chart below.

The research also shows that TPOs are 4.2 times more likely to have capabilities for monitoring employee experience, as compared to all others. As a result, these organizations reported 76% higher ROI from deployments of new enterprise technologies.

True visibility into both customer and employee experience has become a key prerequisite for any effective strategy for managing performance of digital services. However, solutions and capabilities for user experience monitoring come in different flavors and their deployments result in different levels of effectiveness. DEJ’s research reveals key attributes of capabilities that TPOs are more likely to deploy and their impact on operational and business goals.

AIOps and automation

Sixty-three percent of organizations reported “limitations of manual work” as a key challenge for their visibility initiatives. Dealing with large amounts of monitoring data collected by tools that are not designed to work together, understanding interdependencies and anomalies, and creating actionable insights is no longer possible without capabilities that fall under an umbrella of a broad technology category known as AIOps. DEJ’s 2020 study, “Strategies of Top Performing Organizations in Deploying AIOps”, identified that organizations are losing annually $1.27 million, on average, due to incident escalation that could be avoided by deploying AIOps capabilities.

However, since that study was conducted, AIOps deployments in the enterprise went on an eventful journey. DEJ’s 2020 study, “The State of IT Performance Management”, revealed a 2.9x increase in the number of organizations that included AIOps in their IT budgets. That number declined in both 2021 and 2022, while DEJ’s current research shows a renewed interest in these types of solutions. This can be contributed to the following factors:

• Maturing of technology and more targeted focus on its key value areas
• More transparency of AIOps solutions workflows
• Increasing complexity and the number of tasks that can’t be completed by “throwing more people at the problem”

Additionally, the research shows a steady increase in deployments of automation capabilities and TPOs are taking more intelligent approaches when it comes to automating tasks to improve their visibility initiatives. For example, these organizations are looking to automate their incident remediation processes only if they are provided with more transparency and control. In other words, they are looking to automate the path to a resolution, but want to have full control over actions that would be taken.

Proactive approach and resilience

DEJ’s research shows that TPOs are 3.8 times less likely to be using a number of trouble tickets as a performance metric, as compared to all others. For these organizations, just the fact that incidents are not prevented before users are impacted is a sign of failure. Building competencies for preventing performance issues comes down to two areas:

• Organizational approach. 41% reported that their approach for managing IT performance can be described as “addressing issues when incidents occur.”
• Technology capabilities. 31% of organizations reported that their goal is to proactively prevent performance issues, but they are lacking technology capabilities to consistently meet this objective

DEJ’s recent research shows that organizations are losing $36.34 million, annually due to the inability to prevent performance issues and, therefore, taking a proactive approach for visibility into digital service can no longer be optional.

Additionally, DEJ’s research shows that TPOs are 77% more likely to focus their visibility efforts to enable resilience and continuously assure optimal levels of user experience. As a result, these organizations are 60% less likely to experience application slowdowns, as compared to their peers.

Visibility for developers

DEJ’s research for this upcoming study includes insights from more than 400 software developers and engineers. 59% of them reported a lack of visibility capabilities as one of their key challenges. Even though many of their organizations are deploying APM solutions that include a code level visibility, developers are reporting that capabilities of these solutions are not effective and do not fit well into their workflows.

DEJ’s recent research shows that the business impact of issues with developers’ productivity and experience is enormous. Additionally, 30% of organizations in DEJ’s 2022 study, “24 Key Areas Shaping IT Performance Markets in 2022”, reported employee churn due to a lack of capabilities for developing and managing digital services. Therefore, organizations should put more emphasis on identifying and deploying monitoring solutions that are designed to address specific needs of software developers. Given the extent of the business impact as a result of not being effective in this area, enabling developers to identify and resolve issues in a timely manner should be raised to an executive level and included in enterprise-wide visibility strategies.

Business-centric approach

One of the key roles of visibility solutions is to provide insights for decision making. Traditionally, these insights have been used in areas such as root cause analysis and remediation. However, the study shows that TPOs are 3 times more likely to be leveraging these insights for making decisions in a business context. DEJ’s research identified several competencies that TPOs are more likely to be adopting that lead to significant improvements in operational and business metrics.

Data management and observability

Sixty-eight percent of organizations are reporting that the amount, velocity, and cardinality of data is one of the key challenges for managing visibility into the performance of digital services. At the same time, organizations still have gaps in data they need to improve visibility, as 51% and 59% of organizations listed: 1) blind spots in their digital delivery chains and 2) a lack of specific data needed for resolving performance issues as key challenges, respectively.

This DEJ’s upcoming study shows that data management capabilities are one of the key difference makers between top performing organizations and the rest of the market. This is one of the key drivers for an emergence of observability, as DEJ’s 2022 study, “Strategies of Leading Organizations in Adopting Observability”, shows:

• 64% of organizations have deployed, or are looking to deploy Observability capabilities
• “Creating actionable insights at scale” as the #1 value area of Observability deployments (76% of organizations)

However, Observability is still far from being a “silver bullet” for effectively managing IT performance data and improving visibility and business performance. That is due to the following reasons:

• As DEJ’s Observability study shows, not all the solutions that are marketed under the “observability” umbrella are created equal, which leads to market confusion and mixed results from deployments of these solutions. DEJ research shows that 37% of organizations have already replaced, or are looking to replace, Observability solutions which they deployed over the last 24 months.
• The concept of Observability was originally driven by the adoption of a cloud native approach and increased complexity, data management challenges and a novel method for building and managing digital services. DEJ’s research shows that there is still confusion between observability and monitoring, which leads to even more management challenges. The graphic below shows that these two are not competing concepts, but that they complement each other.

The chart above shows that, as compared to the findings of DEJ’s 2022 Observability study: 1) increased adoption of cloud native technologies and managing network performance as two use cases that are driving new adoption of Observability solutions; 2) increased awareness that Observability solutions are not “next generation monitoring”.

Aligning tools, teams, and processes

Even though this upcoming study identified a number of technology capabilities that are allowing organizations to improve visibility and performance of digital services, DEJ’s research shows that the success that TPOs are experiencing goes well beyond the technical functionalities they are deploying.

The study identified attributes of internal workflows, strategies and approaches that serve as key enablers of winning efforts to improve visibility and performance of digital services.

Keeping up with ongoing changes in business and technology environments is hard enough, but becomes even more difficult if organizations don’t put processes in place for continuously improving their ability to deal with new requirements.

Summary

Both DEJ’s trending data and the current research show that the market for visibility solutions is in a difficult position and a rapid, fundamental change is needed. The business impact is growing and starting to include new areas. For that reason, the monetary and competitive impact that used to be described as “a death by a thousand cuts” is rapidly progressing to the point where organizations, if they don’t transform their approaches for managing visibility, won’t be able to execute their core business strategies and could jeopardize their existence.

Performance of digital services is having a strong impact on every major enterprise initiative and key business goals. Therefore, visibility competencies can no longer be perceived as a tactical initiative because they are impacting each of the key business goals and should be a part of enterprise-wide strategies.