Supporting App Metrics In Azure Event Hub Receiver For OpenTelemetry Collector
This article discusses the proposal to add support for application metrics (App Metrics) within the Azure Event Hub receiver for the OpenTelemetry Collector Contrib project. Currently, the receiver primarily supports metrics related to Azure resources. However, there's a growing need to ingest metrics emitted by applications, particularly those leveraging Application Insights and forwarding data to Azure Event Hubs. This article explores the problem, proposed solution, alternatives considered, and next steps.
Problem Statement
The primary challenge addressed is the need for centralized metrics storage and analysis across multiple Azure tenants and subscriptions. Many organizations have applications emitting metrics into their respective Application Insights instances. These metrics are then forwarded to an Azure Event Hub in a remote Azure tenant. To gain a holistic view of application performance and health, it's crucial to collect and consolidate these metrics into a central repository, accessible via a query language like PromQL.
The current Azure Event Hub receiver in the OpenTelemetry Collector Contrib primarily focuses on metrics related to Azure resources. It lacks native support for application-level metrics that originate from sources like Application Insights. This limitation necessitates a solution to parse and ingest these App Metrics effectively.
Gathering Metrics from Multiple Azure Environments
The core problem revolves around the need to gather metrics from multiple Azure tenants and subscriptions. Applications within these diverse environments emit metrics to their own Application Insights instances. These instances then forward a curated set of metrics to an Azure Event Hub located in a remote Azure tenant. This centralized Event Hub serves as a collection point for telemetry data from various applications.
Ingesting Metrics into Azure Managed Prometheus
To effectively analyze and monitor these metrics, the goal is to ingest them into an Azure Managed Prometheus data collection rule. This enables the use of PromQL, a powerful query language, for analyzing metrics and setting up alerts based on Service Level Objectives (SLOs). However, the OpenTelemetry Collector, acting as an intermediary, needs to be able to receive metrics from the Event Hub and transform them into a format suitable for Prometheus ingestion.
Handling Different Metric Formats
Azure employs different formats for metrics, depending on their source. The existing Azure Event Hub receiver is designed to handle metrics specific to Azure resources. However, it needs to be extended to support the format used for application metrics emitted through Application Insights. This requires parsing the JSON payload and extracting relevant metric data.
Avoiding Metric Duplication
Alternative solutions, such as forwarding metrics via Diagnostic Settings to a remote Log Analytics Workspace, introduce the issue of metric duplication. This approach duplicates all metrics and logs in the originating environment. Furthermore, it ingests metrics as logs, making it difficult to leverage PromQL for analysis. Therefore, a dedicated solution for handling App Metrics within the Event Hub receiver is crucial.
Proposed Solution
The proposed solution involves extending the parsing capabilities of the Azure Event Hub receiver to accommodate App Metrics. This entails adding a new structure to handle the specific format of metrics emitted by Application Insights. The core idea is to introduce a modular and extensible approach to parsing, allowing for the addition of support for other metric formats in the future.
Extending JSON Parsing
The primary approach is to extend the parsing of received JSON payloads. This involves creating a new data structure specifically designed to represent the format of App Metrics. This new structure will allow the receiver to correctly interpret the incoming metric data from Application Insights.
Adding a Struct for AppMetrics
A key component of the solution is the addition of a new struct (data structure) tailored for handling "AppMetrics." This struct will define the fields and data types necessary to represent the various metrics emitted by applications through Application Insights. This structured approach ensures that the receiver can accurately extract and process the metric data.
Implementing Extensible Code
To ensure future flexibility, the solution incorporates extensible code that facilitates the addition of support for other metric formats. This is crucial as metric formats may evolve, and new sources of metrics may be integrated into the system. By designing for extensibility, the receiver can adapt to changing requirements without requiring major code overhauls.
Handling Sum, Min, Max, and ItemCount
One particular challenge is the format of the App Metrics payload, which includes a set of values for each metric: Sum, Min, Max, and ItemCount. The initial implementation converts all metrics into gauges and multiplies them by four. While this provides a workaround, it might not be the most optimal approach. Future refinements could explore alternative methods for representing and processing these aggregated values.
Maturation of the Code
The developer has indicated a willingness to allow the code to mature for a few days before submitting a pull request (PR). This provides an opportunity to gather feedback, address any potential issues, and ensure that the solution aligns with the overall goals of the OpenTelemetry Collector Contrib project.
Alternatives Considered
Several alternative approaches were considered before arriving at the proposed solution. Each alternative has its own set of trade-offs, which influenced the decision to extend the Azure Event Hub receiver.
Forwarding Metrics to Log Analytics Workspace
One alternative explored was forwarding metrics with Diagnostic Settings to a remote Log Analytics Workspace. This approach involves configuring Azure services to send their metrics data to a central Log Analytics workspace for analysis. However, this method has significant drawbacks.
The primary issue is the duplication of metrics and logs in the originating environment. This duplication can lead to increased storage costs and complexity in managing the telemetry data. Additionally, Log Analytics ingests metrics as logs, which limits the ability to use PromQL for querying and analyzing the data. PromQL is specifically designed for time-series data, and ingesting metrics as logs makes it challenging to leverage the full power of the query language.
Utilizing Existing Azure Resource Metrics Support
Another alternative was to attempt to adapt the existing Azure resource metrics support within the Event Hub receiver. However, this approach was deemed less suitable due to the differences in metric formats between Azure resources and application-level metrics. The existing parsing logic is tailored for the structure of Azure resource metrics, and it would require significant modifications to handle the App Metrics format effectively.
Custom Metric Processing Pipeline
A third alternative considered was building a custom metric processing pipeline outside of the OpenTelemetry Collector. This would involve developing a separate application or service to consume data from the Event Hub, parse the App Metrics, and ingest them into Prometheus. While this approach offers the most flexibility, it also requires a significant investment in development and maintenance effort. It would also deviate from the goal of leveraging the OpenTelemetry Collector as the primary telemetry processing engine.
Additional Context and Next Steps
The proposed solution addresses a critical need for centralized metric collection and analysis in Azure environments. By extending the Azure Event Hub receiver to support App Metrics, organizations can gain better visibility into the performance and health of their applications.
The next steps involve submitting a pull request (PR) for the proposed changes. This will allow the community to review the code, provide feedback, and contribute to its refinement. It's essential to ensure that the solution is well-tested, robust, and aligns with the overall architecture and goals of the OpenTelemetry Collector Contrib project.
Community Feedback and Collaboration
Community feedback is crucial to ensure the solution meets the needs of a broad range of users. Open discussion and collaboration can help identify potential issues, suggest improvements, and ensure the solution is well-integrated into the OpenTelemetry ecosystem.
Testing and Validation
Thorough testing and validation are essential to ensure the solution is reliable and performs as expected. This includes unit tests, integration tests, and end-to-end tests to verify that the receiver correctly parses App Metrics and ingests them into Prometheus.
Long-Term Maintainability
Consideration should be given to the long-term maintainability of the solution. This includes writing clear and concise code, providing adequate documentation, and ensuring that the code is easily extensible to support future metric formats and requirements.
Conclusion
Adding support for App Metrics in the Azure Event Hub receiver is a valuable enhancement to the OpenTelemetry Collector Contrib project. It addresses a critical need for centralized metric collection and analysis in Azure environments. The proposed solution, which involves extending the JSON parsing capabilities and adding a new struct for AppMetrics, offers a flexible and extensible approach. By incorporating community feedback, conducting thorough testing, and focusing on long-term maintainability, this enhancement can significantly improve the observability of applications running in Azure.