Enhance OpenTelemetry Span Filtering For AsyncPG With Attribute Hydration

In the realm of distributed tracing, OpenTelemetry stands as a powerful tool for gaining insights into the performance and behavior of applications. When working with asynchronous PostgreSQL databases using the AsyncPG library, instrumenting your application with OpenTelemetry can provide valuable visibility into database interactions. However, a challenge arises when filtering spans created by the AsyncPG instrumentation due to the timing of attribute addition. This article delves into the problem, proposes a solution, explores alternatives, and discusses the implications of enhancing OpenTelemetry span filtering for AsyncPG.

The Problem: Delayed Attribute Hydration

The core issue lies in how spans are created and populated with attributes in the current AsyncPG instrumentation. Spans are initially created without attributes, and the attributes, which contain crucial information about the database operation, such as the table name and action, are added immediately after using span.set_attribute. This approach presents a significant hurdle for samplers, which are responsible for determining which spans should be recorded and exported. Samplers typically rely on request attributes that are known at span creation time to make informed decisions.

Because attributes are added after span creation, samplers have no access to these attributes during the sampling decision process. In practice, this means that there is currently no effective way to filter spans based on specific database operations, such as queries on particular tables or specific actions like inserts or updates. This limitation hinders the ability to focus on relevant spans and can lead to an overwhelming amount of tracing data, making it difficult to identify and address performance bottlenecks or issues.

The inability to filter spans effectively impacts several key areas:

• Performance Analysis: Identifying slow queries or specific database operations that contribute to performance degradation becomes challenging.
• Error Monitoring: Pinpointing database-related errors and their context is more difficult without filtering capabilities.
• Cost Optimization: Exporting a large volume of unfiltered spans can increase tracing costs.
• Targeted Debugging: Focusing on specific areas of interest during debugging is hampered by the lack of filtering.

The Proposed Solution: Attribute Hydration During Span Creation

To address this challenge, the proposed solution is to perform attribute hydration, the process of adding attributes to a span, during span creation itself. Instead of creating a span without attributes and then adding them immediately afterward, the attributes would be included when the span is initially created. This approach ensures that samplers have access to the necessary information for making sampling decisions.

Benefits of Attribute Hydration During Span Creation

• Effective Span Filtering: Samplers can leverage request attributes, such as table names and actions, to filter spans based on specific criteria.
• Accurate Timing: Including attribute hydration in the span creation process provides a more accurate representation of the actual request time, as it encompasses the time taken to gather and add the attributes.
• Improved Performance Analysis: Filtering spans based on database operations enables targeted analysis of performance bottlenecks.
• Enhanced Error Monitoring: Pinpointing database-related errors and their context becomes easier with filtering capabilities.
• Cost Optimization: Reducing the volume of exported spans through filtering can lower tracing costs.

The key to implementing this solution lies in the efficiency of the attribute hydration function. The assessment is that the function does not incur a meaningful performance penalty, making it a viable option for inclusion in the span creation process. By moving attribute hydration to span creation, the timing is also more accurate, reflecting the actual request time more closely.

Exploring Alternatives

While the proposed solution offers a direct and effective way to address the span filtering issue, it's essential to consider alternative approaches. One alternative involves relying on other logic, such as contextvars and code changes, to filter spans. However, this approach has limitations and drawbacks.

Limitations of Alternative Approaches

• Complexity: Implementing filtering logic using contextvars and code changes can introduce complexity and make the code harder to maintain.
• Intrusiveness: This approach often requires modifications to the application code, which can be undesirable.
• Limited Scope: Contextvars may not always be available or suitable for capturing all the necessary information for filtering.
• Inconsistency: Relying on manual filtering logic can lead to inconsistencies in how spans are filtered across different parts of the application.

Without a mechanism to filter spans based on attributes known at span creation, users are forced to resort to these less-than-ideal alternatives, which can be cumbersome and error-prone. The proposed solution of attribute hydration during span creation offers a more robust and consistent approach to span filtering.

The Importance of Accurate Request Timing

Another significant advantage of performing attribute hydration during span creation is the improved accuracy of request timing. The current approach, where attributes are added after span creation, may not fully capture the time taken to gather the attributes themselves. By including attribute hydration in the span creation process, the resulting span duration more accurately reflects the total time spent on the database operation.

Accurate request timing is crucial for several reasons:

• Performance Monitoring: Precise timing data is essential for identifying performance bottlenecks and optimizing database interactions.
• Service Level Agreements (SLAs): Accurate timing is necessary for monitoring and meeting SLAs related to database performance.
• Root Cause Analysis: When investigating performance issues or errors, accurate timing information can help pinpoint the source of the problem.

By ensuring that spans accurately represent the duration of database operations, the proposed solution enhances the value of OpenTelemetry tracing data for performance analysis and troubleshooting.

Implementing the Solution

The implementation of this solution would involve modifying the AsyncPG instrumentation to perform attribute hydration during span creation. This would likely entail updating the code that creates spans to include the logic for gathering and adding attributes before the span is started. The specific steps involved would depend on the structure of the existing instrumentation code.

Key Considerations for Implementation

• Performance Impact: While the assessment is that the attribute hydration function does not incur a significant performance penalty, it's essential to thoroughly evaluate the impact of the change on application performance.
• Compatibility: The implementation should maintain compatibility with existing OpenTelemetry APIs and configurations.
• Testability: The changes should be thoroughly tested to ensure that spans are created correctly and that attributes are added as expected.
• Maintainability: The implementation should be designed to be maintainable and easy to update in the future.

By carefully considering these factors, the implementation can be carried out in a way that minimizes risks and maximizes the benefits of the solution.

Conclusion

Enhancing OpenTelemetry span filtering for AsyncPG by performing attribute hydration during span creation offers a significant improvement in the usability and effectiveness of tracing for asynchronous PostgreSQL databases. This approach addresses the current limitations of span filtering, provides more accurate request timing, and simplifies the process of analyzing and troubleshooting database-related issues. By enabling samplers to make informed decisions based on request attributes, the proposed solution empowers users to gain deeper insights into their applications and optimize their database interactions. The shift towards attribute hydration during span creation not only aligns with best practices for span management but also enhances the overall observability of applications utilizing AsyncPG, making it a crucial step forward for OpenTelemetry instrumentation.