Automate Issue Labeling With Claude Code Triage Workflow

by gitftunila 57 views
Iklan Headers

Overview

This document outlines the plan to vendor and adapt Claude Code's issue triage workflow for automatically labeling new issues based on content correlation, leveraging our existing MCP infrastructure. By automating this process, we aim to improve consistency, discoverability, and efficiency in issue management.

Parent Issue

This task is a sub-task of #864 (SPIKE Evaluate Claude GitHub Actions agent), which focuses on evaluating the potential of Claude GitHub Actions agents within our workflow.

Reference Implementation

This implementation will be based on the existing workflow found in https://github.com/anthropics/claude-code/blob/main/.github/workflows/claude-issue-triage.yml, providing a solid foundation for our automated labeling system.

Goal

The primary goal is to create a system that automatically labels new issues created in this repository. This system will:

  1. Analyze the Issue Content: Thoroughly examine the content of the newly created issue.
  2. Apply Relevant Labels: Automatically apply the most relevant existing labels based on the content analysis.
  3. Utilize GitHub MCP Server: Employ our GitHub MCP server for all operations, ensuring consistency and control.
  4. Maintain Full Control and Customization: Allow for complete control and customization of the labeling process.

Implementation Plan

The implementation will follow a structured approach, encompassing the following key steps:

1. Vendor the Workflow

  • Create a new workflow file: .github/workflows/issue-auto-label.yml.
  • Properly credit Anthropic's original implementation to acknowledge their contribution.
  • Adapt the workflow to align with our specific tooling and conventions.

2. Key Adaptations

  • Use Our GitHub MCP Server: Instead of making direct API calls, the workflow will utilize our GitHub MCP server for all interactions.
  • Integrate with Our Principles: Ensure the workflow aligns with our core principles of transparency and the snowball method.
  • Add _meta Field Support: Implement support for the _meta field to enhance observability and tracking of the labeling process.
  • Link Labeled Issues: Automatically link labeled issues to related past issues to create a network of connected information.

3. Enhanced Features

  • Analyze Against Knowledge Directory: Extend the analysis to include content from our knowledge/ directory, enabling more context-aware labeling.
  • Suggest Relevant Principles: Propose relevant principles that might apply to the issue, promoting adherence to our guiding principles.
  • Auto-Link Similar Issues: Automatically link to similar past issues, facilitating knowledge sharing and reducing redundant efforts.
  • Create Audit Trail: Maintain a comprehensive audit trail of all labeling decisions, ensuring transparency and accountability.

Example Workflow Structure

name: Auto-label Issues
# Adapted from https://github.com/anthropics/claude-code/.github/workflows/claude-issue-triage.yml

on:
  issues:
    types: [opened]

jobs:
  label-issue:
    runs-on: ubuntu-latest
  steps:
    - name: Analyze and Label Issue
      # Use our MCP server to:
      # 1. Get issue content
      # 2. Compare against existing labels
      # 3. Apply most relevant labels
      # 4. Add comment explaining labeling

This YAML structure provides a basic outline of the workflow, showcasing the key steps involved in automatically labeling issues.

Benefits

Implementing this automated issue labeling system offers several significant benefits:

  • Consistency: Issues will be labeled consistently, adhering to a standardized approach.
  • Discoverability: Related issues will surface automatically, improving knowledge sharing and collaboration.
  • Snowball Effect: Each labeled issue contributes to improving future labeling accuracy and efficiency, creating a positive feedback loop. The snowball effect ensures that the system becomes more intelligent and effective over time, as it learns from past labeling decisions and adapts to new information. This continuous improvement is crucial for maintaining the relevance and accuracy of the labels, especially as the project evolves and new issues emerge.
  • Time Savings: Manual triaging efforts will be significantly reduced, freeing up valuable time for other tasks.

Success Criteria

The success of this implementation will be measured against the following criteria:

  • [ ] Labeling Speed: Issues should be labeled within 1 minute of creation, ensuring timely categorization.
  • [ ] Label Relevance: Labels should be relevant 90% or more of the time, demonstrating the accuracy of the system. Achieving a high level of label relevance is crucial for the system's effectiveness. When labels accurately reflect the content and context of an issue, it becomes easier for team members to find and address relevant tasks. This not only improves the overall efficiency of issue management but also ensures that important issues receive the attention they deserve. The 90%+ target ensures that the automated labeling system is not only fast but also highly accurate, providing a reliable foundation for issue triage and workflow management.
  • [ ] Transparency: Labeling decisions should be transparent, with clear explanations provided in comments.
  • [ ] Compatibility: The system should work seamlessly with our existing label taxonomy, ensuring consistency across the repository.

Implementation Steps

The implementation will proceed through the following steps:

  1. [ ] Study Claude Code's Workflow: Thoroughly examine Claude Code's workflow to understand its structure and functionality. This step involves a detailed review of the YAML configuration, the logic behind the labeling process, and any dependencies or external integrations. By gaining a comprehensive understanding of the reference implementation, we can identify best practices, potential challenges, and areas for improvement or adaptation. This foundational knowledge will be crucial for successfully vendorizing and customizing the workflow to meet our specific needs.
  2. [ ] Create Adapted Version: Develop an adapted version of the workflow in our repository, incorporating our specific requirements and conventions. This includes modifying the YAML configuration, integrating with our MCP server, and adding support for our principles and metadata. The goal is to create a customized version that aligns seamlessly with our existing infrastructure and workflows, while also leveraging the core functionality of the Claude Code implementation. This step requires careful attention to detail and a thorough understanding of our repository's structure and conventions.
  3. [ ] Set Up GitHub Action: Configure the GitHub Action with our MCP server, ensuring proper integration and communication. Setting up the GitHub Action involves configuring the necessary environment variables, authentication credentials, and permissions to allow the workflow to interact with our MCP server. Proper integration is essential for the workflow to function correctly, as the MCP server is responsible for analyzing issue content, comparing it against existing labels, and applying the most relevant labels. This step requires a solid understanding of GitHub Actions and our MCP server's API.
  4. [ ] Test with Variety of Issue Types: Rigorously test the workflow with a variety of issue types to ensure its effectiveness and accuracy. Testing is a critical step in the implementation process, as it allows us to identify and address any potential issues or bugs before deploying the workflow to production. By testing with a diverse range of issue types, including bug reports, feature requests, and documentation updates, we can ensure that the workflow can handle different scenarios and provide accurate labels. This step also involves validating that the labeling decisions are consistent with our existing label taxonomy and that the comments explaining the labeling process are clear and informative.
  5. [ ] Add Observability/Metrics: Implement observability and metrics to monitor the performance and effectiveness of the workflow. Adding observability and metrics is essential for the long-term success of the automated labeling system. By monitoring key performance indicators (KPIs) such as labeling speed, label relevance, and error rates, we can identify areas for improvement and ensure that the workflow continues to function optimally. Observability also allows us to track the overall impact of the system on issue management, such as the reduction in manual triaging efforts and the improvement in issue discoverability. This step involves integrating monitoring tools and dashboards to provide real-time insights into the workflow's performance.

Future Enhancements

Future enhancements to the system could include:

  • Suggesting issue templates based on content analysis.
  • Automatically assigning issues to relevant team members.
  • Creating a knowledge graph of issue relationships to visualize connections and dependencies.
  • Feeding labeling patterns back into procedures to continuously improve the system's accuracy and efficiency.

Principles: transparency-in-agent-work, snowball-method, systems-stewardship Labels: automation, github-actions, enhancement