Incorrect Candidate Price Issue In Karpenter Consolidation With Karpenter-oci

Jul 23, 2025 by gitftunila 78 views

Incorrect Candidate Price Populated During Karpenter Consolidation

Introduction

This document analyzes an issue encountered during Karpenter consolidation where an incorrect candidate price is populated, leading to the replacement of existing VM instances with instances of the same shape but a higher price. This problem was observed while testing the consolidation feature in karpenter-oci. The investigation revealed that the BuildNodePoolMap() function in karpenter 1.4.0's pkg/controllers/disruption/helpers.go returns a nodePoolToInstanceTypesMap with a middle key that doesn't point to a unique instance type. This results in Karpenter selecting an unexpected instance type with a higher price during the NewCandidate() invocation within getCandidate(), causing unnecessary instance replacements.

Problem Description

During testing of Karpenter's consolidation feature with the Oracle Cloud Infrastructure (OCI) provider, it was observed that existing virtual machine (VM) instances were being replaced by new instances with the same shape, leading to increased costs. The core issue lies within the logic of Karpenter's consolidation process, specifically how it determines the optimal candidates for replacement. The investigation focused on identifying the root cause of this behavior, and pinpointed the BuildNodePoolMap() function in karpenter 1.4.0's pkg/controllers/disruption/helpers.go as the source of the problem. This function is responsible for constructing a map of node pools to available instance types. However, the structure of the map created by this function can lead to ambiguity in instance type selection, resulting in the choice of a more expensive instance.

The nodePoolToInstanceTypesMap is structured in such a way that the middle key, which is intended to represent a unique instance type, can map to multiple instance type objects. This ambiguity arises because the key used doesn't fully capture the unique characteristics of each instance type, leading to collisions when multiple instance types share the same key. For instance, different instance types might have the same shape but vary in other attributes such as CPU or memory, which are not included in the key. As a result, when Karpenter's consolidation logic uses this map to select a candidate instance type for replacement, it may inadvertently choose an instance type with a higher price due to the lack of a unique identifier.

Root Cause Analysis

The investigation revealed that the function BuildNodePoolMap() in karpenter1.4.0/pkg/controllers/disruption/helpers.go is returning a nodePoolToInstanceTypesMap where the middle key does not point to a unique instance type. For example, the following scenario was observed:

nodePoolToInstanceTypesMap[nodepoolname][VM.Standard.A2.Flex] -->
    &{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [8], .....}
    &{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [10], .....}
    &{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [12], .....}

This means that multiple instance types with the same shape (e.g., VM.Standard.A2.Flex) are being mapped to the same key in the nodePoolToInstanceTypesMap. This ambiguity causes issues when the NewCandidate() function, invoked by getCandidate(), uses the following code to retrieve an instance type:

instanceType := instanceTypeMap[node.Labels()[corev1.LabelInstanceTypeStable]]

Because the key node.Labels()[corev1.LabelInstanceTypeStable] (which represents the instance type label on the node) is not unique within the instanceTypeMap, Karpenter may select an unexpected instance type, potentially one with a higher price. This leads to the replacement of existing instances with more expensive ones, defeating the purpose of consolidation.

The issue stems from the way instance types are constructed in karpenter-oci. While the code attempts to generate a unique key and value using shape, CPU, and memory in the listInstanceType() function, the initial construction of the instance type uses an incorrect name. This leads to the ambiguity observed in the nodePoolToInstanceTypesMap.

Detailed Code Examination

The problem is further illustrated by examining the code snippets provided. The first image shows the output of the nodePoolToInstanceTypesMap, highlighting the issue of multiple instance types being associated with the same key (VM.Standard.A2.Flex). The second image shows the invocation of NewCandidate() within getCandidate(), where the non-unique instance type mapping leads to the selection of a potentially incorrect candidate.

The third and fourth images focus on the karpenter-oci code, specifically the listInstanceType() function. These images reveal that the instance type name is initially constructed using *shape.Shape.Shape, which is not sufficient to guarantee uniqueness. Although the code later attempts to create a unique key using shape, CPU, and memory, the initial incorrect name assignment creates the ambiguity that manifests in the nodePoolToInstanceTypesMap.

Proposed Solution and Attempts

To address this issue, an attempt was made to modify the instance type name construction in karpenter-oci to use a more comprehensive format:

fmt.Sprintf("%s-%s-%s", *shape.Shape.Shape, cpu(shape.CalcCpu), resources.Quantity(fmt.Sprint(shape.CalMemInGBs)))

This change aimed to create a unique name by incorporating the shape, CPU, and memory of the instance type. However, this modification introduced other issues, indicating that further investigation and a more refined solution are required.

Challenges and Future Directions

While the attempted solution provided a direction for resolving the issue, it also highlighted the complexity of the problem. The interaction between Karpenter's core consolidation logic and the cloud provider-specific implementation in karpenter-oci requires careful consideration to ensure accurate instance type selection. Future efforts should focus on:

Ensuring Unique Instance Type Identification: Implement a robust mechanism for generating unique instance type identifiers within karpenter-oci. This may involve incorporating all relevant attributes of the instance type, such as shape, CPU, memory, and any other distinguishing characteristics.
Refining the BuildNodePoolMap() Function: Review the logic of the BuildNodePoolMap() function in Karpenter core to ensure that it correctly handles the unique instance type identifiers provided by the cloud provider. This may involve adjusting the structure of the nodePoolToInstanceTypesMap or modifying the key generation process.
Comprehensive Testing: Develop a comprehensive suite of tests to validate the consolidation functionality in karpenter-oci, including scenarios with diverse instance types and pricing models. This will help ensure that the consolidation process accurately selects the most cost-effective instances.

Conclusion

The issue of incorrect candidate price population during Karpenter consolidation in karpenter-oci is a significant concern that can lead to increased costs and inefficient resource utilization. The root cause lies in the non-unique identification of instance types within the nodePoolToInstanceTypesMap, which results in the selection of more expensive instances during consolidation. While an initial attempt to address this issue introduced other problems, the analysis provides valuable insights into the complexities of Karpenter's consolidation process and the importance of accurate instance type identification. Further investigation and a refined solution are necessary to ensure the proper functioning of Karpenter's consolidation feature in the OCI environment.

This article details a specific issue encountered while using Karpenter with the Oracle Cloud Infrastructure (OCI) provider. The problem involves the incorrect pricing of candidate instances during the consolidation process, leading to suboptimal instance replacements. By understanding the root cause and potential solutions, users can better troubleshoot similar issues and contribute to the ongoing development of Karpenter.

Appendix

Relevant Code Snippets

karpenter1.4.0/pkg/controllers/disruption/helpers.go - BuildNodePoolMap() function
karpenter-oci/pkg/providers/oci/cloudprovider.go - listInstanceType() function

Images

Image 1: nodePoolToInstanceTypesMap output showing non-unique instance type mapping
Image 2: NewCandidate() invocation within getCandidate()
Image 3: karpenter-oci code snippet for instance type construction
Image 4: karpenter-oci code snippet for unique key generation