Karpenter OCI Consolidation Issue Incorrect Candidate Price And VM Replacements

Jul 23, 2025 by gitftunila 80 views

Incorrect Candidate Price Populated During Karpenter Consolidation

Introduction

This article addresses an issue encountered during the consolidation process in Karpenter, specifically when using the Oracle Cloud Infrastructure (OCI) provider. Karpenter, a Kubernetes Node autoscaling project, aims to efficiently manage cluster resources by provisioning and deprovisioning nodes based on workload demands. The consolidation feature in Karpenter helps to further optimize resource utilization by identifying opportunities to migrate workloads onto fewer nodes, thereby reducing overall costs. However, a bug in Karpenter version 1.4.0, particularly within the karpenter-oci provider, can lead to incorrect pricing calculations during consolidation, resulting in unnecessary node replacements. This document will delve into the details of the issue, its root cause, and potential solutions.

The Karpenter consolidation process aims to optimize resource utilization within a Kubernetes cluster by identifying opportunities to migrate workloads onto a smaller set of nodes. This process involves evaluating the current node pool configuration, identifying underutilized nodes, and determining if workloads can be consolidated onto fewer instances without impacting performance or availability. During this evaluation, Karpenter calculates the cost of running workloads on different instance types and compares these costs to the current node configuration. The goal is to identify a consolidation plan that reduces overall spending while maintaining the necessary resources for running applications.

The problem arises when Karpenter incorrectly calculates the price of candidate instances during the consolidation process. In certain scenarios, Karpenter may identify a candidate instance as having a higher price than the existing instance, even if they have the same specifications or the candidate instance is actually cheaper. This miscalculation can lead to Karpenter replacing existing, perfectly adequate instances with new ones of the same shape, defeating the purpose of consolidation and potentially increasing costs. This issue was observed specifically within the karpenter-oci provider in Karpenter version 1.4.0, indicating a potential problem in how instance types and pricing are handled within the Oracle Cloud Infrastructure environment. Understanding the root cause of this issue is crucial for ensuring the effective and cost-efficient operation of Karpenter in OCI.

Problem Description

During testing of the Karpenter-OCI consolidation feature, it was observed that existing virtual machine (VM) instances were being replaced by new VM instances with the same shape. This behavior was unexpected, as the consolidation process should ideally aim to reduce costs by migrating workloads to fewer instances or more cost-effective instance types. Further investigation, involving printing logs from the consolidation.go file within the Karpenter 1.4.0 module, revealed that Karpenter was identifying the new instances as having a higher price than the existing ones, even though they were essentially the same. This incorrect pricing comparison was the trigger for the unnecessary replacements.

The root cause of this issue lies within the BuildNodePoolMap() function in the karpenter1.4.0/pkg/controllers/disruption/helpers.go file. This function is responsible for creating a map of node pools to instance types, which is used during the consolidation process to evaluate potential candidate instances. The problem is that the intermediate key in the nodePoolToInstanceTypesMap (specifically, the instance type name) is not guaranteed to be unique. This means that multiple InstanceType objects with the same name (e.g., VM.Standard.A2.Flex) can be associated with a single node pool in the map.

For example, the nodePoolToInstanceTypesMap might contain entries like this:

nodePoolToInstanceTypesMap[nodepoolname][VM.Standard.A2.Flex] -->
    &{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [8], .....}
    &{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [10], .....}
    &{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [12], .....}

As you can see, the key VM.Standard.A2.Flex maps to multiple InstanceType objects, each potentially with different pricing information. This ambiguity leads to issues later in the consolidation process. When the NewCandidate() function is invoked by getCandidate(), Karpenter uses the instance type label from the node (node.Labels()[corev1.LabelInstanceTypeStable]) to look up the corresponding InstanceType in the map. However, because the map contains multiple entries for the same instance type name, Karpenter might select an InstanceType with an unexpectedly high price, leading to the incorrect consolidation decisions.

Root Cause Analysis

The investigation pinpointed the issue to the BuildNodePoolMap() function within karpenter1.4.0/pkg/controllers/disruption/helpers.go. This function constructs a map that links node pools to available instance types. The structure of this map, nodePoolToInstanceTypesMap, uses the instance type name as a key. However, the logic within the karpenter-oci provider doesn't guarantee the uniqueness of these instance type names, leading to multiple entries for the same instance type within the map.

Specifically, the following structure is observed:

nodePoolToInstanceTypesMap[nodepoolname][VM.Standard.A2.Flex]

This structure can contain multiple entries, such as:

&{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [8], .....}
&{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [10], .....}
&{VM.Standard.A2.Flex karpenter.k8s.oracle/instance-cpu In [12], .....}

This lack of uniqueness becomes problematic when the NewCandidate() function, called by getCandidate(), attempts to retrieve an instance type based on the node's label (instanceType := instanceTypeMap[node.Labels()[corev1.LabelInstanceTypeStable]]). Since multiple InstanceType objects share the same name, Karpenter might select one with an inflated price, triggering an unnecessary node replacement during consolidation.

Further examination of the karpenter-oci code reveals that the instance type names are constructed using the shape name (shape.Shape.Shape). While there are attempts to create unique keys and values by incorporating shape, CPU, and memory information within the listInstanceType() function, the core issue of using the shape name as the primary identifier persists.

This is illustrated in the following code snippet:

// Incorrect name specified when constructing an instance type
// Although there are steps to generate a unique key and value using shape-cpu-memory in function listInstanceType()

The images provided in the original report visually demonstrate this issue. They highlight how the BuildNodePoolMap() function generates the problematic map with non-unique instance type keys and how this leads to incorrect instance type selection during the candidate evaluation process.

Proposed Solution and Challenges

An initial attempt to resolve this issue involved modifying the instance type name generation within the karpenter-oci provider. The proposed change was to construct the name using a combination of the shape, CPU, and memory, specifically using the format fmt.Sprintf("%s-%s-%s", *shape.Shape.Shape, cpu(shape.CalcCpu), resources.Quantity(fmt.Sprint(shape.CalMemInGBs))). The goal was to create unique names for each instance type, thereby preventing the ambiguity in the nodePoolToInstanceTypesMap.

However, this approach introduced other complications. While it addressed the immediate problem of non-unique names, it potentially disrupted other parts of the system that rely on the original naming convention. This highlights the importance of carefully considering the impact of any changes on the overall system architecture.

Therefore, a more comprehensive solution is required. This solution should focus on ensuring the uniqueness of instance type identifiers within the nodePoolToInstanceTypesMap without breaking existing functionality. Possible approaches include:

Introduce a Unique Identifier: Instead of relying solely on the instance type name, a unique identifier could be generated for each instance type. This could be a combination of the shape, CPU, memory, and potentially other attributes.
Modify the Map Structure: The structure of the nodePoolToInstanceTypesMap could be modified to use a more complex key that incorporates additional information beyond just the instance type name. This could involve creating a custom struct that encapsulates the relevant attributes.
Refactor the Instance Type Selection Logic: The logic for selecting candidate instance types could be refactored to take into account the potential for multiple entries with the same name. This might involve iterating over the list of InstanceType objects and comparing their attributes to the node's requirements.

Each of these approaches has its own trade-offs in terms of complexity and potential impact on the system. A thorough analysis is needed to determine the best solution.

Conclusion

The issue of incorrect candidate price population during Karpenter consolidation, specifically within the karpenter-oci provider, highlights the importance of careful instance type management and pricing calculations. The root cause lies in the non-uniqueness of instance type names within the nodePoolToInstanceTypesMap, which leads to Karpenter potentially selecting an InstanceType with an inflated price during the consolidation process. While an initial attempt to address this issue by modifying the instance type name generation introduced new challenges, a more comprehensive solution is needed.

This solution should focus on ensuring the uniqueness of instance type identifiers without disrupting existing functionality. Potential approaches include introducing a unique identifier, modifying the map structure, or refactoring the instance type selection logic. By addressing this issue, Karpenter can effectively consolidate resources in OCI, leading to cost savings and improved resource utilization. Further investigation and testing are crucial to ensure that the chosen solution effectively resolves the problem without introducing any new issues.

Keywords

Karpenter, Karpenter Consolidation, Oracle Cloud Infrastructure (OCI), Instance Type Pricing, Kubernetes Node Autoscaling, Resource Optimization, Incorrect Pricing Calculation, nodePoolToInstanceTypesMap, karpenter-oci, VM Instance Replacement, Cost Optimization