What is Karpenter?
Karpenter is an open-source, high-performance Kubernetes node autoscaler that automatically provisions and deprovisions nodes in response to changing application demands. Originally developed by AWS for Amazon EKS, Karpenter has evolved into a multi-provider project under the Kubernetes Autoscaling SIG. Unlike the traditional Cluster Autoscaler, Karpenter operates at the pod-level granularity, making scaling decisions in seconds rather than minutes.
Karpenter works by directly watching for pods that are unschedulable due to resource constraints, then dynamically launching the optimal compute instance type and size for those pods. It continuously monitors node utilization and consolidates workloads onto fewer nodes when possible, reducing waste and driving infrastructure costs down. Think of it as an intelligent, just-in-time capacity provisioner that eliminates the need for pre-defined node groups and static scaling configurations.
Core Concepts
- NodePool — Defines constraints and policies for node provisioning, including instance families, capacity types (on-demand/spot), architecture, and taints.
- NodeClaim — Represents an individual provisioning request. Karpenter creates NodeClaims based on pending pods and NodePool configurations.
- Consolidation — An optimization mechanism that periodically replaces underutilized nodes with fewer, better-packed nodes or terminates empty nodes entirely.
- Drift Detection — Automatically detects when node settings deviate from the desired NodePool specification and triggers replacement.
- Interruption Handling — Gracefully handles spot instance termination notices, scheduled maintenance events, and voluntary node disruptions.
Why Karpenter Matters
🚀 Deploy your AI agent in 10 minutes
Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.
Try it free →The traditional Cluster Autoscaler relies on node groups with predefined instance types and sizes. This model introduces several pain points: you must forecast capacity needs, manually tune group sizes, and accept that scaling actions take minutes due to cloud provider API polling intervals. Workloads with diverse resource requirements often lead to stranded capacity — large nodes running small pods, or pods stuck pending because no single node group matches their specific needs.
Karpenter solves these problems by making provisioning decisions at the pod level, in real time. Here's why teams adopt it:
Immediate Scaling Response
Instead of waiting for a cluster-level controller loop to notice pending pods (typically 30 seconds to several minutes), Karpenter intercepts pod scheduling events directly via a webhook. Combined with direct cloud API calls for instance launch, new nodes can be ready in as little as 30–60 seconds.
Dynamic Right-Sizing
Karpenter evaluates the exact resource requests of pending pods and selects the most cost-effective instance type that satisfies those constraints. If a pod needs 3 vCPUs and 7GiB of memory, Karpenter won't launch a 16-core behemoth — it picks an instance that fits precisely, often from hundreds of possible SKUs across multiple families.
Spot Instance Optimization
Karpenter natively integrates with spot capacity pools and gracefully handles spot interruptions. It can prioritize spot instances while falling back to on-demand when spot capacity is scarce, and it automatically replaces nodes when a spot termination notice arrives, often before the workload is disrupted.
Consolidation and Cost Efficiency
Beyond scaling up, Karpenter continuously evaluates whether the cluster is running efficiently. If workloads can be packed onto fewer or cheaper nodes, Karpenter will cordon, drain, and replace those nodes automatically. This "defragmentation" behavior often yields 20–40% cost reductions compared to static node groups.
How Karpenter Works: Architecture Deep Dive
Karpenter runs as a controller inside your Kubernetes cluster. It consists of several core controllers that reconcile different resource types. Understanding this architecture helps you troubleshoot and tune the system effectively.
The Scheduling Hook
Karpenter registers a mutating webhook with the Kubernetes scheduler. When a pod enters the scheduling queue and no existing node satisfies its requirements, the scheduler emits an event. Karpenter intercepts this, evaluates the pod's constraints, and creates a NodeClaim resource. This tight integration avoids the polling delay inherent in the Cluster Autoscaler model.
NodeClaim Lifecycle
A NodeClaim is a custom resource that represents a provisioning intent. Karpenter's nodeclaim controller reconciles NodeClaims by calling the cloud provider's instance launch API. Once the instance reports healthy via a readiness check, Karpenter binds the NodeClaim to a Kubernetes Node object. The lifecycle then transitions to active management — health monitoring, drift detection, and eventual consolidation or termination.
Provider Interface
Karpenter abstracts cloud-specific logic behind a provider interface. The official AWS provider ships with extensive support for EC2 instance types, launch templates, subnet selection, and security group automation. Community providers exist for Azure, GCP, and on-premises environments. This abstraction ensures Karpenter's core scheduling logic remains provider-agnostic.
Complete Implementation Guide
Let's walk through a production-ready Karpenter deployment on Amazon EKS, covering every step from initial setup to advanced configuration. You'll need an existing EKS cluster, kubectl, helm, and the AWS CLI configured.
Step 1: Install Karpenter via Helm
Add the Karpenter Helm repository and install the controller. The Helm chart creates the CRDs, service accounts, and the core controller deployment.
# Add the Karpenter Helm repository
helm repo add karpenter https://charts.karpenter.sh
helm repo update
# Install Karpenter in the karpenter namespace
helm upgrade --install karpenter karpenter/karpenter \
--namespace karpenter \
--create-namespace \
--set serviceAccount.annotations.eks.amazonaws.com/role-arn="arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole" \
--set settings.clusterName="${CLUSTER_NAME}" \
--set settings.clusterEndpoint="${CLUSTER_ENDPOINT}" \
--set controller.resources.requests.cpu=1 \
--set controller.resources.requests.memory=1Gi \
--wait
The installation requires an IAM role for the Karpenter controller. Create this using the provided policy that grants permissions for EC2 instance lifecycle management, IAM profile passthrough, and SQS queue interaction for interruption handling.
Step 2: Configure IAM Permissions
Karpenter needs broad EC2 permissions to discover instance types, manage launch templates, and handle instance lifecycle. Create an IAM role with the following trust policy and attach the required permissions.
# Create the IAM policy document
cat > karpenter-controller-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypes",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeLaunchTemplateVersions",
"ec2:RunInstances",
"ec2:TerminateInstances",
"ec2:CreateLaunchTemplate",
"ec2:DeleteLaunchTemplate",
"ec2:CreateTags",
"ec2:DescribeImages",
"iam:PassRole",
"ssm:GetParameter",
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueUrl",
"sqs:GetQueueAttributes"
],
"Resource": "*"
}
]
}
EOF
# Create the IAM policy
aws iam create-policy \
--policy-name KarpenterControllerPolicy \
--policy-document file://karpenter-controller-policy.json
# Attach the policy to the Karpenter controller role
aws iam put-role-policy \
--role-name KarpenterControllerRole \
--policy-name KarpenterControllerPolicy \
--policy-document file://karpenter-controller-policy.json
Step 3: Define Your First NodePool
The NodePool is Karpenter's central configuration object. It defines which nodes Karpenter is allowed to provision. A well-designed NodePool balances cost, performance, and availability requirements. Start with a general-purpose NodePool that covers most workloads.
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
metadata:
labels:
node-type: karpenter
spec:
# Node taints for scheduling isolation
taints:
- key: node.karpenter.sh/capacity-type
value: spot
effect: NoSchedule
# Node labels applied to the instance
labels:
node.karpenter.sh/capacity-type: spot
node.karpenter.sh/owner: platform-team
# Requirements constrain instance selection
requirements:
- key: kubernetes.io/arch
operator: In
values: ["amd64", "arm64"]
- key: kubernetes.io/os
operator: In
values: ["linux"]
- key: node.karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: node.karpenter.sh/instance-category
operator: In
values: ["c", "m", "r", "t"]
- key: node.karpenter.sh/instance-generation
operator: Gt
values: ["2"]
# NodeClass reference (provider-specific)
nodeClassRef:
name: default
# Limits to prevent runaway scaling
limits:
cpu: "1000"
memory: "4000Gi"
# Disruption settings
disruption:
consolidationPolicy: WhenUnderutilized
expireAfter: 720h
# Weighted preferences for cost optimization
weight: 100
Each requirement uses an operator (In, Gt, Lt, Exists) and a list of values. The Gt: ["2"] on instance generation ensures only modern instance generations are used. The consolidation policy WhenUnderutilized tells Karpenter to replace nodes when packing efficiency drops below a threshold.
Step 4: Create the EC2NodeClass
The EC2NodeClass is the AWS-specific configuration object that specifies networking, security, and AMI details for provisioned nodes. This is where you define the infrastructure template.
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: default
spec:
# AMI family discovery - automatically selects the latest EKS-optimized AMI
amiFamily: Bottlerocket
# Subnet selection via tags
subnetSelectorTerms:
- tags:
kubernetes.io/cluster/${CLUSTER_NAME}: owned
subnet-type: private
# Security group selection
securityGroupSelectorTerms:
- tags:
kubernetes.io/cluster/${CLUSTER_NAME}: owned
aws:eks:cluster-name: ${CLUSTER_NAME}
# Instance profile for the node
instanceProfile: KarpenterNodeInstanceProfile
# Detailed monitoring for CloudWatch
detailedMonitoringDisabled: false
# Metadata service configuration
metadataOptions:
httpEndpoint: enabled
httpProtocolIPv6: disabled
httpPutResponseHopLimit: 2
httpTokens: required
# Block device mappings for root volume
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeSize: 80Gi
volumeType: gp3
iops: 3000
throughput: 125
deleteOnTermination: true
The subnetSelectorTerms and securityGroupSelectorTerms use tag-based discovery, so Karpenter automatically finds the right networking resources without hard-coded IDs. The AMI family Bottlerocket selects Amazon's purpose-built container operating system, which boots faster and has a smaller attack surface than standard Linux distributions.
Step 5: Test Provisioning with a Sample Workload
Deploy a workload that requests specific resources to trigger Karpenter provisioning. The pod's resource requests must be unsatisfiable by existing nodes for Karpenter to act.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test
spec:
replicas: 3
selector:
matchLabels:
app: nginx-test
template:
metadata:
labels:
app: nginx-test
spec:
# Node affinity to guide scheduling
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values: ["karpenter"]
containers:
- name: nginx
image: nginx:1.25
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "4"
memory: "8Gi"
Watch the provisioning process with these commands:
# Watch NodeClaims being created
kubectl get nodeclaims --watch
# Watch nodes appearing
kubectl get nodes --watch
# Watch Karpenter controller logs
kubectl logs -f -n karpenter deployment/karpenter
You'll see Karpenter create a NodeClaim, call EC2, and within 60–90 seconds a new node joins the cluster. The pending pods then schedule onto it.
Step 6: Configure Consolidation and Disruption
Consolidation is Karpenter's cost optimization superpower. It evaluates nodes periodically and replaces them when doing so reduces overall cost without disrupting workloads. Configure disruption policies carefully to balance cost against stability.
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: consolidated-pool
spec:
template:
spec:
nodeClassRef:
name: default
requirements:
- key: node.karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
disruption:
# Consolidation policy options:
# WhenUnderutilized - replace when pods can fit on fewer nodes
# WhenEmpty - only remove empty nodes
consolidationPolicy: WhenUnderutilized
# Maximum lifetime before node is replaced regardless of utilization
expireAfter: 720h
# Budget controls the rate of voluntary disruptions
budgets:
- nodes: "10%"
schedule: "0 2 * * *"
duration: 1h
- nodes: "2"
schedule: "* * * * *"
limits:
cpu: "500"
memory: "2000Gi"
The budgets field limits how many nodes Karpenter can disrupt simultaneously. The cron-style schedule lets you define different disruption rates for maintenance windows versus peak traffic hours. For example, allow 10% disruption during a nightly maintenance window but only 2 nodes per operation during business hours.
Step 7: Implement Spot Best Practices with Interruption Handling
Karpenter natively handles spot instance interruptions. It subscribes to an SQS queue that receives EC2 spot termination notifications and scheduled maintenance events. When a termination notice arrives, Karpenter cordons and drains the affected node before the instance is reclaimed, then provisions a replacement if needed.
# Create the SQS queue for spot interruption notifications
aws sqs create-queue \
--queue-name KarpenterInterruptionQueue \
--region ${AWS_REGION}
# Configure EventBridge rule for EC2 spot interruptions
aws events put-rule \
--name KarpenterSpotInterruptionRule \
--event-pattern '{"source":["aws.ec2"],"detail-type":["EC2 Spot Instance Interruption Warning"]}'
# Connect EventBridge to SQS
aws events put-targets \
--rule KarpenterSpotInterruptionRule \
--targets "Id"="1","Arn"="arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:KarpenterInterruptionQueue"
# Grant SQS permissions to the Karpenter controller role
aws iam put-role-policy \
--role-name KarpenterControllerRole \
--policy-name KarpenterSQSInterruptionPolicy \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueUrl",
"sqs:GetQueueAttributes"
],
"Resource": "arn:aws:sqs:'${AWS_REGION}':'${AWS_ACCOUNT_ID}':KarpenterInterruptionQueue"
}
]
}'
To maximize spot usage while maintaining reliability, create separate NodePools with weighted preferences:
# Spot-optimized pool with high weight
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: spot-pool
spec:
weight: 100
template:
metadata:
labels:
capacity-type: spot
spec:
requirements:
- key: node.karpenter.sh/capacity-type
operator: In
values: ["spot"]
nodeClassRef:
name: default
disruption:
consolidationPolicy: WhenUnderutilized
expireAfter: 168h
---
# On-demand fallback pool with lower weight
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: ondemand-fallback
spec:
weight: 10
template:
metadata:
labels:
capacity-type: on-demand
spec:
requirements:
- key: node.karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
- key: node.karpenter.sh/instance-category
operator: In
values: ["c", "m"]
nodeClassRef:
name: default
When spot capacity is available, the higher-weighted spot pool provisions nodes. If spot capacity is constrained, Karpenter falls back to the on-demand pool automatically. This gives you spot economics with on-demand reliability.
Advanced Configuration Patterns
Multi-Architecture Clusters
Karpenter can provision both x86 and ARM64 (Graviton) instances simultaneously. Graviton processors offer up to 40% better price-performance for many workloads. Configure your NodePool to prefer ARM64 but fall back to x86:
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: multi-arch
spec:
template:
spec:
requirements:
- key: kubernetes.io/arch
operator: In
values: ["arm64", "amd64"]
- key: node.karpenter.sh/instance-category
operator: In
values: ["c", "m", "r"]
nodeClassRef:
name: default
Use pod-level node affinity or nodeSelector to pin specific workloads to ARM64. For workloads that benefit from Graviton (web servers, microservices, stateless applications), add:
nodeSelector:
kubernetes.io/arch: arm64
GPU Workload Provisioning
Karpenter supports GPU instance provisioning for AI/ML workloads. Define a dedicated NodePool with GPU instance requirements and appropriate taints to prevent non-GPU workloads from landing on expensive GPU nodes:
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: gpu-pool
spec:
template:
metadata:
labels:
workload-type: gpu
spec:
taints:
- key: nvidia.com/gpu
value: "true"
effect: NoSchedule
requirements:
- key: node.karpenter.sh/instance-category
operator: In
values: ["g", "p"]
- key: node.karpenter.sh/instance-generation
operator: Gt
values: ["3"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
nodeClassRef:
name: gpu-node-class
limits:
cpu: "200"
disruption:
consolidationPolicy: WhenEmpty
The GPU NodeClass should specify an AMI with NVIDIA drivers installed (like the EKS GPU-optimized AMI) and potentially larger EBS volumes for model storage.
Zone-Aware Topology Spread
For high availability, Karpenter respects topology spread constraints and can distribute nodes across availability zones. Configure zone requirements in the NodePool and use pod topology spread constraints:
# NodePool with zone requirements
spec:
requirements:
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a", "us-east-1b", "us-east-1c"]
---
# Pod specification with topology spread
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: my-app
- maxSkew: 1
topologyKey: node-type
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: my-app
Karpenter provisions nodes across the specified zones to satisfy the topology spread constraints. The combination of zone requirements and pod-level constraints ensures workloads remain resilient to zone failures.
Monitoring and Observability
Karpenter emits Prometheus metrics and structured logs. Set up monitoring to track provisioning latency, consolidation effectiveness, and error rates. Key metrics include:
# Prometheus metrics exposed by Karpenter
karpenter_nodes_created_total
karpenter_nodes_terminated_total
karpenter_nodeclaims_created_total
karpenter_nodeclaims_failed_total
karpenter_consolidation_attempts_total
karpenter_consolidation_replacements_total
karpenter_provisioning_duration_seconds
karpenter_interruption_actions_total
Configure alerts for provisioning failures, high consolidation churn, or approaching capacity limits:
# Example Prometheus alert rule
- alert: KarpenterHighProvisioningFailureRate
expr: |
rate(karpenter_nodeclaims_failed_total[5m]) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "Karpenter is failing to provision nodes"
description: "NodeClaim failure rate exceeds 10% over 5 minutes. Check cloud provider capacity and IAM permissions."
- alert: KarpenterCapacityLimitApproaching
expr: |
karpenter_nodes_usage_ratio > 0.85
for: 10m
labels:
severity: critical
annotations:
summary: "Karpenter approaching capacity limits"
description: "NodePool utilization exceeds 85% of defined limits. Consider increasing limits or checking for stuck pods."
Best Practices
1. Start Simple, Then Refine
Begin with a single general-purpose NodePool covering common instance families (C, M, R). Observe provisioning patterns for a week before creating specialized pools. Over-segmentation early on leads to stranded capacity and complex troubleshooting.
2. Set Realistic Limits
Always configure limits on every NodePool. Without limits, a buggy deployment requesting impossible resources (like 500 vCPUs per pod) could trigger an infinite provisioning loop that exhausts your cloud account's capacity. Set limits slightly above your expected peak usage with a 20% buffer.
3. Use Consolidation Budgets
Unrestricted consolidation can cause excessive node churn during traffic spikes. Apply disruption budgets to limit the number of simultaneous node replacements. For production clusters, start with nodes: "20%" and adjust based on your workload's tolerance for rescheduling events.
4. Implement Pod Disruption Budgets
Karpenter respects PodDisruptionBudget objects. Define PDBs for critical deployments to prevent consolidation from violating availability guarantees. Without PDBs, Karpenter might drain a node hosting the last replica of a critical service.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: critical-app-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: critical-app
5. Tag Resources Consistently
Karpenter's tag-based resource discovery relies on consistent tagging across subnets, security groups, and launch templates. Create a tagging convention and enforce it through infrastructure-as-code pipelines. Missing tags cause provisioning failures that are hard to debug.
6. Monitor Spot Interruption Rates
Track how often spot instances are reclaimed. If interruption rates exceed 5% of your fleet per hour, consider increasing the on-demand fallback pool weight or switching instance families to those with more stable spot availability.
7. Version Pin Your AMI
While amiFamily: Bottlerocket auto-discovers the latest AMI, in production you should pin to a specific AMI version to prevent unexpected node image changes during a deployment window. Use amiSelectorTerms with explicit AMI IDs for controlled rollouts.
8. Separate System and Application NodePools
Create dedicated NodePools for system workloads (CoreDNS, ingress controllers, metrics servers) with on-demand capacity and smaller instance sizes. Keep application workloads in separate pools where spot and consolidation can operate more aggressively.
9. Test Failover Behavior
Regularly simulate spot interruptions, zone failures, and capacity constraints. Use tools like aws:ec2-spot-interruption EventBridge events injected manually or chaos engineering frameworks. Validate that critical workloads reschedule within acceptable timeframes.
10. Keep Karpenter Updated
Karpenter's release cadence is rapid — roughly monthly minor versions. Subscribe to the GitHub releases page and test upgrades in staging environments. Version drift beyond 2–3 months often introduces subtle compatibility issues with newer Kubernetes versions or cloud provider API changes.
Troubleshooting Common Issues
Nodes Stuck Pending
If pods remain pending despite Karpenter running, check NodeClaim status first:
kubectl get nodeclaims -o wide
kubectl describe nodeclaim <claim-name>
Common causes include subnet selectors not matching any subnets, security group tags missing, IAM permissions insufficient for instance profile passthrough, or instance capacity shortages in the requested availability zone.
Consolidation Not Triggering
Consolidation requires nodes to be underutilized below a threshold (typically 50% of allocatable resources). Check node utilization:
kubectl top nodes
If nodes are packed densely but consolidation isn't happening, verify the NodePool's consolidationPolicy is set to WhenUnderutilized and that no PodDisruptionBudgets are blocking the drain operation.
High Provisioning Latency
If new nodes take more than 2 minutes to become ready, investigate the AMI boot time, instance initialization scripts, and container image pull times. Bottlerocket AMIs typically boot in under 30 seconds. Pre-warm container images using a DaemonSet or configure image pull-through caches.
Conclusion
Karpenter represents a fundamental shift in Kubernetes node management — from static, human-configured node groups to dynamic, workload-driven provisioning. By operating at pod-level granularity, it eliminates the capacity planning guesswork that plagues traditional autoscaling approaches. The combination of just-in-time provisioning, continuous consolidation, and native spot instance handling delivers both operational simplicity and significant cost savings.
Implementing Karpenter requires thoughtful NodePool design, proper IAM configuration, and a monitoring framework that surfaces provisioning health. Start with the patterns described here — a general-purpose pool, spot with on-demand fallback, consolidation budgets, and PodDisruptionBudgets — then evolve your configuration as workload patterns emerge. The investment in learning Karpenter's abstractions pays dividends in cluster efficiency, developer velocity, and cloud infrastructure cost reduction.