GKE Best Practices: Cost, Security, and Performance

Introduction to GKE Best Practices: Cost, Security, and Performance

Google Kubernetes Engine (GKE) is a managed Kubernetes service that abstracts away the complexity of cluster management while providing deep integration with Google Cloud's ecosystem. As organizations increasingly adopt GKE for production workloads, understanding how to optimize for cost, security, and performance becomes critical. These three pillars are deeply interconnected — a cost-efficient cluster often benefits from performance tuning, while a secure cluster prevents costly breaches and downtime. This tutorial walks through actionable best practices across all three domains, complete with code examples and configuration snippets you can apply directly to your GKE environments.

1. Cost Optimization on GKE

GKE costs stem from compute resources (nodes), networking (egress traffic, load balancers), and ancillary services (logging, monitoring, storage). Without deliberate optimization, clusters can easily accumulate waste — idle nodes, over-provisioned pods, unused load balancers, and excessive logs. The following practices help you keep your GKE spend under control.

1.1 Right-Sizing Node Pools with Machine Types and Autoscaling

Choosing the right machine type and enabling autoscaling ensures you pay only for the compute your workloads actually need. GKE supports both standard and custom machine types, as well as E2 shared-core machines for burstable, low-CPU workloads.

Best practices:

• Use e2-standard- or e2-highmem- families for general workloads; they offer a good balance of cost and performance
• Leverage node auto-provisioning (NAP) to let GKE automatically create and delete node pools based on pod requirements
• Set appropriate min and max node counts in autoscaling to prevent runaway scaling
• Use gke-usage-metering to track per-namespace or per-label cost breakdowns

# Example: Create a cluster with node auto-provisioning and E2 machine family
gcloud container clusters create my-cluster \
  --region=us-central1 \
  --enable-autoprovisioning \
  --min-cpu=2 --max-cpu=32 \
  --min-memory=8 --max-memory=128 \
  --autoprovisioning-scales-up-cool-down=2m \
  --autoprovisioning-scales-down-cool-down=10m

# Create a specific node pool with E2 standard machines and autoscaling
gcloud container node-pools create e2-pool \
  --cluster=my-cluster \
  --machine-type=e2-standard-4 \
  --num-nodes=2 \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=10 \
  --zone=us-central1-a

1.2 Leveraging Spot VMs for Non-Critical Workloads

Spot VMs offer significant discounts (up to 60-90%) compared to on-demand instances, making them ideal for batch jobs, CI/CD runners, or stateless services that can tolerate interruptions. GKE gracefully handles node preemption by draining pods and rescheduling them.

# Create a node pool using Spot VMs
gcloud container node-pools create spot-pool \
  --cluster=my-cluster \
  --machine-type=e2-standard-4 \
  --spot \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=20 \
  --zone=us-central1-a

# Pod template with toleration for Spot node taints and affinity
apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-processor
spec:
  replicas: 3
  selector:
    matchLabels:
      app: batch-processor
  template:
    metadata:
      labels:
        app: batch-processor
    spec:
      tolerations:
      - key: "cloud.google.com/gke-spot"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: "cloud.google.com/gke-spot"
                operator: "In"
                values:
                - "true"
      containers:
      - name: processor
        image: my-batch-image:latest
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"

1.3 Optimizing Resource Requests and Limits

Over-provisioning resources is one of the most common sources of waste. Setting accurate requests and limits allows the Kubernetes scheduler to pack pods efficiently onto nodes, reducing the number of nodes needed.

# Example: Well-defined resource specifications for a microservice
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: api
        image: my-api:v2.1
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Use Vertical Pod Autoscaling (VPA) in recommendation mode to analyze actual usage and suggest optimal resource values over time:

# VPA in recommendation mode (doesn't auto-adjust, just logs suggestions)
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-service
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
    - containerName: "api"
      minAllowed:
        cpu: "100m"
        memory: "128Mi"
      maxAllowed:
        cpu: "2"
        memory: "2Gi"

1.4 Reducing Networking Costs

Inter-zone and inter-region egress traffic can become a significant cost driver. Keep communication within the same zone or region whenever possible, and use internal load balancers instead of external ones for intra-cluster or VPC-internal traffic.

# Use topology-aware routing to prefer same-zone endpoints
apiVersion: v1
kind: Service
metadata:
  name: internal-api
  annotations:
    networking.gke.io/topology-aware-routing: "true"
spec:
  type: ClusterIP
  selector:
    app: api-service
  ports:
  - port: 80
    targetPort: 8080

1.5 Cleaning Up Unused Resources

Orphaned load balancers, persistent volumes, and static IPs continue to incur charges. Implement regular cleanup automation:

# Script to find and delete unused external load balancers in GCP
# List all forwarding rules with no associated backends
gcloud compute forwarding-rules list --format="table(name,region)" | while read -r name region; do
  echo "Checking forwarding rule: $name in $region"
done

# GKE cleanup: delete services with type LoadBalancer that are no longer needed
kubectl get svc --all-namespaces | grep LoadBalancer | awk '{print $1, $2}' | while read -r ns name; do
  kubectl delete svc -n "$ns" "$name" --dry-run=client
done

2. Security Best Practices on GKE

GKE security spans the cluster control plane, node configuration, network policies, workload identity, image supply chain, and runtime protection. A layered defense-in-depth approach is essential.

2.1 Enabling GKE Control Plane Features: Private Clusters and Authorized Networks

By default, GKE clusters have a public endpoint. For production, use private clusters with authorized networks to restrict access to the control plane, drastically reducing the attack surface.

# Create a private GKE cluster with authorized networks
gcloud container clusters create secure-cluster \
  --region=us-central1 \
  --enable-private-nodes \
  --enable-master-authorized-networks \
  --master-authorized-networks=10.0.0.0/8,203.0.113.0/24 \
  --enable-private-endpoint \
  --network=my-vpc \
  --subnetwork=my-subnet

2.2 Workload Identity: Eliminating Service Account Key Management

Workload Identity maps a Kubernetes service account to a Google Cloud IAM service account, allowing pods to authenticate to GCP services without storing long-lived credentials. This is a cornerstone of zero-trust on GKE.

# Step 1: Enable Workload Identity on the cluster (if not already enabled)
gcloud container clusters update my-cluster \
  --workload-pool=PROJECT_ID.svc.id.goog

# Step 2: Create a GCP service account and grant IAM roles
gcloud iam service-accounts create gke-sa --display-name="GKE Pod SA"
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:gke-sa@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

# Step 3: Bind Kubernetes SA to GCP SA via annotation
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  namespace: default
  annotations:
    iam.gke.io/gcp-service-account: gke-sa@PROJECT_ID.iam.gserviceaccount.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
spec:
  template:
    spec:
      serviceAccountName: my-app-sa
      containers:
      - name: app
        image: my-app:latest

2.3 Network Policies and Firewall Rules

Kubernetes network policies enforce pod-level micro-segmentation, limiting lateral movement. Combine with GCP firewall rules for node-level restrictions.

# Network policy: allow only specific ingress to API pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-service
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432
  - to:  # Allow DNS resolution
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

2.4 Binary Authorization and Image Integrity

Binary Authorization ensures only trusted container images, signed by verified attestors, can be deployed. This prevents supply chain attacks where compromised images reach production.

# Enable Binary Authorization on the cluster
gcloud container clusters update my-cluster \
  --enable-binauthz \
  --region=us-central1

# Create a Binary Authorization policy (example: allow only specific trusted paths)
gcloud beta container binauthz policies create \
  --project=PROJECT_ID \
  --default-deny \
  --allowlist-pattern="us-central1-docker.pkg.dev/PROJECT_ID/trusted-repo/**"

2.5 CIS Benchmarks and GKE Security Posture Management

GKE automatically applies many CIS Kubernetes benchmarks. You can use Security Posture (a built-in feature) to assess cluster configuration against industry standards and detect misconfigurations.

# Enable Security Posture on a new cluster
gcloud container clusters create compliant-cluster \
  --region=us-central1 \
  --enable-security-posture \
  --workload-pool=PROJECT_ID.svc.id.goog

# View security posture findings
gcloud container clusters describe compliant-cluster \
  --region=us-central1 --format="json(securityPostureConfig)"

2.6 Runtime Security with GKE Sandbox (gVisor)

For multi-tenant or untrusted workloads, GKE Sandbox provides an additional isolation layer using gVisor, which intercepts system calls and limits the attack surface against the host kernel.

# Create a node pool with GKE Sandbox enabled
gcloud container node-pools create sandbox-pool \
  --cluster=my-cluster \
  --machine-type=e2-standard-4 \
  --sandbox=type=gvisor \
  --zone=us-central1-a

# Pod that requires sandbox via runtimeClassName
apiVersion: v1
kind: Pod
metadata:
  name: isolated-workload
spec:
  runtimeClassName: gvisor
  containers:
  - name: sandboxed-app
    image: untrusted-third-party-app:latest

3. Performance Optimization on GKE

Performance on GKE is about ensuring low latency, high throughput, and efficient resource utilization. This involves node selection, networking tuning, storage optimization, and intelligent scheduling.

3.1 Choosing the Right Node Architecture and Machine Series

GKE offers multiple machine families optimized for different workloads. For compute-intensive tasks, use C2D (Milan) or C3 instances; for memory-heavy workloads, use M3 instances; for general-purpose, E2 or N2D with balanced specs.

# Node pool with C3 high-CPU machines for compute-bound services
gcloud container node-pools create compute-pool \
  --cluster=my-cluster \
  --machine-type=c3-standard-8 \
  --num-nodes=3 \
  --zone=us-central1-a

# Node pool with M3 high-memory machines for in-memory caches
gcloud container node-pools create memory-pool \
  --cluster=my-cluster \
  --machine-type=m3-highmem-16 \
  --num-nodes=2 \
  --zone=us-central1-a

3.2 Optimizing Container Networking with Dataplane V2 and Native CNI

GKE Dataplane V2 (based on eBPF and Cilium) provides enhanced networking performance, lower latency, and better observability compared to the legacy iptables-based kube-proxy. It also supports native load balancing features.

# Create a cluster with Dataplane V2 enabled
gcloud container clusters create fast-cluster \
  --region=us-central1 \
  --enable-dataplane-v2 \
  --enable-master-authorized-networks \
  --network=my-vpc

For pods that require high network throughput, use the native CNI with dedicated IPs to bypass node-level iptables rules:

# Pod with native CNI and dedicated IP
apiVersion: v1
kind: Pod
metadata:
  name: high-throughput-app
  annotations:
    networking.gke.io/native-cni: "true"
spec:
  nodeSelector:
    cloud.google.com/gke-network: "dedicated"
  containers:
  - name: app
    image: high-throughput-app:latest
    resources:
      requests:
        cpu: "4"
        memory: "8Gi"

3.3 Horizontal Pod Autoscaling with Custom Metrics

HPA scales pods based on CPU/memory or custom metrics from Cloud Monitoring. Tuning scaling behavior prevents thrashing and ensures responsiveness.

# HPA with CPU target and scaling behavior tuning
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 120

3.4 Node Affinity, Anti-Affinity, and Topology Spread Constraints

Strategic pod placement reduces latency and improves resilience. Use node affinity to target hardware-optimized nodes, pod anti-affinity to spread replicas across nodes for high availability, and topology spread constraints to distribute pods evenly across zones.

# Deployment with topology spread across zones and anti-affinity
apiVersion: apps/v1
kind: Deployment
metadata:
  name: distributed-api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: distributed-api
  template:
    metadata:
      labels:
        app: distributed-api
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: distributed-api
      - maxSkew: 2
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: distributed-api
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: distributed-api
              topologyKey: kubernetes.io/hostname
      containers:
      - name: api
        image: my-api:v2.1

3.5 Persistent Disk and Storage Performance Tuning

Storage I/O can become a bottleneck for stateful workloads. Choose the right disk type, enable SSD persistent disks, and use regional PDs for replicated storage with zone-level failover.

# StorageClass for SSD persistent disk with regional replication
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd-regional
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  replication-type: regional-pd
  zones: "us-central1-a,us-central1-b"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
# StatefulSet using the high-performance StorageClass
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: fast-db
spec:
  serviceName: "fast-db"
  replicas: 3
  selector:
    matchLabels:
      app: fast-db
  template:
    metadata:
      labels:
        app: fast-db
    spec:
      containers:
      - name: db
        image: postgres:15
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: ssd-regional
      resources:
        requests:
          storage: 100Gi

3.6 Monitoring and Observability with GKE Dashboard and Cloud Monitoring

Proactive monitoring helps identify performance regressions before they impact users. GKE integrates natively with Cloud Monitoring and Cloud Logging, providing pre-built dashboards and SLO-based alerting.

# Enable managed Prometheus and Cloud Monitoring integration
gcloud container clusters update my-cluster \
  --enable-managed-prometheus \
  --region=us-central1

# Deploy a PodMonitoring resource to scrape custom metrics
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: api-metrics
  namespace: default
spec:
  selector:
    matchLabels:
      app: api-service
  endpoints:
  - port: 8080
    interval: 30s
    path: /metrics

4. Bringing It All Together: A Holistic Approach

The most effective GKE implementations treat cost, security, and performance as an integrated discipline. A cluster that scales down aggressively (cost) should still maintain security posture through network policies and workload identity. A high-performance cluster using Dataplane V2 should also leverage spot VMs for non-critical components to offset costs. The table below summarizes the interplay:

• Cost ↔ Performance: Autoscaling (HPA, VPA, NAP) reduces waste while maintaining performance SLOs
• Security ↔ Cost: Workload Identity eliminates secret management overhead; private clusters reduce exposure without adding compute cost
• Performance ↔ Security: Dataplane V2 improves both network throughput and observability for threat detection; Sandbox isolation enables safe multi-tenancy without performance penalties when properly node-pooled

Conclusion

Optimizing GKE for cost, security, and performance is an ongoing process, not a one-time setup. Start by auditing your current clusters: identify over-provisioned resources, public endpoints, and latency bottlenecks. Implement the practices covered in this tutorial incrementally — enable Workload Identity to eliminate credential sprawl, adopt Spot VMs for batch workloads to cut compute costs, migrate to Dataplane V2 for lower network latency, and enforce network policies for defense-in-depth. Leverage GKE's built-in features like node auto-provisioning, security posture management, and managed Prometheus to automate much of this work. With these foundations in place, your GKE clusters will be resilient, efficient, and ready to scale with your business needs — all while keeping your cloud bill predictable and your attack surface minimized.