Monitoring GKE: Metrics, Alarms, and Dashboards

Effective monitoring is the backbone of operating Kubernetes clusters in production. Google Kubernetes Engine (GKE) integrates deeply with Google Cloud's operations suite—Cloud Monitoring, Cloud Logging, and Cloud Dashboards—to give you visibility into every layer of your infrastructure. This tutorial walks through the complete monitoring lifecycle: collecting metrics, building meaningful alarms, and visualizing everything in dashboards. Every concept is accompanied by practical code you can deploy immediately.

What Monitoring Means for GKE

Monitoring GKE means observing and understanding three distinct layers:

Infrastructure metrics — CPU, memory, disk usage of nodes and pods, network throughput, and Kubernetes control plane activity
Application metrics — custom business KPIs exposed by your workloads, such as request latency, error rates, or queue depth
Platform signals — Kubernetes events, audit logs, and system component health (kubelet, API server, scheduler)

Google Cloud Monitoring (formerly Stackdriver) automatically collects infrastructure metrics from GKE clusters without requiring any instrumentation. For application and platform signals, you combine auto-collected data with custom metrics you define, log-based metrics derived from structured logs, and Kubernetes-native metric endpoints scraped by the Managed Prometheus service.

Why Monitoring Matters in GKE

Without monitoring, you operate blind. GKE abstracts many infrastructure concerns, but that abstraction can hide problems until they become outages. A well-monitored cluster lets you:

Detect anomalies before users notice — spot memory leaks, CPU saturation, or network bottlenecks early
Optimize cost — identify over-provisioned workloads and right-size node pools
Troubleshoot faster — correlate metrics, logs, and events during incidents
Enforce SLOs — measure availability and latency against your promises to users
Automate response — trigger horizontal pod autoscaling (HPA), node auto-provisioning, or Cloud Functions for remediation

GKE's deep integration with Cloud Monitoring means you get rich dashboards and alerting with minimal effort, but the real power comes when you tailor the system to your workloads.

Built-in GKE Metrics: What You Get for Free

When you create a GKE cluster with Cloud Monitoring enabled (the default), the system deploys a gke-metrics-agent DaemonSet on every node. This agent collects and forwards metrics to Cloud Monitoring. The following categories are captured automatically:

Node Metrics

kubernetes/node/cpu/core_usage_time — CPU time consumed
kubernetes/node/memory/allocatable_bytes — memory available for pods
kubernetes/node/ephemeral_storage/allocatable_bytes — ephemeral storage limits
kubernetes/node/network/received_bytes_count and sent_bytes_count

Pod and Container Metrics

kubernetes/container/cpu/core_usage_time — per-container CPU
kubernetes/container/memory/used_bytes — memory usage including cache
kubernetes/container/restart_count — cumulative restart counter
kubernetes/container/network/received_bytes_count — per-pod network I/O

API Server and Control Plane Metrics

kubernetes/apiserver/request_count — API calls by verb and resource
kubernetes/apiserver/request_latencies — latency distribution
kubernetes/scheduler/schedule_attempts — scheduling successes and failures

You can query these directly in Cloud Monitoring's Metrics Explorer. For example, to list all pods with high CPU usage:

# In Cloud Monitoring Query Language (MQL)
fetch k8s_container
| metric 'kubernetes/container/cpu/core_usage_time'
| rate(1m)
| group_by [resource.project_id, resource.cluster_name, resource.namespace_name, resource.pod_name]
| top 10

Managed Prometheus: Scraping Custom Metrics

For application metrics, GKE offers Managed Prometheus, which scrapes Prometheus-compatible endpoints and stores the data in Cloud Monitoring without you running your own Prometheus servers. This is the recommended approach for custom metrics because it unifies storage, eliminates Prometheus maintenance, and lets you use Cloud Monitoring's alerting and dashboarding.

Enabling Managed Prometheus

Managed Prometheus requires the gmp-system components. You enable it by deploying the PodMonitoring custom resource, which tells the collector what endpoints to scrape and how.

First, apply the Managed Prometheus operator (if not already present in your cluster). For Autopilot clusters, it is pre-installed. For Standard clusters, install it via:

# Apply the Prometheus operator manifests
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/main/manifests/operator-setup.yaml

# Deploy the collector
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/main/manifests/collector-setup.yaml

Exposing Application Metrics

Your application must expose a Prometheus metrics endpoint. Below is a simple Go application that serves metrics on port 8080:

package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requestCount = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "app_requests_total",
            Help: "Total number of processed requests",
        },
        []string{"method", "status"},
    )
    requestLatency = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "app_request_duration_seconds",
            Help:    "Request latency histogram",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method"},
    )
)

func main() {
    // Expose metrics
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

Creating a PodMonitoring Resource

Now define a PodMonitoring resource to tell Managed Prometheus to scrape this endpoint every 30 seconds:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: my-app-monitoring
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-application
  endpoints:
    - port: metrics-port
      interval: 30s
      path: /metrics
  targetLabels:
    - app
    - version

Apply this with kubectl apply -f podmonitoring.yaml. Within minutes, your custom metrics appear in Cloud Monitoring under the prefix prometheus.googleapis.com/app_requests_total.

Querying Prometheus-Backed Metrics

Use PromQL-compatible queries in Cloud Monitoring:

# Rate of requests per second over 5 minutes
rate(prometheus.googleapis.com/app_requests_total:counter{method="GET"}[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(prometheus.googleapis.com/app_request_duration_seconds:histogram[5m]))

Log-Based Metrics: Extracting Signals from Logs

Not all useful signals come from instrumentation. Log-based metrics let you extract numeric values from structured logs and treat them as first-class Cloud Monitoring metrics. This is ideal for tracking audit events, error occurrences in unstructured output, or business events logged as JSON.

Creating a Log-Based Metric

Suppose your application logs structured JSON that includes a order_value field. You want to track order values as a distribution metric. First, ensure your logs are in a format Cloud Logging can parse:

// Go application logging structured JSON to stdout
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Info("order processed",
    "order_id", "ord-12345",
    "order_value", 299.95,
    "customer_tier", "premium",
)

In Cloud Logging, these entries appear with the JSON payload fields exposed. Now create a log-based metric via the Cloud Monitoring API or the console:

# Using gcloud to create a log-based distribution metric
gcloud logging metrics create order_value_distribution \
  --description="Distribution of order values from application logs" \
  --log-filter='resource.type="k8s_container" jsonPayload.order_value:*' \
  --value-extractor='$.jsonPayload.order_value' \
  --bucket-boundaries=10,50,100,250,500,1000 \
  --units="USD"

This metric now streams into Cloud Monitoring and can be used in dashboards and alerts just like any other metric. Log-based metrics update within a few minutes of log ingestion.

Common Log-Based Metric Patterns

Error counts: Extract 1 for each log entry matching severity=ERROR
Audit trails: Count specific API operations from Cloud Audit logs
Slow query detection: Extract query duration from database logs
User activity: Track login events or feature usage from application logs

Alarms: Creating Alerting Policies

Metrics alone are not enough—you need alerting policies that notify you when things go wrong. Cloud Monitoring supports conditions based on metric thresholds, absence of data, forecasts, and complex MQL expressions.

Basic Threshold Alert

This example creates an alert when pod memory usage exceeds 90% of the limit for more than 5 minutes:

# gcloud command to create a memory alert
gcloud alpha monitoring policies create \
  --display-name="Pod Memory Above 90% Limit" \
  --condition='resource.type="k8s_container"
               metric.type="kubernetes/container/memory/limit_utilization"
               filter="resource.namespace_name=~\"production-.*\""
               aggregations: per_resource
               condition-threshold: value>0.9 duration=300s' \
  --notification-channels="projects/my-project/notificationChannels/1234567890" \
  --documentation="Pod memory utilization has exceeded 90% of its limit for 5 minutes. Check for memory leaks or consider increasing limits."

Alert on Metric Absence (Heartbeat)

Sometimes the absence of data is the problem. Create an alert that fires when a critical cron job's success metric stops reporting:

# Alert when no success metric received for 30 minutes
gcloud alpha monitoring policies create \
  --display-name="Cron Job Not Completing" \
  --condition='resource.type="prometheus_target"
               metric.type="prometheus.googleapis.com/cron_success_total:counter"
               absent-for=1800s' \
  --notification-channels="projects/my-project/notificationChannels/1234567890"

Forecast-Based Alert

Cloud Monitoring can predict future metric values using time-series forecasting. This alert fires when disk is predicted to fill within 24 hours:

# Forecast-based alert for node disk space
gcloud alpha monitoring policies create \
  --display-name="Node Disk Forecasted Full" \
  --condition='resource.type="k8s_node"
               metric.type="kubernetes/node/ephemeral_storage/limit_utilization"
               condition-threshold: forecast-value>0.95 horizon=86400s duration=300s' \
  --notification-channels="projects/my-project/notificationChannels/1234567890"

Multi-Condition and Composite Alerts

For nuanced alerting, combine conditions. This alert fires only when both CPU is high AND a recent deployment was detected:

apiVersion: monitoring.googleapis.com/v1
kind: AlertPolicy
metadata:
  name: high-cpu-after-deploy
spec:
  displayName: "High CPU After Deployment"
  combiner: AND
  conditions:
    - conditionThreshold:
        filter: 'resource.type="k8s_container" AND metric.type="kubernetes/container/cpu/limit_utilization"'
        aggregations:
          - alignmentPeriod: 300s
            perSeriesAligner: ALIGN_MEAN
        comparison: COMPARISON_GT
        thresholdValue: 0.85
        duration: 600s
        trigger:
          count: 2
    - conditionAbsent:
        filter: 'metric.type="logging.googleapis.com/user/deployment_event"'
        duration: 900s
  notificationChannels:
    - projects/my-project/notificationChannels/1234567890

Notification Channels

Alerting policies route notifications through channels you configure once. Supported channels include:

Email and SMS
PagerDuty, Slack, and Webhooks
Cloud Pub/Sub for custom automation
Mobile push notifications via the Cloud Monitoring mobile app

# Create an email notification channel
gcloud alpha monitoring channels create \
  --display-name="On-Call Email" \
  --type=email \
  --channel-labels=email_address=oncall@example.com

# Create a Slack notification channel
gcloud alpha monitoring channels create \
  --display-name="Slack Alerts" \
  --type=slack \
  --channel-labels=channel_name="#alerts",token="xoxb-your-slack-token"

Dashboards: Visualizing Metrics

Cloud Monitoring dashboards let you build curated views of your cluster health. GKE ships with a pre-built Kubernetes Engine Dashboard that shows cluster-level metrics, node utilization, and workload status. You can create custom dashboards that mix infrastructure and application metrics.

Using the Built-in GKE Dashboard

Navigate to Cloud Monitoring → Dashboards → Kubernetes Engine. This dashboard includes:

Cluster CPU and memory utilization
Node pool breakdowns
Pod restarts and phase counts
API server request rates and latencies
Network throughput

You can clone and customize this dashboard to add your own charts.

Creating Custom Dashboards via Code

For infrastructure-as-code workflows, define dashboards as YAML and deploy them with Terraform or directly via the API. Here is a complete custom dashboard definition:

apiVersion: monitoring.googleapis.com/v1
kind: Dashboard
metadata:
  name: custom-gke-dashboard
spec:
  displayName: "Production GKE Overview"
  dashboardFilters:
    - filterKey: resource.cluster_name
      labelName: Cluster
      defaultValue: prod-cluster-us-central1
  mosaicLayout:
    columns: 12
    tiles:
      - title: "Pod CPU Usage - Top 10"
        height: 4
        width: 6
        widget:
          xyChart:
            dataSets:
              - timeSeriesQuery:
                  timeSeriesQueryLanguage: >
                    fetch k8s_container
                    | metric 'kubernetes/container/cpu/core_usage_time'
                    | filter resource.cluster_name = 'prod-cluster-us-central1'
                    | rate(1m)
                    | group_by [resource.pod_name]
                    | top 10
                plotType: LINE
            chartOptions:
              mode: COLOR
            yAxis:
              label: "CPU cores"
      - title: "Memory Utilization %"
        height: 4
        width: 6
        widget:
          xyChart:
            dataSets:
              - timeSeriesQuery:
                  timeSeriesQueryLanguage: >
                    fetch k8s_container
                    | metric 'kubernetes/container/memory/limit_utilization'
                    | filter resource.cluster_name = 'prod-cluster-us-central1'
                    | group_by [resource.namespace_name], mean(val())
                    | window 5m
                plotType: LINE
            yAxis:
              label: "Utilization"
              minValue: 0
              maxValue: 1
      - title: "Request Rate by Status"
        height: 4
        width: 6
        widget:
          xyChart:
            dataSets:
              - timeSeriesQuery:
                  timeSeriesQueryLanguage: >
                    fetch prometheus_target
                    | metric 'prometheus.googleapis.com/app_requests_total:counter'
                    | filter metadata.system_labels.app = 'my-application'
                    | rate(1m)
                    | group_by [metric.status], sum(val())
                plotType: LINE
            chartOptions:
              mode: COLOR
      - title: "P95 Request Latency"
        height: 4
        width: 6
        widget:
          xyChart:
            dataSets:
              - timeSeriesQuery:
                  timeSeriesQueryLanguage: >
                    fetch prometheus_target
                    | metric 'prometheus.googleapis.com/app_request_duration_seconds:histogram'
                    | rate(1m)
                    | histogram_quantile(0.95)
                    | window 5m
                plotType: LINE
            yAxis:
              label: "Seconds"
      - title: "Container Restarts"
        height: 4
        width: 12
        widget:
          timeSeriesTable:
            columnSettings:
              - column: resource.pod_name
                displayName: "Pod"
              - column: metric.restart_count
                displayName: "Restarts"
            dataSets:
              - timeSeriesQuery:
                  timeSeriesQueryLanguage: >
                    fetch k8s_container
                    | metric 'kubernetes/container/restart_count'
                    | filter resource.cluster_name = 'prod-cluster-us-central1'
                    | group_by [resource.pod_name], max(val())
                    | filter val() > 0
                tableDisplayOptions:
                  showFullTable: true

Apply this dashboard using gcloud monitoring dashboards create --config-from-file=dashboard.yaml or via Terraform's google_monitoring_dashboard resource.

Using Terraform for Dashboard as Code

resource "google_monitoring_dashboard" "gke_overview" {
  dashboard_json = jsonencode({
    displayName = "Production GKE Overview"
    mosaicLayout = {
      columns = 12
      tiles = [
        {
          title  = "Pod CPU Usage - Top 10"
          height = 4
          width  = 6
          widget = {
            xyChart = {
              dataSets = [
                {
                  timeSeriesQuery = {
                    timeSeriesQueryLanguage = <



Custom Dashboard Widget Types

Cloud Monitoring supports multiple widget types beyond XY charts:

  Scorecard — show a single value (e.g., current error rate) with an optional threshold color
  TimeSeriesTable — tabular view of recent metric values across resources
  AlertChart — embed the status of alerting policies directly in the dashboard
  Text — add Markdown-formatted notes, links to runbooks, or SLO definitions
  CollapsibleGroup — organize widgets into expandable sections for large dashboards


Here is an example Scorecard widget showing current error rate:

# Scorecard widget definition within a dashboard tile
- title: "Current Error Rate"
  height: 2
  width: 3
  widget:
    scorecard:
      scorecardType: LATEST_VALUE
      thresholds:
        - label: "OK"
          value: 0.01
          color: GREEN
        - label: "Warning"
          value: 0.05
          color: YELLOW
        - label: "Critical"
          value: 0.1
          color: RED
      timeSeriesQuery:
        timeSeriesQueryLanguage: >
          fetch prometheus_target
          | metric 'prometheus.googleapis.com/app_requests_total:counter'
          | filter metric.status = '500'
          | rate(5m)
          | group_by [], sum(val())

Best Practices for GKE Monitoring

1. Use Managed Prometheus for Custom Metrics
Avoid running standalone Prometheus servers inside GKE. Managed Prometheus stores data in Cloud Monitoring, which provides unlimited retention (with appropriate pricing tier), seamless alerting integration, and eliminates the operational burden of maintaining Prometheus instances. Migrate existing Prometheus scrape configurations to PodMonitoring and ClusterPodMonitoring resources.

2. Define SLOs and Alert on Error Budgets
Rather than alerting on every transient spike, define Service Level Objectives (SLOs) and alert when your error budget is being consumed too quickly. Cloud Monitoring supports SLO definitions with burn-rate alerts:

# Define an SLO and burn-rate alert via gcloud
gcloud monitoring slo create \
  --service="my-service" \
  --slo-id="availability-slo" \
  --display-name="99.9% Availability" \
  --goal=0.999 \
  --calendar-period="rolling-30d" \
  --good-service-monitoring-filter='metric.type="prometheus.googleapis.com/app_requests_total:counter"
      metric.status!="500"' \
  --total-service-monitoring-filter='metric.type="prometheus.googleapis.com/app_requests_total:counter"'

3. Group Metrics by Meaningful Labels
When designing PodMonitoring resources, include targetLabels that carry business meaning—like app, version, environment, region. Avoid overly granular labels (like pod name) for aggregate metrics, as they create high-cardinality time series that increase cost and slow queries.

4. Build Layered Alerting
Structure alerts in tiers:

  Paging alerts (P1): Immediate human action required—routed to on-call via PagerDuty, firing within minutes
  Warning alerts (P2): Potential issues—routed to Slack or email, firing after a longer duration to reduce noise
  Info alerts (P3): Long-term trends—captured in dashboards, no notification; used for capacity planning


5. Monitor the Monitoring System
Ensure your metrics pipeline is healthy. Create alerts on:

  Managed Prometheus collector DaemonSet pod restarts
  Metric absence for critical custom metrics (heartbeat alert)
  Cloud Monitoring API quota usage approaching limits


6. Use Log-Based Metrics for Legacy or Third-Party Workloads
When you cannot modify an application to expose Prometheus metrics, extract signals from its structured logs. This works well for COTS software, legacy monoliths, or workloads you don't control. Invest time in ensuring logs are consistently structured (JSON) to make extraction reliable.

7. Dashboard Organization
Maintain at least three dashboard types per cluster:

  High-level service dashboard: SLO status, error rates, latency—for product teams and stakeholders
  Infrastructure dashboard: Node health, disk, network—for platform engineers
  Troubleshooting dashboard: Deep-dive charts with drill-downs, recent deployments, pod restarts—for on-call responders


8. Automate Dashboard Deployment
Treat dashboards as code using Terraform or Cloud Deployment Manager. Store dashboard definitions in your infrastructure repository alongside cluster configurations. This ensures dashboards are versioned, reviewable, and automatically deployed with new clusters.

9. Set Up Cost-Aware Monitoring
Monitor resource utilization against requests and limits. Create dashboards showing:

  CPU and memory request vs. actual usage — to identify over-provisioning
  Unused node capacity — to right-size node pools
  Pod density per node — to optimize bin packing


# Query comparing requested CPU vs actual usage
fetch k8s_container
| metric 'kubernetes/container/cpu/limit_utilization'
| group_by [resource.namespace_name, resource.pod_name], mean(val())
| window 30m
| filter val() < 0.1
| map [val(), "Under 10% utilized"]

10. Leverage Cloud Monitoring API for Automation
Use the Cloud Monitoring API to programmatically create alerts, dashboards, and SLOs in CI/CD pipelines. This enables:

  Automatically creating alerts for new services upon deployment
  Generating per-environment dashboards from templates
  Integrating metric queries into deployment health checks


# Python example: Query metrics via Cloud Monitoring API
from google.cloud import monitoring_v3
from datetime import datetime, timedelta
import google.auth

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{project_id}"

# Query last 15 minutes of container CPU
now = datetime.utcnow()
interval = monitoring_v3.TimeInterval(
    end_time={"seconds": int(now.timestamp())},
    start_time={"seconds": int((now - timedelta(minutes=15)).timestamp())}
)

request = monitoring_v3.ListTimeSeriesRequest(
    name=project_name,
    filter='resource.type="k8s_container" AND metric.type="kubernetes/container/cpu/core_usage_time"',
    interval=interval,
    view=monitoring_v3.ListTimeSeriesRequest.TimeSeriesView.FULL
)

for ts in client.list_time_series(request):
    print(f"Pod: {ts.resource.labels['pod_name']}")
    for point in ts.points:
        print(f"  {point.interval.end_time}: {point.value.double_value}")

Conclusion

Monitoring GKE effectively means layering Google Cloud's built-in capabilities with your own application-specific signals. Start with the automatic infrastructure metrics that require zero configuration, then add Managed Prometheus scraping for your custom application metrics, supplement with log-based metrics for signals you cannot instrument directly, and tie everything together with thoughtful alerting policies and well-organized dashboards. The combination of Cloud Monitoring's scalable time-series database, PromQL-compatible querying, and infrastructure-as-code support lets you build a monitoring system that grows with your clusters. Treat dashboards and alerts as code, define SLOs with error-budget-based alerting, and continuously refine your observability stack as your workloads evolve. With the practices and code examples in this tutorial, you have a complete foundation for production-grade GKE monitoring.