Monitoring Spanner: Metrics, Alarms, and Dashboards

Monitoring Spanner: Metrics, Alarms, and Dashboards

Google Cloud Spanner is a fully managed, mission-critical relational database service that offers global consistency and horizontal scalability. While Spanner handles much of the operational heavy lifting automatically, effective monitoring remains essential for cost control, performance tuning, and early problem detection. This tutorial covers the metrics landscape, how to build alarms, and how to construct informative dashboards for your Spanner instances.

What Monitoring Surfaces Are Available

Spanner exposes monitoring data through three primary channels:

• Cloud Monitoring (formerly Stackdriver) – The central hub for time-series metrics, dashboards, and alerting policies. Spanner automatically publishes over 60 metrics here.
• Cloud Console Spanner Overview Page – A built-in dashboard showing instance-level CPU utilization, throughput, and storage metrics at a glance.
• Spanner Audit Logs – Administrative and data-access logs that feed into Cloud Logging, which can also generate log-based metrics and alerts.

Key Metrics You Need to Know

Spanner metrics fall into several logical groups. Understanding these categories helps you build targeted dashboards and alarms.

Compute and Capacity Metrics

• Instance CPU utilization – spanner.googleapis.com/instance/cpu/utilization. Tracks CPU usage against allocated capacity. Sustained values above 70% indicate you should consider scaling up.
• Node count – spanner.googleapis.com/instance/node_count. The number of nodes currently allocated to the instance.
• Autoscaling state – Available via the instance/storage_per_node_bytes and autoscaler-specific log entries. Watch for throttling events.

Throughput and Latency Metrics

• Read operations per second – spanner.googleapis.com/instance/read_ops.
• Write operations per second – spanner.googleapis.com/instance/write_ops.
• Average latency per operation type – Metrics like spanner.googleapis.com/instance/read_latency and spanner.googleapis.com/instance/write_latency track p50, p95, and p99 latencies in milliseconds.

Storage Metrics

• Total storage used – spanner.googleapis.com/instance/storage/used_bytes.
• Storage per node – spanner.googleapis.com/instance/storage_per_node_bytes. Critical for capacity planning.

Transaction and Lock Metrics

• Transaction conflict rate – spanner.googleapis.com/instance/transaction/conflicts. A spike here signals contention issues that may require schema or query redesign.
• Lock wait time – Derived from transaction latency breakdowns and lock statistics in the Spanner introspection tables.

Why Monitoring Matters for Spanner

Unlike self-managed databases where you might monitor disk I/O or replication lag directly, Spanner abstracts away much of the infrastructure. Yet monitoring remains critical for three reasons:

• Cost optimization – Spanner charges per node-hour. Without visibility into CPU utilization and storage-per-node ratios, you may be paying for over-provisioned capacity. Monitoring lets you right-size instances or fine-tune autoscaler parameters.
• Performance regression detection – A gradual increase in p99 latency or transaction conflict rate can silently degrade user experience. Early alarms let you investigate before customers notice.
• Capacity planning – Storage grows over time. Monitoring storage per node helps you predict when you'll hit limits and need to scale, avoiding emergency migrations.

How to Use Cloud Monitoring with Spanner

1. Accessing Built-in Spanner Dashboards

The Cloud Console provides a per-instance overview. Navigate to Spanner → Instances → [Your Instance] → Metrics. You'll see pre-built charts for CPU, operations, latency, and storage. This is the quickest way to spot-check instance health.

For a broader view, open Cloud Monitoring → Dashboards → Spanner. This dashboard aggregates metrics across all your Spanner instances, giving you a fleet-wide perspective.

2. Building Custom Dashboards with MQL

Monitoring Query Language (MQL) lets you fetch, filter, and aggregate Spanner metrics with precision. You can create custom dashboards that surface exactly what your team needs.

Here's an MQL query that retrieves CPU utilization for a specific instance, aggregated over 1-minute windows:

# Fetch CPU utilization for a specific Spanner instance
fetch spanner_instance
| metric 'spanner.googleapis.com/instance/cpu/utilization'
| filter resource.instance_id == 'my-production-instance'
| group_by 1m, [value_utilization_mean: mean(value.utilization)]
| every 1m
| window 5m

To create this dashboard via the gcloud CLI, first define the dashboard JSON:

{
  "displayName": "Spanner CPU Overview",
  "dashboardFilters": [],
  "mosaicLayout": {
    "columns": 2,
    "tiles": [
      {
        "title": "CPU Utilization - Prod Instance",
        "widget": {
          "timeSeriesQuery": {
            "timeSeriesQueryLanguage": "fetch spanner_instance\n| metric 'spanner.googleapis.com/instance/cpu/utilization'\n| filter resource.instance_id == 'my-production-instance'\n| group_by 1m, [value_utilization_mean: mean(value.utilization)]\n| every 1m\n| window 5m"
          },
          "plotType": "LINE"
        }
      }
    ]
  }
}

Then apply it with:

gcloud monitoring dashboards create --config dashboard.json

3. Querying Spanner Introspection Tables for Deeper Insights

Spanner exposes built-in introspection tables that provide transaction-level detail, lock statistics, and query performance data. These are queried directly via SQL and complement the Cloud Monitoring metrics.

-- Find the most frequently executed queries in the last 7 days
SELECT 
  query_text,
  request_count,
  avg_latency_seconds,
  avg_cpu_seconds
FROM spanner_sys.query_stats_top_mins(
  60,  -- minutes of history to analyze
  10   -- top N results
)
ORDER BY request_count DESC;

-- Inspect active lock conflicts in real time
SELECT 
  lock_table,
  lock_column,
  transaction_tag,
  wait_time_seconds
FROM spanner_sys.lock_stats_top_tables
ORDER BY wait_time_seconds DESC;

You can schedule a Cloud Function or Dataflow job to periodically query these tables and push derived metrics into Cloud Monitoring as custom metrics, enabling alerting on query-level regressions.

4. Creating Alarms (Alerting Policies)

Alerts turn passive monitoring into active protection. Spanner alerts are configured in Cloud Monitoring under Alerting → Policies. The most impactful alarms for Spanner fall into three tiers:

Tier 1 – Resource Saturation Alarms

• High CPU utilization (sustained above 80% for 10+ minutes)
• Storage per node approaching limits (e.g., above 4 TB per node for standard instances)

Tier 2 – Performance Degradation Alarms

• p99 read latency exceeding a baseline threshold (e.g., 200ms)
• Transaction conflict rate spiking above 1% of total transactions

Tier 3 – Availability Alarms

• Instance-level error rate (monitor spanner.googleapis.com/instance/error metrics)
• Sudden drop in operation count (indicates client connectivity issues)

Here's how to create a CPU utilization alert using gcloud:

gcloud alpha monitoring policies create \
  --display-name="Spanner High CPU - Production" \
  --condition="resource.type=\"spanner_instance\" metric.name=\"spanner.googleapis.com/instance/cpu/utilization\" filter=\"resource.instance_id==\"my-production-instance\"\" aggregations=\"duration=300s,alignment=mean,perSeriesAligner=mean\" comparison=\"COMPARISON_GT\" threshold=0.8 trigger=\"trigger_count=1,trigger_fraction=0.0\" duration=\"300s\"" \
  --notification-channels="projects/my-project/notificationChannels/123456789" \
  --alert-strategy="notification-prompts=OPENED,CLOSED"

For a more readable approach, define the alert in YAML and apply it:

# alert-policy.yaml
displayName: "Spanner High CPU - Production"
conditions:
  - displayName: "CPU utilization above 80% for 5 minutes"
    conditionThreshold:
      filter: >
        resource.type="spanner_instance"
        AND metric.name="spanner.googleapis.com/instance/cpu/utilization"
        AND resource.labels.instance_id="my-production-instance"
      aggregations:
        - alignmentPeriod: 300s
          perSeriesAligner: ALIGN_MEAN
      comparison: COMPARISON_GT
      thresholdValue: 0.8
      duration: 300s
      trigger:
        count: 1
notificationChannels:
  - "projects/my-project/notificationChannels/123456789"
alertStrategy:
  notificationPrompts:
    - OPENED
    - CLOSED

Apply with:

gcloud alpha monitoring policies create --policy-from-file=alert-policy.yaml

5. Setting Up Log-Based Metrics and Alerts

Some Spanner events appear only in audit logs, not in standard metrics. For example, autoscaler decisions, schema change completions, or IAM policy modifications. You can create log-based metrics to track and alert on these.

# Create a log-based metric for autoscaler scale-up events
gcloud logging metrics create spanner-autoscaler-scale-up \
  --description="Count of Spanner autoscaler scale-up events" \
  --log-filter='resource.type="spanner_instance" 
    protoPayload.methodName="google.spanner.admin.instance.v1.InstanceAdmin.Merge" 
    AND protoPayload.entity.metadata.instance.autoscaling="true"'

Once the metric exists, you can create an alerting policy on it just like any built-in metric. This lets you get notified when the autoscaler triggers, helping you correlate cost changes with capacity events.

Building a Comprehensive Spanner Dashboard

A well-designed dashboard tells a story. For Spanner, organize tiles into rows that answer key operational questions:

• Row 1: Is the instance healthy? – CPU utilization, error rate, and node count sparklines.
• Row 2: What's the performance profile? – p50/p95/p99 latency for reads and writes, transaction conflict rate.
• Row 3: What's the throughput? – Operations per second (reads, writes, mutations) stacked as area charts.
• Row 4: What's the storage trajectory? – Storage used, storage per node, and projected growth line.
• Row 5: What are the top queries? – A table widget backed by a Dataflow job that populates a custom metric from introspection table queries.

Here's an MQL snippet for a latency heatmap tile showing p95 read latency across multiple instances:

fetch spanner_instance
| metric 'spanner.googleapis.com/instance/read_latency'
| filter 
    resource.instance_id == 'instance-a' ||
    resource.instance_id == 'instance-b' ||
    resource.instance_id == 'instance-c'
| group_by 1m, [value_p95: percentile(95, value.latency)]
| every 1m
| window 10m

Best Practices for Spanner Monitoring

• Alert on trends, not spikes – Use a duration window of at least 5 minutes for threshold conditions. Transient spikes in CPU or latency are normal during commit bursts; sustained anomalies indicate real problems.
• Separate production and non-production alerts – Tag instances with labels like env:production and env:staging, then filter alerting policies accordingly. Nobody wants to be paged for a staging instance under load test.
• Build a runbook alongside each alarm – Document the triage steps directly in the alert notification (e.g., link to a wiki page). For CPU alarms, the runbook might say: "Check introspection tables for heavy queries, verify autoscaler status, consider manual node increase if throttling persists."
• Monitor cost per operation – Create a custom dashboard tile that divides total node cost (from Cloud Billing metrics) by operation count. A rising cost-per-operation ratio signals inefficiency.
• Use SLO-based alerting where possible – Instead of raw threshold alerts, define Service Level Objectives (SLOs) on latency and error rate, then alert on budget burn rate. This reduces alert fatigue by focusing on user-impacting conditions.
• Retain introspection table data – The spanner_sys tables hold data for a limited window (typically 7 days). Export interesting snapshots to BigQuery for long-term trend analysis of query performance.
• Version-control your dashboards and alerts – Store dashboard JSON and alert policy YAML in your infrastructure repository. Changes can be reviewed in pull requests and applied via CI/CD pipelines, preventing drift between environments.

Automating Dashboard and Alert Deployment with Terraform

Managing monitoring configuration manually becomes unsustainable as your Spanner footprint grows. Terraform's Google Cloud provider supports both dashboard and alerting policy resources:

# terraform spanner-monitoring.tf

resource "google_monitoring_dashboard" "spanner_fleet" {
  dashboard_json = jsonencode({
    displayName = "Spanner Fleet Overview"
    mosaicLayout = {
      columns = 2
      tiles = [
        {
          title   = "CPU Utilization - All Prod"
          widget = {
            timeSeriesQuery = {
              timeSeriesQueryLanguage = <