How to Monitor Server Performance with Prometheus and Grafana

What Are Prometheus and Grafana?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It collects and stores time-series data as key-value pairs with timestamps, using a pull-based model where it scrapes metrics from HTTP endpoints exposed by instrumented applications or exporters. Grafana is a visualization and analytics platform that connects to Prometheus (and many other data sources) to create rich, interactive dashboards. Together, they form the de facto standard for observability in modern infrastructure, from bare-metal servers to Kubernetes clusters.

Why Monitoring Server Performance Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Servers are the backbone of any application. Without proper monitoring, you risk:

Unnoticed resource exhaustion — CPU saturation, memory leaks, or disk space filling up silently
Slow incident response — no historical data means no baseline, making it impossible to determine what "normal" looks like
Capacity planning guesswork — without trend data, you cannot predict when to scale up or replace hardware
Mean-time-to-resolution (MTTR) inflation — engineers waste time SSH-ing into servers to run top, df, or iostat manually during an outage

A well-configured Prometheus + Grafana stack gives you immediate visibility into CPU load, memory usage, disk I/O, network throughput, and hundreds of other metrics — all queryable, graphable, and alertable.

Architecture Overview

The typical monitoring pipeline looks like this:

┌─────────────┐     scrape     ┌───────────────┐     query     ┌──────────────┐
│  Node       │◄──────────────│   Prometheus   │◄─────────────│   Grafana    │
│  Exporter   │  HTTP/metrics │   (time-series  │              │   (dashboards,
│  (on server)│               │    database)    │              │    alerts)   │
└─────────────┘               └───────┬─────────┘              └──────────────┘
                                      │
                                      │ alerts
                                      ▼
                               ┌─────────────────┐
                               │  Alertmanager   │
                               │  (routing,      │
                               │   silencing)    │
                               └─────────────────┘
                                      │
                                      ▼
                              Email / Slack / PagerDuty

Node Exporter — a small daemon you install on each server that exposes system metrics (CPU, memory, disk, network) on port 9100
Prometheus — scrapes the exporter every 15–30 seconds, stores the data, and evaluates alerting rules
Grafana — queries Prometheus via its HTTP API and renders dashboards
Alertmanager — receives alerts from Prometheus, deduplicates, groups, and routes them to notification channels

Step 1 — Installing and Configuring Prometheus

Download and Extract

First, download the latest Prometheus binary for your architecture. The following example uses Linux amd64; adjust the URL from the official download page as needed.

# On the monitoring server (or a dedicated VM)
cd /opt
PROM_VERSION="2.53.0"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xzf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
mv prometheus-${PROM_VERSION}.linux-amd64 prometheus
cd prometheus

Create a System User

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /var/lib/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus /opt/prometheus

Configuration File

Create /opt/prometheus/prometheus.yml — the heart of Prometheus. This file defines global settings, scrape intervals, and scrape targets:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'primary'

# Alertmanager configuration (optional for now)
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

# Load alerting rules from files
rule_files:
  - "alert_rules.yml"

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter on all servers
  - job_name: 'node_exporter'
    static_configs:
      - targets:
          - 'server1.example.com:9100'
          - 'server2.example.com:9100'
          - 'server3.example.com:9100'
        labels:
          environment: 'production'
          datacenter: 'us-east'
    # Optional: add TLS and auth if exporters are behind a reverse proxy
    # scheme: https
    # tls_config:
    #   ca_file: /etc/prometheus/ca.pem
    # basic_auth:
    #   username: exporter_user
    #   password: exporter_pass

Key fields explained:

scrape_interval — how often Prometheus fetches metrics from targets (15s is a good default; for high-resolution needs, 5s works but increases storage)
evaluation_interval — how often alerting rules are evaluated
static_configs — a hardcoded list of targets; for dynamic infrastructure, use file_sd_configs, consul_sd_configs, or kubernetes_sd_configs instead
labels — additional key-value pairs attached to all metrics from this job, useful for filtering in queries

Run Prometheus as a Systemd Service

sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring Service
After=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/prometheus \
  --config.file=/opt/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/opt/prometheus/consoles \
  --web.console.libraries=/opt/prometheus/console_libraries \
  --web.enable-lifecycle \
  --web.external-url=http://monitoring.example.com:9090/
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Verify Prometheus is running by visiting http://YOUR_SERVER_IP:9090. You should see the expression browser and the Status → Targets page showing the Prometheus self-scrape target as green.

Step 2 — Installing Node Exporter on Every Server

Node Exporter exposes system-level metrics that Prometheus scrapes. Install it on every server you want to monitor.

# On each target server
cd /opt
NODE_VERSION="1.8.2"
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_VERSION}/node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
tar xzf node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
mv node_exporter-${NODE_VERSION}.linux-amd64 node_exporter
cd node_exporter

sudo useradd --no-create-home --shell /bin/false node_exporter

Systemd Unit for Node Exporter

sudo tee /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/opt/node_exporter/node_exporter \
  --collector.cpu \
  --collector.meminfo \
  --collector.diskstats \
  --collector.filesystem \
  --collector.netstat \
  --collector.netdev \
  --collector.loadavg \
  --collector.systemd \
  --collector.processes \
  --web.listen-address=:9100
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Verify it's exposing metrics:

curl http://localhost:9100/metrics | head -30
# Should print metrics like:
# node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67
# node_memory_MemTotal_bytes 1.644e+10

Firewall Considerations

Ensure port 9100 is reachable from the Prometheus server. On the target server:

# Using ufw
sudo ufw allow from PROMETHEUS_SERVER_IP to any port 9100 proto tcp

# Using firewalld
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="PROMETHEUS_SERVER_IP" port port="9100" protocol="tcp" accept'
sudo firewall-cmd --reload

Back on the Prometheus server, check Status → Targets in the Prometheus UI. The node_exporter job should show all your servers as UP with green status.

Step 3 — Installing Grafana

Install via Official Repository (Debian/Ubuntu)

sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://packages.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Install via Docker (Alternative)

docker run -d \
  --name=grafana \
  --restart=always \
  -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  -v grafana-provisioning:/etc/grafana/provisioning \
  grafana/grafana:latest

Grafana listens on port 3000 by default. Navigate to http://YOUR_SERVER_IP:3000 and log in with admin / admin (you will be prompted to change the password).

Add Prometheus as a Data Source

Go to Configuration → Data Sources → Add data source
Select Prometheus
Set URL to http://PROMETHEUS_SERVER_IP:9090
Click Save & Test — you should see a green "Data source is working" banner

Step 4 — Building Your First Dashboard

Import the Official Node Exporter Dashboard

The fastest way to get a comprehensive server dashboard is to import the battle-tested community dashboard with ID 1860 (Node Exporter Full) or 11074 (Node Exporter for Prometheus).

Go to Dashboards → Import
Enter 1860 in the "Import via grafana.com" field
Select your Prometheus data source from the dropdown
Click Import

You now have a full server overview showing CPU, memory, disk, network, and more — all populated from Node Exporter metrics.

Understanding the Key Dashboard Panels

Let's break down what the imported dashboard shows and the PromQL queries behind each panel. Understanding these queries is essential for customizing your own dashboards later.

CPU Usage Panel — typically uses a query like:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

This calculates the percentage of time the CPU was not idle over a 5-minute sliding window. The rate() function converts the monotonically increasing counter into a per-second rate, and avg by (instance) averages across all CPU cores for each server.

Memory Usage Panel:

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

Or more commonly, showing percentage used:

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Disk Usage Panel — for each mountpoint:

100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|fuse.*"} / node_filesystem_size_bytes) * 100)

Network Traffic Panel — bytes per second:

rate(node_network_receive_bytes_total{device!="lo"}[5m])

For transmit, replace receive with transmit.

Creating a Custom Dashboard from Scratch

Let's build a simple dashboard manually to understand the process.

Go to Dashboards → New Dashboard
Click Add visualization
Select your Prometheus data source

For a CPU graph panel, enter this PromQL query in the query editor:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Set Panel title to "CPU Usage"
Under Panel options → Legend, enable {{instance}} to show server names
Set Unit (in the right panel under Standard options) to Percent (0-100)
Set Min to 0 and Max to 100
Configure thresholds: green below 70, orange 70–90, red above 90

Repeat for memory, disk, and network panels. Save the dashboard with a descriptive name like "Production Server Overview".

Step 5 — Writing Alerting Rules

Dashboards are reactive; alerts are proactive. Prometheus has a built-in alerting engine that evaluates rules and fires alerts to Alertmanager.

Create Alert Rules File

Create /opt/prometheus/alert_rules.yml (referenced in our prometheus.yml earlier):

groups:
  - name: server_alerts
    rules:
      # High CPU usage
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage has been above 80% for 5 minutes. Current value: {{ $value }}%"
          runbook_url: "https://wiki.example.com/alerts/high-cpu"

      # High memory usage
      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage has exceeded 85% for 10 minutes. Current: {{ $value }}%"

      # Low disk space
      - alert: LowDiskSpace
        expr: 100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|fuse.*"} / node_filesystem_size_bytes) * 100) > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Disk space critical on {{ $labels.instance }}"
          description: "Root filesystem usage is above 90%. Only {{ (query \"node_filesystem_avail_bytes{mountpoint='/'}\") | humanize }} remaining."

      # Server unreachable
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Server {{ $labels.instance }} is DOWN"
          description: "Prometheus cannot scrape {{ $labels.instance }}. The server or exporter may be unreachable."

      # High load average
      - alert: HighLoadAverage
        expr: node_load15 / count by (instance) (node_cpu_seconds_total{mode="system"}) > 1.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High 15-minute load average on {{ $labels.instance }}"
          description: "Load15 exceeds 1.5x CPU count. System may be overloaded."

      # Network errors
      - alert: NetworkErrors
        expr: rate(node_network_receive_errors_total{device!="lo"}[5m]) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Network receive errors on {{ $labels.instance }}"
          description: "Interface {{ $labels.device }} is experiencing receive errors."

Key concepts in alerting rules:

expr — the PromQL expression that triggers the alert when it evaluates to true (non-zero)
for — the duration the expression must remain true before the alert fires; prevents flapping
labels — used for routing in Alertmanager (e.g., severity: critical pages on-call, severity: warning goes to Slack)
annotations — human-readable metadata attached to the alert; {{ $labels.instance }} and {{ $value }} are template variables

Reload Prometheus Configuration

Since we started Prometheus with --web.enable-lifecycle, we can reload without restarting:

curl -X POST http://localhost:9090/-/reload

Verify alerts are loaded at http://PROMETHEUS_IP:9090/alerts. You'll see all defined rules with their current state (inactive, pending, or firing).

Step 6 — Setting Up Alertmanager

Alertmanager handles the routing, deduplication, grouping, and silencing of alerts. Without it, Prometheus would fire individual alerts with no aggregation.

Install Alertmanager

cd /opt
AM_VERSION="0.27.0"
wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar xzf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
mv alertmanager-${AM_VERSION}.linux-amd64 alertmanager
cd alertmanager

sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir -p /var/lib/alertmanager
sudo chown -R alertmanager:alertmanager /var/lib/alertmanager /opt/alertmanager

Alertmanager Configuration

Create /opt/alertmanager/alertmanager.yml:

global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'alertmanager@example.com'
  smtp_auth_password: 'YOUR_APP_PASSWORD'
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

# Routing tree
route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'critical-pager'
      continue: true
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'default-receiver'
    email_configs:
      - to: 'ops-team@example.com'
        send_resolved: true

  - name: 'critical-pager'
    email_configs:
      - to: 'oncall@example.com'
        send_resolved: true
    pagerduty_configs:
      - routing_key: 'YOUR_PAGERDUTY_ROUTING_KEY'
        send_resolved: true

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#ops-alerts'
        send_resolved: true
        title: '{{ .GroupLabels.alertname }} - {{ .CommonLabels.severity }}'
        text: |
          *Alert:* {{ .CommonAnnotations.summary }}
          *Description:* {{ .CommonAnnotations.description }}
          *Runbook:* {{ .CommonAnnotations.runbook_url }}
          *Affected instances:* {{ range .Alerts }}{{ .Labels.instance }} {{ end }}

# Inhibition rules prevent redundant alerts
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['instance']

Systemd Unit for Alertmanager

sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager Service
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
ExecStart=/opt/alertmanager/alertmanager \
  --config.file=/opt/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager/
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager

Verify Alertmanager is running at http://ALERTMANAGER_IP:9093. Now update prometheus.yml to point to Alertmanager (already configured in our example above — just ensure the target is correct) and reload Prometheus.

Step 7 — Advanced PromQL Queries for Server Monitoring

PromQL (Prometheus Query Language) is where the real power lies. Here are essential queries every operator should know:

CPU Saturation

# Per-core CPU utilization (as a decimal fraction)
rate(node_cpu_seconds_total{mode!="idle"}[5m]) / ignoring(mode) group_left
  count without(mode) (node_cpu_seconds_total{mode="system"})

# CPU iowait percentage — indicates disk bottlenecks
avg by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m]) * 100)

Memory Deep Dive

# Available memory in human-readable format via Grafana
node_memory_MemAvailable_bytes

# Swap usage trend
(node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / node_memory_SwapTotal_bytes * 100

# Memory pressure — rate of OOM kills
rate(node_vmstat_oom_kill_total[5m])

Disk I/O Latency

# Disk read latency (seconds per operation)
rate(node_disk_read_time_seconds_total{device=~"sd[a-z]|nvme.*"}[5m]) / 
  rate(node_disk_reads_completed_total{device=~"sd[a-z]|nvme.*"}[5m])

# Disk write latency
rate(node_disk_write_time_seconds_total{device=~"sd[a-z]|nvme.*"}[5m]) / 
  rate(node_disk_writes_completed_total{device=~"sd[a-z]|nvme.*"}[5m])

# Disk utilization percentage (how busy the disk is)
rate(node_disk_io_time_seconds_total{device=~"sd[a-z]|nvme.*"}[5m]) * 100

Network Deep Dive

# TCP connections by state
node_netstat_Tcp_CurrEstab  # Established connections
rate(node_netstat_Tcp_ActiveOpens[5m])  # New connections per second

# Dropped packets (potential NIC saturation or firewall issues)
rate(node_network_receive_drop_total{device!="lo"}[5m])

System Load vs CPU Count

# Load average per CPU — values > 1 indicate saturation
node_load1 / count by (instance) (node_cpu_seconds_total{mode="idle"})

Step 8 — Grafana Dashboard Provisioning and Templating

Using Dashboard Variables

Grafana variables make dashboards dynamic and reusable. Create a variable for instance selection:

Go to Dashboard Settings → Variables → Add variable
Name: instance
Type: Query
Data source: Prometheus
Query: label_values(node_cpu_seconds_total{mode="idle"}, instance)
Multi-value: enabled
Include All option: enabled

Now modify your panel queries to use the variable. Instead of:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Use:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])) * 100)

The $instance variable dynamically filters to the selected servers. With "All" selected, instance=~"$instance" expands to match all instances.

Provisioning Dashboards as Code

For GitOps workflows, Grafana supports provisioning dashboards and data sources via YAML files. Create /etc/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: "15s"

And /etc/grafana/provisioning/dashboards/dashboards.yml:

apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    options:
      path: /var/lib/grafana/dashboards

Place exported dashboard JSON files in /var/lib/grafana/dashboards/ and Grafana will load them on startup. This makes your entire monitoring stack reproducible and version-controllable.

Step 9 — Recording Rules for Performance

When dashboards become complex with many panels all computing rate() on the same raw metrics, query latency increases. Recording rules precompute expensive expressions and store them as new time series.

Add a recording rules section to prometheus.yml:

# Under rule_files: add another file
rule_files:
  - "alert_rules.yml"
  - "recording_rules.yml"

Create /opt/prometheus/recording_rules.yml:

groups:
  - name: server_recording_rules
    interval: 60s
    rules:
      # Precompute per-instance CPU usage
      - record: instance:node_cpu_usage:rate5m
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

      # Precompute memory usage percentage
      - record: instance:node_memory_usage:percent
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

      # Precompute disk usage percentage per mountpoint
      - record: instance_device:node_disk_usage:percent
        expr: 100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|fuse.*"} / node_filesystem_size_bytes) * 100)

      # Precompute network bytes per second
      - record: instance_device:node_network_recv_bytes:rate5m
        expr: rate(node_network_receive_bytes_total{device!="lo"}[5m])
      
      - record: instance_device:node_network_trans_bytes:rate5m
        expr: rate(node_network_transmit_bytes_total{device!="lo"}[5m])

Now your dashboard queries can use the simpler, precomputed metric names like instance:node_cpu_usage:rate5m instead of the raw PromQL. This dramatically speeds up dashboard loading and reduces Prometheus CPU usage during query peaks.

Best Practices

Start with the USE and RED methodologies:
- USE (Utilization, Saturation, Errors) — for infrastructure like servers: monitor CPU utilization, memory saturation, disk errors
- RED (Rate, Errors, Duration) — for services: request rate, error rate, latency
Right-size scrape intervals — 15s is a sweet spot. Faster scraping (5s) gives better resolution but increases storage and CPU. Slower (30–60s) is acceptable for stable infrastructure with hundreds of targets.
Set appropriate retention — Prometheus defaults to 15 days of retention in its TSDB. For long-term trends, configure remote_write to stream data to a long-term storage backend like Thanos, Cortex, or Grafana Mimir.
```
# In prometheus.yml for remote write
remote_write:
  - url: "http://mimir.example.com/api/v1/push"
```
Use external labels for federation — the external_labels in the global config uniquely identify each Prometheus instance. When aggregating data from multiple Prometheuses, these labels prevent metric collisions.
Secure your endpoints — Node Exporter exposes all system metrics without authentication by default. Place it behind a reverse proxy (nginx, Caddy) with TLS and basic auth, or use --web.config.file for native TLS support in newer versions.
Template alerts with runbook URLs — every alert annotation should include a runbook_url linking to a playbook that describes exactly what to check and how to resolve the issue. This reduces MTTR dramatically.
Version-control everything — prometheus.yml, alert rules, recording rules