Monitoring Event Grid: Metrics, Alarms, and Dashboards

Understanding Event Grid Monitoring

Azure Event Grid is a fully managed event routing service that enables you to build reactive, event-driven applications. To ensure reliability, performance, and cost optimization, you need to monitor its operations using metrics, configure proactive alarms (alerts), and visualize data with dashboards. This tutorial walks you through the entire process.

What Is Event Grid Monitoring?

Event Grid monitoring involves tracking key performance indicators like delivery success rates, latency, throttling, and dead-letter counts. These metrics are surfaced through Azure Monitor, providing a unified view of your event-driven pipeline. You can then set up alarms (metric alerts) to notify you when anomalies occur and build dashboards to visualize trends.

Why It Matters

Without proper monitoring, failures in event delivery can go unnoticed, leading to lost data, processing delays, and frustrated downstream consumers. Monitoring helps you:

Detect delivery failures and take corrective action before impact.
Optimize performance by identifying bottlenecks like high latency or throttling.
Control costs by tracking events that exceed quotas or are unnecessarily routed.
Maintain SLAs with real-time visibility and historical reporting.

Key Metrics for Event Grid

Azure Event Grid publishes a set of platform metrics out-of-the-box. These are available in the Azure portal under the Metrics blade, and programmatically via Azure Monitor REST APIs, CLI, or PowerShell. The most important metrics include:

Delivery Success Count – Number of events successfully delivered to endpoints.
Delivery Failure Count – Events that could not be delivered (excluding dead-lettered).
Matched Events Count – Events that matched a subscription filter.
Publish Success Count – Events successfully published to a topic.
Publish Failure Count – Failed publish attempts (e.g., due to throttling).
Destination Delivery Latency (ms) – End-to-end latency from publish to delivery.
Dead Letter Count – Events sent to dead-letter storage after exhausting retries.
Throttled Events Count – Events rejected due to rate limiting.

These metrics are aggregated over 1-minute intervals and retained for 93 days by default. You can query them directly in the Azure Monitor Metrics explorer or via Kusto queries.

Querying Metrics with Azure CLI

Here's how to retrieve the delivery success count for a specific Event Grid topic using the Azure CLI:

az monitor metrics list \
  --resource "/subscriptions/{subscriptionId}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topicName}" \
  --metric "DeliverySuccessCount" \
  --aggregation "Total" \
  --interval 5min \
  --start-time 2025-03-01T00:00:00Z \
  --end-time 2025-03-02T00:00:00Z

This returns time-series data with the total successful deliveries per 5-minute window. You can replace Total with Average, Maximum, or Minimum depending on your needs.

Using Azure Monitor Metrics in Code (C# Example)

If you're building a custom monitoring solution, you can fetch metrics programmatically using Azure Monitor Query libraries:

using Azure.Monitor.Query;
using Azure.Monitor.Query.Models;

var client = new MetricsQueryClient(new DefaultAzureCredential());
var resourceId = "/subscriptions/.../topics/mytopic";

var response = await client.QueryMetricsAsync(
    resourceId,
    new[] { "DeliveryFailureCount" },
    new MetricsQueryOptions
    {
        Granularity = TimeSpan.FromMinutes(5),
        TimeRange = new DateTimeRange(DateTime.UtcNow.AddHours(-6), DateTime.UtcNow)
    });

foreach (var metric in response.Metrics)
{
    Console.WriteLine($"Metric: {metric.Name}");
    foreach (var series in metric.TimeSeries)
    {
        foreach (var value in series.Values)
        {
            Console.WriteLine($"Time: {value.TimeStamp}, Failures: {value.Total}");
        }
    }
}

Setting Up Alarms (Metric Alerts)

Proactive alerting ensures you're notified immediately when a metric breaches a defined threshold. Azure Monitor supports static and dynamic thresholds. For Event Grid, you commonly want alerts for delivery failures, high latency, or throttling.

Creating a Metric Alert for Delivery Failures

You can create alerts via the portal, ARM templates, or CLI. Below is an Azure CLI example that fires when the delivery failure count exceeds 10 over a 5-minute window:

az monitor metrics alert create \
  --name "HighDeliveryFailures" \
  --resource-group "EventGridRG" \
  --scopes "/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topicName}" \
  --condition "avg DeliveryFailureCount > 10" \
  --window-size 5m \
  --evaluation-frequency 5m \
  --description "Alert when delivery failures exceed 10 in 5 minutes" \
  --action "/subscriptions/{subId}/resourceGroups/{rg}/providers/microsoft.insights/actionGroups/{actionGroupName}" \
  --severity 2

Key parameters:

--condition: The threshold expression. Use avg, max, min, or total.
--window-size: The period over which the metric is evaluated.
--evaluation-frequency: How often the rule is evaluated (must be >= window-size).
--action: The Action Group ID (e.g., email, SMS, webhook) to notify.

Dynamic Thresholds for Anomaly Detection

Instead of fixed numbers, you can enable dynamic thresholds that automatically learn the metric's normal baseline and alert on deviations. This is ideal for unpredictable workloads. Use the portal or ARM template with dynamicThreshold condition type. CLI support is limited; use ARM/Bicep:

{
  "type": "Microsoft.Insights/metricAlerts",
  "apiVersion": "2023-01-01",
  "name": "DynamicLatencyAlert",
  "properties": {
    "description": "Alert on anomalous destination delivery latency",
    "severity": 2,
    "enabled": true,
    "scopes": [
      "/subscriptions/.../topics/myTopic"
    ],
    "evaluationFrequency": "PT15M",
    "windowSize": "PT15M",
    "criteria": {
      "odata.type": "Microsoft.Azure.Monitor.MultipleResourceMultipleMetricCriteria",
      "allOf": [
        {
          "criterionType": "DynamicThresholdCriterion",
          "name": "LatencyCheck",
          "metricName": "DestinationDeliveryLatency",
          "operator": "GreaterThan",
          "alertSensitivity": "Medium",
          "failingPeriods": {
            "numberOfEvaluationPeriods": 3,
            "minFailingPeriodsToAlert": 2
          },
          "timeAggregation": "Average"
        }
      ]
    },
    "actions": [
      {
        "actionGroupId": "/subscriptions/.../actionGroups/ops-team"
      }
    ]
  }
}

This alert fires if the average latency exceeds the learned dynamic threshold in 2 out of 3 consecutive evaluation periods.

Building Dashboards for Visual Monitoring

Azure Monitor dashboards let you pin metric charts, logs, and alerts into a single pane of glass. You can create custom dashboards using the portal, or programmatically via the Dashboard API. For Event Grid, a typical dashboard includes:

Delivery success/failure over time.
Latency percentile (P95, P99).
Dead-lettered events trend.
Throttled events heatmap.

Creating a Dashboard Tile with CLI

Here's how to add a metric chart to an existing dashboard using the Azure CLI (requires the portal extension):

az portal dashboard create \
  --name "EventGridOverview" \
  --resource-group "dashboards-rg" \
  --location "westus" \
  --tags "project=eventmonitoring" \
  --definition '{
    "lenses": [
      {
        "order": 0,
        "parts": [
          {
            "position": { "x": 0, "y": 0, "rowSpan": 4, "colSpan": 6 },
            "metadata": {
              "inputs": [
                {
                  "name": "ResourceId",
                  "value": "/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topicName}"
                }
              ],
              "type": "Extension/Microsoft_Azure_Monitoring/PartType/MetricsChartPart",
              "chart": {
                "metrics": [
                  {
                    "resourceId": "/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topicName}",
                    "metric": "DeliveryFailureCount",
                    "aggregation": "Sum",
                    "timeGrain": "PT1H"
                  },
                  {
                    "resourceId": "/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topicName}",
                    "metric": "DeliverySuccessCount",
                    "aggregation": "Sum",
                    "timeGrain": "PT1H"
                  }
                ],
                "title": "Delivery Outcomes (Hourly)",
                "timeRange": {
                  "duration": "PT24H"
                }
              }
            }
          }
        ]
      }
    ]
  }'

This adds a combined chart for successes and failures over the last 24 hours. You can pin similar tiles for latency, dead-letter, etc.

Using Azure Workbooks for Advanced Visualization

Workbooks provide rich, interactive reports combining metrics, logs, and custom queries. They are ideal for drilling down into Event Grid behavior. You can create a workbook that shows delivery latency distribution using Kusto:

// Query from AzureDiagnostics for Event Grid topic
AzureDiagnostics
| where ResourceType == "TOPICS"
| where Category == "Delivery"
| project TimeGenerated, LatencyMs = toint(Properties_LatencyMs), OperationName
| summarize percentiles(LatencyMs, 50, 95, 99) by bin(TimeGenerated, 15m)
| render timechart

Save this query in a Workbook and pin it to a dashboard for continuous visibility.

Best Practices for Event Grid Monitoring

Alert on failures, not just successes. Delivery failures can silently drop events. Set alerts with low thresholds.
Monitor dead-letter queues. Dead-lettered events indicate retry exhaustion; monitor counts and set alerts.
Track latency percentiles. Average latency can hide spikes. Use P95 and P99 in dashboards and dynamic alerts.
Combine metrics and diagnostic logs. Enable diagnostic settings to capture delivery attempts and failures in Log Analytics for deeper investigation.
Use Action Groups effectively. Route alerts to appropriate teams via email, SMS, PagerDuty, or webhooks to trigger automated remediation (e.g., scaling endpoints).
Create separate dashboards for development and production. Production dashboards should focus on reliability; development on throughput and debugging.
Regularly review and tune alert thresholds. As workloads change, adjust static thresholds or rely on dynamic thresholds to reduce noise.
Leverage Azure Policy to enforce monitoring configuration. Ensure every Event Grid topic has diagnostic settings and alert rules deployed via policy.

Conclusion

Monitoring Azure Event Grid is not just about observing metrics—it's about building a resilient event-driven architecture. By leveraging built-in metrics, proactive alerts, and rich dashboards, you can detect issues early, optimize performance, and maintain high availability. Start by identifying the key metrics that matter for your workload, set up intelligent alerts using static or dynamic thresholds, and build visual dashboards that give your team an at-a-glance understanding of system health. With the code samples and best practices in this tutorial, you're now equipped to implement a comprehensive monitoring strategy for Event Grid.