Flannel Networking: Complete Setup and Configuration Guide

Successful ping responses indicate Flannel is working correctly. If pings fail, proceed to the troubleshooting section below.

Customizing Flannel Configuration

Modifying the kube-flannel ConfigMap

Flannel reads its configuration from a ConfigMap named kube-flannel-cfg in the kube-system namespace. To customize the network CIDR, backend type, or other parameters, edit this ConfigMap before deploying or update it and restart the flanneld pods:

# View the current configuration
kubectl -n kube-system get configmap kube-flannel-cfg -o yaml

# Edit the configuration
kubectl -n kube-system edit configmap kube-flannel-cfg

Here is a complete example of a custom ConfigMap using the host-gw backend with a different pod network CIDR:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "172.16.0.0/16",
      "Backend": {
        "Type": "host-gw"
      }
    }

After updating the ConfigMap, restart the Flannel DaemonSet to apply changes:

# Delete existing pods to force recreation with new config
kubectl -n kube-system delete pods -l app=flannel

# Watch pods come back online
kubectl -n kube-system get pods -l app=flannel -w

Enabling Direct Server Return (DSR) with host-gw

When using host-gw backend in environments where nodes have multiple network interfaces, you may want to specify the exact external interface for routing. Use the "DirectRouting" option combined with explicit node IP annotations:

net-conf.json: |
  {
    "Network": "10.244.0.0/16",
    "Backend": {
      "Type": "host-gw",
      "DirectRouting": true
    }
  }

Advanced Flannel Configuration

Configuring Flannel with External etcd

While the default deployment uses the Kubernetes API server for subnet management, Flannel can also use an external etcd cluster. This is useful in bootstrap scenarios or when running Flannel outside of Kubernetes:

# Start flanneld with etcd backend
/opt/bin/flanneld \
  --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379 \
  --etcd-prefix=/coreos.com/network \
  --etcd-keyfile=/etc/ssl/etcd/key.pem \
  --etcd-certfile=/etc/ssl/etcd/cert.pem \
  --etcd-cafile=/etc/ssl/etcd/ca.pem

# Pre-configure etcd with network settings
etcdctl --endpoints=http://10.0.0.1:2379 \
  set /coreos.com/network/config \
  '{"Network":"10.244.0.0/16","Backend":{"Type":"vxlan","Port":8472}}'

WireGuard Backend for Encrypted Tunnels

The WireGuard backend encrypts all node-to-node traffic, which is essential in multi-tenant or untrusted network environments. To enable it, you must have WireGuard kernel modules loaded on all nodes:

# Install WireGuard tools on each node
sudo apt-get install wireguard-tools

# Verify WireGuard kernel module
lsmod | grep wireguard

# Configure Flannel for WireGuard
net-conf.json: |
  {
    "Network": "10.244.0.0/16",
    "Backend": {
      "Type": "wireguard",
      "PersistentKeepaliveInterval": 25
    }
  }

The PersistentKeepaliveInterval option (in seconds) helps maintain tunnels through NAT gateways by periodically sending keepalive packets.

Specifying Node Network Interface

In multi-homed nodes with several network interfaces, flanneld may select the wrong interface for tunnel traffic. Override this by setting the --iface or --public-ip flags in the DaemonSet pod spec:

# Relevant section of the DaemonSet container args
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=ens192
- --public-ip=$(NODE_IP)

IP Masquerade Configuration

Flannel includes an IP masquerade rule by default (the --ip-masq flag) that NATs pod traffic exiting the node to the node's primary IP. This is essential for pods to reach external networks. You can disable it if you have other NAT rules or if you want to route pod IPs directly:

# Disable masquerade in the DaemonSet
args:
- --kube-subnet-mgr
# Remove --ip-masq from args list

When disabling masquerade, ensure your upstream network knows how to route pod IP ranges back to the respective nodes.

Troubleshooting Flannel Networking

Common Issues and Diagnostic Commands

Flannel problems typically manifest as pod-to-pod communication failures across nodes. Here is a systematic diagnostic approach:

# 1. Verify Flannel pods are healthy
kubectl -n kube-system get pods -l app=flannel
kubectl -n kube-system describe pod -l app=flannel

# 2. Check flanneld logs for subnet allocation issues
kubectl -n kube-system logs -l app=flannel --tail=100 | grep -i error

# 3. Verify the flannel virtual interface exists
ip link show flannel.1
ip addr show flannel.1

# 4. Check routing table entries managed by Flannel
ip route | grep flannel
# Expected output shows routes to remote pod subnets via flannel.1

# 5. Verify VXLAN UDP port is open (for VXLAN backend)
sudo tcpdump -i eth0 udp port 8472 -c 10

# 6. Check node subnet annotations
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

# 7. Inspect Flannel's internal network configuration
kubectl -n kube-system exec -it $(kubectl -n kube-system get pod -l app=flannel -o jsonpath='{.items[0].metadata.name}') -- cat /run/flannel/subnet.env

Diagnosing VXLAN Encapsulation Issues

If you suspect VXLAN forwarding problems, capture traffic on both the source and destination nodes:

# On source node: capture VXLAN traffic to destination node IP
sudo tcpdump -i eth0 -n "udp port 8472 and host " -w flannel-vxlan.pcap

# On destination node: capture incoming VXLAN traffic
sudo tcpdump -i eth0 -n "udp port 8472 and host " -w flannel-incoming.pcap

# Analyze the captures
tcpdump -r flannel-vxlan.pcap -n -v

Fixing Stale Subnet Allocations

When a node is removed from the cluster without proper cleanup, its subnet allocation may remain reserved, preventing new nodes from getting subnets. Clear stale allocations:

# List all subnet allocations stored in Kubernetes
kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR

# If a node no longer exists but its subnet is still allocated,
# check Flannel's lease cleanup mechanism or manually remove
# the node object if it's truly gone
kubectl delete node 

# Force flanneld to re-allocate by restarting pods
kubectl -n kube-system delete pods -l app=flannel

MTU Considerations

VXLAN encapsulation adds overhead (approximately 50 bytes for the VXLAN header plus UDP and IP headers). If your physical network MTU is 1500, pods will see a reduced MTU. Flannel automatically sets the correct MTU on the flannel.1 interface, but verify it matches your expectations:

# Check the MTU on the flannel interface
ip link show flannel.1 | grep mtu

# Verify pod interfaces have correct MTU
kubectl exec -it  -- ip link show eth0 | grep mtu

If you're using jumbo frames (MTU 9000) on your physical network, you can configure Flannel to take advantage of them by setting the MTU in the backend configuration.

Flannel in Production: Best Practices

Choose the Right Backend for Your Environment

• On-premises bare metal with flat network — use host-gw for maximum throughput and minimal overhead
• Cloud environments or routed networks — use VXLAN for universal compatibility
• Security-sensitive deployments — use WireGuard backend to encrypt all inter-node traffic
• AWS deployments — consider aws-vpc backend to offload routing to VPC route tables

Dedicated Pod Network CIDR Planning

Allocate a pod network CIDR that is large enough for your cluster's growth. A /16 provides 255 node subnets; if you expect more than 200 nodes, use a /12 or /8. Reserve this CIDR exclusively for pods — never let it overlap with service CIDRs, node IP ranges, or VPN address spaces. Document the allocation in your network architecture diagrams.

Monitoring Flannel Health

Set up monitoring for Flannel-specific metrics and logs. Key indicators to watch:

• Flannel pod restarts — frequent restarts indicate configuration or resource issues
• Subnet allocation failures — visible in flanneld logs as "failed to register subnet" errors
• VXLAN packet loss — monitor UDP port 8472 traffic for drops
• Node pod CIDR assignment — every node should have a non-empty podCIDR in its spec

# Example Prometheus alert for Flannel pod status
alert: FlannelDaemonSetUnhealthy
expr: |
  sum(kube_daemonset_status_number_ready{
    namespace="kube-system",
    daemonset="kube-flannel-ds"
  }) < sum(kube_node_info)
for: 5m
labels:
  severity: critical
annotations:
  summary: "Flannel pods missing on some nodes"

Securing Flannel Traffic

By default, VXLAN traffic is unencrypted. If your nodes communicate over untrusted networks, implement one of these strategies:

• Use the WireGuard backend for built-in encryption
• Apply IPsec transport mode encryption between nodes at the system level
• Restrict VXLAN UDP port 8472 to only the cluster node IPs using firewall rules

# Example iptables rule to restrict VXLAN traffic to cluster nodes
iptables -A INPUT -p udp --dport 8472 \
  -s  -j ACCEPT
iptables -A INPUT -p udp --dport 8472 -j DROP

# Example firewall rules for WireGuard port
iptables -A INPUT -p udp --dport 51820 \
  -s  -j ACCEPT
iptables -A INPUT -p udp --dport 51820 -j DROP

Upgrading Flannel

Flannel upgrades are typically handled by applying the updated manifest. The DaemonSet's update strategy should be set to RollingUpdate with appropriate maxUnavailable values to ensure zero-downtime upgrades:

# Check current update strategy
kubectl -n kube-system get daemonset kube-flannel-ds -o yaml | grep -A5 updateStrategy

# Apply new manifest
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

# Monitor rollout
kubectl -n kube-system rollout status daemonset/kube-flannel-ds

# Verify all pods updated
kubectl -n kube-system get pods -l app=flannel -o wide

During upgrades, pods on a node may experience brief network interruptions while the local flanneld restarts. Schedule upgrades during maintenance windows for production clusters and test the new version in a staging environment first.

Resource Limits and Node Sizing

Flanneld is lightweight but should have appropriate resource limits to prevent it from consuming excessive resources under abnormal conditions:

# Example resource limits in DaemonSet pod spec
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 256Mi

Migrating Between Flannel Backends

Changing from one backend to another (e.g., VXLAN to host-gw) requires careful planning because it affects routing for all pods. The safest approach is a rolling migration:

# 1. Update the ConfigMap with the new backend configuration
kubectl -n kube-system apply -f - <