go-pmtud

go-pmtud is a simplified implementation of cloudflare/pmtud in Go.

Problem

Using ECMP (Equal Cost Multi Path) on bare metal Kubernetes clusters makes load sharing of traffic possible (e.g. by using service addresses of type ExternalIP).

Hosts (and Pods) try to leverage full MTU size that is derived from their interface configuration (e.g. 9000 bytes).

If (1) MTU is smaller somewhere in the path between sender and receiver and (2) packet has a DF (do-not-fragment) bit set, router sends ICMP Destination Unreachable message (type 3 code 4 message) to sender (originator of too large packets).

In case of ECMP it may not reach the original sender, thus breaking the communication.

More details in this blog post by Cloudflare: Path MTU discovery in practice.

go-pmtud replicates ICMP Destination Unreachable packets to all nodes in same Kubernetes cluster, so that the sender gets awareness that it has to use smaller packets for a particular destination.

Architecture

iptables (NFLOG group 33)
        │
        ▼
  nflog controller          ← captures…