After a couple of weeks running IPSec in transport mode between two dedicated servers (i.e. on their respective public IP addresses), things started running amok, with the following messages in the syslogs:

Mar 16 16:37:29 host1 kernel: [5985158.107191] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:37:42 host1 kernel: [5985171.146710] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:37:43 host1 kernel: [5985172.458608] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:38:10 host1 kernel: [5985199.337572] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:38:12 host1 kernel: [5985201.193461] pmtu discovery on SA ESP/4f014d9c/ddaec880

This appears to happen when big enough packets are trying to get through, and things go wrong because of the overhead induced by ESP packets. Also, it would appear my iptables rules are breaking PMTU by refusing certain ICMP packets, which are used in PMTU discovery. I will need to dig deeper into that.

In the meantime, here’s a quick (and dirty?) hack to clamp the Maximum Segment Size for TCP sessions and avoid overloading the packet size:

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

If you’re using PF, it may look something like this (though I haven’t tested it myself, but that’s the general idea):

match on em0 scrub (max-mss 1440)