After a couple of weeks running IPSec in transport mode between two dedicated servers (i.e. on their respective public IP addresses), things started running amok, with the following messages in the syslogs:
This appears to happen when big enough packets are trying to get through, and things go wrong because of the overhead induced by ESP packets. Also, it would appear my iptables rules are breaking PMTU by refusing certain ICMP packets, which are used in PMTU discovery. I will need to dig deeper into that.
In the meantime, here’s a quick (and dirty?) hack to clamp the Maximum Segment Size for TCP sessions and avoid overloading the packet size:
If you’re using PF, it may look something like this (though I haven’t tested it myself, but that’s the general idea):