[AusNOG] Upstream PMTUD broken? Packets blackhole

[Tim Warnock]

Just to add to the case:

I had two dsl services and an nbn service where we had to drop the dialer ip mtu to 1460 (the services had worked forever(tm) on the old settings) because we don't know what changed.

Things that worked and that had worked forever suddenly didn't and that was the fix.

Go figure.

Extra info: 2x ADSL2+ on TW DSLAM, 1x NBN via AAPT NWB.

-----Original Message-----
From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of s.s.o.n.i.k+a.u.s.n.o.g at gmail.com
Sent: Thursday, 15 September 2016 1:42 AM
To: 'Mark Newton' <newton at atdot.dotat.org>
Cc: 'Cameron Ferdinands' <cameron at jferdinands.com>; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Upstream PMTUD broken? Packets blackhole

I’ve performed packet capture on the traffic passing both directions through LAN side interfaces and WAN side interfaces, including capturing intermediary packets (ie. packet fragments before and after reassembly). The MRU value does mirror the MTU value, and I haven’t overridden the default behaviour, nor renegotiated PPP (as it would defeat the purpose if the other endpoint doesn’t send packets larger than the MRU here). The firewall generates and transmits ICMP messages WAN side when packets arrive with lengths between 1457 bytes and 1464 bytes. No larger packets arrive, as they are dropped due to the MTU on the dialer being set to 1492. The dialer also generates ICMP messages, and I specifically requested that a public IP be configured, as that device borders two different MTU domains. The dialer’s ICMP messages never reach the sending device either. I’ve also verified that I’m seeing correct behaviour when testing between two of the ISPs tails.
 
Believe me, I’m aware of how noobish this sounds. I’ve been in the industry 15 years, and make a point of training incoming techs on the importance of the MTU value, what it’s for, the differences between MTU/MRU and MSS. (I skip the explanation about L2MTU and L3 MTU. If they’re working with that equipment and need me to explain that the L2MTU has to be large enough to hold packets of sizes the L3 MTU specifies, then there’s been a hiring error).
 
If this had only affected one client, I certainly wouldn’t embarrass myself posting about it here. However it’s affecting many clients simultaneously, and these were devices that all worked fine before. Even clients with 1500 MTU links are having trouble – the divide is smaller, but there still should not be a 4 byte gap between the largest successful packet transmission and receipt of an ICMP message indicating fragmentation required.
 
I’m wondering if someone forgot to tune the ICMP rate limit somewhere. The default limit of one or two ICMP 3:4 messages per interface per second is functionally equivalent to blocking it altogether. Especially if the device borders MTU domains and carries countless thousands of flows.
 
 
Kind Regards,
 
-Dave
 
 
Unplug the firewall's outside interface.
Plug in your laptop. 
Give it the firewall's outside IP address, don't mess with the MTU.
Test.
Reconnect the firewall.
 
My suspicion is that the tests will be just fine, and that the reason it isn't working is because your firewall silently lowered its MRU to 1484 too when you lowered the MTU, and its TCP stack only sends ICMP messages when it can't SEND frames, not when it can't receive them; which means everything works fine until the Internet tries to send you a 1500 byte packet, then it all goes to hell in a hand basket because you never receive it and never tell the sender.
 
This is an extremely well trodden path.
 
   - mark