[AusNOG] Sensible MTU For Core Routers

Sun Apr 19 12:41:27 EST 2015

Hmm, you've opened the MTU can of worms.
It's wise to set your internal interface MTUs to be both consistent and large enough for the outset to support any sort of tunnelling you might possibly perform across your network internally at any time in the future e.g., GRE, L2TP, IPsec, MPLS. Having a too small MTU is can be a much worse problem to have than a too big MTUs, so in the past I've just set it to the maximum supported by the device. Changing an MTU at a later date can either be very disruptive to the traffic flowing across the interface (because it may have to redo memory buffer allocations, or renegotiate it with the other end e.g., PPP) or may even require a reboot of a device (so service outage, out of hours work etc. and I like to say it takes 10 times the effort to work out of hours, so you avoid it everywhere you can.)
A few other rules about MTUs:
- know what the MTU value actually is describing. Is it a payload MTU (e.g. 1500 octets for Ethernet), a "whole of frame" MTU, which includes overhead fields of the frame or packet (e.g., for Ethernet, 12 octets Dst Addr. & Src. Addr. + 2 octet type/length + 1500 octet payload + 4 octet FCS (checksum) = 1518 octets. (Not that you'll likely to be exposed to it when setting an MTU, however there is also 8 octets of per-frame preamble in Ethernet, as well as an interframe gap that is also measured in bits, if you happen to want to calculate the maximum frames per second a layer 1 link speed of e.g., 10Mbps supports. ))
- VLAN tags or other tunnelling headers/fields may or may not be included in the MTU that you set. For example, if you're setting the "whole of frame MTU" on an interface, the vendor might be limiting it to allow 1 or more VLAN tags to be added. IOW, you're not actually setting the "whole of frame MTU".

- Layer 2 payload MTUs are layer 3 whole of packet MTUs. This becomes important to know because if you use the term "IPv4 MTU" are you describing the IPv4 whole of packet MTU, or the IPv4 payload MTU? The IPv4 payload can vary because of IPv4 options (they're rare, however the one you're likely to see is the Router Alert option), as can the IPv6 payload because of Extension Headers.

- We're now in a multiprotocol world, so it may be possible to set different payload and/or whole of frame MTUs for different protocols. Typically you'll be setting the "Layer 2 MTU", either the payload or whole of frame MTU. However, on routers it is likely to be possible to set different MTUs for different layer 3 protocols, and they'll be the "whole of frame MTU" for the layer 3 protocol. This can be useful to know in a few scenarios, but also explains why there might be multiple per-interface MTU parameters you can set on a router.
- Some links can support different "MTUs" in different directions e.g., PPP. PPP supports setting Max Receive Unit and Maximum Transmit Unit values, and they're announced to each other (or negotiatied, I'm getting a bit rusty)
- in general make try to make sure all devices attached to the same link have the same MTU, otherwise one of the devices can generate a frame that is too big for other devices to receive, and it will be silently dropped. You can be a bit loose with this rule as long as you ensure that all frames that will ever appear on the link will be no larger than the smallest MTU that any of the attached devices (routers, switches, hosts.) support. This is a bit in conflict with my practice of setting it to as large as the device will support. I've figured that a bit of effort to ensure too large frames are originated or forwarded is a lesser effort than the possible effort (out of hours reboot etc.) involved in changing an MTU. I would strictly follow this rule when hosts are attached to the link that I don't administer.
- avoid "dumbell MTU" scenarios i.e., large/small/large MTU values. They'll likely trigger packet fragmentation or path MTU discovery. Fragmentation can't or is costly to do in hardware, and PMTUD can be unreliable either because the ICMP Packet Too Big messages are dropped by overly zealous security people or are rate limited. In the later case, the rate limits can be increased, however they're still generated by the router control plane, which taxes the general purpose control plane CPU. Examples of dumbell MTU scenarios are residential PPPoE access without large PPPoE MTU support (i.e., customer LAN MTU:1500, PPPoE MTU:1492, ISP MTU:1500) or VPN scenarios (e.g., branch LAN MTU:1500, IPsec/L2TP tunnel MTU (much < 1500), central LAN MTU:1500). Lowering the outer MTUs to suit the middle of the dumbell MTU is better, and in IPv4 can be achieved with the DHCP MTU option (assuming the hosts support it) or the IPv6 RA MTU option. (The other option to 'fix' this is the TCP MSS hack, however that only works for TCP, not UDP, IPsec, GRE etc., etc., etc., and is ugly when you look at what it is actually having to do to pull it off (because the TCP MSS option doesn't have to be in a fixed location in the TCP segment.))
- IPv4 minimum whole of packet MTU is 68 octets, IPv6 minimum whole of packet MTU is 1280 octets.

With all those worms, it's probably time to go fishing.

      From: James Mcintosh <james.mcintosh at rocketmail.com>
 To: "ausnog at lists.ausnog.net" <ausnog at lists.ausnog.net> 
 Sent: Sunday, 19 April 2015, 10:53
 Subject: [AusNOG] Sensible MTU For Core Routers

Hey Noggers,
With router MTU's being configurable up to a whopping 9216 what are people setting for their interface MTU's on internet connected core and PE routers?
Is it is still just the default of 1500? What about core switches? What are you setting there to get get interoperability with various ethernet providers?

-James
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20150419/6e9dd7d2/attachment.html>