[AusNOG] MTU debugging? (Or possibly just a fault with Amazon SES?)

Wed Jun 3 14:47:49 EST 2015

Hi Gents,

Quite a bit of tcpdumping at various places now has me somewhat confused..

When I trace a connection, all is fine until it gets to this packet going
OUT the tunnel interface on the US end of the tunnel:

21:15:47.374650 IP (tos 0x0, ttl 53, id 58671, offset 0, flags [DF], proto
TCP (6), length 1702)
    a13-18.smtp-out.amazonses.com.43433 > cpanel02.rendrag.net.au.smtp:
Flags [P.], cksum 0xe77f (incorrect -> 0x8ad4), seq 2409:4059, ack 4391,
win 441, options [nop,nop,TS val 671045654 ecr 1901069108], length 1650
        0x0000:  4500 06a6 e52f 4000 3506 793b 36f0 0d12  E..../@.5.y;6...

I'm a little confused WHY this packet is being sent out the tunnel though.
The interface MTU is 1452.  But sure enough, as expected the router IS
sending the correct ICMP reply back to amazon:

21:15:47.374664 IP (tos 0xc0, ttl 64, id 27466, offset 0, flags [none],
proto ICMP (1), length 576)
    rtr01.lax01.ca.rendrag.net.au > a13-18.smtp-out.amazonses.com: ICMP
cpanel02.rendrag.net.au unreachable - need to frag (mtu 1452), length 556

However the next packet in is then again over sized:
        IP (tos 0x0, ttl 53, id 58671, offset 0, flags [DF], proto TCP (6),
length 1500)
    a13-18.smtp-out.amazonses.com.43433 > cpanel02.rendrag.net.au.smtp:
Flags [.], seq 2409:3857, ack 4391, win 441, options [nop,nop,TS val 671
045654 ecr 1901069108], length 1448
        0x0000:  45c0 0240 6b4a 0000 4001 adf6 ae88 6c32  E.. at kJ..@.....l2

So the router again sends a frag needed:
21:15:47.762629 IP (tos 0xc0, ttl 64, id 27467, offset 0, flags [none],
proto ICMP (1), length 576)
    rtr01.lax01.ca.rendrag.net.au > a13-18.smtp-out.amazonses.com: ICMP
cpanel02.rendrag.net.au unreachable - need to frag (mtu 1452), length 556

And what do you know, another packet that is too large?
        IP (tos 0x0, ttl 54, id 58673, offset 0, flags [DF], proto TCP (6),
length 1500)
    a13-18.smtp-out.amazonses.com.43433 > cpanel02.rendrag.net.au.smtp:
Flags [.], seq 961:2409, ack 4391, win 441, options [nop,nop,TS val
671046043 ecr 1901069329], length 1448
        0x0000:  45c0 0240 6b4b 0000 4001 adf5 ae88 6c32  E.. at kK..@.....l2

The question then is - am I reading this wrong? My router is sending
frag-needed with max mtu 1452, but then amazon continues to send too-large
packets? Is amazon perhaps seeing this as 'payload max length of 1452' ?
 or should they be seeing it as max total packet length of 1452?

Thanks,

Damien

On 3 June 2015 at 12:31, Mark Andrews <marka at isc.org> wrote:

>
> In message <287364545.4594356.1433297845041.JavaMail.yahoo at mail.yahoo.com>,
> Mar
> k ZZZ Smith writes:
> >
> > MSS hacking is definitely a hack, it doesn't work on non-TCP protocols,
> > and involves looking far deeper into each and every packet to
> > -  see if they're TCP-  find the TCP header (which in theory may not
> > always be in the same place because of IP options)
> > -  then see of they're TCP SYNs, -  find the TCP MSS option, if it exists
> > (it's optional), and if it does,  find where it is, because it isn't
> > required to be in the same place,-  update the TCP MSS value-  since the
> > contents of the TCP header has been intentionally changed, recalculate
> > the TCP header checksum and update that too.
> > and after all that, you still haven't fixed your PMTUD problem, you've
> > just pasted over it for just one of the possible protocols that can be
> > used end-to-end over the Internet (which could include TCP hidden inside
> > some other protocol e.g. GRE, PPTP, IPsec, so you haven't even fixed it
> > for all TCP traffic either). MSS hacking should really only be used when
> > it isn't possible or feasible to fix or avoid PMTUD issues.
> >
> > If you want to avoid the cost of PMTUD for dumbbell MTU paths, it is
> > better to lower the interface MTU on the source and destination hosts, as
> > it will then work for all protocols the hosts' use, not just TCP. There
> > is a DHCPv4 MTU option that can be used to do this, although some hosts
> > may not support it, so it'd be best to check it, and manually change the
> > MTUs on hosts that don't. IPv6 RAs have an MTU option that can be used to
> > lower hosts' interface MTUs, and support for that option by IPv6 hosts is
> > mandatory. You'll lose some performance for transfers between hosts
> > attached to the same LAN with lower host MTUs, but if the majority of
> > traffic is to or from off-link destinations or sources, as most traffic
> > is for most LANs, the loss of LAN performance won't occur very often, and
> > when it does, it may not be significant anyway.
> >
>
> There are also various socket options that can be used to set transmit
> packet sizes at the application level.  Named does this for IPv6 as
> PMTUD (TCP and UDP) as bigger packets really don't provide a benefit
> for DNS.
>
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742                 INTERNET: marka at isc.org
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>

-- 

Damien Gardner Jnr
VK2TDG. Dip EE. GradIEAust
rendrag at rendrag.net -  http://www.rendrag.net/
--
We rode on the winds of the rising storm,
 We ran to the sounds of thunder.
We danced among the lightning bolts,
 and tore the world asunder
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20150603/c41cc10a/attachment.html>