<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span lang=EN-AU>Hi all,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>Starting about four weeks ago, many of my customers began reporting weird problems with their internet connections. Multiple ISPs, multiple upstream providers, no config changes to routers/firewalls – yet “heavy” websites stop loading, SSL errors show up continuously, downloads and uploads fail, SMTP delays, email vanishing in transit, frequent timeouts, VPN failures, YouTube and Netflix video quality degraded through the floor, and just generally slow and dodgy feeling internet connections.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>It feels like an MTU issue, or overzealous ICMP 3:4 filtering.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>I’ve done as much testing and investigation as I can do, and the every ISPs has pointed to the cause of the problem being with their Tier 1 / Tier 2 providers. Traceroute from affected customers pass through Vocus or Telstra, and seem to all pass through IXs in NSW. If I’m lucky the traces complete, but “request timed out” *** often shows up where it should not.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>Results of MTU testing show gaps of varying sizes between the largest successfully transmitted packet, and the size where ICMP 3:4 “frag needed” response is returned.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>Right now, I’m pinging an affected client’s router from New York with three different sized packets. All are a stream of “request timed out” errors. All have the DF bit set, and use lengths one byte larger than the MTU each device expected to accept.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>Client firewall MTU set to 1484: Successful reply at length 1456. Unsuccessful at 1457, however echo request is seen, and unreachable 3:4 message is returned by firewall, but never arrives at the source.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>PPPoE Dialer MTU set to 1492: Echo requests are seen hitting the firewall at lengths up to 1464 (expected), yet no 3:4 replies make it back to the source. At 1465, no longer see packets (also expected) as PPPoE Dialer is now sending unreachable 3:4 responses. These are also failing to make it back to the source device.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>Regular Ethernet MTU 1500: Completely unsuccessful right now. Mainly to test upper bounds of this packet blackhole.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>As for Telstra connections, they’re also experiencing the blackhole issue, with additional intermittent periods of heavy packet loss. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>Curiously, I’m seeing the odd echo response from Telstra make it all the way back to the source server:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>....<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>Request timed out.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>Request timed out.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>Reply from 123.123.123.123: bytes=1469 time=284ms TTL=51<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>Request timed out.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>Request timed out.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>....<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>(Note: 1468 byte packets reply successfully every single time from this IP).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>Has anyone else been seeing similar issues recently?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU>Any help would be greatly appreciated.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-AU>-Dave<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p></div></body></html>