<div dir="ltr">I saw degradation on one of my PIPE internet services at around 10:17 as well, but it really only seemed to affect traffic from us to Salesforce in Europe. At least, thats what most of the complaints were about.<div><br></div><div>traceroute to salesforce europe showed YUGE latency, in the order of 3+ seconds rtt, and ping showed stacks of timeouts.</div><div><br></div><div>and then at about 11:05 for us it instantly came good.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 24, 2017 at 11:24 AM, Sam McLeod <span dir="ltr"><<a href="mailto:samm@infoxchange.org" target="_blank">samm@infoxchange.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 24/1/17, 11:18 am, "AusNOG on behalf of Sam McLeod" <<a href="mailto:ausnog-bounces@lists.ausnog.net">ausnog-bounces@lists.ausnog.<wbr>net</a> on behalf of <a href="mailto:samm@infoxchange.org">samm@infoxchange.org</a>> wrote:<br>
<br>
[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at <a href="http://aka.ms/LearnAboutSpoofing" rel="noreferrer" target="_blank">http://aka.ms/<wbr>LearnAboutSpoofing</a>]<br>
<br>
On 24/1/17, 11:03 am, "AusNOG on behalf of Sam McLeod" <<a href="mailto:ausnog-bounces@lists.ausnog.net">ausnog-bounces@lists.ausnog.<wbr>net</a> on behalf of <a href="mailto:samm@infoxchange.org">samm@infoxchange.org</a>> wrote:<br>
<br>
[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at <a href="http://aka.ms/LearnAboutSpoofing" rel="noreferrer" target="_blank">http://aka.ms/<wbr>LearnAboutSpoofing</a>]<br>
<br>
There is a major packet loss and routing issue occurring with TPG/PIPE (and perhaps AAPT)<br>
Symptoms are 40-100% which has essentially taken all our applications and services offline on this link.<br>
<br>
We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re denying any outage and stating they can’t detecting any problems and have no alerts which is the usual response for major outages from PIPE/TPG, we’re going to keep pushing them to fix the issue but are left waiting for a response in the mean time.<br>
<br>
<br>
~| mtr 8.8.8.8 --report-wide -c 40<br>
<br>
HOST: Loss% Snt Last Avg Best Wrst StDev<br>
<br>
4.|-- <a href="http://ve100.rv-376flinders-mlx-01.tpg-telecom.net" rel="noreferrer" target="_blank">ve100.rv-376flinders-mlx-01.<wbr>tpg-telecom.net</a> 0.0% 40 10.6 10.5 10.4 10.9 0.0<br>
5.|-- <a href="http://eth10-7.rn-400harris-mlx-03.tpg-telecom.net" rel="noreferrer" target="_blank">eth10-7.rn-400harris-mlx-03.<wbr>tpg-telecom.net</a> 0.0% 40 10.6 10.6 10.4 13.0 0.3<br>
6.|-- <a href="http://eth2-1.rn-400harris-cer-03.tpg-telecom.net" rel="noreferrer" target="_blank">eth2-1.rn-400harris-cer-03.<wbr>tpg-telecom.net</a> 0.0% 40 10.5 10.9 10.5 17.9 1.3<br>
7.|-- <a href="http://as15169.sydney.pipenetworks.com" rel="noreferrer" target="_blank">as15169.sydney.pipenetworks.<wbr>com</a> 15.0% 40 76.4 49.6 11.0 79.2 30.9<br>
8.|-- 216.239.51.218 42.5% 40 66.3 38.0 11.3 76.8 31.2<br>
9.|-- 216.239.40.31 12.5% 40 174.6 152.0 109.1 195.5 32.6<br>
10.|-- 209.85.246.221 30.0% 40 190.6 159.1 125.1 191.9 31.1<br>
11.|-- 209.85.142.132 17.5% 40 190.6 161.9 118.8 207.3 32.0<br>
12.|-- 72.14.232.129 15.0% 40 211.5 185.2 146.3 212.0 30.7<br>
13.|-- 209.85.245.16 22.5% 40 211.2 182.3 146.1 214.6 30.7<br>
14.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0<br>
<br>
--<br>
Sam McLeod | Systems Lead<br>
Please contact me via email when possible.<br>
Infoxchange - Technology For Social Justice | Infoxchange.org | 33 Elizabeth Street, Richmond Victoria 3121<br>
<br>
<br>
Update as of 11.10AM AEDT<br>
<br>
TPG/PIPE still unaware of any issues although I’ve heard directly from several other customers that are experiencing major issues.<br>
Their L3 support team said that they need to wait 30 minutes for the NOC to respond even if it’s a P1 issue.<br>
<br>
Current traceroute:<br>
<br>
~| mtr 8.8.8.8 --report-wide -c 40<br>
<br>
HOST: Loss% Snt Last Avg Best Wrst StDev<br>
<br>
4.|-- <a href="http://ve100.rv-376flinders-mlx-01.tpg-telecom.net" rel="noreferrer" target="_blank">ve100.rv-376flinders-mlx-01.<wbr>tpg-telecom.net</a> 0.0% 100 10.6 10.6 10.4 20.6 1.0<br>
5.|-- <a href="http://eth10-7.rn-400harris-mlx-03.tpg-telecom.net" rel="noreferrer" target="_blank">eth10-7.rn-400harris-mlx-03.<wbr>tpg-telecom.net</a> 0.0% 100 10.5 10.6 10.3 12.9 0.4<br>
6.|-- <a href="http://eth2-1.rn-400harris-cer-03.tpg-telecom.net" rel="noreferrer" target="_blank">eth2-1.rn-400harris-cer-03.<wbr>tpg-telecom.net</a> 0.0% 100 10.6 10.7 10.4 12.5 0.3<br>
7.|-- <a href="http://as15169.sydney.pipenetworks.com" rel="noreferrer" target="_blank">as15169.sydney.pipenetworks.<wbr>com</a> 13.0% 100 11.3 28.7 11.1 78.3 27.7<br>
8.|-- 216.239.51.218 5.0% 100 11.3 32.6 11.2 77.1 30.0<br>
9.|-- 216.239.40.31 14.0% 100 110.3 128.5 109.1 200.6 28.2<br>
10.|-- 209.85.142.130 13.0% 100 119.6 136.3 119.3 186.1 27.9<br>
11.|-- 209.85.249.53 9.0% 100 147.3 166.1 147.0 213.6 28.6<br>
12.|-- 64.233.175.209 12.0% 100 146.8 163.6 145.7 221.3 27.8<br>
13.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0<br>
<br>
<br>
Update as of 11:15AM AEDT<br>
<br>
TPG/PIPE L3 support have provided the information to their NOC team that is unaware of any issues.<br>
They’ll investigate it when they’re able to.<br>
<br>
We have noticed some route changes and now some national traffic such as Google cached / CDN services and 8.8.8.8 are now routing from Australia to the USA rather than to Sydney.<br>
<br>
No ETA from PIPE/TPG<br>
<br>
<br>
</div></div>Update as of 11:22AM AEDT<br>
<br>
PIPE/TPG still have no acknowledged the issue however it does appear that they have failed over some routes from Sydney.<br>
<br>
We are no seeing no packet loss; however, some IPs are routing internationally.<br>
<br>
~| mtr 8.8.8.8 --report-wide -c 10<br>
<span class=""><br>
HOST: Loss% Snt Last Avg Best Wrst StDev<br>
<br>
<br>
</span> 4.|-- <a href="http://ve100.rv-376flinders-mlx-01.tpg-telecom.net" rel="noreferrer" target="_blank">ve100.rv-376flinders-mlx-01.<wbr>tpg-telecom.net</a> 0.0% 10 10.6 10.6 10.4 10.7 0.0<br>
5.|-- <a href="http://eth10-7.rn-400harris-mlx-03.tpg-telecom.net" rel="noreferrer" target="_blank">eth10-7.rn-400harris-mlx-03.<wbr>tpg-telecom.net</a> 0.0% 10 10.5 10.4 10.3 10.7 0.0<br>
6.|-- <a href="http://eth2-1.rn-400harris-cer-03.tpg-telecom.net" rel="noreferrer" target="_blank">eth2-1.rn-400harris-cer-03.<wbr>tpg-telecom.net</a> 0.0% 10 10.6 10.6 10.4 10.7 0.0<br>
7.|-- <a href="http://as15169.sydney.pipenetworks.com" rel="noreferrer" target="_blank">as15169.sydney.pipenetworks.<wbr>com</a> 0.0% 10 11.2 11.3 11.1 12.2 0.0<br>
8.|-- 216.239.51.218 0.0% 10 11.4 11.4 11.2 11.7 0.0<br>
9.|-- 216.239.40.31 0.0% 10 109.4 111.5 109.2 129.7 6.4<br>
10.|-- 209.85.142.130 0.0% 10 119.4 119.8 119.0 123.8 1.3<br>
11.|-- 209.85.249.53 0.0% 10 147.3 147.2 147.0 147.5 0.0<br>
12.|-- 64.233.175.209 0.0% 10 147.2 146.1 145.6 147.2 0.3<br>
13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<wbr>_________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/<wbr>mailman/listinfo/ausnog</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Kind Regards,<br><br>Christopher Barnes<br><br>e. <a href="mailto:chris.p.barnes@gmail.com" target="_blank">chris.p.barnes@gmail.com</a><br></div>
</div>