[AusNOG] Major packet loss and routing issues with TPG/PIPE
Brad Peczka
brad at bradpeczka.com
Tue Jan 24 12:21:49 EST 2017
It seems like the suggestion from the other day, of a dedicated AAPT/TPG/PIPE/et al outages list, is becoming more and more necessary.
Regards,
-Brad.
________________________________________
From: AusNOG [ausnog-bounces at lists.ausnog.net] On Behalf Of Sam McLeod [samm at infoxchange.org]
Sent: Tuesday, 24 January 2017 8:35 AM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] Major packet loss and routing issues with TPG/PIPE
There is a major packet loss and routing issue occurring with TPG/PIPE (and perhaps AAPT)
Symptoms are 40-100% which has essentially taken all our applications and services offline on this link.
We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re denying any outage and stating they can’t detecting any problems and have no alerts which is the usual response for major outages from PIPE/TPG, we’re going to keep pushing them to fix the issue but are left waiting for a response in the mean time.
~| mtr 8.8.8.8 --report-wide -c 40
HOST: Loss% Snt Last Avg Best Wrst StDev
4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 40 10.6 10.5 10.4 10.9 0.0
5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 40 10.6 10.6 10.4 13.0 0.3
6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 40 10.5 10.9 10.5 17.9 1.3
7.|-- as15169.sydney.pipenetworks.com 15.0% 40 76.4 49.6 11.0 79.2 30.9
8.|-- 216.239.51.218 42.5% 40 66.3 38.0 11.3 76.8 31.2
9.|-- 216.239.40.31 12.5% 40 174.6 152.0 109.1 195.5 32.6
10.|-- 209.85.246.221 30.0% 40 190.6 159.1 125.1 191.9 31.1
11.|-- 209.85.142.132 17.5% 40 190.6 161.9 118.8 207.3 32.0
12.|-- 72.14.232.129 15.0% 40 211.5 185.2 146.3 212.0 30.7
13.|-- 209.85.245.16 22.5% 40 211.2 182.3 146.1 214.6 30.7
14.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0
--
Sam McLeod | Systems Lead
Please contact me via email when possible.
Infoxchange - Technology For Social Justice | Infoxchange.org | 33 Elizabeth Street, Richmond Victoria 3121
Update as of 11.10AM AEDT
TPG/PIPE still unaware of any issues although I’ve heard directly from several other customers that are experiencing major issues.
Their L3 support team said that they need to wait 30 minutes for the NOC to respond even if it’s a P1 issue.
Current traceroute:
~| mtr 8.8.8.8 --report-wide -c 40
HOST: Loss% Snt Last Avg Best Wrst StDev
4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 100 10.6 10.6 10.4 20.6 1.0
5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 100 10.5 10.6 10.3 12.9 0.4
6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 100 10.6 10.7 10.4 12.5 0.3
7.|-- as15169.sydney.pipenetworks.com 13.0% 100 11.3 28.7 11.1 78.3 27.7
8.|-- 216.239.51.218 5.0% 100 11.3 32.6 11.2 77.1 30.0
9.|-- 216.239.40.31 14.0% 100 110.3 128.5 109.1 200.6 28.2
10.|-- 209.85.142.130 13.0% 100 119.6 136.3 119.3 186.1 27.9
11.|-- 209.85.249.53 9.0% 100 147.3 166.1 147.0 213.6 28.6
12.|-- 64.233.175.209 12.0% 100 146.8 163.6 145.7 221.3 27.8
13.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
Update as of 11:15AM AEDT
TPG/PIPE L3 support have provided the information to their NOC team that is unaware of any issues.
They’ll investigate it when they’re able to.
We have noticed some route changes and now some national traffic such as Google cached / CDN services and 8.8.8.8 are now routing from Australia to the USA rather than to Sydney.
No ETA from PIPE/TPG
Update as of 11:22AM AEDT
PIPE/TPG still have no acknowledged the issue however it does appear that they have failed over some routes from Sydney.
We are no seeing no packet loss; however, some IPs are routing internationally.
~| mtr 8.8.8.8 --report-wide -c 10
HOST: Loss% Snt Last Avg Best Wrst StDev
4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 10 10.6 10.6 10.4 10.7 0.0
5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 10 10.5 10.4 10.3 10.7 0.0
6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 10 10.6 10.6 10.4 10.7 0.0
7.|-- as15169.sydney.pipenetworks.com 0.0% 10 11.2 11.3 11.1 12.2 0.0
8.|-- 216.239.51.218 0.0% 10 11.4 11.4 11.2 11.7 0.0
9.|-- 216.239.40.31 0.0% 10 109.4 111.5 109.2 129.7 6.4
10.|-- 209.85.142.130 0.0% 10 119.4 119.8 119.0 123.8 1.3
11.|-- 209.85.249.53 0.0% 10 147.3 147.2 147.0 147.5 0.0
12.|-- 64.233.175.209 0.0% 10 147.2 146.1 145.6 147.2 0.3
13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
Update 11:30AM AEDT
TPG/PIPE’s NOC have stated they don’t know what the issue was, what caused it and would not state what they did (if anything) to resolve the issue.
Related note: We’re still waiting (and asking every day) for a report from them on the 14-hour outage that TPG/PIPE had when they didn’t realise they had broken fibre in Melbourne…
Needless to say, I cannot say a good word about PIPE/TPG based on the quality of service or support we’ve received over the past years.
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog
More information about the AusNOG
mailing list