[AusNOG] Major packet loss and routing issues with TPG/PIPE
Sam McLeod
samm at infoxchange.org
Tue Jan 24 11:24:16 EST 2017
On 24/1/17, 11:18 am, "AusNOG on behalf of Sam McLeod" <ausnog-bounces at lists.ausnog.net on behalf of samm at infoxchange.org> wrote:
[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
On 24/1/17, 11:03 am, "AusNOG on behalf of Sam McLeod" <ausnog-bounces at lists.ausnog.net on behalf of samm at infoxchange.org> wrote:
[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
There is a major packet loss and routing issue occurring with TPG/PIPE (and perhaps AAPT)
Symptoms are 40-100% which has essentially taken all our applications and services offline on this link.
We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re denying any outage and stating they can’t detecting any problems and have no alerts which is the usual response for major outages from PIPE/TPG, we’re going to keep pushing them to fix the issue but are left waiting for a response in the mean time.
~| mtr 8.8.8.8 --report-wide -c 40
HOST: Loss% Snt Last Avg Best Wrst StDev
4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 40 10.6 10.5 10.4 10.9 0.0
5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 40 10.6 10.6 10.4 13.0 0.3
6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 40 10.5 10.9 10.5 17.9 1.3
7.|-- as15169.sydney.pipenetworks.com 15.0% 40 76.4 49.6 11.0 79.2 30.9
8.|-- 216.239.51.218 42.5% 40 66.3 38.0 11.3 76.8 31.2
9.|-- 216.239.40.31 12.5% 40 174.6 152.0 109.1 195.5 32.6
10.|-- 209.85.246.221 30.0% 40 190.6 159.1 125.1 191.9 31.1
11.|-- 209.85.142.132 17.5% 40 190.6 161.9 118.8 207.3 32.0
12.|-- 72.14.232.129 15.0% 40 211.5 185.2 146.3 212.0 30.7
13.|-- 209.85.245.16 22.5% 40 211.2 182.3 146.1 214.6 30.7
14.|-- ??? 100.0 40 0.0 0.0 0.0 0.0 0.0
--
Sam McLeod | Systems Lead
Please contact me via email when possible.
Infoxchange - Technology For Social Justice | Infoxchange.org | 33 Elizabeth Street, Richmond Victoria 3121
Update as of 11.10AM AEDT
TPG/PIPE still unaware of any issues although I’ve heard directly from several other customers that are experiencing major issues.
Their L3 support team said that they need to wait 30 minutes for the NOC to respond even if it’s a P1 issue.
Current traceroute:
~| mtr 8.8.8.8 --report-wide -c 40
HOST: Loss% Snt Last Avg Best Wrst StDev
4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 100 10.6 10.6 10.4 20.6 1.0
5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 100 10.5 10.6 10.3 12.9 0.4
6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 100 10.6 10.7 10.4 12.5 0.3
7.|-- as15169.sydney.pipenetworks.com 13.0% 100 11.3 28.7 11.1 78.3 27.7
8.|-- 216.239.51.218 5.0% 100 11.3 32.6 11.2 77.1 30.0
9.|-- 216.239.40.31 14.0% 100 110.3 128.5 109.1 200.6 28.2
10.|-- 209.85.142.130 13.0% 100 119.6 136.3 119.3 186.1 27.9
11.|-- 209.85.249.53 9.0% 100 147.3 166.1 147.0 213.6 28.6
12.|-- 64.233.175.209 12.0% 100 146.8 163.6 145.7 221.3 27.8
13.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
Update as of 11:15AM AEDT
TPG/PIPE L3 support have provided the information to their NOC team that is unaware of any issues.
They’ll investigate it when they’re able to.
We have noticed some route changes and now some national traffic such as Google cached / CDN services and 8.8.8.8 are now routing from Australia to the USA rather than to Sydney.
No ETA from PIPE/TPG
Update as of 11:22AM AEDT
PIPE/TPG still have no acknowledged the issue however it does appear that they have failed over some routes from Sydney.
We are no seeing no packet loss; however, some IPs are routing internationally.
~| mtr 8.8.8.8 --report-wide -c 10
HOST: Loss% Snt Last Avg Best Wrst StDev
4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 10 10.6 10.6 10.4 10.7 0.0
5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 10 10.5 10.4 10.3 10.7 0.0
6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 10 10.6 10.6 10.4 10.7 0.0
7.|-- as15169.sydney.pipenetworks.com 0.0% 10 11.2 11.3 11.1 12.2 0.0
8.|-- 216.239.51.218 0.0% 10 11.4 11.4 11.2 11.7 0.0
9.|-- 216.239.40.31 0.0% 10 109.4 111.5 109.2 129.7 6.4
10.|-- 209.85.142.130 0.0% 10 119.4 119.8 119.0 123.8 1.3
11.|-- 209.85.249.53 0.0% 10 147.3 147.2 147.0 147.5 0.0
12.|-- 64.233.175.209 0.0% 10 147.2 146.1 145.6 147.2 0.3
13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
More information about the AusNOG
mailing list