[AusNOG] Major packet loss and routing issues with TPG/PIPE
Chris Barnes
chris.p.barnes at gmail.com
Tue Jan 24 11:33:05 EST 2017
I saw degradation on one of my PIPE internet services at around 10:17 as
well, but it really only seemed to affect traffic from us to Salesforce in
Europe. At least, thats what most of the complaints were about.
traceroute to salesforce europe showed YUGE latency, in the order of 3+
seconds rtt, and ping showed stacks of timeouts.
and then at about 11:05 for us it instantly came good.
On Tue, Jan 24, 2017 at 11:24 AM, Sam McLeod <samm at infoxchange.org> wrote:
> On 24/1/17, 11:18 am, "AusNOG on behalf of Sam McLeod" <
> ausnog-bounces at lists.ausnog.net on behalf of samm at infoxchange.org> wrote:
>
> [This sender failed our fraud detection checks and may not be who they
> appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
>
> On 24/1/17, 11:03 am, "AusNOG on behalf of Sam McLeod" <
> ausnog-bounces at lists.ausnog.net on behalf of samm at infoxchange.org> wrote:
>
> [This sender failed our fraud detection checks and may not be who
> they appear to be. Learn about spoofing at http://aka.ms/
> LearnAboutSpoofing]
>
> There is a major packet loss and routing issue occurring with
> TPG/PIPE (and perhaps AAPT)
> Symptoms are 40-100% which has essentially taken all our
> applications and services offline on this link.
>
> We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re
> denying any outage and stating they can’t detecting any problems and have
> no alerts which is the usual response for major outages from PIPE/TPG,
> we’re going to keep pushing them to fix the issue but are left waiting for
> a response in the mean time.
>
>
> ~| mtr 8.8.8.8 --report-wide -c 40
>
> HOST:
> Loss% Snt Last Avg Best Wrst StDev
>
> 4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 40
> 10.6 10.5 10.4 10.9 0.0
> 5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 40
> 10.6 10.6 10.4 13.0 0.3
> 6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 40
> 10.5 10.9 10.5 17.9 1.3
> 7.|-- as15169.sydney.pipenetworks.com 15.0% 40
> 76.4 49.6 11.0 79.2 30.9
> 8.|-- 216.239.51.218 42.5% 40
> 66.3 38.0 11.3 76.8 31.2
> 9.|-- 216.239.40.31 12.5% 40
> 174.6 152.0 109.1 195.5 32.6
> 10.|-- 209.85.246.221 30.0% 40
> 190.6 159.1 125.1 191.9 31.1
> 11.|-- 209.85.142.132 17.5% 40
> 190.6 161.9 118.8 207.3 32.0
> 12.|-- 72.14.232.129 15.0% 40
> 211.5 185.2 146.3 212.0 30.7
> 13.|-- 209.85.245.16 22.5% 40
> 211.2 182.3 146.1 214.6 30.7
> 14.|-- ??? 100.0 40
> 0.0 0.0 0.0 0.0 0.0
>
> --
> Sam McLeod | Systems Lead
> Please contact me via email when possible.
> Infoxchange - Technology For Social Justice | Infoxchange.org | 33
> Elizabeth Street, Richmond Victoria 3121
>
>
> Update as of 11.10AM AEDT
>
> TPG/PIPE still unaware of any issues although I’ve heard directly from
> several other customers that are experiencing major issues.
> Their L3 support team said that they need to wait 30 minutes for the
> NOC to respond even if it’s a P1 issue.
>
> Current traceroute:
>
> ~| mtr 8.8.8.8 --report-wide -c 40
>
> HOST: Loss% Snt
> Last Avg Best Wrst StDev
>
> 4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 100
> 10.6 10.6 10.4 20.6 1.0
> 5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 100
> 10.5 10.6 10.3 12.9 0.4
> 6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 100
> 10.6 10.7 10.4 12.5 0.3
> 7.|-- as15169.sydney.pipenetworks.com 13.0% 100
> 11.3 28.7 11.1 78.3 27.7
> 8.|-- 216.239.51.218 5.0% 100
> 11.3 32.6 11.2 77.1 30.0
> 9.|-- 216.239.40.31 14.0% 100
> 110.3 128.5 109.1 200.6 28.2
> 10.|-- 209.85.142.130 13.0% 100
> 119.6 136.3 119.3 186.1 27.9
> 11.|-- 209.85.249.53 9.0% 100
> 147.3 166.1 147.0 213.6 28.6
> 12.|-- 64.233.175.209 12.0% 100
> 146.8 163.6 145.7 221.3 27.8
> 13.|-- ??? 100.0 100
> 0.0 0.0 0.0 0.0 0.0
>
>
> Update as of 11:15AM AEDT
>
> TPG/PIPE L3 support have provided the information to their NOC team
> that is unaware of any issues.
> They’ll investigate it when they’re able to.
>
> We have noticed some route changes and now some national traffic such
> as Google cached / CDN services and 8.8.8.8 are now routing from Australia
> to the USA rather than to Sydney.
>
> No ETA from PIPE/TPG
>
>
> Update as of 11:22AM AEDT
>
> PIPE/TPG still have no acknowledged the issue however it does appear that
> they have failed over some routes from Sydney.
>
> We are no seeing no packet loss; however, some IPs are routing
> internationally.
>
> ~| mtr 8.8.8.8 --report-wide -c 10
>
> HOST: Loss% Snt Last
> Avg Best Wrst StDev
>
>
> 4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 10 10.6
> 10.6 10.4 10.7 0.0
> 5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 10 10.5
> 10.4 10.3 10.7 0.0
> 6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 10 10.6
> 10.6 10.4 10.7 0.0
> 7.|-- as15169.sydney.pipenetworks.com 0.0% 10 11.2
> 11.3 11.1 12.2 0.0
> 8.|-- 216.239.51.218 0.0% 10 11.4
> 11.4 11.2 11.7 0.0
> 9.|-- 216.239.40.31 0.0% 10 109.4
> 111.5 109.2 129.7 6.4
> 10.|-- 209.85.142.130 0.0% 10 119.4
> 119.8 119.0 123.8 1.3
> 11.|-- 209.85.249.53 0.0% 10 147.3
> 147.2 147.0 147.5 0.0
> 12.|-- 64.233.175.209 0.0% 10 147.2
> 146.1 145.6 147.2 0.3
> 13.|-- ??? 100.0 10 0.0
> 0.0 0.0 0.0 0.0
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
--
Kind Regards,
Christopher Barnes
e. chris.p.barnes at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170124/7cf79224/attachment-0001.html>
More information about the AusNOG
mailing list