[AusNOG] Major packet loss and routing issues with TPG/PIPE
Daniel Hood
dsmhood at gmail.com
Tue Jan 24 11:37:29 EST 2017
In Europe at the moment connecting to the office in Australia, which is on
PIPE internet. I had some decent packet loss. It stopped at approx 11:15am
though.
Regards,
Daniel
On Tue, Jan 24, 2017 at 11:35 AM, Sam McLeod <samm at infoxchange.org> wrote:
> There is a major packet loss and routing issue occurring with
> TPG/PIPE (and perhaps AAPT)
> Symptoms are 40-100% which has essentially taken all our
> applications and services offline on this link.
>
> We have escalated with PIPE/TPG from levels 1,2 and 3 and
> they’re denying any outage and stating they can’t detecting any problems
> and have no alerts which is the usual response for major outages from
> PIPE/TPG, we’re going to keep pushing them to fix the issue but are left
> waiting for a response in the mean time.
>
>
> ~| mtr 8.8.8.8 --report-wide -c 40
>
> HOST:
> Loss% Snt Last Avg Best Wrst StDev
>
> 4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0%
> 40 10.6 10.5 10.4 10.9 0.0
> 5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0%
> 40 10.6 10.6 10.4 13.0 0.3
> 6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0%
> 40 10.5 10.9 10.5 17.9 1.3
> 7.|-- as15169.sydney.pipenetworks.com 15.0%
> 40 76.4 49.6 11.0 79.2 30.9
> 8.|-- 216.239.51.218 42.5%
> 40 66.3 38.0 11.3 76.8 31.2
> 9.|-- 216.239.40.31 12.5%
> 40 174.6 152.0 109.1 195.5 32.6
> 10.|-- 209.85.246.221 30.0%
> 40 190.6 159.1 125.1 191.9 31.1
> 11.|-- 209.85.142.132 17.5%
> 40 190.6 161.9 118.8 207.3 32.0
> 12.|-- 72.14.232.129 15.0%
> 40 211.5 185.2 146.3 212.0 30.7
> 13.|-- 209.85.245.16 22.5%
> 40 211.2 182.3 146.1 214.6 30.7
> 14.|-- ??? 100.0
> 40 0.0 0.0 0.0 0.0 0.0
>
> --
> Sam McLeod | Systems Lead
> Please contact me via email when possible.
> Infoxchange - Technology For Social Justice | Infoxchange.org
> | 33 Elizabeth Street, Richmond Victoria 3121
>
>
> Update as of 11.10AM AEDT
>
> TPG/PIPE still unaware of any issues although I’ve heard directly
> from several other customers that are experiencing major issues.
> Their L3 support team said that they need to wait 30 minutes for
> the NOC to respond even if it’s a P1 issue.
>
> Current traceroute:
>
> ~| mtr 8.8.8.8 --report-wide -c 40
>
> HOST: Loss% Snt
> Last Avg Best Wrst StDev
>
> 4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 100
> 10.6 10.6 10.4 20.6 1.0
> 5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 100
> 10.5 10.6 10.3 12.9 0.4
> 6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 100
> 10.6 10.7 10.4 12.5 0.3
> 7.|-- as15169.sydney.pipenetworks.com 13.0% 100
> 11.3 28.7 11.1 78.3 27.7
> 8.|-- 216.239.51.218 5.0% 100
> 11.3 32.6 11.2 77.1 30.0
> 9.|-- 216.239.40.31 14.0% 100
> 110.3 128.5 109.1 200.6 28.2
> 10.|-- 209.85.142.130 13.0% 100
> 119.6 136.3 119.3 186.1 27.9
> 11.|-- 209.85.249.53 9.0% 100
> 147.3 166.1 147.0 213.6 28.6
> 12.|-- 64.233.175.209 12.0% 100
> 146.8 163.6 145.7 221.3 27.8
> 13.|-- ??? 100.0 100
> 0.0 0.0 0.0 0.0 0.0
>
>
> Update as of 11:15AM AEDT
>
> TPG/PIPE L3 support have provided the information to their NOC
> team that is unaware of any issues.
> They’ll investigate it when they’re able to.
>
> We have noticed some route changes and now some national traffic
> such as Google cached / CDN services and 8.8.8.8 are now routing from
> Australia to the USA rather than to Sydney.
>
> No ETA from PIPE/TPG
>
>
> Update as of 11:22AM AEDT
>
> PIPE/TPG still have no acknowledged the issue however it does appear
> that they have failed over some routes from Sydney.
>
> We are no seeing no packet loss; however, some IPs are routing
> internationally.
>
> ~| mtr 8.8.8.8 --report-wide -c 10
>
> HOST: Loss% Snt
> Last Avg Best Wrst StDev
>
>
> 4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net 0.0% 10
> 10.6 10.6 10.4 10.7 0.0
> 5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net 0.0% 10
> 10.5 10.4 10.3 10.7 0.0
> 6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net 0.0% 10
> 10.6 10.6 10.4 10.7 0.0
> 7.|-- as15169.sydney.pipenetworks.com 0.0% 10
> 11.2 11.3 11.1 12.2 0.0
> 8.|-- 216.239.51.218 0.0% 10
> 11.4 11.4 11.2 11.7 0.0
> 9.|-- 216.239.40.31 0.0% 10
> 109.4 111.5 109.2 129.7 6.4
> 10.|-- 209.85.142.130 0.0% 10
> 119.4 119.8 119.0 123.8 1.3
> 11.|-- 209.85.249.53 0.0% 10
> 147.3 147.2 147.0 147.5 0.0
> 12.|-- 64.233.175.209 0.0% 10
> 147.2 146.1 145.6 147.2 0.3
> 13.|-- ??? 100.0 10
> 0.0 0.0 0.0 0.0 0.0
>
>
> Update 11:30AM AEDT
>
> TPG/PIPE’s NOC have stated they don’t know what the issue was, what caused
> it and would not state what they did (if anything) to resolve the issue.
>
> Related note: We’re still waiting (and asking every day) for a report from
> them on the 14-hour outage that TPG/PIPE had when they didn’t realise they
> had broken fibre in Melbourne…
>
> Needless to say, I cannot say a good word about PIPE/TPG based on the
> quality of service or support we’ve received over the past years.
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170124/d82f28ef/attachment.html>
More information about the AusNOG
mailing list