[AusNOG] Major packet loss and routing issues with TPG/PIPE

Sam McLeod samm at infoxchange.org
Tue Jan 24 12:27:55 EST 2017


Yes, it's almost weekly for us especially with poor peering / upstream routes where our Vocus link is rock solid and their support is so much better.

--
Sam

> On 24 Jan 2017, at 12:24 pm, Brad Peczka <brad at bradpeczka.com> wrote:
> 
> It seems like the suggestion from the other day, of a dedicated AAPT/TPG/PIPE/et al outages list, is becoming more and more necessary.
> 
> Regards,
> -Brad.
> ________________________________________
> From: AusNOG [ausnog-bounces at lists.ausnog.net] On Behalf Of Sam McLeod [samm at infoxchange.org]
> Sent: Tuesday, 24 January 2017 8:35 AM
> To: ausnog at ausnog.net
> Subject: Re: [AusNOG] Major packet loss and routing issues with TPG/PIPE
> 
>            There is a major packet loss and routing issue occurring with TPG/PIPE (and perhaps AAPT)
>            Symptoms are 40-100% which has essentially taken all our applications and services offline on this link.
> 
>            We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re denying any outage and stating they can’t detecting any problems and have no alerts which is the usual response for major outages from PIPE/TPG, we’re going to keep pushing them to fix the issue but are left waiting for a response in the mean time.
> 
> 
>            ~| mtr 8.8.8.8 --report-wide -c 40
> 
>            HOST:                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
> 
>              4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%    40   10.6  10.5  10.4  10.9   0.0
>              5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%    40   10.6  10.6  10.4  13.0   0.3
>              6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%    40   10.5  10.9  10.5  17.9   1.3
>              7.|-- as15169.sydney.pipenetworks.com              15.0%    40   76.4  49.6  11.0  79.2  30.9
>              8.|-- 216.239.51.218                               42.5%    40   66.3  38.0  11.3  76.8  31.2
>              9.|-- 216.239.40.31                                12.5%    40  174.6 152.0 109.1 195.5  32.6
>             10.|-- 209.85.246.221                               30.0%    40  190.6 159.1 125.1 191.9  31.1
>             11.|-- 209.85.142.132                               17.5%    40  190.6 161.9 118.8 207.3  32.0
>             12.|-- 72.14.232.129                                15.0%    40  211.5 185.2 146.3 212.0  30.7
>             13.|-- 209.85.245.16                                22.5%    40  211.2 182.3 146.1 214.6  30.7
>             14.|-- ???                                          100.0    40    0.0   0.0   0.0   0.0   0.0
> 
>            --
>            Sam McLeod | Systems Lead
>            Please contact me via email when possible.
>            Infoxchange - Technology For Social Justice | Infoxchange.org | 33 Elizabeth Street, Richmond Victoria 3121
> 
> 
>        Update as of 11.10AM AEDT
> 
>        TPG/PIPE still unaware of any issues although I’ve heard directly from several other customers that are experiencing major issues.
>        Their L3 support team said that they need to wait 30 minutes for the NOC to respond even if it’s a P1 issue.
> 
>        Current traceroute:
> 
>            ~| mtr 8.8.8.8 --report-wide -c 40
> 
>            HOST:                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
> 
>          4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%   100   10.6  10.6  10.4  20.6   1.0
>          5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%   100   10.5  10.6  10.3  12.9   0.4
>          6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%   100   10.6  10.7  10.4  12.5   0.3
>          7.|-- as15169.sydney.pipenetworks.com              13.0%   100   11.3  28.7  11.1  78.3  27.7
>          8.|-- 216.239.51.218                                5.0%   100   11.3  32.6  11.2  77.1  30.0
>          9.|-- 216.239.40.31                                14.0%   100  110.3 128.5 109.1 200.6  28.2
>         10.|-- 209.85.142.130                               13.0%   100  119.6 136.3 119.3 186.1  27.9
>         11.|-- 209.85.249.53                                 9.0%   100  147.3 166.1 147.0 213.6  28.6
>         12.|-- 64.233.175.209                               12.0%   100  146.8 163.6 145.7 221.3  27.8
>         13.|-- ???                                          100.0   100    0.0   0.0   0.0   0.0   0.0
> 
> 
>        Update as of 11:15AM AEDT
> 
>        TPG/PIPE L3 support have provided the information to their NOC team that is unaware of any issues.
>        They’ll investigate it when they’re able to.
> 
>        We have noticed some route changes and now some national traffic such as Google cached / CDN services and 8.8.8.8 are now routing from Australia to the USA rather than to Sydney.
> 
>        No ETA from PIPE/TPG
> 
> 
>    Update as of 11:22AM AEDT
> 
>    PIPE/TPG still have no acknowledged the issue however it does appear that they have failed over some routes from Sydney.
> 
>    We are no seeing no packet loss; however, some IPs are routing internationally.
> 
>            ~| mtr 8.8.8.8 --report-wide -c 10
> 
>            HOST:                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
> 
> 
>      4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%    10   10.6  10.6  10.4  10.7   0.0
>      5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%    10   10.5  10.4  10.3  10.7   0.0
>      6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%    10   10.6  10.6  10.4  10.7   0.0
>      7.|-- as15169.sydney.pipenetworks.com               0.0%    10   11.2  11.3  11.1  12.2   0.0
>      8.|-- 216.239.51.218                                0.0%    10   11.4  11.4  11.2  11.7   0.0
>      9.|-- 216.239.40.31                                 0.0%    10  109.4 111.5 109.2 129.7   6.4
>     10.|-- 209.85.142.130                                0.0%    10  119.4 119.8 119.0 123.8   1.3
>     11.|-- 209.85.249.53                                 0.0%    10  147.3 147.2 147.0 147.5   0.0
>     12.|-- 64.233.175.209                                0.0%    10  147.2 146.1 145.6 147.2   0.3
>     13.|-- ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
> 
> 
> Update 11:30AM AEDT
> 
> TPG/PIPE’s NOC have stated they don’t know what the issue was, what caused it and would not state what they did (if anything) to resolve the issue.
> 
> Related note: We’re still waiting (and asking every day) for a report from them on the 14-hour outage that TPG/PIPE had when they didn’t realise they had broken fibre in Melbourne…
> 
> Needless to say, I cannot say a good word about PIPE/TPG based on the quality of service or support we’ve received over the past years.
> 
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog


More information about the AusNOG mailing list