[AusNOG] Major packet loss and routing issues with TPG/PIPE

[Sam McLeod]

On 24/1/17, 11:03 am, "AusNOG on behalf of Sam McLeod" <ausnog-bounces at lists.ausnog.net on behalf of samm at infoxchange.org> wrote:

    [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
    
    There is a major packet loss and routing issue occurring with TPG/PIPE (and perhaps AAPT)
    Symptoms are 40-100% which has essentially taken all our applications and services offline on this link.
    
    We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re denying any outage and stating they can’t detecting any problems and have no alerts which is the usual response for major outages from PIPE/TPG, we’re going to keep pushing them to fix the issue but are left waiting for a response in the mean time.
    
    
    ~| mtr 8.8.8.8 --report-wide -c 40
    
    HOST:                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
    
      4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%    40   10.6  10.5  10.4  10.9   0.0
      5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%    40   10.6  10.6  10.4  13.0   0.3
      6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%    40   10.5  10.9  10.5  17.9   1.3
      7.|-- as15169.sydney.pipenetworks.com              15.0%    40   76.4  49.6  11.0  79.2  30.9
      8.|-- 216.239.51.218                               42.5%    40   66.3  38.0  11.3  76.8  31.2
      9.|-- 216.239.40.31                                12.5%    40  174.6 152.0 109.1 195.5  32.6
     10.|-- 209.85.246.221                               30.0%    40  190.6 159.1 125.1 191.9  31.1
     11.|-- 209.85.142.132                               17.5%    40  190.6 161.9 118.8 207.3  32.0
     12.|-- 72.14.232.129                                15.0%    40  211.5 185.2 146.3 212.0  30.7
     13.|-- 209.85.245.16                                22.5%    40  211.2 182.3 146.1 214.6  30.7
     14.|-- ???                                          100.0    40    0.0   0.0   0.0   0.0   0.0
    
    --
    Sam McLeod | Systems Lead
    Please contact me via email when possible.
    Infoxchange - Technology For Social Justice | Infoxchange.org | 33 Elizabeth Street, Richmond Victoria 3121
    

Update as of 11.10AM AEDT

TPG/PIPE still unaware of any issues although I’ve heard directly from several other customers that are experiencing major issues.
Their L3 support team said that they need to wait 30 minutes for the NOC to respond even if it’s a P1 issue.

Current traceroute:

    ~| mtr 8.8.8.8 --report-wide -c 40
    
    HOST:                                             Loss%   Snt   Last   Avg  Best  Wrst StDev

  4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%   100   10.6  10.6  10.4  20.6   1.0
  5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%   100   10.5  10.6  10.3  12.9   0.4
  6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%   100   10.6  10.7  10.4  12.5   0.3
  7.|-- as15169.sydney.pipenetworks.com              13.0%   100   11.3  28.7  11.1  78.3  27.7
  8.|-- 216.239.51.218                                5.0%   100   11.3  32.6  11.2  77.1  30.0
  9.|-- 216.239.40.31                                14.0%   100  110.3 128.5 109.1 200.6  28.2
 10.|-- 209.85.142.130                               13.0%   100  119.6 136.3 119.3 186.1  27.9
 11.|-- 209.85.249.53                                 9.0%   100  147.3 166.1 147.0 213.6  28.6
 12.|-- 64.233.175.209                               12.0%   100  146.8 163.6 145.7 221.3  27.8
 13.|-- ???                                          100.0   100    0.0   0.0   0.0   0.0   0.0


Update as of 11:15AM AEDT

TPG/PIPE L3 support have provided the information to their NOC team that is unaware of any issues.
They’ll investigate it when they’re able to.

We have noticed some route changes and now some national traffic such as Google cached / CDN services and 8.8.8.8 are now routing from Australia to the USA rather than to Sydney.

No ETA from PIPE/TPG