[AusNOG] Major packet loss and routing issues with TPG/PIPE

[Sam McLeod]

            There is a major packet loss and routing issue occurring with TPG/PIPE (and perhaps AAPT)
            Symptoms are 40-100% which has essentially taken all our applications and services offline on this link.
    
            We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re denying any outage and stating they can’t detecting any problems and have no alerts which is the usual response for major outages from PIPE/TPG, we’re going to keep pushing them to fix the issue but are left waiting for a response in the mean time.
    
    
            ~| mtr 8.8.8.8 --report-wide -c 40
    
            HOST:                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
    
              4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%    40   10.6  10.5  10.4  10.9   0.0
              5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%    40   10.6  10.6  10.4  13.0   0.3
              6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%    40   10.5  10.9  10.5  17.9   1.3
              7.|-- as15169.sydney.pipenetworks.com              15.0%    40   76.4  49.6  11.0  79.2  30.9
              8.|-- 216.239.51.218                               42.5%    40   66.3  38.0  11.3  76.8  31.2
              9.|-- 216.239.40.31                                12.5%    40  174.6 152.0 109.1 195.5  32.6
             10.|-- 209.85.246.221                               30.0%    40  190.6 159.1 125.1 191.9  31.1
             11.|-- 209.85.142.132                               17.5%    40  190.6 161.9 118.8 207.3  32.0
             12.|-- 72.14.232.129                                15.0%    40  211.5 185.2 146.3 212.0  30.7
             13.|-- 209.85.245.16                                22.5%    40  211.2 182.3 146.1 214.6  30.7
             14.|-- ???                                          100.0    40    0.0   0.0   0.0   0.0   0.0
    
            --
            Sam McLeod | Systems Lead
            Please contact me via email when possible.
            Infoxchange - Technology For Social Justice | Infoxchange.org | 33 Elizabeth Street, Richmond Victoria 3121
    
    
        Update as of 11.10AM AEDT
    
        TPG/PIPE still unaware of any issues although I’ve heard directly from several other customers that are experiencing major issues.
        Their L3 support team said that they need to wait 30 minutes for the NOC to respond even if it’s a P1 issue.
    
        Current traceroute:
    
            ~| mtr 8.8.8.8 --report-wide -c 40
    
            HOST:                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
    
          4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%   100   10.6  10.6  10.4  20.6   1.0
          5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%   100   10.5  10.6  10.3  12.9   0.4
          6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%   100   10.6  10.7  10.4  12.5   0.3
          7.|-- as15169.sydney.pipenetworks.com              13.0%   100   11.3  28.7  11.1  78.3  27.7
          8.|-- 216.239.51.218                                5.0%   100   11.3  32.6  11.2  77.1  30.0
          9.|-- 216.239.40.31                                14.0%   100  110.3 128.5 109.1 200.6  28.2
         10.|-- 209.85.142.130                               13.0%   100  119.6 136.3 119.3 186.1  27.9
         11.|-- 209.85.249.53                                 9.0%   100  147.3 166.1 147.0 213.6  28.6
         12.|-- 64.233.175.209                               12.0%   100  146.8 163.6 145.7 221.3  27.8
         13.|-- ???                                          100.0   100    0.0   0.0   0.0   0.0   0.0
    
    
        Update as of 11:15AM AEDT
    
        TPG/PIPE L3 support have provided the information to their NOC team that is unaware of any issues.
        They’ll investigate it when they’re able to.
    
        We have noticed some route changes and now some national traffic such as Google cached / CDN services and 8.8.8.8 are now routing from Australia to the USA rather than to Sydney.
    
        No ETA from PIPE/TPG
    
    
    Update as of 11:22AM AEDT
    
    PIPE/TPG still have no acknowledged the issue however it does appear that they have failed over some routes from Sydney.
    
    We are no seeing no packet loss; however, some IPs are routing internationally.
    
            ~| mtr 8.8.8.8 --report-wide -c 10
    
            HOST:                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
    
    
      4.|-- ve100.rv-376flinders-mlx-01.tpg-telecom.net   0.0%    10   10.6  10.6  10.4  10.7   0.0
      5.|-- eth10-7.rn-400harris-mlx-03.tpg-telecom.net   0.0%    10   10.5  10.4  10.3  10.7   0.0
      6.|-- eth2-1.rn-400harris-cer-03.tpg-telecom.net    0.0%    10   10.6  10.6  10.4  10.7   0.0
      7.|-- as15169.sydney.pipenetworks.com               0.0%    10   11.2  11.3  11.1  12.2   0.0
      8.|-- 216.239.51.218                                0.0%    10   11.4  11.4  11.2  11.7   0.0
      9.|-- 216.239.40.31                                 0.0%    10  109.4 111.5 109.2 129.7   6.4
     10.|-- 209.85.142.130                                0.0%    10  119.4 119.8 119.0 123.8   1.3
     11.|-- 209.85.249.53                                 0.0%    10  147.3 147.2 147.0 147.5   0.0
     12.|-- 64.233.175.209                                0.0%    10  147.2 146.1 145.6 147.2   0.3
     13.|-- ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
    

Update 11:30AM AEDT

TPG/PIPE’s NOC have stated they don’t know what the issue was, what caused it and would not state what they did (if anything) to resolve the issue.

Related note: We’re still waiting (and asking every day) for a report from them on the 14-hour outage that TPG/PIPE had when they didn’t realise they had broken fibre in Melbourne…

Needless to say, I cannot say a good word about PIPE/TPG based on the quality of service or support we’ve received over the past years.