<div>Then find a new provider :)</div><div><br></div><div>There is plenty of good ones out there. Vocus have been rock solid lately</div><div><br></div><div><br></div><div><div class="gmail_quote"><div>On Tue, 24 Jan 2017 at 11:36 am, Sam McLeod <<a href="mailto:samm@infoxchange.org">samm@infoxchange.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">            There is a major packet loss and routing issue occurring with TPG/PIPE (and perhaps AAPT)<br class="gmail_msg"><br>            Symptoms are 40-100% which has essentially taken all our applications and services offline on this link.<br class="gmail_msg"><br><br class="gmail_msg"><br>            We have escalated with PIPE/TPG from levels 1,2 and 3 and they’re denying any outage and stating they can’t detecting any problems and have no alerts which is the usual response for major outages from PIPE/TPG, we’re going to keep pushing them to fix the issue but are left waiting for a response in the mean time.<br class="gmail_msg"><br><br class="gmail_msg"><br><br class="gmail_msg"><br>            ~| mtr 8.8.8.8 --report-wide -c 40<br class="gmail_msg"><br><br class="gmail_msg"><br>            HOST:                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev<br class="gmail_msg"><br><br class="gmail_msg"><br>              4.|-- <a href="http://ve100.rv-376flinders-mlx-01.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">ve100.rv-376flinders-mlx-01.tpg-telecom.net</a>   0.0%    40   10.6  10.5  10.4  10.9   0.0<br class="gmail_msg"><br>              5.|-- <a href="http://eth10-7.rn-400harris-mlx-03.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">eth10-7.rn-400harris-mlx-03.tpg-telecom.net</a>   0.0%    40   10.6  10.6  10.4  13.0   0.3<br class="gmail_msg"><br>              6.|-- <a href="http://eth2-1.rn-400harris-cer-03.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">eth2-1.rn-400harris-cer-03.tpg-telecom.net</a>    0.0%    40   10.5  10.9  10.5  17.9   1.3<br class="gmail_msg"><br>              7.|-- <a href="http://as15169.sydney.pipenetworks.com" rel="noreferrer" class="gmail_msg" target="_blank">as15169.sydney.pipenetworks.com</a>              15.0%    40   76.4  49.6  11.0  79.2  30.9<br class="gmail_msg"><br>              8.|-- 216.239.51.218                               42.5%    40   66.3  38.0  11.3  76.8  31.2<br class="gmail_msg"><br>              9.|-- 216.239.40.31                                12.5%    40  174.6 152.0 109.1 195.5  32.6<br class="gmail_msg"><br>             10.|-- 209.85.246.221                               30.0%    40  190.6 159.1 125.1 191.9  31.1<br class="gmail_msg"><br>             11.|-- 209.85.142.132                               17.5%    40  190.6 161.9 118.8 207.3  32.0<br class="gmail_msg"><br>             12.|-- 72.14.232.129                                15.0%    40  211.5 185.2 146.3 212.0  30.7<br class="gmail_msg"><br>             13.|-- 209.85.245.16                                22.5%    40  211.2 182.3 146.1 214.6  30.7<br class="gmail_msg"><br>             14.|-- ???                                          100.0    40    0.0   0.0   0.0   0.0   0.0<br class="gmail_msg"><br><br class="gmail_msg"><br>            --<br class="gmail_msg"><br>            Sam McLeod | Systems Lead<br class="gmail_msg"><br>            Please contact me via email when possible.<br class="gmail_msg"><br>            Infoxchange - Technology For Social Justice | Infoxchange.org | 33 Elizabeth Street, Richmond Victoria 3121<br class="gmail_msg"><br><br class="gmail_msg"><br><br class="gmail_msg"><br>        Update as of 11.10AM AEDT<br class="gmail_msg"><br><br class="gmail_msg"><br>        TPG/PIPE still unaware of any issues although I’ve heard directly from several other customers that are experiencing major issues.<br class="gmail_msg"><br>        Their L3 support team said that they need to wait 30 minutes for the NOC to respond even if it’s a P1 issue.<br class="gmail_msg"><br><br class="gmail_msg"><br>        Current traceroute:<br class="gmail_msg"><br><br class="gmail_msg"><br>            ~| mtr 8.8.8.8 --report-wide -c 40<br class="gmail_msg"><br><br class="gmail_msg"><br>            HOST:                                             Loss%   Snt   Last   Avg  Best  Wrst StDev<br class="gmail_msg"><br><br class="gmail_msg"><br>          4.|-- <a href="http://ve100.rv-376flinders-mlx-01.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">ve100.rv-376flinders-mlx-01.tpg-telecom.net</a>   0.0%   100   10.6  10.6  10.4  20.6   1.0<br class="gmail_msg"><br>          5.|-- <a href="http://eth10-7.rn-400harris-mlx-03.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">eth10-7.rn-400harris-mlx-03.tpg-telecom.net</a>   0.0%   100   10.5  10.6  10.3  12.9   0.4<br class="gmail_msg"><br>          6.|-- <a href="http://eth2-1.rn-400harris-cer-03.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">eth2-1.rn-400harris-cer-03.tpg-telecom.net</a>    0.0%   100   10.6  10.7  10.4  12.5   0.3<br class="gmail_msg"><br>          7.|-- <a href="http://as15169.sydney.pipenetworks.com" rel="noreferrer" class="gmail_msg" target="_blank">as15169.sydney.pipenetworks.com</a>              13.0%   100   11.3  28.7  11.1  78.3  27.7<br class="gmail_msg"><br>          8.|-- 216.239.51.218                                5.0%   100   11.3  32.6  11.2  77.1  30.0<br class="gmail_msg"><br>          9.|-- 216.239.40.31                                14.0%   100  110.3 128.5 109.1 200.6  28.2<br class="gmail_msg"><br>         10.|-- 209.85.142.130                               13.0%   100  119.6 136.3 119.3 186.1  27.9<br class="gmail_msg"><br>         11.|-- 209.85.249.53                                 9.0%   100  147.3 166.1 147.0 213.6  28.6<br class="gmail_msg"><br>         12.|-- 64.233.175.209                               12.0%   100  146.8 163.6 145.7 221.3  27.8<br class="gmail_msg"><br>         13.|-- ???                                          100.0   100    0.0   0.0   0.0   0.0   0.0<br class="gmail_msg"><br><br class="gmail_msg"><br><br class="gmail_msg"><br>        Update as of 11:15AM AEDT<br class="gmail_msg"><br><br class="gmail_msg"><br>        TPG/PIPE L3 support have provided the information to their NOC team that is unaware of any issues.<br class="gmail_msg"><br>        They’ll investigate it when they’re able to.<br class="gmail_msg"><br><br class="gmail_msg"><br>        We have noticed some route changes and now some national traffic such as Google cached / CDN services and 8.8.8.8 are now routing from Australia to the USA rather than to Sydney.<br class="gmail_msg"><br><br class="gmail_msg"><br>        No ETA from PIPE/TPG<br class="gmail_msg"><br><br class="gmail_msg"><br><br class="gmail_msg"><br>    Update as of 11:22AM AEDT<br class="gmail_msg"><br><br class="gmail_msg"><br>    PIPE/TPG still have no acknowledged the issue however it does appear that they have failed over some routes from Sydney.<br class="gmail_msg"><br><br class="gmail_msg"><br>    We are no seeing no packet loss; however, some IPs are routing internationally.<br class="gmail_msg"><br><br class="gmail_msg"><br>            ~| mtr 8.8.8.8 --report-wide -c 10<br class="gmail_msg"><br><br class="gmail_msg"><br>            HOST:                                         Loss%   Snt   Last   Avg  Best  Wrst StDev<br class="gmail_msg"><br><br class="gmail_msg"><br><br class="gmail_msg"><br>      4.|-- <a href="http://ve100.rv-376flinders-mlx-01.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">ve100.rv-376flinders-mlx-01.tpg-telecom.net</a>   0.0%    10   10.6  10.6  10.4  10.7   0.0<br class="gmail_msg"><br>      5.|-- <a href="http://eth10-7.rn-400harris-mlx-03.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">eth10-7.rn-400harris-mlx-03.tpg-telecom.net</a>   0.0%    10   10.5  10.4  10.3  10.7   0.0<br class="gmail_msg"><br>      6.|-- <a href="http://eth2-1.rn-400harris-cer-03.tpg-telecom.net" rel="noreferrer" class="gmail_msg" target="_blank">eth2-1.rn-400harris-cer-03.tpg-telecom.net</a>    0.0%    10   10.6  10.6  10.4  10.7   0.0<br class="gmail_msg"><br>      7.|-- <a href="http://as15169.sydney.pipenetworks.com" rel="noreferrer" class="gmail_msg" target="_blank">as15169.sydney.pipenetworks.com</a>               0.0%    10   11.2  11.3  11.1  12.2   0.0<br class="gmail_msg"><br>      8.|-- 216.239.51.218                                0.0%    10   11.4  11.4  11.2  11.7   0.0<br class="gmail_msg"><br>      9.|-- 216.239.40.31                                 0.0%    10  109.4 111.5 109.2 129.7   6.4<br class="gmail_msg"><br>     10.|-- 209.85.142.130                                0.0%    10  119.4 119.8 119.0 123.8   1.3<br class="gmail_msg"><br>     11.|-- 209.85.249.53                                 0.0%    10  147.3 147.2 147.0 147.5   0.0<br class="gmail_msg"><br>     12.|-- 64.233.175.209                                0.0%    10  147.2 146.1 145.6 147.2   0.3<br class="gmail_msg"><br>     13.|-- ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0<br class="gmail_msg"><br><br class="gmail_msg"><br><br class="gmail_msg"><br>Update 11:30AM AEDT<br class="gmail_msg"><br><br class="gmail_msg"><br>TPG/PIPE’s NOC have stated they don’t know what the issue was, what caused it and would not state what they did (if anything) to resolve the issue.<br class="gmail_msg"><br><br class="gmail_msg"><br>Related note: We’re still waiting (and asking every day) for a report from them on the 14-hour outage that TPG/PIPE had when they didn’t realise they had broken fibre in Melbourne…<br class="gmail_msg"><br><br class="gmail_msg"><br>Needless to say, I cannot say a good word about PIPE/TPG based on the quality of service or support we’ve received over the past years.<br class="gmail_msg"><br><br class="gmail_msg"><br><br class="gmail_msg"><br>_______________________________________________<br class="gmail_msg"><br>AusNOG mailing list<br class="gmail_msg"><br><a href="mailto:AusNOG@lists.ausnog.net" class="gmail_msg" target="_blank">AusNOG@lists.ausnog.net</a><br class="gmail_msg"><br><a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br class="gmail_msg"><br></blockquote></div></div>