[AusNOG] Telstra 3G/4G Citrix, VPN Issues

[Ross Wheeler]

On Thu, 26 Oct 2017, Jarryd Sullivan wrote:

> G'day, wondering if anyone else here has recently been experiencing 
> issues with any of their customers running Telstra 3G/4G services using 
> Citrix or VPN's? We've got a number of customers experiencing issues 
> where these no longer work, Citrix apps won't launch, VPN's won't 
> establish etc. and we've been discussing on the phone with Telstra no 
> one seems to be aware of any issues so trying to find out if it's just 
> isolated to our customers. I've got a few customers at remote sites 
> which rely upon 3G/4G services as their only internet connection method 
> as nothing else is available in the area. If we test using other 
> providers 3G/4G services there seem to be no issues.

Few weeks ago (13/Oct) I had a bunch of 3G/4G devices on telstra all 
failed at 2am. Only affected devices on one particular netblock (1.129/16 
from memory).

Devices on 1.144/16 were unaffected.

Symptoms were VPN tunnels made from the devices to my servers all dropped 
and failed to re-establish. Each device required a powercycle to fix (a 
right royal PITA when many were in difficult-to-access locations).

Telstra confirmed with me that this was a "planned outage" for an upgrade 
to one of two Sydney POPs. I never got to the bottom of the problem, but 
telstra said they did find some issue and were going to try to fix that.

Then again a week ago (19/Oct) at 3am, the same issue but this time only 
affecting devices on 1.144/16 and not affecting 1.129/16. Telstra have 
failed to respond to my email to confirm if this was them updating the 
second POP or not, but it seems highly likely. This time I had a sneeky 
backdoor to my devices and was able to instruct them remotely to kill and 
restart the VPN. Most came back after that, a small number required site 
visits.

R.