[AusNOG] AAPT Ethernet outage

[Matt Perkins]

On 3/07/12 1:09 PM, Brent Paddon wrote:
>
>     minimum 30 second intervals. Dont Lie. We will know. Just tell us
>     the truth you will find it will be welcomed by your
>
>
> I presume you mean 30 minutes - 30 seconds might be a tad too short... :-)
Yep minute my bad. 30 seconds may piss people of some what.
>
>     Here's a hint. Dont blame the vendor. I cant blame you to my
>     customers they dont care all they care about is they were down and
>     how it wont happen again. Take charge of it.  Here's a free
>     technical hint for your last outage. You need a hard power
>
>
> You want an incident report but you don't want them to tell you what 
> happened (ie the vendors equipment had a bug).  Sorry, you can't have 
> it both ways.
My poorly worded request. to be  clear. I want them to tell the truth 
(if thats the vendor so be it) but the resolution moving forward should 
not be "Ask the vendor to be more careful next time" I want to see 
something like a watchdog or remote access that's a good resolution. 
That's what im getting at here. I dont want it's out of our control. 
It's not. I am also open to there is no fix at this price point.
>
> If your customers are super sensitive to outages, they should have 
> redundancy - either of their own or have you install a second tail 
> from as diverse a path as you can muster into their sites.  Offering 
> this to customers and then having them refuse the added cost of the 
> redundancy MAY be a good way of diffusing potential situations in the 
> future.  All systems/networks have their moments, no matter who runs them.
We allays offer redundancy. It's rarely taken and always our fault when 
there is an outage.  Rubbing you did not take the redundancy so na -na 
nana . In the face of your customer is a approach likely to not win 
friends and influence people.  Customers these days are price sensitive 
it's a hard enough sell to try and get them to take business grade 
services rather then trot down the home grade path. (If i had a $ for 
every time a sales guy has said to me. The customer said "but i can get 
20Meg's unlimited from blah blah")

AAPT's business model is business grade wholesale there needs to be a 
delimiter. It needs to better then dodo. If that be in support, 
communication or SLA.  Not just in price.  Only Telstra gets away with 
that and only because they have access everywhere. The CAN is dead long 
live then NBN.
>
>     watchdog on the switch. A device that will hard reboot the power
>     on your switch when it cant be seen for 5 minutes it needs to be
>     in the pop and self contained.  Im sure you could afford them
>     after all the money you saved on those non mainstream switches.
>
>
> I'm not sure a watchdog system is what I would deploy, but I will say 
> that I think its poor form for AAPT to not have some kind of remote 
> access via another mechanism (eg 3G or ADSL from another carrier) into 
> those POPs and for a completely OOB emergency access mechanism which 
> removed the need for anyone to visit the site.  The cost of something 
> like this should be ridiculously low in almost every case and probably 
> would have bought this particular instance down in its length (and 
> therefore impact on clients) considerably.
>
3G cards in a cisco work well true. Team that with a serial power switch 
unit on the aux port and you have a reboot/management out of band back 
channel. Punters will note that basement signal strength can often be 
problematic. Yes an ADSL on another carrier would also be nice. Hell you 
could use a dialup modem and pretend it's the 80's it would still do the 
job.

Out of band management is Prudent for any ISP.

>  Brent

There portal indicates they were using Ciena Switches. Does anyone have 
any experience with Ciena Switch's. I know we have been down the cheap 
switch path a number of times we always end up back with a well known 
brand saying why do we do this every time. It always bites us in the ass.


Matt.


-- 
/* Matt Perkins
         Direct 1300 137 379     Spectrum Networks Ptd. Ltd.
         Office 1300 133 299     matt at spectrum.com.au
         Fax    1300 133 255     Level 6, 350 George Street Sydney 2000
         SIP 1300137379 at sip.spectrum.com.au
         PGP/GNUPG Public Key can be found at  http://pgp.mit.edu
*/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20120703/dc1061b9/attachment.html>