[AusNOG] Speaking of DNS...

[Mark Foster]

Responses to at least 3 folks from this thread below...

>> To be fair - I wouldn't expect a 3rd party 'user' to be calling and 
>> informing of problems within my network. I'd expect perhaps another 
>> network operator to contact, possibly asking the helpdesk to speak 
>> with NOC regarding the issue, which would be handled differently.
> ROFL
> helpdesk doing something other than reading the script.
> That's a great one.
> I may need to lay down for a while.

Yeah, tell me about it.

I did my time in technical support roles and the trick was to ensure 
that there is infact, scope to escalate a call that goes 'off script'.  
Rather than calltakers who simply say 'sorry, we can't help you'.

As an engineer, If someone's ringing me to advise me of a problem - or 
potential problem - in my network, I wan't to know about it - but let's 
also qualify this; I am expecting the caller to know what they're 
talking about (so they're not joe-user, they're another network operator 
or someone with appropriate cloo-level) and I expect my support staff to 
be able to determine this before they attempt to refer the call to my 
NOC. If the front-line can't do it, the senior / escalation tier should 
be able to manage it, so by the time it reaches me, it is infact 
something i'm likely to care about.

Some years ago I did a presentation at an NZNOG conference called 
'bridging the gap between support and the NOC' - a big part of this is 
ensuring that there's a good link between them, a career progression and 
an open ability to ensure that the right information flows in both 
directions.
>
>
>
> I wound up sending an email to ausnog and it was sorted in half an hour.

Having to resort to broadcasting on a public mailing list is sad, though 
I myself have also done it.

In New Zealand we operate a 'NOC List' on the NZNOG website which 
provides useful names and contact information for network operators to 
use.  For the most part it's successful in providing an 'out of band' 
way to get hold of a local operator. I've used it several times.
Where i've struggled is mainly in dealing with offshore operators, 
usually to report a compromised server, a DDoS or a broken bogon filter, 
but sometimes weird MSS problems, broken SPF or ICMP filters gone wrong.

> It was only a year or two ago the person who answered the helpdesk 
> phone had cli access to all the managed routers and could trouble 
> shoot and fix on the spot. 

This is the reason many players prefer to deal with smaller 
organisations; you're closer to all-tiers in the one call. I'm a firm 
believer in companies losing their ability to provide decent technical 
support when they reach a certain size.
Some do better than others in mitigating this.

In response to Noel's numbered points:

> 1/ public facing phone numbers of most ISP's are for existing or 
> potential customers, so ringing ISP-A customer support to report a 
> perceived problem on ISP-A's network will likely go right over the 
> head of the call taker
>
Public facing phone numbers should simply be ways to contact the 
organisation, for _whatever purpose_.  IMHO.  Yes the majority will be 
sales/marketing/support type calls, but engineering's also important. 
Even if you have tech support staff 'fronting' it, you should have 
someone accessible who can understand enough to not have everything 
simply go over their head, who can determine if it's appropriate to 
refer the call upwards (or not).

> 2/ existing customers, or external persons, with most problems think 
> it is the ISP's fault, not their own, in all but a few rare events it 
> is the customers own fault, or unrelated to ISP-A's network.
>

Customers, i'd agree with you. External persons bothering to take the 
time, should be able to be split into those who're meddling in something 
not their affair, and those who're describing an actual, serious issue.

> 3/ I dont know of any ISP (except small owner/oper vISP's maybe) that 
> will permit front line call takers to transfer a call to NOC, in fact 
> it is likely specifically forbidden to do so, also it is likely taboo 
> in most places to give out the NOC email address (even though in most 
> cases the whois is the right address, or at least will get there) for 
> much same reasons as above, else you;d soon see custoemr support with 
> 5 calls a day and NOC with 15K a day.
>

I don't expect to ring a third party organisation and simply say 
'transfer me to your NOC'.  But I would like to be able to say 'I have 
detected a serious problem with a server/service on your network, which 
poses a substantial risk to your business (or mine), and I would like to 
speak with someone technical that I can explain the situation to' and 
get a solid answer.  This isn't necessarily your NOC, it depends on your 
size and structure internally.

> 4/ As above, network staff from ISP-B who have a clue on how to 
> contact ISP-A, there is not only whois, but other methods known to 
> most here, which I wont link to because although its publicly 
> accessible, it has direct numbers and addresses :)
>

Works great domestically with players that I know, or who are perhaps 
2-degrees separated. Doesn't work well when fishing around with offshore 
parties or those with whom we don't have standing peering or business 
relationships.
By the time you're resorting to broadcast on NZNOG/AusNOG/NANOG/etc 
you've already demonstrated a problem.

All IMHO.

Mark.