[AusNOG] SPAM-LOW: Pipe peering issue - Equinix

Mon Nov 14 17:48:03 EST 2011

Hi Matt,

Thanks for the Info, would have been good to have this in my ticket though…
I am still waiting for a reply on a few questions, in summary they are
below.

Why did it take 3 hours to shut down the offending port? I called the NOC 1
hour after it started, got put through to a messaging service and I only
received a call back 2 hours later, I also submitted a ticket to which I
received a reply 2 hours later also.

I was not too concerned that it had happened, as these things do happen,
but the inability to contact someone at PIPE to notify them of the issue is
a larger concern to me.

Last time this happened it started at 2am again, but I shut our port down
and waited till morning to see if PIPE resolved it but they hadn’t, so I
called up 5 hours later and someone answered and had NO idea it was
happening, he placed me on hold, had a look and then said to me that they
can see an issue. Within minutes the issue was fixed. I’m not trying to
cause a stir here, but the process / escalation of your monitoring really
needs to be looked at. As many have said this is the 5th or 6th time it has
happened, and EVERYTIME I have called your NOC and they have not been aware
of the issue. As stated above, I am still yet to receive a response from my
open ticket as to the issue.

Kindest Regards,
Jared Hirst
*
Servers Australia Pty Ltd
*
Direct:                   (02) 4307 4205
Fax:                        (02) 4307 4201
Network Ops:        (02) 9037 4343
Web:                      *http://www.serversaustralia.com.au
*Office Address:    2/2 Teamster Close, Tuggerah NSW 2259
Postal Address:    PO Box 3187, Tuggerah NSW 2259

*From:* ausnog-bounces at lists.ausnog.net [mailto:
ausnog-bounces at lists.ausnog.net] *On Behalf Of *Matt Whitlock
*Sent:* Monday, 14 November 2011 5:39 PM
*To:* Andrew Fort; Sean K. Finn
*Cc:* ausnog at ausnog.net
*Subject:* Re: [AusNOG] SPAM-LOW: Pipe peering issue - Equinix

Hi everyone,

I just wanted to make a clear statement about where we are up to on this
issue.

In the past few months we have seen problems with unknown unicast flooding
on the PIPE Sydney IX. This issue affects a subset of peers connected to
just one of the switches that make up the Sydney IX.

We use a combination of port security and layer 2 ACL's on all peering
interfaces to prevent this type of problem. Over the years these mechanisms
have prevented many broadcast storms and loops on the network. For those
concerned, our monitoring system does detect this and other similar
problems for reporting to the NOC. They are monitored and escalated as
required.

With these particular issues, the controls do not appear to be effective.
We suspect a hardware or software problem to be the cause, but it has
proven difficult to reproduce. The issue has been narrowed to traffic
injected by a specific peer and some additional protections are now in
place to minimise the risk while we work with the peer to find the best
permanent solution.

We have had a migration plan in progress for a while now to replace the
switch platform; I’m sure you can all imagine it is not a quick or easy
task to complete. We have an expectation of contacting you all in the next
few days to continue this work with a view to setting a mutually agreed
migration date.

Ta,

Matt Whitlock
Operations Manager, PIPE

------------------------------

*From:* ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] *On Behalf Of *Andrew Fort
*Sent:* Monday, 14 November 2011 11:43 AM
*To:* Sean K. Finn
*Cc:* ausnog at ausnog.net
*Subject:* Re: [AusNOG] SPAM-LOW: Pipe peering issue - Equinix

IXes for years have done mac port filtering to mitigate this issue.
One port, one mac. No tagging. If a port is lit up, it should point to the
issue (if filtered frames hit l2 int counters). A faulty transceiver
spewing a valid source mac?

Is mac filtering not being done there, or is it not working?

On Nov 13, 2011 5:33 PM, "Sean K. Finn" <sean.finn at ozservers.com.au> wrote:

When a CAM Table is maxed-out a switch becomes a full-duplex hub. (Cool
hey).

If it doesn’t know which mac address to send the traffic to, it sends it
EVERYWHERE.

If the traffic is deliberately being injected into a port knowing that the
mac address doesn’t exist on the peering fabric, EVERYONE gets it, even if
the CAM table isn’t full.

I’m not on PIPE-IX-SYD, (Only on Equinix-SYD, O Hai) but I can imagine
that..

*There’s a customer connected to the IX who’s own network has become a hub
so they are smashing L2 traffic into the IX not on purpose.*

At least I’d hope that’s the case, I wouldn’t imagine anyone would
deliberately sabotage the IX.

If someone can sniff the packets you may be able to get the MAC address,
then do an Arping on all of the IX IP’s and see which mac addresses match
the IP’s in question, or look at your routers ARP table for a corresponding
MAC address to get the offending networks PEER-IP. (Or look on the graphs
and see who is inverse to everyone else).

Who has more peers, by the way? Equinix-SYD or PIPE-SYD?

S.

*From:* ausnog-bounces at lists.ausnog.net [mailto:
ausnog-bounces at lists.ausnog.net] *On Behalf Of *ZoneNetworks - Joel
*Sent:* Sunday, November 13, 2011 9:35 AM
*To:* Skeeve Stevens
*Cc:* ausnog at ausnog.net
*Subject:* Re: [AusNOG] SPAM-LOW: Pipe peering issue - Equinix

To my knowledge it's a hardware/firmware issue not traffic related
...though looking at the graphs it seems like traffic/flood which is
misleading

If more customers gave them a hard time it might get fixed sooner, we were
told (in august) they had a long term fix in plan but no ETA when it will
be done

Not something you would hear from the "old pipe"

Joel

Sent from my iPad

On 13/11/2011, at 10:06 AM, Skeeve Stevens <Skeeve at eintellego.net> wrote:

Do we know where the traffic is coming from?

…Skeeve

--

Skeeve Stevens, CEO - eintellego Pty Ltd

skeeve at eintellego.net ; www.eintellego.net

Phone: 1300 753 383 ; Fax: (+612) 8572 9954

Cell +61 (0)414 753 383 ; skype://skeeve

facebook.com/eintellego

twitter.com/networkceoau ; www.linkedin.com/in/skeeve

PO Box 7726, Baulkham Hills, NSW 1755 Australia

--

eintellego - The Experts Who The Experts Call

Juniper - HP Networking - Cisco - Brocade

On 13/11/11 10:04 AM, "Jared Hirst" <jared.hirst at serversaustralia.com.au>
wrote:

Yeah every few weeks it happens is what I mean, they start sending

hundreds of mbits of useless broadcast traffic to all customers in

Equinix. It's getting frustrating and annoying as it always seems to

happen at 2:00on a Saturday morning :(

Kindest Regards,

Jared Hirst

Servers Australia Pty Ltd

Phone: 02 4307 4200

Fax: 02 4307 4201

Web: http://www.serversaustralia.com.au

On 13/11/2011, at 4:26 AM, "joel at zonenetworks.com.au"

<joel at zonenetworks.com.au> wrote:

It's been happening for months now not few weeks

Last time we were told they need to fix switch/router but they can't do it
for some reason

Sent from my HTC

----- Reply message -----

From: "Jared Hirst" <jared.hirst at serversaustralia.com.au>

To: "ausnog at ausnog.net" <ausnog at ausnog.net>

Subject: SPAM-LOW:  [AusNOG] Pipe peering issue - Equinix

Date: Sun, Nov 13, 2011 3:30 am

Guys,

Just as an FYI there appears to be some form of broadcast storm on the

pipe peering fabric in Equinix Sydney, looks to be affecting a few

people and filling links from the looks of their online graphs.

We have had to shut our pipe peering down until they resolve it, which

is my next question... Do pipe have a 24/7 NOC still? I called over 30

minutes ago and went to a messaging service but have not had a call

back, have also created a ticket with no response.

So if anyone is wondering why their pipe link is full, this is why.

Appears to be happening every few weeks in Equinix lately.

Kindest Regards,

Jared Hirst

Servers Australia Pty Ltd

Phone: 02 4307 4200

Fax: 02 4307 4201

Web: http://www.serversaustralia.com.au

_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net

http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net

http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20111114/3500d32a/attachment.html>