[AusNOG] SPAM-LOW: Pipe peering issue - Equinix

Mon Nov 14 17:39:18 EST 2011

Hi everyone,

I just wanted to make a clear statement about where we are up to on this
issue.

In the past few months we have seen problems with unknown unicast
flooding on the PIPE Sydney IX. This issue affects a subset of peers
connected to just one of the switches that make up the Sydney IX.

We use a combination of port security and layer 2 ACL's on all peering
interfaces to prevent this type of problem. Over the years these
mechanisms have prevented many broadcast storms and loops on the
network. For those concerned, our monitoring system does detect this and
other similar problems for reporting to the NOC. They are monitored and
escalated as required.

With these particular issues, the controls do not appear to be
effective. We suspect a hardware or software problem to be the cause,
but it has proven difficult to reproduce. The issue has been narrowed to
traffic injected by a specific peer and some additional protections are
now in place to minimise the risk while we work with the peer to find
the best permanent solution.

We have had a migration plan in progress for a while now to replace the
switch platform; I'm sure you can all imagine it is not a quick or easy
task to complete. We have an expectation of contacting you all in the
next few days to continue this work with a view to setting a mutually
agreed migration date.

Ta,

Matt Whitlock 
Operations Manager, PIPE 

________________________________

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Andrew Fort
Sent: Monday, 14 November 2011 11:43 AM
To: Sean K. Finn
Cc: ausnog at ausnog.net
Subject: Re: [AusNOG] SPAM-LOW: Pipe peering issue - Equinix

IXes for years have done mac port filtering to mitigate this issue.
One port, one mac. No tagging. If a port is lit up, it should point to
the issue (if filtered frames hit l2 int counters). A faulty transceiver
spewing a valid source mac?

Is mac filtering not being done there, or is it not working?

On Nov 13, 2011 5:33 PM, "Sean K. Finn" <sean.finn at ozservers.com.au>
wrote:

When a CAM Table is maxed-out a switch becomes a full-duplex hub. (Cool
hey).

If it doesn't know which mac address to send the traffic to, it sends it
EVERYWHERE.

If the traffic is deliberately being injected into a port knowing that
the mac address doesn't exist on the peering fabric, EVERYONE gets it,
even if the CAM table isn't full.

I'm not on PIPE-IX-SYD, (Only on Equinix-SYD, O Hai) but I can imagine
that..

There's a customer connected to the IX who's own network has become a
hub so they are smashing L2 traffic into the IX not on purpose.

At least I'd hope that's the case, I wouldn't imagine anyone would
deliberately sabotage the IX.

If someone can sniff the packets you may be able to get the MAC address,
then do an Arping on all of the IX IP's and see which mac addresses
match the IP's in question, or look at your routers ARP table for a
corresponding MAC address to get the offending networks PEER-IP. (Or
look on the graphs and see who is inverse to everyone else).

Who has more peers, by the way? Equinix-SYD or PIPE-SYD? 

S.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of ZoneNetworks -
Joel
Sent: Sunday, November 13, 2011 9:35 AM
To: Skeeve Stevens
Cc: ausnog at ausnog.net
Subject: Re: [AusNOG] SPAM-LOW: Pipe peering issue - Equinix

To my knowledge it's a hardware/firmware issue not traffic related
...though looking at the graphs it seems like traffic/flood which is
misleading

If more customers gave them a hard time it might get fixed sooner, we
were told (in august) they had a long term fix in plan but no ETA when
it will be done

Not something you would hear from the "old pipe"

Joel

Sent from my iPad

On 13/11/2011, at 10:06 AM, Skeeve Stevens <Skeeve at eintellego.net>
wrote:

	Do we know where the traffic is coming from?

	...Skeeve

	--

	Skeeve Stevens, CEO - eintellego Pty Ltd

	skeeve at eintellego.net ; www.eintellego.net

	Phone: 1300 753 383 ; Fax: (+612) 8572 9954
<tel:%28%2B612%29%208572%209954> 

	Cell +61 (0)414 753 383 <tel:%2B61%20%280%29414%20753%20383>  ;
skype://skeeve

	facebook.com/eintellego

	twitter.com/networkceoau ; www.linkedin.com/in/skeeve

	PO Box 7726, Baulkham Hills, NSW 1755 Australia

	--

	eintellego - The Experts Who The Experts Call

	Juniper - HP Networking - Cisco - Brocade

	On 13/11/11 10:04 AM, "Jared Hirst"
<jared.hirst at serversaustralia.com.au> wrote:

		Yeah every few weeks it happens is what I mean, they
start sending

		hundreds of mbits of useless broadcast traffic to all
customers in

		Equinix. It's getting frustrating and annoying as it
always seems to

		happen at 2:00on a Saturday morning :(

		Kindest Regards,

		Jared Hirst

		Servers Australia Pty Ltd

		Phone: 02 4307 4200

		Fax: 02 4307 4201

		Web: http://www.serversaustralia.com.au

		On 13/11/2011, at 4:26 AM, "joel at zonenetworks.com.au"

		<joel at zonenetworks.com.au> wrote:

			It's been happening for months now not few weeks

			Last time we were told they need to fix
switch/router but they can't do it for some reason

			Sent from my HTC

			----- Reply message -----

			From: "Jared Hirst"
<jared.hirst at serversaustralia.com.au>

			To: "ausnog at ausnog.net" <ausnog at ausnog.net>

			Subject: SPAM-LOW:  [AusNOG] Pipe peering issue
- Equinix

			Date: Sun, Nov 13, 2011 3:30 am

			Guys,

			Just as an FYI there appears to be some form of
broadcast storm on the

			pipe peering fabric in Equinix Sydney, looks to
be affecting a few

			people and filling links from the looks of their
online graphs.

			We have had to shut our pipe peering down until
they resolve it, which

			is my next question... Do pipe have a 24/7 NOC
still? I called over 30

			minutes ago and went to a messaging service but
have not had a call

			back, have also created a ticket with no
response.

			So if anyone is wondering why their pipe link is
full, this is why.

			Appears to be happening every few weeks in
Equinix lately.

			Kindest Regards,

			Jared Hirst

			Servers Australia Pty Ltd

			Phone: 02 4307 4200

			Fax: 02 4307 4201

			Web: http://www.serversaustralia.com.au

			_______________________________________________

			AusNOG mailing list

			AusNOG at lists.ausnog.net

			http://lists.ausnog.net/mailman/listinfo/ausnog

		_______________________________________________

		AusNOG mailing list

		AusNOG at lists.ausnog.net

		http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20111114/2946094b/attachment.html>