[AusNOG] 10GBs

Thu Nov 9 10:31:57 EST 2017

Intel NICs are very twitchy indeed about non-Intel optics. If you're using non-Intel optics you typically need to tell the driver to enable unsupported optic mode. 

Since you say some packets are disappearing rather than the link not coming up at all, it's probably not this, but I'd strongly suggest reviewing the *specific* Intel NIC list of compatibility, and checking whether the Cisco module you're using is within parameters of the NIC. There's a lot that can go wrong here - thermal and power budgets are a problem for many optics in compute. 

As others have said, DACs might be the way to go but be ware that many switch vendors can't or don't test against the zillions of DACs available in the market. 

If you're running a 10G optical switch, forget 10GBASE-T SFP+ as the power budget for electrical is a nightmare. On the other hand you can now find 10GBASE-T switches available. 

> On Nov 8, 2017, at 5:59 PM, Burt Mascareigne <Burt at stormnetwork.com.au> wrote:
> 
> Hey All
>  
> No, we have not done optical loss testing.  We’re a IaaS and thought instead of Etherchanneling 4 x 1GB, we’d go crazy and try 10GB.
>  
> We’re not a multimillion $ setup, it is something we wanted to dabble in, as all the documents seem to suggest Plug and Play.  I’m not really sure anymore, it seems AusNog is just for huge companies only. 
>  
> Just to clarify, it is in Intel Card with Cisco SFPs connecting to a ZyXEL off MM with a 3meter range. 
>  
> Based on the feed back:
>  
> Intel don’t play nice with anything but Intel, but it “should” work.
> It’s probably one of the tech cleaning the fibre endpoints with their fingers
> And no one has even mentioned ZyXEL, which speaks volumes…
>  
> We keep seeing “Mellanox” everywhere.  Looking at going end to end with them, new MM cables etc.
>  
> Thanks all, it was not quite plug and play as I thought it would be, but we’re learning.
>  
>  
>  
> Regards,
>  
>  
> Burt Mascareigne
> Mobile 0414 450 962   Office (02) 9965 5422
> Address Level 19, 1 O’Connell Street, Sydney NSW 2000
> Web http://www.stormnetwork.com.au
> <image004.png>
>  
>  
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Robert Hudson
> Sent: Thursday, 9 November 2017 9:42 AM
> To: Mike Taylor <mtaylor at totalteam.co.nz>
> Cc: ausnog at lists.ausnog.net
> Subject: Re: [AusNOG] 10GBs
>  
> Grab an optical fibre scope while you're at it. It's amazing how dirty your cable ends and recepticals can actually be.
>  
> And for the love of $deity, make sure nobody's done something as stupid (been there, done that - I had an excuse because I was new to working with fibre, the major carrier datacentre operators less so - yes, they did it too) as plugging SC cables into SCA recepticals or vice versa.
>  
> Have you done optical loss testing? Some network gear can tell you directly what your tx and rx power levels are.
>  
> On 9 Nov. 2017 7:03 am, "Mike Taylor" <mtaylor at totalteam.co.nz> wrote:
> So, yeah, Layer1 first, apply your fibre cleaning tools to the fibres, SFPs, patch leads etc.
> 
> 10gig gets very very twitchy if the cables and SFPs aren't cleaned properly...
> 
> Better still, swap out all the patch leads for brand new ones and see what happens...
> 
> Mike
> 
>  
> 
> Mike Taylor
> The Total Team
>  
> On 08/11/17 15:59, Robert Hudson wrote:
> There is no trial and error. Start at Layer 1, and work up the stack.
>  
> Correct setup and good logging should resolve it.
>  
> There are lots of potential issues. You can cut some of them down immediately by selecting hardware on the hypervisor supplier's HCL - they should have good drivers for the 10Gbps NICs at that point.
>  
> Speed/duplex, frame size, etc, all need to be consistent across the entire environment. Get wireshark or an equivalent involved (using a mirror port if necessary).
>  
> If your hardware and software are all under full support, involve your vendors in the issue.
>  
> On 8 Nov. 2017 1:23 pm, "Burt Mascareigne" <Burt at stormnetwork.com.au> wrote:
> Hi Noggers
>  
> We’re having “fun” with 10GB NICS, Switches and SFPs.
>  
> We can’t work out which is causing drop out, packets.. we are somewhat certain packets are vanishing into the Ether..
>  
> We don’t know which is causing what.
>  
> I’m not after troubleshooting, I am curious what people’s view on this. 
>  
> Do you buy 1 Vendor for all your 10GBs, A-Z, is there some hidden rule about mixing vendors, is this all trial and error (and very expensive at best)?
>  
> Any recommendations on set ups?  Preferred Vendors, Vender to avoid like the plague.  
>  
> We’re talking Servers to Switch kind of thing.  Not SANs etc.  Good ol’ HyperVisor to the switch.
>  
> Thanks
>  
>  
>  
>  
>  
> Regards,
>  
>  
> Burt Mascareigne
> Mobile 0414 450 962   Office (02) 9965 5422
> Address Level 19, 1 O’Connell Street, Sydney NSW 2000
> Web http://www.stormnetwork.com.au
> <image003.png>
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
> 
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
> 
>  
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20171108/18e432f4/attachment.html>