<div dir="ltr">You're over-thinking the costs associated with it.<div><br></div><div>My work with 10Gbps fibre was for a small company (200 staff in Australia, 5000 globally). We used it locally within DCs, and also between DCs, so I was working with SR and LR optics (and multi-mode and single-mode respectively).</div><div><br></div><div>You can pick up a fibre scope and cleaning kit for < $500.</div><div><br></div><div>If you're talking about work within a server room/datacentre, you may also want to consider 10Gbps over copper (either CAT6 or Twinax).</div><div><br></div><div>If you stick with fibre, you also need to make sure that you're running the right type of cable. Single-mode is for long range or extended range (generally between DCs), multi-mode is for short range (within a server room/data centre).</div><div><br></div><div>Fibre isn't expensive - but nor is it "simple" or for people who don't understand what they're doing with it. You can beat that aspect by simply doing lots of reading (and asking questions - you'll get plenty of help here - off list if you wish. :)</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 November 2017 at 09:59, Burt Mascareigne <span dir="ltr"><<a href="mailto:Burt@stormnetwork.com.au" target="_blank">Burt@stormnetwork.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-AU" link="blue" vlink="purple">
<div class="m_-2728467743866041085WordSection1">
<p class="MsoNormal"><span>Hey All<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>No, we have not done optical loss testing. We’re a IaaS and thought instead of Etherchanneling 4 x 1GB, we’d go crazy and try 10GB.
<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>We’re not a multimillion $ setup, it is something we wanted to dabble in, as all the documents seem to suggest Plug and Play. I’m not really sure anymore, it seems AusNog is just for huge companies
only. <u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Just to clarify, it is in Intel Card with Cisco SFPs connecting to a ZyXEL off MM with a 3meter range.
<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Based on the feed back:<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Intel don’t play nice with anything but Intel, but it “should” work.
<u></u><u></u></span></p>
<p class="MsoNormal"><span>It’s probably one of the tech cleaning the fibre endpoints with their fingers<u></u><u></u></span></p>
<p class="MsoNormal"><span>And no one has even mentioned ZyXEL, which speaks volumes…<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>We keep seeing “</span>Mellanox” everywhere. Looking at going end to end with them, new MM cables etc.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks all, it was not quite plug and play as I thought it would be, but we’re learning.<span><u></u><u></u></span></p><span class="">
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Burt Mascareigne<br>
Mobile</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> 0414 450 962 <b>Office</b> (02) 9965 5422<br>
<b>Address</b> <a href="https://maps.google.com/?q=Level+19,+1+O%E2%80%99Connell+Street,+Sydney+NSW+2000&entry=gmail&source=g">Level 19, 1 O’Connell Street, Sydney NSW 2000</a><br>
<b>Web</b> <a href="http://www.stormnetwork.com.au/" target="_blank"><span style="color:#1f497d">http://www.stormnetwork.<wbr>com.au</span></a><br>
</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"><img border="0" width="133" height="31" style="width:1.3888in;height:.3194in" id="m_-2728467743866041085Picture_x0020_1" src="cid:image004.png@01D35941.670DF2B0" alt="01-300x70"></span><span lang="EN-US" style="color:black"><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
</span><p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> AusNOG [mailto:<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.<wbr>ausnog.net</a>]
<b>On Behalf Of </b>Robert Hudson<br>
<b>Sent:</b> Thursday, 9 November 2017 9:42 AM<br>
<b>To:</b> Mike Taylor <<a href="mailto:mtaylor@totalteam.co.nz" target="_blank">mtaylor@totalteam.co.nz</a>><br>
<b>Cc:</b> <a href="mailto:ausnog@lists.ausnog.net" target="_blank">ausnog@lists.ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] 10GBs<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Grab an optical fibre scope while you're at it. It's amazing how dirty your cable ends and recepticals can actually be.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">And for the love of $deity, make sure nobody's done something as stupid (been there, done that - I had an excuse because I was new to working with fibre, the major carrier datacentre operators less so - yes, they did it too) as plugging
SC cables into SCA recepticals or vice versa.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Have you done optical loss testing? Some network gear can tell you directly what your tx and rx power levels are.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 9 Nov. 2017 7:03 am, "Mike Taylor" <<a href="mailto:mtaylor@totalteam.co.nz" target="_blank">mtaylor@totalteam.co.nz</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p>So, yeah, Layer1 first, apply your fibre cleaning tools to the fibres, SFPs, patch leads etc.<u></u><u></u></p>
<p>10gig gets very very twitchy if the cables and SFPs aren't cleaned properly...<u></u><u></u></p>
<p>Better still, swap out all the patch leads for brand new ones and see what happens...<u></u><u></u></p>
<p>Mike<u></u><u></u></p>
<p><span style="color:#888888"><u></u> <u></u></span></p>
<pre><span style="color:#888888">Mike Taylor<u></u><u></u></span></pre>
<pre><span style="color:#888888">The Total Team<u></u><u></u></span></pre>
<pre><span style="color:#888888"><u></u> <u></u></span></pre>
<div>
<div>
<p class="MsoNormal">On 08/11/17 15:59, Robert Hudson wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">There is no trial and error. <span style="font-family:"Arial",sans-serif">Start at Layer 1, and work up the stack.</span>
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Correct setup and good logging should resolve it.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">There are lots of potential issues. You can cut some of them down immediately by selecting hardware on the hypervisor supplier's HCL - they should have good drivers for the 10Gbps NICs at that point.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Speed/duplex, frame size, etc, all need to be consistent across the entire environment. Get wireshark or an equivalent involved (using a mirror port if necessary).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If your hardware and software are all under full support, involve your vendors in the issue.<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 8 Nov. 2017 1:23 pm, "Burt Mascareigne" <<a href="mailto:Burt@stormnetwork.com.au" target="_blank">Burt@stormnetwork.com.au</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal">Hi Noggers<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We’re having “fun” with 10GB NICS, Switches and SFPs.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We can’t work out which is causing drop out, packets.. we are somewhat certain packets are vanishing into the Ether..<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We don’t know which is causing what.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I’m not after troubleshooting, I am curious what people’s view on this.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Do you buy 1 Vendor for all your 10GBs, A-Z, is there some hidden rule about mixing vendors, is this all trial and error (and very expensive at best)?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Any recommendations on set ups? Preferred Vendors, Vender to avoid like the plague. <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We’re talking Servers to Switch kind of thing. Not SANs etc. Good ol’ HyperVisor to the switch.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Regards,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Burt Mascareigne<br>
Mobile</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> 0414 450 962 <b>Office</b> (02) 9965 5422<br>
<b>Address</b> <a href="https://maps.google.com/?q=Level+19,+1+O%E2%80%99Connell+Street,+Sydney+NSW+2000&entry=gmail&source=g">Level 19, 1 O’Connell Street, Sydney NSW 2000</a><br>
<b>Web</b> <a href="http://www.stormnetwork.com.au/" target="_blank"><span style="color:#1f497d">http://www.stormnetwork.<wbr>com.au</span></a><br>
</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"><img border="0" width="133" height="31" style="width:1.3888in;height:.3263in" id="m_-2728467743866041085m_3483318052147019316m_-5962761642232150388Picture_x0020_1" src="cid:image003.png@01D35940.85D11ED0" alt="01-300x70"></span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/<wbr>mailman/listinfo/ausnog</a><u></u><u></u></p>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<pre>______________________________<wbr>_________________<u></u><u></u></pre>
<pre>AusNOG mailing list<u></u><u></u></pre>
<pre><a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><u></u><u></u></pre>
<pre><a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/<wbr>mailman/listinfo/ausnog</a><u></u><u></u></pre>
</blockquote>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/<wbr>mailman/listinfo/ausnog</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>