[AusNOG] Open Networking

[Lincoln Dale]

g'day,

my AU$0.02 worth, been involved in working with the hardware of all of
these:

On Tue, Jul 5, 2016 at 9:08 AM, Nik Geyer <nik at neko.id.au> wrote:

> Broadcom Trident 2 runs 12MB packet buffer, the original Trident was less
> from memory.
>

Trident+ and Trident was 9MB.


>
> The Broadcom Tomahawk chipset (25/50/100GE cheap switches) run a 7.6MB
> packet buffer.
>

 Tomahawk is 16MB not 7.6MB though strictly speaking its 4 quads of 4MB.

The Broadcom Dune (Jericho) chipset uses external DDR4/GDDR5 memory for
> packet buffering, so you'll see various sizing from different vendors
> depending on how much they throw at it. For example, the new Arista
> 7280CR-48 which is based on this chipset has 32GB of packet buffer. The
> downside to external memory is that it isn't necessarily as fast as
> on-ASIC/SoC packet buffers, so if you're in a latency sensitive environment
> it may not be the best option. Weapon of a storage switch though!
>

There's no real downside to the buffers on Jericho, heads of queues are
kept on-chip so there is no latency difference for on-chip vs off-chip
buffer.
Its not a cut-through switch (for other reasons) but latency is sub 3.5
usec, which is just noise in the scheme of things.


>
> Again, it all comes down to due diligence and selecting the right hardware
> for your requirements. Various features can differ greatly between the
> chipsets also, e.g. if you want to VXLAN route at line rate without packet
> recirculation, don't buy a Trident 2 switch (unless it's a Nexus 9300 with
> it's Merchant+ Insieme ASIC wizardry).
>

Trident2 can do VXLAN Bridging in a single pass, and can do VXLAN Routing
via recirculation.
Whether that recirc is an issue or not depends on whether all of the ports
are exposed on the front panel, e.g. 128x10G / 32x40G or not.

If e.g. its a 48x10G + 6x40G switch, there's 540Gbps of recirc available,
which is more than enough for no downside.

Bigger issue is for the most part I don't think many people actually got
around to enabling VXLAN Routing on T2. Only a handful of clueful folks
AFAIK.


Its a misnomer to think of N9K3 as 'superior' here when its functional in
that manner, as its really no different than recirculation on T2 except
always requiring all traffic to go over an internal 6x40G/12x40G LAG
interface.



> Trident 2+ is better though with single pass encap/decap and double the
> VXLAN performance of the Trident 2. Tomahawk and Jericho are better again,
> but Tomahawk has truly terrible buffers and arguably some hardware
> deficiencies vendors are still trying to sort out around arbitration of
> packets when congested, Jericho switches are expensive.
>

Qumran-MX based switches aren't a significant price premium to that of T2+.
e.g. Arista 7280R-48C6.


>
> Sent from my iPhone
>
> On 5 Jul 2016, at 10:40 AM, Andrew Yager <andrew at rwts.com.au> wrote:
>
> We've been running it for almost 18 months now in production and have done
> a fair bit of work with a number of the vendors building OS platforms.
>
> IMO the one to watch in the ISP/"Traditional Network" space is a product
> called OCNOS by IP Infusion. It's a very Cisco like CLI with a range of
> "carrier" technologies including MPLS, RSVP, LDP implemented. We have found
> a couple of limitations with FRR, but still quite acceptable once tested.
> We've also done a fair bit of interop testing with Juniper and Cisco and
> are still running it in our labs on some S6000s and have had no major
> issues. It also has a rudimentary HQOS implementation that we haven't
> really drilled into yet.
>
>
As far as the comments on the 'open' side of things, I'd suggest you test
your use-cases and specifically the failure cases very very carefully.
Caveat Emptor applies here, I still stick by my adage of if it was so easy
to write the control-plane of a switch, why aren't there dozens of vendors
to choose from?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160705/ff66b9b7/attachment-0001.html>