<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 07/15/2013 10:03 AM, Lincoln Dale
wrote:<br>
</div>
<blockquote
cite="mid:CAAhBB_CAOzgQhZtJ_UF5gUCM6PVp=vnWYid6LtnHmP0r2B1syQ@mail.gmail.com"
type="cite">
<meta http-equiv="Context-Type" content="text/html;
charset=ISO-8859-1">
On Mon, Jul 15, 2013 at 9:29 AM, Paul Gear <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:ausnog@libertysys.com.au"
target="_blank">ausnog@libertysys.com.au</a>></span> wrote:<br>
<div class="gmail_quote">...
<blockquote class="gmail_quote">
<div>On a recent Packet Pushers show where Arista were talking
about their new switches, they pointed out that their
buffers seemed overly large, but at the high bandwidths they
were serving, this was only 250 ms or thereabouts (my memory
is a bit hazy, but i think it was about 512 MB per 10 Gbps
port). </div>
</blockquote>
<div><br>
The podcast you're talking about was about switches that have
~125MB / 10G port, 4x/10x that for 40G/100G ports.. In all
cases if you actually had something consuming all those
buffers, its ~12msec.<br>
<br>
Reality is that its not quite that simple, as the switches in
question are VoQ based, so that buffer that is physically on
ingress representing queueing on output and is in fact
distributed queuing.<br>
Switch buffers are also never 100% effective utilization
either. (silicon stores packets in 'cells' and those are not
variable-sized cells.)<br>
</div>
</div>
</blockquote>
<br>
Pardon my ignorance, but VoQ == Virtual Output Queuing? Is the
Wikipedia description more or less accurate?
<a class="moz-txt-link-freetext" href="https://en.wikipedia.org/wiki/Virtual_Output_Queues">https://en.wikipedia.org/wiki/Virtual_Output_Queues</a><br>
<br>
<blockquote
cite="mid:CAAhBB_CAOzgQhZtJ_UF5gUCM6PVp=vnWYid6LtnHmP0r2B1syQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div>
I could talk for days in this topic having done all analysis
and simulation on the 'right' about of buffer on switches but
suffice to say what is 'right' depends on the place in the
network and # of simultaneous TCP flows going thru the box and
degree of incast/oversubscription.<br>
</div>
</div>
</blockquote>
<br>
Feel free to wax lyrical. Today's my study day and i'm trying to
learn something while i wait for CSU to fix their phones. <span
class="moz-smiley-s1"><span> :-) </span></span><br>
<br>
<blockquote
cite="mid:CAAhBB_CAOzgQhZtJ_UF5gUCM6PVp=vnWYid6LtnHmP0r2B1syQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div>
In the case of the company I work for yes we have done a lot
of analysis on this, both by having telemetry data of actual
buffer queue depths in production environments but also
testing of various workflows of modern applications and
traffic flows.<br>
Its how we determined (for example) to use 2Gbit DDR3-2166
rather than 1Gbit DDR3-2166 parts when building said switch.<br>
</div>
</div>
</blockquote>
<br>
What sort of testing process/software did you use for those sorts of
test flows? I'm guessing the licensing costs and the time spent
would vastly outweigh the budget for switching of a project like the
one for which Joseph started this thread.<br>
<br>
<blockquote
cite="mid:CAAhBB_CAOzgQhZtJ_UF5gUCM6PVp=vnWYid6LtnHmP0r2B1syQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div>If you were interested in theory/simulation/practice on
this, its actually something I gave a talk at CAIA (<a
moz-do-not-send="true" href="http://caia.swin.edu.au/">http://caia.swin.edu.au/</a>)
last year. More than happy to share the slides/content if
there is no video recording of it.<br>
</div>
</div>
</blockquote>
<br>
It appears they don't keep much at all:
<a class="moz-txt-link-freetext" href="http://caia.swin.edu.au/seminars/details/120223A.html">http://caia.swin.edu.au/seminars/details/120223A.html</a> Share away! <span
class="moz-smiley-s1"><span> :-) </span></span><br>
<br>
<blockquote
cite="mid:CAAhBB_CAOzgQhZtJ_UF5gUCM6PVp=vnWYid6LtnHmP0r2B1syQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div>
<br>
</div>
<blockquote class="gmail_quote">
<div>How does one determine the optimal buffer size (and hence
switch selection) for a particular environment? Is there a
rule of thumb based on the bandwidth of the link and the
maximum time one wants a packet to queue? (And how would
one even determine what this maximum might be? I would
think that it varies depending upon the application.) I
guess this paragraph's questions are mostly directed to Greg
& Lincoln - in the cases you've mentioned, how did you
know that small buffers were the problem?<br>
</div>
</blockquote>
<div><br>
Unfortunately most ethernet switches simply haven't provided
the telemetry to indicate how close they are to running out of
buffers until its too late and you've fallen off the cliff and
have large numbers of drops going on.<br>
(its actually worse than this: many switches don't even
provide accurate drop counters when they are dropping packets)<br>
Historically many switches had 'dedicated' buffers/port and
didn't have overflow/shared pools for dealing with <br>
<br>
Even if accurate drop counters are available, how many people
actually monitor those?<br>
<br>
Thing about TCP is it still works even if you have drops. Just
for many environments "working" isn't good enough, they want
"working well." The DataSift blog I pointed to is a good
example.<br>
</div>
</div>
</blockquote>
<br>
My fear with all of this as that for most environments working <i>is</i>
good enough, and working well costs more than they are prepared to
spend.<br>
<br>
<blockquote
cite="mid:CAAhBB_CAOzgQhZtJ_UF5gUCM6PVp=vnWYid6LtnHmP0r2B1syQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div>The short answer is that there is no single 'right' answer
for what is appropriate. It depends on the traffic load,
distribution and what the underlying apps/hosts/servers are
doing. Which may change over time too.<br>
</div>
</div>
</blockquote>
<br>
Are there any quick wins available to the budget end of town? Are
there rules of thumb or quick estimates that can help determine if
it's an issue?<br>
<br>
Paul<br>
<br>
</body>
</html>