[AusNOG] Bufferbloat - Queue management vs Buffer filling

[Mark Smith]

On Sun, 16 Jan 2011 14:46:26 +1100
Andrew Fort <afort at choqolat.org> wrote:

> On Sun, Jan 16, 2011 at 12:02 PM, John Edwards <john at netniche.com.au> wrote:
> > On 16/01/2011, at 2:47 AM, Andrew Cox wrote:
> >
> > One of the places I have seen this occur is where there is some kind of
> > peering standoff between major carriers. The link is congested, but the
> > dance of peering politics has not yet resulted in a link upgrade for
> > whatever reason.
> 
> Sometimes it's politics, sometimes it's bureaucracy, but whatever the
> cause, this always helps:
> 
> interface chocka 11/0
>  random-detect
> 
> While I won't say everyone has that turned on, it's been a best
> practice for a long time.  I know we made sure it was in "tier 1"
> eyeball or content world when I lived in such a hell :-).  You can
> also influence RED with DSCP or precedence marks on most platforms.
> 
> If the router buffer is always full (say as on our chocka interface),
> then it will be dropping from somewhere (by default, the tail).  TCP
> will be affected negatively without RED, because the ACK has to get
> all the way to the end of the buffer before it gets dropped, meaning
> the far end TCP speaker has to resend more traffic.  This is where big
> buffers have a negative side effect, because they mess most with TCP
> traffic, making it bursty and reducing overall throughput.  Just
> making the buffer smaller may give you more problems than turning RED
> on, because it is more likely to drop if there is a pause on any of
> the interfaces that traffic is going to, which will tend to influence
> TCP by limiting the maximum window size (i.e., maximum possible
> throughput), because there are a lot of drops.
> 
> The takeaway from Gettys' work is that one needs to consider buffers
> and importantly, their management.  There's no point managing a 1-byte
> or 1-packet buffer, so naturally you tend to think of it when the
> buffers are large.
> 
> To me, the fact that a networking problem recently got slashdotted
> means it's been a problem for at least a decade.  Corollary: Ask the
> vendors how well their software engineers understand networking. :-)
> 
> > It's hard to see it becoming relevant on most switched NBN networks. The
> > likely risk will occur where some smaller players use Linux routers to
> > terminate NBN connections, which will have gobs of memory and 64 bit
> > architectures that can handle and auto-tune big buffers to treat every
> > packet as sacred - all while re-arranging other application memory to swap
> > space.
> 
> To paraphrase Mr. Bush: "I encourage competitors to build their
> routers in this way".  Does this actually happen?  I could be wrong, I
> thought space used for sk_buff buffers was pre-carved and finite?
> 

skbs are dynamically allocated, however interface queues aren't. By
default, the Linux interface queues for ethernets are 1000 packets, for
ppp interfaces it is 3.

You can view their size via the "ip link show" command - the qlen
parameter -


4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff

and change it via the "ip link set <ifname> txqueuelen <value>" command.

Apparently only a couple of the Linux interface queuing disciplines use
the standard interface queues, others implement their own due to their
complexity/methods of operation.



Regards,
Mark.