[AusNOG] the state of bufferbloat awareness and remediation in australia?

Mon Jan 27 02:57:32 EST 2020

On Sat, Jan 25, 2020 at 7:18 PM Mark Smith <markzzzsmith at gmail.com> wrote:
>
>
>
> On Sun, 26 Jan 2020, 13:38 Rob Thomas, <xrobau at gmail.com> wrote:
>>
>> Unsurprisingly to most, I'm the guy who played one of the VoIP packets
>> (I'm in the blue Clearly IP shirt).

And what an unruly packet you were.

>>
>> After his talk at LCA, I was discussing the way to handle VoIP, and I
>> mentioned that the way I do most VoIP client connections is via a
>> tunnel. This allows me to put a proper queue on the tunnel ITSELF, and
>> just hope that the actual outgoing link of the customer isn't
>> congested.
>>
>> This seems to work reasonably well, and customers usually end up with
>> good audio (as well as bypassing all the NAT nightmares).

It would be interesting to gather voip jitter, latency and loss
statistics from various networking edges there, under load. Tracking
that in any way? What sort of voip termination software do you use
(asterisk? freeswitch?).

Applying fq_codel or cake to the edge router gives short queues
overall, and a dedicated queue for the tunnel, which so long as it is
sparse (e.g. voip packets only), automatically gets nearly zero
latency.

do you also put a diffserv mark on the tunnel?

Applying sqm-scripts or an equivalent also can take advantage of
diffserv markings on the tunnel on the way out from the home/office
and (at least on some ISPs) the way in, for an even higher level of
certainty. Do you have any data on how often dscp is remarked in
transit in australia? In most of my travels dscps are no longer peed
on that much, the major exception being comcast which remarks nearly
everything to CS1. I've seen dscps survive from as far away as
argentina -> australia....

the pie aqm is also being rolled out in docsis 3.1 - it works pretty
good on the cablemodems in the upstream direction only.

The docsis 3.0 cable deployment was much, much worse than this
http://www.dslreports.com/speedtest/results/bufferbloat?up=1 two years
ago before those upgrades began to roll out.

>>
>> I'm hoping that Dave comes back next year to LCA (and for those that
>> missed the announcement, it's in Canberra), and can do some more talks
>> next year 8)

As much as I enjoyed the trip and conference and y'all and a chance to
hang out at APNIC, I'd prefer to arrange a longer (and at least
somewhat paid) trip next time. Karmically fixing bufferbloat is +1B,
financially... -2m... for me.

I really liked the keynotes at linux.conf.au - the one on "drop your
tools" was really profound and has made me think a lot.

https://www.youtube.com/playlist?list=PLD8dAKx4J2I6CoeaU6F-2Zs7XmWjUBLlT

>
>
> Networking talks will get a larger audience at a networking conference.

I have done quite a few of those...

The nature of this particular talk was intended to be light and funny,
conveying intuition rather than deep technical details. Most of the
people in the room had some grasp of the subject at hand but me making
it funny I hope made the core concepts more memorable, and people more
inclined to rush home and try a fq-codel based shaper out. (sch_cake
has this down to one line of configuration for egress, 4 for ingress,
the sqm-scripts automate this for most linuxes for the older htb +
fq_codel implementation, the edgerouter and pfsense both have "smart
queue management" scripts as well as dozens of other products  (and no
configuration is required for things running at line rate)

I'm sure the talk would go down well at a networking conference also,
but I certainly do do far, far more technical talks (see youtube) and
my work in general is more focused on fixing cpe and the edges of the
internet... Right now (on a tiny budget) I'm trying to get wifi 6
working better on the ax200 chipset, and helping duke it out in the
IETF in the L4S vs SCE dispute over the meaning of the last half a bit
in the IP ECN header (on no budget).

Oh, my, that debate is no fun at all...

https://datatracker.ietf.org/meeting/105/materials/slides-105-tsvwg-sessa-61-some-congestion-experienced-00
https://www.youtube.com/watch?v=FDK88vdE5r0&t=1h15m

https://lwn.net/Articles/783673/

and a zillion drafts and talks on l4s as well.

> I doubt many on the AusNOG mailing list knew there would be a buffer bloat related presentation at LCA, and even if so, network engineers are going to have a harder time convincing management to pay for them to go to a week long conference just for one or just a few relevant talks.

Is there another networking conference in australia worth doing on the
bufferbloat front?

A hallway track is generally worthwhile at nearly any conference.

>
>
>
>>
>> --Rob
>>
>> On Sun, 26 Jan 2020 at 12:29, Dave Taht <dave.taht at gmail.com> wrote:
>> >
>> > Ladies and gentlemen! Ryan Mounce has entered the room! Ryan
>> > contributed the ack-filtering code in sch_cake several years ago. All
>> > Hail Ryan! Cake ( https://lwn.net/Articles/758353/ )has been available
>> > out of tree for linux for 5 years (with maintained backports all the
>> > way back to linux 3.10) and it finally went upstream in linux 4.19.
>> > Please note that as much as I like cake, the sqm scripts go back even
>> > further (and also allow for configuring not just fq_codel but pie)
>> >
>> > On Sat, Jan 25, 2020 at 5:52 PM Ryan Mounce <ryan at mounce.com.au> wrote:
>> > >
>> > > NBN Co do inject options into DHCP and PPPoE requests with the down/up sync rate for their wholesale xDSL services so that ISPs (RSPs in NBN lingo) can shape individual services correctly. I have no sense how widely ISPs are actually taking advantage of this.
>> >
>> > Well, I try to keep kicking em along. As noted in my previous message
>> > we're seeing a couple ISPs in germany move on this (free.fr in france
>> > adopted this stuff in 2012, everybody else is lagging somewhat :/)
>> > It's only a few line of shell script hook into the negotiation phases
>> > at this point. A days worth of work. Or less. And as for "correctly",
>> > well, I keep hoping folk will leverage the sqm-scripts and/or cake -
>> > when available. Aside from the fritzbox (which I know has fq_codel in
>> > it), are there any other common home routers with a reasonably modern
>> > linux or freebsd os in 'em?
>> >
>> > Do you have any insight into the DHCP message?
>> >
>> > who "owns" the end-user router in australia now?
>> >
>> > anyway, my talk at linux.conf.au (and blatant plug #2) is reviewed now
>> > at: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/
>> >
>> > great to see you here, ryan,
>> >
>> > > -Ryan
>> > >
>> > > On Fri, 17 Jan 2020 at 11:24, Dave Taht <dave.taht at gmail.com> wrote:
>> > >>
>> > >> Hi, all. I'm here at linux.conf.au having just given a talk about how
>> > >> tcp works in the bufferbloated age[1].
>> > >>
>> > >> On my way here I stopped in for a few days with geoff huston and
>> > >> george michaelson who filled my ears with the chaos of the NBN rollout
>> > >> and other issues in the australian network infrastructure... and gave
>> > >> me a shot at some data they had on bloat, and ecn usage in the wild
>> > >> which I hope to write up over the next month or so.
>> > >>
>> > >> I was wondering about a few things:
>> > >>
>> > >> How y'all doing on eliminating bufferbloat from your networks? Using
>> > >> things like fq_codel, sch_fq + bbr, etc?
>> > >>
>> > >> Do you have any awareness from your regulatorium?
>> > >>
>> > >> I see, from sites like whirlpool, that some consumer hardware here,
>> > >> like the fritzbox, have fq_codel now, but it's not clear if ISPs are
>> > >> actively configuring it (or what we call "sqm") yet. (theres a PPPoe
>> > >> message now in use in parts of germany for up/down and frame rate
>> > >> seeing increasing deployment).
>> > >>
>> > >> Lastly:
>> > >>
>> > >> Anyone need a wayward network researcher/theorist for a few weeks or
>> > >> months to help address their bufferbloat issues in their stacks and
>> > >> hw? - maybe not this trip but on some other occasion?
>> > >> (I rather like hanging in australia)
>> > >>
>> > >> [1] blatant plug  http://youtu.be/ZeCIbCzGY6k
>> > >>
>> > >> --
>> > >> Make Music, Not War
>> > >>
>> > >> Dave Täht
>> > >> CTO, TekLibre, LLC
>> > >> http://www.teklibre.com
>> > >> Tel: 1-831-435-0729
>> > >> _______________________________________________
>> > >> AusNOG mailing list
>> > >> AusNOG at lists.ausnog.net
>> > >> http://lists.ausnog.net/mailman/listinfo/ausnog
>> > >
>> > > --
>> > > Regards,
>> > > Ryan Mounce
>> > >
>> > > ryan at mounce.com.au
>> > > 0415 799 929
>> > >
>> > > Sent from mobile
>> >
>> >
>> >
>> > --
>> > Make Music, Not War
>> >
>> > Dave Täht
>> > CTO, TekLibre, LLC
>> > http://www.teklibre.com
>> > Tel: 1-831-435-0729
>> > _______________________________________________
>> > AusNOG mailing list
>> > AusNOG at lists.ausnog.net
>> > http://lists.ausnog.net/mailman/listinfo/ausnog
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729