[AusNOG] Spreading the load of ISP customers at Layer2

[Dave Taht]

On Mon, Sep 13, 2021 at 6:15 AM Damian Ivereigh <damo at launtel.net.au> wrote:
>
> I would love to use fq_codel, but right now we just use Mikrotik's red.
> The biggest hassle is that NBN shape down at 10ms which is pretty hard
> on the shaping software!
>
> Does make me wonder if we should ditch the Mikrotiks and use an open
> source solution.

LibreQos has come a long way so far. Also preseem's commercial product
is pretty good, especially
in the WISP market.

Microtiks beta has full fq_codel and cake support. I keep hoping more
folk will test cake, in particular, on their
wide range of hardware, as it only requires a single line of
configuration to use (and can be dynamically adjusted
on the fly), but as it essentially has per packet resolution (10ms!?
Pah!) and a raft of other (optional) features,
can be more cpu-intensive than the normal "SQM" htb + fq_codel
implementations. (but if you are in a position
where line rate backpressure is enough, rather than software shaping,
the cpu usage is very moderate).

We pitched it, in the linux mainline commit, somewhat snarkily - "as
simple enough that even an ISP can configure it". :)

It was kind of my hope that we would see cake move into the slower
parts of the market (DSL for example is where it's
most common today) and that as cpus got better or offloads got better,
we'd see more and more of it.

discussion here: https://lwn.net/Articles/758353/

paper here: https://arxiv.org/pdf/1804.07617.pdf

Anyway, all of this is a distraction from your original question, and
I'm sorry for that. I do wish - cake was targetted primarily at home
routers - that we understood the actual needs of the ISP headend
portions of the market, better, 'cause then we'd go back to work
trying to address those. LibreQos's main limitation, in my mind, is
that it
is limited to about 1000 devices per instance, and that limit comes
from an ancient dependence on some 16 bit code in the Linux kernel
that perhaps we can
get around by switching to ebpf. (preseem is using ebpf)



Anyway
> Damian
>
> On 9/13/21 10:33 PM, Dave Taht wrote:
> > Wow. You live in such a different world than I. I would really like to
> > better understand problems such as these, but where
> > you are worried about arp at this low level, I worry about good queue
> > and subscriber bandwidth management like that in this:
> >
> > https://github.com/rchac/LibreQoS
> >
> > (leveraging sch_cake in some releases)
> >
> > so, and I know it's kind of off topic from the problem you have... how
> > the heck do you do bandwidth and queue management
> > in either scenario below?
> >
> > On Mon, Sep 13, 2021 at 2:24 AM Damian Ivereigh <damo at launtel.net.au> wrote:
> >> Hi guys,
> >>
> >> We have built all our ISP infrastructure based on the NBN style doubled
> >> tagging of services - in other words each subscriber circuit comes
> >> through on it's own ctag. This makes separating everything really easy
> >> because we pipe each vlan through to different BNG's. However we are now
> >> presented with a wholesaler who does not separate each circuit, but
> >> instead just bridges them all together into a single circuit. We can
> >> distinguish each circuit only by inspecting the DHCP Option82 so that we
> >> can allocate the right IP address, which is fine, but it is hard to
> >> allocate them to use a particular BNG to send and receive traffic.
> >>
> >> By the way I am not talking dynamic load balancing just having multiple
> >> BNG with a subsection of the customers on each one - load sharing?
> >>
> >> Until now with double tagging, we can reuse the same gateway IP address
> >> (i.e. the side facing the customer) on all the BNG and because each BNG
> >> only sees it's circuits, it will only respond to arps that it should do
> >> on the vlans assigned to it. However with all the customers on the same
> >> circuit it is impossible for multiple BNG to have the same IP address
> >> without creating all sorts of duplicate arps etc. We could turn off arp
> >> on all but one of the BNG and then put up with the asymmetric routing
> >> (makes reverse path filtering impossible) - i.e. send all upload traffic
> >> through a single BNG, but download comes from different ones (according
> >> to what BNG they are allocated to).
> >>
> >> I have come up with another hack by using essentially using arp spoofing
> >> where we get a separate box to respond to the arp requests based on what
> >> the source IP is, but I can't help wondering how others have handled
> >> this. The wholesaler tells me there are other ISPs with 5000+ services
> >> on the single circuit (feels like a recipe for a broadcast storm to me).
> >>
> >> Oh and no we don't want to use PPPoE :-)
> >>
> >> Ideas anyone?
> >>
> >> Damian
> >>
> >> --
> >> Launtel - We're at your call
> >> Tel: 1800LAUNTEL (1800528683)
> >> Mob: 0418217582
> >> Fax: 1300784109
> >> http://www.launtel.net.au
> >>
> >> _______________________________________________
> >> AusNOG mailing list
> >> AusNOG at lists.ausnog.net
> >> http://lists.ausnog.net/mailman/listinfo/ausnog
> >
> >
> --
> Launtel - We're at your call
> Tel: 1800LAUNTEL (1800528683)
> Mob: 0418217582
> Fax: 1300784109
> http://www.launtel.net.au
>


-- 
Fixing Starlink's Latencies: https://www.youtube.com/watch?v=c9gLo6Xrwgw

Dave Täht CEO, TekLibre, LLC