[AusNOG] So who's read an RFC or Internet Draft?

[Mark Smith]

Hi Geoff,

On 11 October 2015 at 12:59, Geoff Huston <gih at apnic.net> wrote:
> I have no idea is this is interesting or not for others - in some sense its
> just re-telling old history, but maybe if you have an interest in the
> design of networking protocols you may find this learned experience useful.
>

Actually, a fair bit if not all of my thinking on this has come from
practical experiences. I tried to bridge Wifi once and when it didn't
work, that is how I found out about the IEEE not including source
addresses :-(

>
>> On 10 Oct 2015, at 1:22 PM, Mark Smith <markzzzsmith at gmail.com> wrote:
>>
>> Hi Geoff,
>>
>> On 7 October 2015 at 06:42, Geoff Huston <gih at apnic.net> wrote:
>>>> "Addresses

<snip>

>>>
>>>
>>> Yes, at one point IP played with the concept of variable length addresses,
>>> which also appeared later in the OSI NSAP address structure, as I recall.
>>>
>>
>> Yes, OSI NSAPs are variable length. Going by what Christian Huitema
>> said in his "IPv6: The New Internet Protocol" book, one of the reasons
>> for fixed length IPv6 addresses was that even though OSI NSAPs could
>> be variable length, people opted for the simplicity of using the same
>> OSI NSAP length anyway. I think most humans tend towards simplicity if
>> they can (a.k.a. laziness).
>
> I recall it as one of those big debate topics at the time in the IETF. The outcome
> was the argument that every variable system has a maximum length they have to cope with,
> and every host has to cope with a maximal length address. So all variable length addresses
> save are just bits on the wire, not bits in the hosts. Given that there was no great
> desire to pursue header compression at the time then the outcome in IPver design
> was to eschew variable length in 6 and go big.
>
> (and then we got sucked in by what turned out to be a specious 8+8 approach
> which never worked, so we chopped off the lower 64 bits anyway! Sod’s Law
> I suppose, that now that IPv6 carries 128 bits everywhere but only 64 of them
> are significant.)
>

I think what we've been able to do with those lower 64 bits will end
up being quite valuable, even if the origins of them didn't work e.g.,
privacy addresses.

>>
>>> There is a story here I was told by Jon about the involvement of a gentleman from
>>> Digital in the who was adamant that Digital’s equipment could not process
>>> variable length addresses at wire speed and he pushed hard for fixed
>>> boundaries in the packet. The compromise was 32 bits fixed size addresses.
>>>
>>> (The same argument resurfaced in the IPv6 chained extensions header structure.
>>> What goes around come around!)
>>>
>>
>> (following for the general AusNOG audience, I'm sure I don't need to
>> tell you how to suck eggs :-) )
>>

<snip>

>> Only one of the IPv6 EH's is intended to be processed in the network -
>> the Hop-by-Hop EH - and that is why it is required to be directly
>> after the IPv6 header. The rest are to be processed by the end or host
>> destination of the packet, which means that other than the HBH EH, all
>> the other EHs are "end" fields rather than "network" fields. EH
>> processing in the network at high speed may be costly, hard and/or
>> impossible. Even HBHs are in some cases being punted to software
>> processing in the control plane of routers, and that can make the
>> control plane vulnerable to a denial of service attack, so in some
>> cases packets with HBHs are being intentionally dropped rather than
>> punted to software by some networks.
>
>
> That’s all good computer science no doubt. These days all network gear
> looks deeply into every packet and pulls out whatever hints about flow state
> it can. And because its a chained structure their introspection costs cycles.
>
> https://www.nanog.org/sites/default/files/monday_general_freedman_flow.pdf
>
> So while the protocol designers _thought_ that the network switch would
> never peek into end-to-end signalling information, they were just plain wrong.
> Today’s networks glom up data however and wherever they can, and they perform
> packet introspection well down into TCP and UDP protocol headers, and at times
> further into the application protocol.
>

I think this is an example of where choosing how and where to solve
the problem can make it easier and probably cheaper or harder and
probably more expensive to solve.

Using Netflow/Sflow collecting as an example, it seem obvious to push
that collection function towards the centre of the network, because
devices towards the centre of the network will have greater visibility
to traffic sources and destinations, as traffic is more aggregated
towards the centre of the network. You'll also have less Netflow/Sflow
sources (i.e., routers/switches) to receive collection data from and
to aggregate data together from. Sounds all good and simple.

However, now that you're dealing with much higher packet per second
rates, you're going to need faster and more expensive hardware to
perform that collection. If your packet per second rates go up over
time as traffic increases over time, you'll either need to buy even
more expensive hardware to perform collection, or start sacrificing
collection accuracy by lowering sampling rates. This is an example of
trying to solve the problem using vertical scaling, and as the
"problem" gets bigger as the traffic volume becomes bigger, it
constantly becomes harder and more expensive to solve. You'll also
incur increased collector costs because you have to buy bigger ones of
those too as the volume of collection data increases, because you
can't spread the collection data across a number of collectors.

I think most people, once their ability to scale vertically runs out,
due to it either becoming too expensive, or literally impossible
because the technology doesn't exist (can you do useful Netflow/Sflow
at 100Gbps rates yet?), will revert to thinking about how to solve the
problem by horizontally scaling. In the Netflow/Sflow case that would
mean pushing the collection out towards the edge of the network and
having more collectors.

The thing I've realised is that if it is inevitable to have to
overcome vertical scaling limitations by resorting to horizontal
scaling, then it is probably better to start with a horizontal scaling
approach in the first place.

I think there are a number of advantages of starting out with a
horizontal scaling approach. Because you're dividing up the problem
into lots of smaller problems, then they individually are usually
easier and cheaper to solve. The equipment you perform the function
with tends to be more commodity, and therefore cheaper individually
because they're less specialised and more easily available (e.g., on
the shelf at your distributor, and in your hands tomorrow, rather than
needing to be manufactured on demand and in your hands in 3 months
time). You do have increased numbers of points of failure, however the
consequence of failure is reduced. As the "units of capacity" are
smaller, it becomes easier to more accurately target capacity upgrades
to where they're needed, without inadvertently providing more capacity
where it isn't needed.

One argument used against this is that the number of devices to manage
is increased. That is already a solved problem - we use software to
manage our devices rather than manually logging into them and managing
them individually (and as somebody who started out looking after
desktop PCs, it is a much easier problem to solve with networking gear
- they're not usually hardware from a different supplier than the
operating system running applications from yet other parties).

I think the main question to answer with horizontal scaling is what is
the size of the units you're dividing the problem up into. It depends
a lot on the problem, however I think there are two things to
consider. Firstly, what is the capacity of common, commodity and
off-the-shelf devices that could be used to solve the problem?
Secondly, what sort of impact is tolerable in the event of failure, as
your unit of expansion, when horizontally scaling is also likely to be
your unit of failure?



>
>
>>
>>> At the _IP layer_ who needs source? I’ve forgotten who pushed for the
>>> source to be added to the IP packet, if I ever knew, but in IPv4 the model
>>> was that the IP information state was forward, and nothing was meant to head
>>> backward, so the source address was unnecessary.
>>>
>>> (You could argue that IPv6 PTB treatment is a basic violation of this
>>> ‘forward flow’ principle, and you could well be correct!)
>>>
>>
>> I think for pure destination based delivery, certainly you're right.
>> Having source addresses in packets certainly helps with
>> troubleshooting though!
>>
>> It is a while since I looked into it, however it is my understanding
>> that the IEEE dropped carrying of source addresses in 802.11 frames
>> (i.e., IIRC, those under the 802.3 frames) because technically they're
>> not needed, as all traffic is (or was) either between a station and an
>> access point, so each receiving device inherently knows what device
>> sent the frame, because there is no ambiguity - it is a point-to-point
>> link. However, since then the IEEE have had to define a "4 address
>> format" mechanism, which puts source addresses back in frames, so that
>> things such as wifi bridges could be built. Dropping the source
>> addresses may have saved half a dozen bytes in a frame, but it has now
>> created compatibility issues.
>>
>> So it is probably better to always include a source address in frames
>> or packets regardless for troubleshooting and possible other
>> unforeseeable future reasons.
>
>
> heh heh
>
> protocol design is full of compromises when you are dealing with intangibles
> to compromise on. As you say source addresses in IP “helps”. It sure “helps”
> with NATs, and onces it was out into the header, it was used in ways nobody ever
> imagined at the time. Today’s Internet crucially depends on having source IP
> addresses in the IP header, even though if you jumped back into the time machine
> and popped out in the early 1980s, you’d be very hard pressed to justify why
> the additional 32 bits of IP header was ‘useful’ at the IP level!
>

Hmm, so if we didn't have source addresses in packets, we probably
wouldn't have NATs. I'm starting to come around :-)

>
> (I’m happy to continue this conversation on the list, but if you find it off topic
> tpo ausnog just send me (or Mark) personal mail and we’ll continue it privately!)
>
>

Regards,
Mark.


> cheers,
>
>    Geoff
>
>