[AusNOG] So who's read an RFC or Internet Draft?

[Philip Loenneker]

Perhaps I’m missing something in your explanation, but we can put in additional collectors at will by essentially cloning the VM and adding them to the pool – isn’t that horizontal scaling? The netflow traffic is distributed across the pool, not replicated to each one, so adding extra servers reduces the load on the existing servers. The primary goal of this design was more to reduce the risk of a single failure breaking everything rather than to enable horizontal scaling, but I think it achieves both.

On 12 Oct 2015 9:19 am, "Philip Loenneker" <Philip.Loenneker at tasmanet.com.au<mailto:Philip.Loenneker at tasmanet.com.au>> wrote:
>
> <snip>
> You'll also incur increased collector costs because you have to buy bigger ones of those too as the volume of collection data increases, because you can't spread the collection data across a number of collectors.
> </snip>
>
> You can spread the load, you just have to be a little creative :)
>

You've only spread the load across a number of collectors, and you've done that by adding another element for which the only way to scale is to buy a bigger one, so you're still pursuing vertical scaling. You've also added a another point of failure to the whole system, because it is another element inline (in serial) with the netflow traffic path.

A simple test to work out if you're vertically or horizontally scaling is can you buy another device of the *same* size/capacity and it adds processing capacity to the system? If it does, then your scaling model is horizontal.

> A few weeks ago we successfully got a proof-of-concept up and running where we have NetFlow feeds going to a load balancer (NetScaler in our case), which then used source-IP persistence to spread the load over 4 collectors. The collectors run as stateless UDP servers of course, which makes it difficult to check whether the services are running, so one of our guys set up an additional service running on TCP that would give a response code with the status of the collector, so we have health-checks and auto-failover working. The collectors we are using are nProbe on Linux, which allows us to dump the raw flows directly into MySQL, so we load balanced that too and set up a cluster of MySQL servers as the destination. Tests have been quite promising, and we get redundancy in a system that wasn't designed to. All of those servers are virtual, which makes scaling even simpler.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20151012/3057f191/attachment.html>