[AusNOG] So who's read an RFC or Internet Draft?

[Philip Loenneker]

<snip>
You'll also incur increased collector costs because you have to buy bigger ones of those too as the volume of collection data increases, because you can't spread the collection data across a number of collectors.
</snip>

You can spread the load, you just have to be a little creative :)

A few weeks ago we successfully got a proof-of-concept up and running where we have NetFlow feeds going to a load balancer (NetScaler in our case), which then used source-IP persistence to spread the load over 4 collectors. The collectors run as stateless UDP servers of course, which makes it difficult to check whether the services are running, so one of our guys set up an additional service running on TCP that would give a response code with the status of the collector, so we have health-checks and auto-failover working. The collectors we are using are nProbe on Linux, which allows us to dump the raw flows directly into MySQL, so we load balanced that too and set up a cluster of MySQL servers as the destination. Tests have been quite promising, and we get redundancy in a system that wasn't designed to. All of those servers are virtual, which makes scaling even simpler.