[AusNOG] Best Open-Source Flow analyzer tools

[Mark Smith]

On Tue, 14 Dec 2010 09:53:50 +1030
John Edwards <john at netniche.com.au> wrote:

> On 13/12/2010, at 11:30 PM, Sean K. Finn wrote:
> 
> > Hence why I'm looking for an open-source alternative that I can rip the guts out of.
> 
> I've had good results before ripping the guts out of this netflow collector:
> 
> http://iagu.net/software/netflow-collector.html
> 
> It's written in Perl, but since it compiles once and then runs in a loop the performance is ok - in my application I added in per-IP-address in-memory 64-bit counters for a variety of subnet and ToS matching rules, and deprecated the logging/writing features to take disk out of the equation. On a pentium-4 class machine, it could handle the throughput of 2x Cisco 7301 broadband aggregation routers.
> 
> I added in a special type of netflow packet to make it answer queries on the counters. We later gave it the ability to dump its in-memory counters to a file and read them back in on restart, so that the software could be upgraded or reconfigured for new subnets with minimal data loss. 
> 
> This was around 2003, in response to the lack of accounting options available for L3 Bridged DSL, and long-running L3 PPP sessions through the same network. I understand that the same system is still running today supporting tens of thousands of broadband users.
> 

It was updated it a few years ago to use the following perl module to
speed up IP address lookups, which knocked more than 60% CPU off of it.
At the time it wasn't and hadn't been coping for quite a while with the
Netflow traffic load that a single 7301 could generate during peak
periods, with the netflow collector host's CPU flatlining at 100% - a
significant concern when you're using netflow for customer traffic
accounting. Changing to an IP specific lookup routine also allowed for
disk logging to be switched back on, which alerted to netflow packets
that had been lost due to non-contiguous netflow packet sequence
identifiers.

After it was moved to a virtualised Linux host, with a faster CPU, it
would have been possible to have 6 7301s sending their netflow traffic
to it - however that would have been too much accounting information to
lose if that host went down for any length of time.

http://search.cpan.org/~philipp/Net-Patricia-1.19/Patricia.pm


> John
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog