[AusNOG] How are you handling metadata?

[Greg Markey]

We are already doing this at scale for internal use only; albeit likely to be smaller than what ISPs are deploying in the field. This isn’t me trying to make other people do my homework for me; the content of the talk is what Optiver is *already* doing.

I think a good topic of discussion for the user group would be to talk about options other than what we have implemented, and how ISPs are facing the challenge of capturing this metadata. For example in Elastic-land, how many index nodes would be required to handle indexing of packet metadata for a 1Gbps link? How much bandwidth do we need to set aside for these metadata messages? Do we have enough spare cycles to compress the data?

In order to capture more information than is available from NetFlow, we use the following stack:

·         pmacct

·         Gollum

·         Kafka

·         ElasticSearch

As part of the meetup I’m hoping to commit the glue code into Github for people to experiment with.

Cheers,
Greg

From: Robert Hudson [mailto:hudrob at gmail.com]
Sent: Wednesday, 27 January 2016 9:56 AM
To: Greg Markey
Cc: Geordie Guy; ausnog at ausnog.net
Subject: Re: [AusNOG] How are you handling metadata?


You're offering carrier-style services to other businesses over these WAN links?
On 27 Jan 2016 9:49 am, "Greg Markey" <Greg.Markey at optiver.com.au<mailto:Greg.Markey at optiver.com.au>> wrote:
Yes, we are; we have a use case for capturing metadata from our WAN taps between regions however I would imagine ISPs are doing it on a much larger scale.

I’ll share the slides with the group (once I’ve actually written them :) )

From: Geordie Guy [mailto:elomis at gmail.com<mailto:elomis at gmail.com>]
Sent: Wednesday, 27 January 2016 8:26 AM
To: Greg Markey
Cc: ausnog at ausnog.net<mailto:ausnog at ausnog.net>
Subject: Re: [AusNOG] How are you handling metadata?

Isn't Optiver a financial services business?

G

On Tue, Jan 26, 2016 at 10:33 AM, Greg Markey <Greg.Markey at optiver.com.au<mailto:Greg.Markey at optiver.com.au>> wrote:
Hello everyone,

I'm reaching out to see if anyone on the list is willing to share some high level details around how they have implemented the capture, processing and storage for the metadata retention scheme. I noticed that AGD is unable to provide specific recommendations to ISPs for hardware and software, leading me to believe that the technical implementations are going to potentially vary significantly between organisations.

I'll be talking about what we've built internally at the Sydney ElasticSearch users group on Thursday, but it would be great to have some comparisons if you don't mind me sharing your solutions (anonymously).

Cheers,
Greg

Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright.  If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system.  Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defects or that the integrity of this communication has been maintained.  Any views expressed in this communication are those of the individual sender.  Optiver does not accept liability for any loss or damage caused directly or indirectly by this communication or its use.

Please consider the environment before printing this email.
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog


_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160127/aed366bf/attachment.html>