[AusNOG] How are you handling metadata?

[Joseph Goldman]

I'm a bit confused too, the topic appears to be around Data Retention 
Metadata correct? In which case, I believe it has been known and 
verified by comments and legalese translations that Netflow style 
information is not required for DR purposes. The data that should be 
held should mostly come from RADIUS packets and such, these kinds of 
storage requirements only go up with subscriber numbers and not link 
utilisation (to an extent)

On 27/01/16 10:29, Greg Markey wrote:
>
> We are already doing this at scale for internal use only; albeit 
> likely to be smaller than what ISPs are deploying in the field. This 
> isn’t me trying to make other people do my homework for me; the 
> content of the talk is what Optiver is **already**//doing.
>
> I think a good topic of discussion for the user group would be to talk 
> about options other than what we have implemented, and how ISPs are 
> facing the challenge of capturing this metadata. For example in 
> Elastic-land, how many index nodes would be required to handle 
> indexing of packet metadata for a 1Gbps link? How much bandwidth do we 
> need to set aside for these metadata messages? Do we have enough spare 
> cycles to compress the data?
>
> In order to capture more information than is available from NetFlow, 
> we use the following stack:
>
> ·pmacct
>
> ·Gollum
>
> ·Kafka
>
> ·ElasticSearch
>
> As part of the meetup I’m hoping to commit the glue code into Github 
> for people to experiment with.
>
> Cheers,
>
> Greg
>
> *From:*Robert Hudson [mailto:hudrob at gmail.com]
> *Sent:* Wednesday, 27 January 2016 9:56 AM
> *To:* Greg Markey
> *Cc:* Geordie Guy; ausnog at ausnog.net
> *Subject:* Re: [AusNOG] How are you handling metadata?
>
> You're offering carrier-style services to other businesses over these 
> WAN links?
>
> On 27 Jan 2016 9:49 am, "Greg Markey" <Greg.Markey at optiver.com.au 
> <mailto:Greg.Markey at optiver.com.au>> wrote:
>
> Yes, we are; we have a use case for capturing metadata from our WAN 
> taps between regions however I would imagine ISPs are doing it on a 
> much larger scale.
>
> I’ll share the slides with the group (once I’ve actually written them :) )
>
> *From:*Geordie Guy [mailto:elomis at gmail.com <mailto:elomis at gmail.com>]
> *Sent:* Wednesday, 27 January 2016 8:26 AM
> *To:* Greg Markey
> *Cc:* ausnog at ausnog.net <mailto:ausnog at ausnog.net>
> *Subject:* Re: [AusNOG] How are you handling metadata?
>
> Isn't Optiver a financial services business?
>
> G
>
> On Tue, Jan 26, 2016 at 10:33 AM, Greg Markey 
> <Greg.Markey at optiver.com.au <mailto:Greg.Markey at optiver.com.au>> wrote:
>
> Hello everyone,
>
> I'm reaching out to see if anyone on the list is willing to share some 
> high level details around how they have implemented the capture, 
> processing and storage for the metadata retention scheme. I noticed 
> that AGD is unable to provide specific recommendations to ISPs for 
> hardware and software, leading me to believe that the technical 
> implementations are going to potentially vary significantly between 
> organisations.
>
> I'll be talking about what we've built internally at the Sydney 
> ElasticSearch users group on Thursday, but it would be great to have 
> some comparisons if you don't mind me sharing your solutions 
> (anonymously).
>
> Cheers,
> Greg
>
> Information contained in this communication (including any 
> attachments) is confidential and may be privileged or subject to 
> copyright.  If you have received this communication in error you are 
> not authorised to use the information in any way and Optiver requests 
> that you notify the sender by return email, destroy all copies and 
> delete the information from your system.  Optiver does not represent, 
> warrant or guarantee that this communication is free from computer 
> viruses or other defects or that the integrity of this communication 
> has been maintained.  Any views expressed in this communication are 
> those of the individual sender. Optiver does not accept liability for 
> any loss or damage caused directly or indirectly by this communication 
> or its use.
>
> Please consider the environment before printing this email.
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160127/903f3bfb/attachment.html>