[AusNOG] How are you handling metadata?

[Andrew Cox]

Using RADIUS alone for data retention requirements only really works if
you've got a "one-customer-per-public-ip" setup though.

While I imagine that covers the vast majority of the services out there,
there's a number of circumstances where this doesn't work, in which case
NetFlow (which must include src ip and src port translations) will.

A non-exhaustive list of examples:
Mining camp resident networks
Shared office space networks
University student access networks

Basically anywhere the end users are on a private subnet and aren't covered
by "local area wireless hotspot" DR exemptions.

- Andrew

On Wed, Jan 27, 2016 at 9:37 AM, Joseph Goldman <joe at apcs.com.au> wrote:

> I'm a bit confused too, the topic appears to be around Data Retention
> Metadata correct? In which case, I believe it has been known and verified
> by comments and legalese translations that Netflow style information is not
> required for DR purposes. The data that should be held should mostly come
> from RADIUS packets and such, these kinds of storage requirements only go
> up with subscriber numbers and not link utilisation (to an extent)
>
>
> On 27/01/16 10:29, Greg Markey wrote:
>
> We are already doing this at scale for internal use only; albeit likely to
> be smaller than what ISPs are deploying in the field. This isn’t me trying
> to make other people do my homework for me; the content of the talk is what
> Optiver is **already** doing.
>
>
>
> I think a good topic of discussion for the user group would be to talk
> about options other than what we have implemented, and how ISPs are facing
> the challenge of capturing this metadata. For example in Elastic-land, how
> many index nodes would be required to handle indexing of packet metadata
> for a 1Gbps link? How much bandwidth do we need to set aside for these
> metadata messages? Do we have enough spare cycles to compress the data?
>
>
>
> In order to capture more information than is available from NetFlow, we
> use the following stack:
>
> ·         pmacct
>
> ·         Gollum
>
> ·         Kafka
>
> ·         ElasticSearch
>
>
>
> As part of the meetup I’m hoping to commit the glue code into Github for
> people to experiment with.
>
>
>
> Cheers,
>
> Greg
>
>
>
> *From:* Robert Hudson [mailto:hudrob at gmail.com <hudrob at gmail.com>]
> *Sent:* Wednesday, 27 January 2016 9:56 AM
> *To:* Greg Markey
> *Cc:* Geordie Guy; ausnog at ausnog.net
> *Subject:* Re: [AusNOG] How are you handling metadata?
>
>
>
> You're offering carrier-style services to other businesses over these WAN
> links?
>
> On 27 Jan 2016 9:49 am, "Greg Markey" <Greg.Markey at optiver.com.au> wrote:
>
> Yes, we are; we have a use case for capturing metadata from our WAN taps
> between regions however I would imagine ISPs are doing it on a much larger
> scale.
>
>
>
> I’ll share the slides with the group (once I’ve actually written them :) )
>
>
>
> *From:* Geordie Guy [mailto: <elomis at gmail.com>elomis at gmail.com]
> *Sent:* Wednesday, 27 January 2016 8:26 AM
> *To:* Greg Markey
> *Cc:* ausnog at ausnog.net
> *Subject:* Re: [AusNOG] How are you handling metadata?
>
>
>
> Isn't Optiver a financial services business?
>
>
>
> G
>
>
>
> On Tue, Jan 26, 2016 at 10:33 AM, Greg Markey <
> <Greg.Markey at optiver.com.au>Greg.Markey at optiver.com.au> wrote:
>
> Hello everyone,
>
> I'm reaching out to see if anyone on the list is willing to share some
> high level details around how they have implemented the capture, processing
> and storage for the metadata retention scheme. I noticed that AGD is unable
> to provide specific recommendations to ISPs for hardware and software,
> leading me to believe that the technical implementations are going to
> potentially vary significantly between organisations.
>
> I'll be talking about what we've built internally at the Sydney
> ElasticSearch users group on Thursday, but it would be great to have some
> comparisons if you don't mind me sharing your solutions (anonymously).
>
> Cheers,
> Greg
>
> Information contained in this communication (including any attachments) is
> confidential and may be privileged or subject to copyright.  If you have
> received this communication in error you are not authorised to use the
> information in any way and Optiver requests that you notify the sender by
> return email, destroy all copies and delete the information from your
> system.  Optiver does not represent, warrant or guarantee that this
> communication is free from computer viruses or other defects or that the
> integrity of this communication has been maintained.  Any views expressed
> in this communication are those of the individual sender.  Optiver does not
> accept liability for any loss or damage caused directly or indirectly by
> this communication or its use.
>
> Please consider the environment before printing this email.
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
> _______________________________________________
> AusNOG mailing listAusNOG at lists.ausnog.nethttp://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160127/ce610e3e/attachment.html>