[AusNOG] How are you handling metadata?

Wed Jan 27 14:44:43 EST 2016

RFC 7422 suggests assigning a range of source ports per-user in a CGNAT
environment, specifically to reduce logging requirements.

Cisco have implemented this in ISG with a feature called "Port Bundle Host
Key" (PBHK), which includes a vendor-specific RADIUS attribute for logging
which "bundle" of ports the user used.

A nifty side effect of this strategy is that it can also be used to
identify/authorise users "by IP address" when the web server is on the
outside of the NAT gateway.

John

On 27 January 2016 at 14:00, Andrew Cox <andrew.cox at myport.com.au> wrote:

> Using RADIUS alone for data retention requirements only really works if
> you've got a "one-customer-per-public-ip" setup though.
>
> While I imagine that covers the vast majority of the services out there,
> there's a number of circumstances where this doesn't work, in which case
> NetFlow (which must include src ip and src port translations) will.
>
> A non-exhaustive list of examples:
> Mining camp resident networks
> Shared office space networks
> University student access networks
>
> Basically anywhere the end users are on a private subnet and aren't
> covered by "local area wireless hotspot" DR exemptions.
>
> - Andrew
>
> On Wed, Jan 27, 2016 at 9:37 AM, Joseph Goldman <joe at apcs.com.au> wrote:
>
>> I'm a bit confused too, the topic appears to be around Data Retention
>> Metadata correct? In which case, I believe it has been known and verified
>> by comments and legalese translations that Netflow style information is not
>> required for DR purposes. The data that should be held should mostly come
>> from RADIUS packets and such, these kinds of storage requirements only go
>> up with subscriber numbers and not link utilisation (to an extent)
>>
>>
>> On 27/01/16 10:29, Greg Markey wrote:
>>
>> We are already doing this at scale for internal use only; albeit likely
>> to be smaller than what ISPs are deploying in the field. This isn’t me
>> trying to make other people do my homework for me; the content of the talk
>> is what Optiver is **already** doing.
>>
>>
>>
>> I think a good topic of discussion for the user group would be to talk
>> about options other than what we have implemented, and how ISPs are facing
>> the challenge of capturing this metadata. For example in Elastic-land, how
>> many index nodes would be required to handle indexing of packet metadata
>> for a 1Gbps link? How much bandwidth do we need to set aside for these
>> metadata messages? Do we have enough spare cycles to compress the data?
>>
>>
>>
>> In order to capture more information than is available from NetFlow, we
>> use the following stack:
>>
>> ·         pmacct
>>
>> ·         Gollum
>>
>> ·         Kafka
>>
>> ·         ElasticSearch
>>
>>
>>
>> As part of the meetup I’m hoping to commit the glue code into Github for
>> people to experiment with.
>>
>>
>>
>> Cheers,
>>
>> Greg
>>
>>
>>
>> *From:* Robert Hudson [mailto:hudrob at gmail.com <hudrob at gmail.com>]
>> *Sent:* Wednesday, 27 January 2016 9:56 AM
>> *To:* Greg Markey
>> *Cc:* Geordie Guy; ausnog at ausnog.net
>> *Subject:* Re: [AusNOG] How are you handling metadata?
>>
>>
>>
>> You're offering carrier-style services to other businesses over these WAN
>> links?
>>
>> On 27 Jan 2016 9:49 am, "Greg Markey" <Greg.Markey at optiver.com.au> wrote:
>>
>> Yes, we are; we have a use case for capturing metadata from our WAN taps
>> between regions however I would imagine ISPs are doing it on a much larger
>> scale.
>>
>>
>>
>> I’ll share the slides with the group (once I’ve actually written them :) )
>>
>>
>>
>> *From:* Geordie Guy [mailto: <elomis at gmail.com>elomis at gmail.com]
>> *Sent:* Wednesday, 27 January 2016 8:26 AM
>> *To:* Greg Markey
>> *Cc:* ausnog at ausnog.net
>> *Subject:* Re: [AusNOG] How are you handling metadata?
>>
>>
>>
>> Isn't Optiver a financial services business?
>>
>>
>>
>> G
>>
>>
>>
>> On Tue, Jan 26, 2016 at 10:33 AM, Greg Markey <
>> <Greg.Markey at optiver.com.au>Greg.Markey at optiver.com.au> wrote:
>>
>> Hello everyone,
>>
>> I'm reaching out to see if anyone on the list is willing to share some
>> high level details around how they have implemented the capture, processing
>> and storage for the metadata retention scheme. I noticed that AGD is unable
>> to provide specific recommendations to ISPs for hardware and software,
>> leading me to believe that the technical implementations are going to
>> potentially vary significantly between organisations.
>>
>> I'll be talking about what we've built internally at the Sydney
>> ElasticSearch users group on Thursday, but it would be great to have some
>> comparisons if you don't mind me sharing your solutions (anonymously).
>>
>> Cheers,
>> Greg
>>
>> Information contained in this communication (including any attachments)
>> is confidential and may be privileged or subject to copyright.  If you have
>> received this communication in error you are not authorised to use the
>> information in any way and Optiver requests that you notify the sender by
>> return email, destroy all copies and delete the information from your
>> system.  Optiver does not represent, warrant or guarantee that this
>> communication is free from computer viruses or other defects or that the
>> integrity of this communication has been maintained.  Any views expressed
>> in this communication are those of the individual sender.  Optiver does not
>> accept liability for any loss or damage caused directly or indirectly by
>> this communication or its use.
>>
>> Please consider the environment before printing this email.
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>> _______________________________________________
>> AusNOG mailing listAusNOG at lists.ausnog.nethttp://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160127/4e8f4863/attachment.html>