<div dir="ltr">Using RADIUS alone for data retention requirements only really works if you've got a "one-customer-per-public-ip" setup though.<br><br>While I imagine that covers the vast majority of the services out there, there's a number of circumstances where this doesn't work, in which case NetFlow (which must include src ip and src port translations) will.<div><br></div><div><div>A non-exhaustive list of examples:</div><div><div>Mining camp resident networks<br></div><div>Shared office space networks</div><div>University student access networks</div><div><br></div><div>Basically anywhere the end users are on a private subnet and aren't covered by "local area wireless hotspot" DR exemptions. </div></div><div><div><div><br><div>- Andrew</div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 27, 2016 at 9:37 AM, Joseph Goldman <span dir="ltr"><<a href="mailto:joe@apcs.com.au" target="_blank">joe@apcs.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
I'm a bit confused too, the topic appears to be around Data
Retention Metadata correct? In which case, I believe it has been
known and verified by comments and legalese translations that
Netflow style information is not required for DR purposes. The data
that should be held should mostly come from RADIUS packets and such,
these kinds of storage requirements only go up with subscriber
numbers and not link utilisation (to an extent)<div><div class="h5"><br>
<br>
<div>On 27/01/16 10:29, Greg Markey wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We
are already doing this at scale for internal use only;
albeit likely to be smaller than what ISPs are deploying in
the field. This isn’t me trying to make other people do my
homework for me; the content of the talk is what Optiver is
*<b>already</b>*<i> </i>doing.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I
think a good topic of discussion for the user group would be
to talk about options other than what we have implemented,
and how ISPs are facing the challenge of capturing this
metadata. For example in Elastic-land, how many index nodes
would be required to handle indexing of packet metadata for
a 1Gbps link? How much bandwidth do we need to set aside for
these metadata messages? Do we have enough spare cycles to
compress the data?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">In
order to capture more information than is available from
NetFlow, we use the following stack:<u></u><u></u></span></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman""> </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">pmacct<u></u><u></u></span></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman""> </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Gollum<u></u><u></u></span></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman""> </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Kafka<u></u><u></u></span></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman""> </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ElasticSearch<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As
part of the meetup I’m hoping to commit the glue code into
Github for people to experiment with.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Cheers,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Greg<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> Robert Hudson [<a href="mailto:hudrob@gmail.com" target="_blank">mailto:hudrob@gmail.com</a>] <br>
<b>Sent:</b> Wednesday, 27 January 2016 9:56 AM<br>
<b>To:</b> Greg Markey<br>
<b>Cc:</b> Geordie Guy; <a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] How are you handling metadata?<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p>You're offering carrier-style services to other businesses
over these WAN links?<u></u><u></u></p>
<div>
<p class="MsoNormal">On 27 Jan 2016 9:49 am, "Greg Markey"
<<a href="mailto:Greg.Markey@optiver.com.au" target="_blank">Greg.Markey@optiver.com.au</a>>
wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes,
we are; we have a use case for capturing metadata from
our WAN taps between regions however I would imagine
ISPs are doing it on a much larger scale.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’ll
share the slides with the group (once I’ve actually
written them :) )</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> Geordie Guy [mailto:<a href="mailto:elomis@gmail.com" target="_blank"></a><a href="mailto:elomis@gmail.com" target="_blank">elomis@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, 27 January 2016 8:26 AM<br>
<b>To:</b> Greg Markey<br>
<b>Cc:</b> <a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] How are you handling
metadata?</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Isn't
Optiver a financial services business?<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">G<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On
Tue, Jan 26, 2016 at 10:33 AM, Greg Markey <<a href="mailto:Greg.Markey@optiver.com.au" target="_blank"></a><a href="mailto:Greg.Markey@optiver.com.au" target="_blank">Greg.Markey@optiver.com.au</a>>
wrote:<u></u><u></u></p>
<p class="MsoNormal">Hello
everyone,<br>
<br>
I'm reaching out to see if anyone on the list is
willing to share some high level details around how
they have implemented the capture, processing and
storage for the metadata retention scheme. I noticed
that AGD is unable to provide specific
recommendations to ISPs for hardware and software,
leading me to believe that the technical
implementations are going to potentially vary
significantly between organisations.<br>
<br>
I'll be talking about what we've built internally at
the Sydney ElasticSearch users group on Thursday,
but it would be great to have some comparisons if
you don't mind me sharing your solutions
(anonymously).<br>
<br>
Cheers,<br>
Greg<br>
<br>
Information contained in this communication
(including any attachments) is confidential and may
be privileged or subject to copyright. If you have
received this communication in error you are not
authorised to use the information in any way and
Optiver requests that you notify the sender by
return email, destroy all copies and delete the
information from your system. Optiver does not
represent, warrant or guarantee that this
communication is free from computer viruses or other
defects or that the integrity of this communication
has been maintained. Any views expressed in this
communication are those of the individual sender.
Optiver does not accept liability for any loss or
damage caused directly or indirectly by this
communication or its use.<br>
<br>
Please consider the environment before printing this
email.<br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><u></u><u></u></p>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
AusNOG mailing list
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br></blockquote></div><br></div>