<div dir="ltr">Using RADIUS alone for data retention requirements only really works if you've got a "one-customer-per-public-ip" setup though.<br><br>While I imagine that covers the vast majority of the services out there, there's a number of circumstances where this doesn't work, in which case NetFlow (which must include src ip and src port translations) will.<div><br></div><div><div>A non-exhaustive list of examples:</div><div><div>Mining camp resident networks<br></div><div>Shared office space networks</div><div>University student access networks</div><div><br></div><div>Basically anywhere the end users are on a private subnet and aren't covered by "local area wireless hotspot" DR exemptions. </div></div><div><div><div><br><div>- Andrew</div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 27, 2016 at 9:37 AM, Joseph Goldman <span dir="ltr"><<a href="mailto:joe@apcs.com.au" target="_blank">joe@apcs.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

  
  <div bgcolor="#FFFFFF" text="#000000">
    I'm a bit confused too, the topic appears to be around Data
    Retention Metadata correct? In which case, I believe it has been
    known and verified by comments and legalese translations that
    Netflow style information is not required for DR purposes. The data
    that should be held should mostly come from RADIUS packets and such,
    these kinds of storage requirements only go up with subscriber
    numbers and not link utilisation (to an extent)<div><div class="h5"><br>
    <br>
    <div>On 27/01/16 10:29, Greg Markey wrote:<br>
    </div>
    <blockquote type="cite">
      
      
      
      <div>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We
            are already doing this at scale for internal use only;
            albeit likely to be smaller than what ISPs are deploying in
            the field. This isn’t me trying to make other people do my
            homework for me; the content of the talk is what Optiver is
            *<b>already</b>*<i> </i>doing.<u></u><u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I
            think a good topic of discussion for the user group would be
            to talk about options other than what we have implemented,
            and how ISPs are facing the challenge of capturing this
            metadata. For example in Elastic-land, how many index nodes
            would be required to handle indexing of packet metadata for
            a 1Gbps link? How much bandwidth do we need to set aside for
            these metadata messages? Do we have enough spare cycles to
            compress the data?<u></u><u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">In
            order to capture more information than is available from
            NetFlow, we use the following stack:<u></u><u></u></span></p>
        <p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">         </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">pmacct<u></u><u></u></span></p>
        <p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">         </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Gollum<u></u><u></u></span></p>
        <p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">         </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Kafka<u></u><u></u></span></p>
        <p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">         </span></span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ElasticSearch<u></u><u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As
            part of the meetup I’m hoping to commit the glue code into
            Github for people to experiment with.<u></u><u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Cheers,<u></u><u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Greg<u></u><u></u></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
        <p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> Robert Hudson [<a href="mailto:hudrob@gmail.com" target="_blank">mailto:hudrob@gmail.com</a>] <br>
            <b>Sent:</b> Wednesday, 27 January 2016 9:56 AM<br>
            <b>To:</b> Greg Markey<br>
            <b>Cc:</b> Geordie Guy; <a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a><br>
            <b>Subject:</b> Re: [AusNOG] How are you handling metadata?<u></u><u></u></span></p>
        <p class="MsoNormal"><u></u> <u></u></p>
        <p>You're offering carrier-style services to other businesses
          over these WAN links?<u></u><u></u></p>
        <div>
          <p class="MsoNormal">On 27 Jan 2016 9:49 am, "Greg Markey"
            <<a href="mailto:Greg.Markey@optiver.com.au" target="_blank">Greg.Markey@optiver.com.au</a>>
            wrote:<u></u><u></u></p>
          <div>
            <div>
              <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes,
                  we are; we have a use case for capturing metadata from
                  our WAN taps between regions however I would imagine
                  ISPs are doing it on a much larger scale.</span><u></u><u></u></p>
              <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
              <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’ll
                  share the slides with the group (once I’ve actually
                  written them :) )</span><u></u><u></u></p>
              <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
              <p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> Geordie Guy [mailto:<a href="mailto:elomis@gmail.com" target="_blank"></a><a href="mailto:elomis@gmail.com" target="_blank">elomis@gmail.com</a>]
                  <br>
                  <b>Sent:</b> Wednesday, 27 January 2016 8:26 AM<br>
                  <b>To:</b> Greg Markey<br>
                  <b>Cc:</b> <a href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a><br>
                  <b>Subject:</b> Re: [AusNOG] How are you handling
                  metadata?</span><u></u><u></u></p>
              <p class="MsoNormal"> <u></u><u></u></p>
              <div>
                <p class="MsoNormal">Isn't
                  Optiver a financial services business?<u></u><u></u></p>
                <div>
                  <p class="MsoNormal"> <u></u><u></u></p>
                </div>
                <div>
                  <p class="MsoNormal">G<u></u><u></u></p>
                </div>
              </div>
              <div>
                <p class="MsoNormal"> <u></u><u></u></p>
                <div>
                  <p class="MsoNormal">On
                    Tue, Jan 26, 2016 at 10:33 AM, Greg Markey <<a href="mailto:Greg.Markey@optiver.com.au" target="_blank"></a><a href="mailto:Greg.Markey@optiver.com.au" target="_blank">Greg.Markey@optiver.com.au</a>>
                    wrote:<u></u><u></u></p>
                  <p class="MsoNormal">Hello
                    everyone,<br>
                    <br>
                    I'm reaching out to see if anyone on the list is
                    willing to share some high level details around how
                    they have implemented the capture, processing and
                    storage for the metadata retention scheme. I noticed
                    that AGD is unable to provide specific
                    recommendations to ISPs for hardware and software,
                    leading me to believe that the technical
                    implementations are going to potentially vary
                    significantly between organisations.<br>
                    <br>
                    I'll be talking about what we've built internally at
                    the Sydney ElasticSearch users group on Thursday,
                    but it would be great to have some comparisons if
                    you don't mind me sharing your solutions
                    (anonymously).<br>
                    <br>
                    Cheers,<br>
                    Greg<br>
                    <br>
                    Information contained in this communication
                    (including any attachments) is confidential and may
                    be privileged or subject to copyright.  If you have
                    received this communication in error you are not
                    authorised to use the information in any way and
                    Optiver requests that you notify the sender by
                    return email, destroy all copies and delete the
                    information from your system.  Optiver does not
                    represent, warrant or guarantee that this
                    communication is free from computer viruses or other
                    defects or that the integrity of this communication
                    has been maintained.  Any views expressed in this
                    communication are those of the individual sender. 
                    Optiver does not accept liability for any loss or
                    damage caused directly or indirectly by this
                    communication or its use.<br>
                    <br>
                    Please consider the environment before printing this
                    email.<br>
                    _______________________________________________<br>
                    AusNOG mailing list<br>
                    <a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
                    <a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><u></u><u></u></p>
                </div>
                <p class="MsoNormal"> <u></u><u></u></p>
              </div>
            </div>
          </div>
          <p class="MsoNormal" style="margin-bottom:12.0pt"><br>
            _______________________________________________<br>
            AusNOG mailing list<br>
            <a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
            <a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><u></u><u></u></p>
        </div>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      <pre>_______________________________________________
AusNOG mailing list
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
    </blockquote>
    <br>
  </div></div></div>

<br>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br></blockquote></div><br></div>