<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    I'm a bit confused too, the topic appears to be around Data
    Retention Metadata correct? In which case, I believe it has been
    known and verified by comments and legalese translations that
    Netflow style information is not required for DR purposes. The data
    that should be held should mostly come from RADIUS packets and such,
    these kinds of storage requirements only go up with subscriber
    numbers and not link utilisation (to an extent)<br>
    <br>
    <div class="moz-cite-prefix">On 27/01/16 10:29, Greg Markey wrote:<br>
    </div>
    <blockquote
cite="mid:D39A115F09181C4C9A4C1C77B0A56260031F8AF2F579@OPSYWNSR0152.comp.optiver.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <meta name="Generator" content="Microsoft Word 14 (filtered
        medium)">
      <style><!--
/* Font Definitions */
@font-face
        {font-family:Wingdings;
        panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
        {font-family:宋体;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:宋体;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
        {font-family:"\@宋体";
        panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p
        {mso-style-priority:99;
        mso-margin-top-alt:auto;
        margin-right:0cm;
        mso-margin-bottom-alt:auto;
        margin-left:0cm;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {mso-style-priority:34;
        margin-top:0cm;
        margin-right:0cm;
        margin-bottom:0cm;
        margin-left:36.0pt;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
span.EmailStyle18
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;}
@page WordSection1
        {size:612.0pt 792.0pt;
        margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
        {page:WordSection1;}
/* List Definitions */
@list l0
        {mso-list-id:416902634;
        mso-list-type:hybrid;
        mso-list-template-ids:17053994 397036576 201916419 201916421 201916417 201916419 201916421 201916417 201916419 201916421;}
@list l0:level1
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Symbol;
        mso-fareast-font-family:宋体;
        mso-bidi-font-family:"Times New Roman";}
@list l0:level2
        {mso-level-number-format:bullet;
        mso-level-text:o;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:"Courier New";}
@list l0:level3
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Wingdings;}
@list l0:level4
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Symbol;}
@list l0:level5
        {mso-level-number-format:bullet;
        mso-level-text:o;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:"Courier New";}
@list l0:level6
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Wingdings;}
@list l0:level7
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Symbol;}
@list l0:level8
        {mso-level-number-format:bullet;
        mso-level-text:o;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:"Courier New";}
@list l0:level9
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Wingdings;}
@list l1
        {mso-list-id:1816531902;
        mso-list-type:hybrid;
        mso-list-template-ids:15993794 -1556996356 201916419 201916421 201916417 201916419 201916421 201916417 201916419 201916421;}
@list l1:level1
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Symbol;
        mso-fareast-font-family:宋体;
        mso-bidi-font-family:"Times New Roman";}
@list l1:level2
        {mso-level-number-format:bullet;
        mso-level-text:o;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:"Courier New";}
@list l1:level3
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Wingdings;}
@list l1:level4
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Symbol;}
@list l1:level5
        {mso-level-number-format:bullet;
        mso-level-text:o;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:"Courier New";}
@list l1:level6
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Wingdings;}
@list l1:level7
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Symbol;}
@list l1:level8
        {mso-level-number-format:bullet;
        mso-level-text:o;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:"Courier New";}
@list l1:level9
        {mso-level-number-format:bullet;
        mso-level-text:;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;
        font-family:Wingdings;}
ol
        {margin-bottom:0cm;}
ul
        {margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
      <div class="WordSection1">
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We
            are already doing this at scale for internal use only;
            albeit likely to be smaller than what ISPs are deploying in
            the field. This isn’t me trying to make other people do my
            homework for me; the content of the talk is what Optiver is
            *<b>already</b>*<i> </i>doing.<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I
            think a good topic of discussion for the user group would be
            to talk about options other than what we have implemented,
            and how ISPs are facing the challenge of capturing this
            metadata. For example in Elastic-land, how many index nodes
            would be required to handle indexing of packet metadata for
            a 1Gbps link? How much bandwidth do we need to set aside for
            these metadata messages? Do we have enough spare cycles to
            compress the data?<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">In
            order to capture more information than is available from
            NetFlow, we use the following stack:<o:p></o:p></span></p>
        <p class="MsoListParagraph"
          style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
            style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
              style="mso-list:Ignore">·<span style="font:7.0pt
                "Times New Roman"">         </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">pmacct<o:p></o:p></span></p>
        <p class="MsoListParagraph"
          style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
            style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
              style="mso-list:Ignore">·<span style="font:7.0pt
                "Times New Roman"">         </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Gollum<o:p></o:p></span></p>
        <p class="MsoListParagraph"
          style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
            style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
              style="mso-list:Ignore">·<span style="font:7.0pt
                "Times New Roman"">         </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Kafka<o:p></o:p></span></p>
        <p class="MsoListParagraph"
          style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
            style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
              style="mso-list:Ignore">·<span style="font:7.0pt
                "Times New Roman"">         </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ElasticSearch<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As
            part of the meetup I’m hoping to commit the glue code into
            Github for people to experiment with.<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Greg<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
              lang="EN-US">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
            lang="EN-US"> Robert Hudson [<a class="moz-txt-link-freetext" href="mailto:hudrob@gmail.com">mailto:hudrob@gmail.com</a>] <br>
            <b>Sent:</b> Wednesday, 27 January 2016 9:56 AM<br>
            <b>To:</b> Greg Markey<br>
            <b>Cc:</b> Geordie Guy; <a class="moz-txt-link-abbreviated" href="mailto:ausnog@ausnog.net">ausnog@ausnog.net</a><br>
            <b>Subject:</b> Re: [AusNOG] How are you handling metadata?<o:p></o:p></span></p>
        <p class="MsoNormal"><o:p> </o:p></p>
        <p>You're offering carrier-style services to other businesses
          over these WAN links?<o:p></o:p></p>
        <div>
          <p class="MsoNormal">On 27 Jan 2016 9:49 am, "Greg Markey"
            <<a moz-do-not-send="true"
              href="mailto:Greg.Markey@optiver.com.au">Greg.Markey@optiver.com.au</a>>
            wrote:<o:p></o:p></p>
          <div>
            <div>
              <p class="MsoNormal"
                style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Yes,
                  we are; we have a use case for capturing metadata from
                  our WAN taps between regions however I would imagine
                  ISPs are doing it on a much larger scale.</span><o:p></o:p></p>
              <p class="MsoNormal"
                style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
              <p class="MsoNormal"
                style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ll
                  share the slides with the group (once I’ve actually
                  written them :) )</span><o:p></o:p></p>
              <p class="MsoNormal"
                style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
              <p class="MsoNormal"
                style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
                    lang="EN-US">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
                  lang="EN-US"> Geordie Guy [mailto:<a
                    moz-do-not-send="true"
                    href="mailto:elomis@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:elomis@gmail.com">elomis@gmail.com</a></a>]
                  <br>
                  <b>Sent:</b> Wednesday, 27 January 2016 8:26 AM<br>
                  <b>To:</b> Greg Markey<br>
                  <b>Cc:</b> <a moz-do-not-send="true"
                    href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a><br>
                  <b>Subject:</b> Re: [AusNOG] How are you handling
                  metadata?</span><o:p></o:p></p>
              <p class="MsoNormal"
                style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
              <div>
                <p class="MsoNormal"
                  style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Isn't
                  Optiver a financial services business?<o:p></o:p></p>
                <div>
                  <p class="MsoNormal"
                    style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                </div>
                <div>
                  <p class="MsoNormal"
                    style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">G<o:p></o:p></p>
                </div>
              </div>
              <div>
                <p class="MsoNormal"
                  style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                <div>
                  <p class="MsoNormal"
                    style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
                    Tue, Jan 26, 2016 at 10:33 AM, Greg Markey <<a
                      moz-do-not-send="true"
                      href="mailto:Greg.Markey@optiver.com.au"
                      target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:Greg.Markey@optiver.com.au">Greg.Markey@optiver.com.au</a></a>>
                    wrote:<o:p></o:p></p>
                  <p class="MsoNormal"
                    style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello
                    everyone,<br>
                    <br>
                    I'm reaching out to see if anyone on the list is
                    willing to share some high level details around how
                    they have implemented the capture, processing and
                    storage for the metadata retention scheme. I noticed
                    that AGD is unable to provide specific
                    recommendations to ISPs for hardware and software,
                    leading me to believe that the technical
                    implementations are going to potentially vary
                    significantly between organisations.<br>
                    <br>
                    I'll be talking about what we've built internally at
                    the Sydney ElasticSearch users group on Thursday,
                    but it would be great to have some comparisons if
                    you don't mind me sharing your solutions
                    (anonymously).<br>
                    <br>
                    Cheers,<br>
                    Greg<br>
                    <br>
                    Information contained in this communication
                    (including any attachments) is confidential and may
                    be privileged or subject to copyright.  If you have
                    received this communication in error you are not
                    authorised to use the information in any way and
                    Optiver requests that you notify the sender by
                    return email, destroy all copies and delete the
                    information from your system.  Optiver does not
                    represent, warrant or guarantee that this
                    communication is free from computer viruses or other
                    defects or that the integrity of this communication
                    has been maintained.  Any views expressed in this
                    communication are those of the individual sender. 
                    Optiver does not accept liability for any loss or
                    damage caused directly or indirectly by this
                    communication or its use.<br>
                    <br>
                    Please consider the environment before printing this
                    email.<br>
                    _______________________________________________<br>
                    AusNOG mailing list<br>
                    <a moz-do-not-send="true"
                      href="mailto:AusNOG@lists.ausnog.net"
                      target="_blank">AusNOG@lists.ausnog.net</a><br>
                    <a moz-do-not-send="true"
                      href="http://lists.ausnog.net/mailman/listinfo/ausnog"
                      target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
                </div>
                <p class="MsoNormal"
                  style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
              </div>
            </div>
          </div>
          <p class="MsoNormal" style="margin-bottom:12.0pt"><br>
            _______________________________________________<br>
            AusNOG mailing list<br>
            <a moz-do-not-send="true"
              href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
            <a moz-do-not-send="true"
              href="http://lists.ausnog.net/mailman/listinfo/ausnog"
              target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
AusNOG mailing list
<a class="moz-txt-link-abbreviated" href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a>
<a class="moz-txt-link-freetext" href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>