<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I'm a bit confused too, the topic appears to be around Data
Retention Metadata correct? In which case, I believe it has been
known and verified by comments and legalese translations that
Netflow style information is not required for DR purposes. The data
that should be held should mostly come from RADIUS packets and such,
these kinds of storage requirements only go up with subscriber
numbers and not link utilisation (to an extent)<br>
<br>
<div class="moz-cite-prefix">On 27/01/16 10:29, Greg Markey wrote:<br>
</div>
<blockquote
cite="mid:D39A115F09181C4C9A4C1C77B0A56260031F8AF2F579@OPSYWNSR0152.comp.optiver.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:416902634;
mso-list-type:hybrid;
mso-list-template-ids:17053994 397036576 201916419 201916421 201916417 201916419 201916421 201916417 201916419 201916421;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:宋体;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1816531902;
mso-list-type:hybrid;
mso-list-template-ids:15993794 -1556996356 201916419 201916421 201916417 201916419 201916421 201916417 201916419 201916421;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:宋体;
mso-bidi-font-family:"Times New Roman";}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We
are already doing this at scale for internal use only;
albeit likely to be smaller than what ISPs are deploying in
the field. This isn’t me trying to make other people do my
homework for me; the content of the talk is what Optiver is
*<b>already</b>*<i> </i>doing.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I
think a good topic of discussion for the user group would be
to talk about options other than what we have implemented,
and how ISPs are facing the challenge of capturing this
metadata. For example in Elastic-land, how many index nodes
would be required to handle indexing of packet metadata for
a 1Gbps link? How much bandwidth do we need to set aside for
these metadata messages? Do we have enough spare cycles to
compress the data?<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">In
order to capture more information than is available from
NetFlow, we use the following stack:<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
style="mso-list:Ignore">·<span style="font:7.0pt
"Times New Roman""> </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">pmacct<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
style="mso-list:Ignore">·<span style="font:7.0pt
"Times New Roman""> </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Gollum<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
style="mso-list:Ignore">·<span style="font:7.0pt
"Times New Roman""> </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Kafka<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span
style="mso-list:Ignore">·<span style="font:7.0pt
"Times New Roman""> </span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ElasticSearch<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As
part of the meetup I’m hoping to commit the glue code into
Github for people to experiment with.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Greg<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
lang="EN-US">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
lang="EN-US"> Robert Hudson [<a class="moz-txt-link-freetext" href="mailto:hudrob@gmail.com">mailto:hudrob@gmail.com</a>] <br>
<b>Sent:</b> Wednesday, 27 January 2016 9:56 AM<br>
<b>To:</b> Greg Markey<br>
<b>Cc:</b> Geordie Guy; <a class="moz-txt-link-abbreviated" href="mailto:ausnog@ausnog.net">ausnog@ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] How are you handling metadata?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p>You're offering carrier-style services to other businesses
over these WAN links?<o:p></o:p></p>
<div>
<p class="MsoNormal">On 27 Jan 2016 9:49 am, "Greg Markey"
<<a moz-do-not-send="true"
href="mailto:Greg.Markey@optiver.com.au">Greg.Markey@optiver.com.au</a>>
wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Yes,
we are; we have a use case for capturing metadata from
our WAN taps between regions however I would imagine
ISPs are doing it on a much larger scale.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ll
share the slides with the group (once I’ve actually
written them :) )</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
lang="EN-US">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""
lang="EN-US"> Geordie Guy [mailto:<a
moz-do-not-send="true"
href="mailto:elomis@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:elomis@gmail.com">elomis@gmail.com</a></a>]
<br>
<b>Sent:</b> Wednesday, 27 January 2016 8:26 AM<br>
<b>To:</b> Greg Markey<br>
<b>Cc:</b> <a moz-do-not-send="true"
href="mailto:ausnog@ausnog.net" target="_blank">ausnog@ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] How are you handling
metadata?</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Isn't
Optiver a financial services business?<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">G<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
Tue, Jan 26, 2016 at 10:33 AM, Greg Markey <<a
moz-do-not-send="true"
href="mailto:Greg.Markey@optiver.com.au"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:Greg.Markey@optiver.com.au">Greg.Markey@optiver.com.au</a></a>>
wrote:<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello
everyone,<br>
<br>
I'm reaching out to see if anyone on the list is
willing to share some high level details around how
they have implemented the capture, processing and
storage for the metadata retention scheme. I noticed
that AGD is unable to provide specific
recommendations to ISPs for hardware and software,
leading me to believe that the technical
implementations are going to potentially vary
significantly between organisations.<br>
<br>
I'll be talking about what we've built internally at
the Sydney ElasticSearch users group on Thursday,
but it would be great to have some comparisons if
you don't mind me sharing your solutions
(anonymously).<br>
<br>
Cheers,<br>
Greg<br>
<br>
Information contained in this communication
(including any attachments) is confidential and may
be privileged or subject to copyright. If you have
received this communication in error you are not
authorised to use the information in any way and
Optiver requests that you notify the sender by
return email, destroy all copies and delete the
information from your system. Optiver does not
represent, warrant or guarantee that this
communication is free from computer viruses or other
defects or that the integrity of this communication
has been maintained. Any views expressed in this
communication are those of the individual sender.
Optiver does not accept liability for any loss or
damage caused directly or indirectly by this
communication or its use.<br>
<br>
Please consider the environment before printing this
email.<br>
_______________________________________________<br>
AusNOG mailing list<br>
<a moz-do-not-send="true"
href="mailto:AusNOG@lists.ausnog.net"
target="_blank">AusNOG@lists.ausnog.net</a><br>
<a moz-do-not-send="true"
href="http://lists.ausnog.net/mailman/listinfo/ausnog"
target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
AusNOG mailing list<br>
<a moz-do-not-send="true"
href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a moz-do-not-send="true"
href="http://lists.ausnog.net/mailman/listinfo/ausnog"
target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
AusNOG mailing list
<a class="moz-txt-link-abbreviated" href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a>
<a class="moz-txt-link-freetext" href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
</blockquote>
<br>
</body>
</html>