<html><body><div style="color:#000; background-color:#fff; font-family:Helvetica Neue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;font-size:16px"><div id="yui_3_16_0_1_1433843929015_123718"><span></span></div><br> <div style="font-family: Helvetica Neue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif; font-size: 16px;" id="yui_3_16_0_1_1433843929015_123721"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif; font-size: 16px;" id="yui_3_16_0_1_1433843929015_123720"> <div dir="ltr" id="yui_3_16_0_1_1433843929015_123719"> <hr size="1"> <font size="2" face="Arial" id="yui_3_16_0_1_1433843929015_123722"> <b><span style="font-weight:bold;">From:</span></b> Joseph Goldman <joe@apcs.com.au><br> <b><span style="font-weight: bold;">To:</span></b> Mark Newton <newton@atdot.dotat.org> <br><b><span style="font-weight: bold;">Cc:</span></b> ausnog@lists.ausnog.net <br> <b><span style="font-weight: bold;">Sent:</span></b> Thursday, 11 June 2015, 8:24<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [AusNOG] From the AGD - Data Retention - Starts October 15 2015<br> </font> </div> <div class="y_msg_container" id="yui_3_16_0_1_1433843929015_123723"><br><div id="yiv1137412750"><div id="yui_3_16_0_1_1433843929015_123724">
<br clear="none">
<br clear="none">
<div class="yiv1137412750moz-cite-prefix">On 11/06/15 00:28, Mark Newton wrote:<br clear="none">
</div>
<blockquote type="cite">
</blockquote></div><div id="yui_3_16_0_1_1433843929015_123728">
On Jun 10, 2015, at 9:28 AM, Joseph Goldman <<a rel="nofollow" shape="rect" class="yiv1137412750" ymailto="mailto:joe@apcs.com.au" target="_blank" href="mailto:joe@apcs.com.au">joe@apcs.com.au</a>>
wrote:<br clear="none" class="yiv1137412750">
<div id="yui_3_16_0_1_1433843929015_123727">
<blockquote class="yiv1137412750" type="cite" id="yui_3_16_0_1_1433843929015_123726"><br clear="none" class="yiv1137412750Apple-interchange-newline">
<div class="yiv1137412750">
</div></blockquote></div></div><div id="yui_3_16_0_1_1433843929015_123730"><div class="yiv1137412750" id="yui_3_16_0_1_1433843929015_123729"> I wonder how
much realistically that $131.3 mill will be split and
provided - but ultimately it looks like for the most part,
Netflow data married with IP history to a subscriber will
cover the broad strokes. </div>
<div><br clear="none" class="yiv1137412750">
</div>
<div>Why would you gather netflow data?</div>
That post was early on when there was still some back and forward
about interpretation on if we had to also retain DST-IP for
communications - mixed with the session terminology I interpreted
(likely incorrectly - but still some argument about it it seems due
to vague terms) that we would need to log each session for data from
src-ip to dst-ip through our network - RADIUS wouldn't offer this
level of information where netflow would, but if it is determined by
the special interest group or whoever gets some clear clarifications
from AGD/CAC that its not needed then I'm more than happy to just
keep RADIUS (as we do anyway)<br clear="none">
<blockquote type="cite" id="yui_3_16_0_1_1433843929015_123732">
<div id="yui_3_16_0_1_1433843929015_123731"><br clear="none">
<div id="yui_3_16_0_1_1433843929015_123733">If you’re retaining my netflow data, you’re retaining
information about things I’m communicating with which are
outside AGD’s specified data set. I’m going to get pretty
upset and annoyed about that unlawful snooping, and I’ll
wonder how insane an ISP would have to be to want to get
involved in that level of detail.</div>
</div>
</blockquote>
Well - in a way I already am keeping a lot of netflow data for our
customers. Not 2 years worth, no, but I still retain a fair bit to
look at trends of data movement by interface,AS,protocol etc that
simple SNMP can't give. I'd hazard a guess that most ISP's are the
same. A lot of ISP's actually do accounting from Netflow data rather
than RADIUS accounting from my understanding,</div><div id="yui_3_16_0_1_1433843929015_123730"><br></div><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr"><br></div><div id="yui_3_16_0_1_1433843929015_123730"><br></div><div id="yui_3_16_0_1_1433843929015_123730"> so you can offer
things such as traffic to particular AS or via particular interface
(Peering for example) to not count towards quota or to count
differently towards customers quota. (Think the old FREE PIPE DATA
days, and some ISP's not counting netflix traffic).<div class="qtdSeparateBR"><br><br></div><div class="yiv1137412750yqt2101721113" id="yiv1137412750yqtfd16340"><br clear="none">
</div><div class="yiv1137412750yqt2101721113" id="yiv1137412750yqtfd16340"><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr" class="" style="">/ So having worked for an ISP who was doing it this way for many 10s of 1000s of customers,... yeah, no.</div><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr" class="" style=""><br class="" style=""></div><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr" class="" style="">/ That was being done with software routers as BRASes (i.e., no hardware forwarding plane separate from the control plane), which mean that Netflow generation could have as much of the router's processing resources as necessary - and if I remember correctly we were losing about 20% of the router's forwarding capacity due to Netflow generation (and another 20% for PPPoE/PPP overhead), compared to the throughput that platform could achieve doing simple IP in ethernet forwarding.</div><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr" class="" style=""><br class="" style=""></div><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr" class="" style="">/ I heard that other ISPs who were using Netflow for zero metering instead would do Netflow generation on their routers attached to the zero-metered sources/destinatons, and then subtract that from the quota use reported via RADIUS accounting from their BRASes. That would have been much less resource intensive, however it doesn't allow you to zero-meter arbitrary sources/destinations on the internet. (And in my opinion, I think it would be better and fairer (in a network neutrality context) to not zero-meter anything, getting rid of all of this complexity, and just give _everybody_ an extra e.g., 20% more quota per month when you decide to get in bed with a VoD provider of your choice.)</div><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr" class="" style=""><br class="" style=""></div><div id="yui_3_16_0_1_1433843929015_123730" dir="ltr" class="" style="">/ Once you move to a hardware forwarding platform, you may encounter control plane capacity limits for Netflow generation for all packets, and you can't trade forwarding capacity for Netflow generation like you could with a software BRAS. Sampled Netflow may overcome that, and would produce the data you mentioned (AS, interface etc.), but might not be accurate enough for the AGD's metadata requirements. </div></div><blockquote type="cite" id="yui_3_16_0_1_1433843929015_124099"><div class="yiv1137412750yqt2101721113" id="yiv1137412750yqtfd42183">
</div><div id="yui_3_16_0_1_1433843929015_124098"><div class="yiv1137412750yqt2101721113" id="yiv1137412750yqtfd21150">
<div id="yui_3_16_0_1_1433843929015_124266"><br clear="none" class="yiv1137412750">
</div>
<div id="yui_3_16_0_1_1433843929015_124264"><i class="yiv1137412750" id="yui_3_16_0_1_1433843929015_124265">You people need to get legal advice. </i>If
geeks are making decision about which data gets retained,
you’re comprehensively stuffing this up.</div>
<div id="yui_3_16_0_1_1433843929015_124263"><br clear="none" class="yiv1137412750">
</div>
<div id="yui_3_16_0_1_1433843929015_124097">Get it together, you people. You’ve already squibbed on the
campaign against it, and now you’re paying the price. For
god’s sake, don’t squib the implementation too.</div></div>
</div>
</blockquote>
Its kind of lucky in a way, that they didn't say implement and be
done, they actually have to review and approve your implementation
plan, so those who do misinterpret and build out an incompatible
implementation should be picked up in that review process.
</div></div><br><div class="yqt2101721113" id="yqtfd94576">_______________________________________________<br clear="none">AusNOG mailing list<br clear="none"><a shape="rect" ymailto="mailto:AusNOG@lists.ausnog.net" href="mailto:AusNOG@lists.ausnog.net" id="yui_3_16_0_1_1433843929015_124262">AusNOG@lists.ausnog.net</a><br clear="none"><a shape="rect" href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br clear="none"></div><br><br></div> </div> </div> </div></body></html>