[AusNOG] From the AGD - Data Retention - Starts October 15 2015

Thu Jun 11 13:10:05 EST 2015

      From: Joseph Goldman <joe at apcs.com.au>
 To: Mark Newton <newton at atdot.dotat.org> 
Cc: ausnog at lists.ausnog.net 
 Sent: Thursday, 11 June 2015, 8:24
 Subject: Re: [AusNOG] From the AGD - Data Retention - Starts October 15 2015

 On 11/06/15 00:28, Mark Newton wrote:

 On Jun 10, 2015, at 9:28 AM, Joseph Goldman <joe at apcs.com.au> wrote:

 I wonder how much realistically that $131.3 mill will be split and provided - but ultimately it looks like for the most part, Netflow data married with IP history to a subscriber will cover the broad strokes.  
  Why would you gather netflow data? That post was early on when there was still some back and forward about interpretation on if we had to also retain DST-IP for communications - mixed with the session terminology I interpreted (likely incorrectly - but still some argument about it it seems due to vague terms) that we would need to log each session for data from src-ip to dst-ip through our network - RADIUS wouldn't offer this level of information where netflow would, but if it is determined by the special interest group or whoever gets some clear clarifications from AGD/CAC that its not needed then I'm more than happy to just keep RADIUS (as we do anyway)

 If you’re retaining my netflow data, you’re retaining information about things I’m communicating with which are outside AGD’s specified data set.  I’m going to get pretty upset and annoyed about that unlawful snooping, and I’ll wonder how insane an ISP would have to be to want to get involved in that level of detail.  
 Well - in a way I already am keeping a lot of netflow data for our customers. Not 2 years worth, no, but I still retain a fair bit to look at trends of data movement by interface,AS,protocol etc that simple SNMP can't give. I'd hazard a guess that most ISP's are the same. A lot of ISP's actually do accounting from Netflow data rather  than RADIUS accounting from my understanding,

 so you can offer things such as traffic to particular AS or via particular interface (Peering for example) to not count towards quota or to count differently towards customers quota. (Think the old FREE PIPE DATA days, and some ISP's not counting netflix traffic).

 / So having worked for an ISP who was doing it this way for many 10s of 1000s of customers,... yeah, no.
/ That was being done with software routers as BRASes (i.e., no hardware forwarding plane separate from the control plane), which mean that Netflow generation could have as much of the router's processing resources as necessary - and if I remember correctly we were losing about 20% of the router's forwarding capacity due to Netflow generation (and another 20% for PPPoE/PPP overhead), compared to the throughput that platform could achieve doing simple IP in ethernet forwarding.
/ I heard that other ISPs who were using Netflow for zero metering instead would do Netflow generation on their routers attached to the zero-metered sources/destinatons, and then subtract that from the quota use reported via RADIUS accounting from their BRASes. That would have been much less resource intensive, however it doesn't allow you to zero-meter arbitrary sources/destinations on the internet. (And in my opinion, I think it would be better and fairer (in a network neutrality context) to not zero-meter anything, getting rid of all of this complexity, and just give _everybody_ an extra e.g., 20% more quota per month when you decide to get in bed with a VoD provider of your choice.)
/ Once you move to a hardware forwarding platform, you may encounter control plane capacity limits for Netflow generation for all packets, and you can't trade forwarding capacity for Netflow generation like you could with a software BRAS. Sampled Netflow may overcome that, and would produce the data you mentioned (AS, interface etc.), but might not be accurate enough for the AGD's metadata requirements. 

  You people need to get legal advice. If geeks are making decision about which data gets retained, you’re comprehensively stuffing this up. 
  Get it together, you people. You’ve already squibbed on the campaign against it, and now you’re paying the price. For god’s sake, don’t squib the implementation too.  
  Its kind of lucky in a way, that they didn't say implement and be done, they actually have to review and approve your implementation plan, so those who do misinterpret and build out an incompatible implementation should be picked up in that review process. 
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20150611/31f195a9/attachment.html>