[AusNOG] Welcome to Metadata Retention

[Noel Butler]

On 02/03/2015 08:09, George Fong wrote: 

> The reality IMHO, is not so much what we have to collect, but what we have to strip out of the data we collect. And that process is going to vary significantly depending on what generates the data. Even between MTAs (eg Postfix vs Exim) the log formats are very different. If you run both then you have twice the work to do. And then you need

You're hardly going to bother to strip out a few segments from a maillog
transaction 

> We could retain everything and not parse them for only the relevant data. There will then be a storage issue.

Hardly. 

For example, here, postfix is around 1.9k per transaction, that's
logging connection, amavis stuff, processing and delivery, gzipped
that's about 620 bytes per transaction, now, crazy over estimation, lets
say someone does 10 million email transactions a day, over two years,
that would still work out to be not much more than the average kids
music collection - less than 200 GB and fits on the smallest of drives
that can be picked up for like 60 bucks, so raid 1 it, wow thats a
massive (NOT) 120 bucks worth of storage to retain Email logs. 

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20150302/437dc606/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: george-2014.png
Type: image/png
Size: 20375 bytes
Desc: not available
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20150302/437dc606/attachment.png>