[AusNOG] Encrypt and compress, or compress and encrypt?

[Karl Kloppenborg]

Late to the conversation.
There’s been great replies, I’d just like to add to these if I can.
Yes compress prior to encrypt and sending but also when looking at
compression you mentioned log-files so if these are large and in a standard
format (i.e traffic logs/apache/nginx etc etc) it’s worth looking at
compression with a specific specification of deduplication.
Many standard compression formats deal with compressing text files by
removing whitespace and converting it to a hashing sequence but fail to do
dedup’ing.
I’m sure many on here know what deduplication is (eg. many sans feature it)
but if you don’t read up on wikipedia:
https://en.wikipedia.org/wiki/Data_deduplication

Whilst you can save considerable transmission and storage space with just
removing the excess whitespace in a text file you can save incredible
amounts of space by indexing and removing multiple sequences of data that
repeat.

Ultimately it really depends on your requirements and how much you want to
get this down by. Best of luck!

—Karl.

Kind Regards,
Karl Kloppenborg


On 18 November 2016 at 5:18:23 pm, Mark Smith (markzzzsmith at gmail.com)
wrote:

Compress and then encrypt. If your encryption is any good there will be no
repeating patterns worth compressing.

On 18 Nov. 2016 16:29, "Ross Wheeler" <ausnog at rossw.net> wrote:


Esteemed geeks, I seek your considered input.

As part of "that which we may not discuss", I am seeking arguments as to
precedence.

If I have a plain-text log file on machine (a), that I wish to store
securely for some period on machine (b), is it better to encrypt the file
first and then compress it, or to compress it first and then encrypt it?
Either way, it will be encrypted before it leaves machine (a) over an ssh
link, and will be stored in the (compressed and encrypted) form on the
storage device.

I think compression is likely to be vastly superior for text files than
binary files, so compression first, then encrypt the (binary) file, and
indeed a couple of quick tests shows files are 10-15 times larger if I
encrypt first. (330KB vs 5500KB, 125KB vs 1611KB). Given there will be a
lot of files to copy daily, the savings in transmission time, storage and
possibly CPU (must be easier to encrypt a small file than a large one),
this looks like the clear winner.

I'm not sure there's much difference either way when extracting data, but
if anyone knows of a pitfall in this process, I'd rather hear about it
before I get too far down the road!

Thanks guys, and have a good weekend... it's almost beer oclock!

R.
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog


_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20161118/a1671a85/attachment.html>