[AusNOG] Encrypt and compress, or compress and encrypt?

[Ross Wheeler]

On Fri, 18 Nov 2016, Karl Kloppenborg wrote:

> Whilst you can save considerable transmission and storage space with just
> removing the excess whitespace in a text file you can save incredible
> amounts of space by indexing and removing multiple sequences of data that
> repeat.
>
> Ultimately it really depends on your requirements and how much you want to
> get this down by. Best of luck!

For those playing at home...

The files I'm storing compress pretty well, and are not huge to start with 
(well, not compared to some of my other logs which fortunately are not 
captured by the DR legislation).

My goals were to use tools that are installed by default on pretty much 
any FreeBSD system, and to not require the installation of *ANYTHING* on 
*ANY* of the source systems. (The exception being ssh keys).

The collection system drives this entirely, no script, not programs, 
utilities, extra widgets, nothing whatsoever, on the production hosts.

Between the files being encrypted on the source host, then all rolled into 
a tarball, and returned via ssh to the logging system and the logging 
system and the source hosts all being within "our" network and thus not 
exposing the traffic to anywhere outside our own control, the chance of 
sucessful interception is almost zero.

Because the data is never in unencrypted form outside the source hosts, 
I'm fairly comfortable with the end solution. Compression before 
encryption yields a very substantial savings in resources - CPU for 
encryption and making the tarball, network for transferring files, disk 
for storing them. Sure, various techniques may yield marginally higher 
compression ratios, but at the expense of CPU, but we're not talking 
terrabytes of data here.

My reason for posting was mainly to see if there were any known pitfalls 
eg https://blog.appcanary.com/2016/encrypt-or-compress.html that I should 
take into account.

Again, I thank everyone for their contributions!
R.