[AusNOG] Offsite Storage

[Alan Maher]

This reminds me of a funny story from 15 years ago (back in the Dark Ages).
We had a series of Red Hat Linux servers where each node was 2 boxes that
backed up to each other. If one should crash, we could log into the other,
change the IP address, and away it went again, while we fixed the first box.
And the there was the one lone Win. NT server with a tape drive backup.
My job, as the late shift guy, was to insert the tape, push the go 
button to
start the backup, make sure it was running, then I could turn out the lights
set the alarm and go home.
The early shift guy took the tape out and sent it away for storage.
We had a series of tapes that were rotated around between off site storage
and so on. Fairly normal stuff.
After some time, it was decided to upgrade the NT server to something more
grunty & modern.
The changeover worked like a charm, and the new NT server was up and running
as per previous. It also had the latest greatest HP SCSI tape drive.
The process of backing up each night remained the same, and the rotation of
tapes continued. Each week a new set of tapes arrived to be used, and 
old tapes
in off site storage were retired.

In a spare moment ( something to be treasured, as there weren't many) I had
a look at the old NT server, and grabbed an old backup tape to see what 
was on it.
Yes, you guessed right, absolutely nothing.
Assuming the drive to be faulty, I tried another machine that was not in 
use, but had
a newer tape drive, and it was the same.
So, I began going through some of the other redundant tapes, and found 
that absolutely
nothing had been backed up for some 2 years.
The tape drive sat there blinking its lights, and the software showed a 
status bar
demonstrating that all was going along smoothly, but in reality it was 
all an April Fool's joke.
Sooooo, I had a look at a more recent tape from the new NT server.
Yes, you guessed right again- it also had nothing on it.
Somewhere, somehow, a confusion between an MS driver, and an HP driver , 
had caused
the NT server to believe it was doing ok, but obviously it wasn't.

But wait, there's more.......lol. I checked with our other 2 NZ sites, 
and asked them to courier
me a tape from their excess stock- same story.(Hardly surprising as they 
were mirror setups)
Then I advised Melbourne HQ of my findings, and they discovered the same 
problem around
all 30 of their Aussie sites.
A "small" driver issue had caused many guys to go home each night 
thinking all was fine with
the world, and the managers woke up feeling the same.

Thankfully, the need to recover using backup tapes never arose, but it 
just goes to show.
The best laid plans of mice and men.............


On 26/11/2014 4:49 p.m., Alex Samad - Yieldbroker wrote:
> Hi
>
> Wondering what people are doing for offsite store / backups. Looking at http://it.slashdot.org/story/14/11/26/0044239/is-lto-tape-on-its-way-out LTO tapes sales are down.  People are backing up to disk.
>
> But what are people doing for offsite.  Removable HDs ?
>
> Alex
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog


---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com