[AusNOG] VMWare Snapshot / Delta VMDK removal

Tue Nov 21 07:51:25 EST 2017

On 20 Nov. 2017 10:20 pm, "Damien Gardner Jnr" <rendrag at rendrag.net> wrote:

> I noticed he was talking Veeam - I've often seen a volume run out of space
> while Veeam was trying to sync an off-site replica, and then the snapshots
> get stuck.  Same if Veeam has crashed or network has dropped, while doing
> the sync - the snapshot never gets cleaned up.
>

Yes, various backup apps that interact with the VMFS file system via the
APIs can leave things lying around - but you need to know this, watch out
for it and act appropriately.

He also mentioned Swapfiles.  That reeks of someone not reserving memory
> for the VM.  Would be worth checking that in the settings.  I know there
> are valid reasons for over-committing ram on VM's compared to the
> hypervisor's physical memory, but unless you know exactly what you're
> doing, and are keeping a watch on it, it's easy to totally fill a datastore
> and get yourself into trouble.  If you have the Physical ram to provide all
> your VM's needs, it's better to just reserve that memory in the settings,
> and then you don't need all that extra disk space.
>

Memory reservations are not necessary (or even considered best practice)
for the vast majority of workloads. Sure, reserve memory for VMs running
Java within them (as the hypervisor can't see what the JVM's memory
management is doing), and for specific apps like Oracle and SQL database
servers, but in general, there's really no need, nor substantial benefit.

The swapfile overhead is capped at the virtual RAM assigned per VM. It
really isn't hard to manage. Sure, the swapfiles disappear when the VM is
shut down, but they come back again when the VMs are booted (and the VMs
won't boot if you don't have space for the swap files).

If a VMFS (or NFS for that matter) file system runs out of space, the VMs
will freeze. This is only easy if the environment is not appropriately
managed. Snapshots consume an increasing amount of space as they live
longer (whereas swapfiles do not), and are a key monitoring point in a
virtualisation environment.

Another key monitoring point is thin provisioning - it's great telling your
guest OS it has a 100GB system drive, but only writing ~20GB in the VMFS
volume when you install your OS. But you then need to realise that your
thin provisioned VMDK *will* grow as new data is written within the VM. And
it will grow to up to 100GB, depending on how much data the VM writes.

Monitoring (and alerting, and then acting on the alerts received) is key to
running a virtualisation environment successfully, on anything from a
single small host with a couple of VMs to a massive environment with
hundreds of hosts and thousands of VMs. There are more monitoring point
than are helpful in the vast majority of situations, but then there are
good free tools available that can filter down the noise into usable data
(and some of them not only point out important issues, but then also tell
you what to do about them).

What it reeks of is not understanding the environment and toolset properly,
and making mistakes accordingly. The good thing about this is that it is
easily remedied though education (self or instructor-led) and then applying
what is learned to the environment.

If you don't know what you're doing, you shouldn't be touching
virtualisation. Yep, that's seemingly harsh - but it's absolutely fair -
the repercussions otherwise can be dire.

> On 20 November 2017 at 15:47, Robert Hudson <hudrob at gmail.com> wrote:
>
>> VMs only have a snapshot when something had told VMware (ESXi or vCenter,
>> which tells ESXi) to create a snapshot. They only have large snapshots or
>> chains of snapshots if those snapshots aren't being cleaned up.
>>
>> Snapshots in this case are completely independent of the guest OS, and
>> have nothing to do with where files within those guest OSs may be being
>> written to within the guest OS fike system.
>>
>> On 20 Nov. 2017 1:08 pm, "Rory Jones" <rory.jones.au at gmail.com> wrote:
>>
>> As others have said, I wouldn’t go messing around with those VMware
>> files. You have a very high chance of borking something. Instead, get to
>> the source of the problem. Find out *why* your VMs have a sudden appetite
>> for data. Unfortunately I can’t speak for Linux, but on the off chance the
>> suspect VMs are running Windows, there’s an awesome little tool called
>> WinDirStat. For the record, for Mac the equivalent is Disk Inventory X. Run
>> these tools, or the equivalent for your OS. It’ll give you a pretty picture
>> like you’ve just broken Tetris of the contents of the drive. Go for the big
>> blocks first. Find out what they are. Diagnose from there.
>>
>> Hope this helps someone in the future.
>>
>> Kind regards,
>> Rory
>> ------------------------------
>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Daniel
>> Watson <dgwatson1988 at gmail.com>
>> *Sent:* Monday, November 20, 2017 8:03:19 AM
>> *To:* Burt Mascareigne
>> *Cc:* ausnog at lists.ausnog.net
>>
>> *Subject:* Re: [AusNOG] VMWare Snapshot / Delta VMDK removal
>>
>> Have a QNAP. So moved the pagefile’s to an ISCIS target. And then removed
>> all the Delta’s
>>
>> Freed up 268gb which has allowed the Veeam backups to run
>>
>> Daniel
>>
>> Sent from my iPhone
>>
>> > On 20 Nov 2017, at 8:09 am, Burt Mascareigne <Burt at stormnetwork.com.au>
>> wrote:
>> >
>> > I feel like you dodged a bullet there.
>> >
>> > If you were not able to shut it down, then get a NAS or some device and
>> clone to it at least that will give you a slimmed down clean version.
>> >
>> >
>> >
>> > Regards,
>> >
>> >
>> > Burt Mascareigne
>> > Mobile 0414 450 962   Office (02) 9965 5422
>> > Address Level 19, 1 O’Connell Street, Sydney NSW 2000
>> <https://maps.google.com/?q=Level+19,+1+O%E2%80%99Connell+Street,+Sydney+NSW+2000&entry=gmail&source=g>
>> > Web http://www.stormnetwork.com.au
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net
>> <ausnog-bounces at lists.ausnog.net>] On Behalf Of Daniel Watson
>> > Sent: Sunday, 19 November 2017 10:23 PM
>> > To: ausnog at lists.ausnog.net
>> > Subject: Re: [AusNOG] VMWare Snapshot / Delta VMDK removal
>> >
>> > Thanks to all those fantastic people who have responded
>> >
>> > I have powered off this particular VM for the time being. And have
>> recovered 35gb in swapfile
>> >
>> > I have taken a new successful snapshot. Which then allowed me to delete
>> “all” snapshots. Including those old delta files.  This will take a few
>> hours to complete. But then should give me back 100gb of space
>> >
>> > Appreciate all the assistance. I’ll keep you posted how it goes tomorrow
>> >
>> > Daniel
>> >
>> > Sent from my iPhone
>> >
>> >> On 19 Nov 2017, at 9:47 pm, Daniel Watson <dgwatson1988 at gmail.com>
>> wrote:
>> >>
>> >> Sorry to bother the list but this is critical
>> >>
>> >> Wanted to know from others. If it is safe to remove a delta VMDK file
>> from a running vm?
>> >>
>> >> A server I am now looking after is running out of space like no
>> tomorrow. And there are 3 delta VMDK ‘s taking up 80gb of space
>> >>
>> >> Any information would be appreciated
>> >>
>> >> Daniel
>> >>
>> >> Sent from my iPhone
>> >> _______________________________________________
>> >> AusNOG mailing list
>> >> AusNOG at lists.ausnog.net
>> >> http://lists.ausnog.net/mailman/listinfo/ausnog
>> > _______________________________________________
>> > AusNOG mailing list
>> > AusNOG at lists.ausnog.net
>> > http://lists.ausnog.net/mailman/listinfo/ausnog
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>
>
> --
>
> Damien Gardner Jnr
> VK2TDG. Dip EE. GradIEAust
> rendrag at rendrag.net -  http://www.rendrag.net/
> --
> We rode on the winds of the rising storm,
>  We ran to the sounds of thunder.
> We danced among the lightning bolts,
>  and tore the world asunder
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20171121/61732749/attachment.html>