[AusNOG] VMWare Snapshot / Delta VMDK removal

Tue Nov 21 08:17:19 EST 2017

Yep, absolutely.

Unfortunately, probably 99% of the 'omg my VM's on my server are down'
calls I get are from the folks who don't understand the platform like you
or I.  So someone increasing a VM from 16 to to 30GB of ram on a server
with 32GB of physical ram, and then wondering why they are now getting out
of disk space errors when they had 30GB free before modifying the VM, is a
common call.

And yeah, thin provisioning, argh!

On 21 November 2017 at 07:51, Robert Hudson <hudrob at gmail.com> wrote:

>
>
> On 20 Nov. 2017 10:20 pm, "Damien Gardner Jnr" <rendrag at rendrag.net>
> wrote:
>
>> I noticed he was talking Veeam - I've often seen a volume run out of
>> space while Veeam was trying to sync an off-site replica, and then the
>> snapshots get stuck.  Same if Veeam has crashed or network has dropped,
>> while doing the sync - the snapshot never gets cleaned up.
>>
>
> Yes, various backup apps that interact with the VMFS file system via the
> APIs can leave things lying around - but you need to know this, watch out
> for it and act appropriately.
>
> He also mentioned Swapfiles.  That reeks of someone not reserving memory
>> for the VM.  Would be worth checking that in the settings.  I know there
>> are valid reasons for over-committing ram on VM's compared to the
>> hypervisor's physical memory, but unless you know exactly what you're
>> doing, and are keeping a watch on it, it's easy to totally fill a datastore
>> and get yourself into trouble.  If you have the Physical ram to provide all
>> your VM's needs, it's better to just reserve that memory in the settings,
>> and then you don't need all that extra disk space.
>>
>
> Memory reservations are not necessary (or even considered best practice)
> for the vast majority of workloads. Sure, reserve memory for VMs running
> Java within them (as the hypervisor can't see what the JVM's memory
> management is doing), and for specific apps like Oracle and SQL database
> servers, but in general, there's really no need, nor substantial benefit.
>
> The swapfile overhead is capped at the virtual RAM assigned per VM. It
> really isn't hard to manage. Sure, the swapfiles disappear when the VM is
> shut down, but they come back again when the VMs are booted (and the VMs
> won't boot if you don't have space for the swap files).
>
> If a VMFS (or NFS for that matter) file system runs out of space, the VMs
> will freeze. This is only easy if the environment is not appropriately
> managed. Snapshots consume an increasing amount of space as they live
> longer (whereas swapfiles do not), and are a key monitoring point in a
> virtualisation environment.
>
> Another key monitoring point is thin provisioning - it's great telling
> your guest OS it has a 100GB system drive, but only writing ~20GB in the
> VMFS volume when you install your OS. But you then need to realise that
> your thin provisioned VMDK *will* grow as new data is written within the
> VM. And it will grow to up to 100GB, depending on how much data the VM
> writes.
>
> Monitoring (and alerting, and then acting on the alerts received) is key
> to running a virtualisation environment successfully, on anything from a
> single small host with a couple of VMs to a massive environment with
> hundreds of hosts and thousands of VMs. There are more monitoring point
> than are helpful in the vast majority of situations, but then there are
> good free tools available that can filter down the noise into usable data
> (and some of them not only point out important issues, but then also tell
> you what to do about them).
>
> What it reeks of is not understanding the environment and toolset
> properly, and making mistakes accordingly. The good thing about this is
> that it is easily remedied though education (self or instructor-led) and
> then applying what is learned to the environment.
>
> If you don't know what you're doing, you shouldn't be touching
> virtualisation. Yep, that's seemingly harsh - but it's absolutely fair -
> the repercussions otherwise can be dire.
>
>
>> On 20 November 2017 at 15:47, Robert Hudson <hudrob at gmail.com> wrote:
>>
>>> VMs only have a snapshot when something had told VMware (ESXi or
>>> vCenter, which tells ESXi) to create a snapshot. They only have large
>>> snapshots or chains of snapshots if those snapshots aren't being cleaned up.
>>>
>>> Snapshots in this case are completely independent of the guest OS, and
>>> have nothing to do with where files within those guest OSs may be being
>>> written to within the guest OS fike system.
>>>
>>> On 20 Nov. 2017 1:08 pm, "Rory Jones" <rory.jones.au at gmail.com> wrote:
>>>
>>> As others have said, I wouldn’t go messing around with those VMware
>>> files. You have a very high chance of borking something. Instead, get to
>>> the source of the problem. Find out *why* your VMs have a sudden appetite
>>> for data. Unfortunately I can’t speak for Linux, but on the off chance the
>>> suspect VMs are running Windows, there’s an awesome little tool called
>>> WinDirStat. For the record, for Mac the equivalent is Disk Inventory X. Run
>>> these tools, or the equivalent for your OS. It’ll give you a pretty picture
>>> like you’ve just broken Tetris of the contents of the drive. Go for the big
>>> blocks first. Find out what they are. Diagnose from there.
>>>
>>> Hope this helps someone in the future.
>>>
>>> Kind regards,
>>> Rory
>>> ------------------------------
>>> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Daniel
>>> Watson <dgwatson1988 at gmail.com>
>>> *Sent:* Monday, November 20, 2017 8:03:19 AM
>>> *To:* Burt Mascareigne
>>> *Cc:* ausnog at lists.ausnog.net
>>>
>>> *Subject:* Re: [AusNOG] VMWare Snapshot / Delta VMDK removal
>>>
>>> Have a QNAP. So moved the pagefile’s to an ISCIS target. And then
>>> removed all the Delta’s
>>>
>>> Freed up 268gb which has allowed the Veeam backups to run
>>>
>>> Daniel
>>>
>>> Sent from my iPhone
>>>
>>> > On 20 Nov 2017, at 8:09 am, Burt Mascareigne <Burt at stormnetwork.com.au>
>>> wrote:
>>> >
>>> > I feel like you dodged a bullet there.
>>> >
>>> > If you were not able to shut it down, then get a NAS or some device
>>> and clone to it at least that will give you a slimmed down clean version.
>>> >
>>> >
>>> >
>>> > Regards,
>>> >
>>> >
>>> > Burt Mascareigne
>>> > Mobile 0414 450 962   Office (02) 9965 5422
>>> > Address Level 19, 1 O’Connell Street, Sydney NSW 2000
>>> <https://maps.google.com/?q=Level+19,+1+O%E2%80%99Connell+Street,+Sydney+NSW+2000&entry=gmail&source=g>
>>> > Web http://www.stormnetwork.com.au
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net
>>> <ausnog-bounces at lists.ausnog.net>] On Behalf Of Daniel Watson
>>> > Sent: Sunday, 19 November 2017 10:23 PM
>>> > To: ausnog at lists.ausnog.net
>>> > Subject: Re: [AusNOG] VMWare Snapshot / Delta VMDK removal
>>> >
>>> > Thanks to all those fantastic people who have responded
>>> >
>>> > I have powered off this particular VM for the time being. And have
>>> recovered 35gb in swapfile
>>> >
>>> > I have taken a new successful snapshot. Which then allowed me to
>>> delete “all” snapshots. Including those old delta files.  This will take a
>>> few hours to complete. But then should give me back 100gb of space
>>> >
>>> > Appreciate all the assistance. I’ll keep you posted how it goes
>>> tomorrow
>>> >
>>> > Daniel
>>> >
>>> > Sent from my iPhone
>>> >
>>> >> On 19 Nov 2017, at 9:47 pm, Daniel Watson <dgwatson1988 at gmail.com>
>>> wrote:
>>> >>
>>> >> Sorry to bother the list but this is critical
>>> >>
>>> >> Wanted to know from others. If it is safe to remove a delta VMDK file
>>> from a running vm?
>>> >>
>>> >> A server I am now looking after is running out of space like no
>>> tomorrow. And there are 3 delta VMDK ‘s taking up 80gb of space
>>> >>
>>> >> Any information would be appreciated
>>> >>
>>> >> Daniel
>>> >>
>>> >> Sent from my iPhone
>>> >> _______________________________________________
>>> >> AusNOG mailing list
>>> >> AusNOG at lists.ausnog.net
>>> >> http://lists.ausnog.net/mailman/listinfo/ausnog
>>> > _______________________________________________
>>> > AusNOG mailing list
>>> > AusNOG at lists.ausnog.net
>>> > http://lists.ausnog.net/mailman/listinfo/ausnog
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>
>>
>> --
>>
>> Damien Gardner Jnr
>> VK2TDG. Dip EE. GradIEAust
>> rendrag at rendrag.net -  http://www.rendrag.net/
>> --
>> We rode on the winds of the rising storm,
>>  We ran to the sounds of thunder.
>> We danced among the lightning bolts,
>>  and tore the world asunder
>>
>

-- 

Damien Gardner Jnr
VK2TDG. Dip EE. GradIEAust
rendrag at rendrag.net -  http://www.rendrag.net/
--
We rode on the winds of the rising storm,
 We ran to the sounds of thunder.
We danced among the lightning bolts,
 and tore the world asunder
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20171121/1174823c/attachment.html>