<div dir="ltr">Yep, absolutely. <div><br></div><div>Unfortunately, probably 99% of the 'omg my VM's on my server are down' calls I get are from the folks who don't understand the platform like you or I. So someone increasing a VM from 16 to to 30GB of ram on a server with 32GB of physical ram, and then wondering why they are now getting out of disk space errors when they had 30GB free before modifying the VM, is a common call.<div><br></div><div>And yeah, thin provisioning, argh!</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 21 November 2017 at 07:51, Robert Hudson <span dir="ltr"><<a href="mailto:hudrob@gmail.com" target="_blank">hudrob@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><span class=""><div><br><div class="gmail_extra"><br><div class="gmail_quote">On 20 Nov. 2017 10:20 pm, "Damien Gardner Jnr" <<a href="mailto:rendrag@rendrag.net" target="_blank">rendrag@rendrag.net</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I noticed he was talking Veeam - I've often seen a volume run out of space while Veeam was trying to sync an off-site replica, and then the snapshots get stuck. Same if Veeam has crashed or network has dropped, while doing the sync - the snapshot never gets cleaned up.</div></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto">Yes, various backup apps that interact with the VMFS file system via the APIs can leave things lying around - but you need to know this, watch out for it and act appropriately.</div><span class=""><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>He also mentioned Swapfiles. That reeks of someone not reserving memory for the VM. Would be worth checking that in the settings. I know there are valid reasons for over-committing ram on VM's compared to the hypervisor's physical memory, but unless you know exactly what you're doing, and are keeping a watch on it, it's easy to totally fill a datastore and get yourself into trouble. If you have the Physical ram to provide all your VM's needs, it's better to just reserve that memory in the settings, and then you don't need all that extra disk space.</div></div></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto">Memory reservations are not necessary (or even considered best practice) for the vast majority of workloads. Sure, reserve memory for VMs running Java within them (as the hypervisor can't see what the JVM's memory management is doing), and for specific apps like Oracle and SQL database servers, but in general, there's really no need, nor substantial benefit.</div><div dir="auto"><br></div><div dir="auto">The swapfile overhead is capped at the virtual RAM assigned per VM. It really isn't hard to manage. Sure, the swapfiles disappear when the VM is shut down, but they come back again when the VMs are booted (and the VMs won't boot if you don't have space for the swap files).</div><div dir="auto"><br></div><div dir="auto">If a VMFS (or NFS for that matter) file system runs out of space, the VMs will freeze. This is only easy if the environment is not appropriately managed. Snapshots consume an increasing amount of space as they live longer (whereas swapfiles do not), and are a key monitoring point in a virtualisation environment.<br></div><div dir="auto"><br></div><div dir="auto">Another key monitoring point is thin provisioning - it's great telling your guest OS it has a 100GB system drive, but only writing ~20GB in the VMFS volume when you install your OS. But you then need to realise that your thin provisioned VMDK *will* grow as new data is written within the VM. And it will grow to up to 100GB, depending on how much data the VM writes.</div><div dir="auto"><br></div><div dir="auto">Monitoring (and alerting, and then acting on the alerts received) is key to running a virtualisation environment successfully, on anything from a single small host with a couple of VMs to a massive environment with hundreds of hosts and thousands of VMs. There are more monitoring point than are helpful in the vast majority of situations, but then there are good free tools available that can filter down the noise into usable data (and some of them not only point out important issues, but then also tell you what to do about them).</div><div dir="auto"><br></div><div dir="auto"><span style="font-family:sans-serif">What it reeks of is not understanding the environment and toolset properly, and making mistakes accordingly. The good thing about this is that it is easily remedied though education (self or instructor-led) and then applying what is learned to the environment.</span><br></div><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto"><span style="font-family:sans-serif">If you don't know what you're doing, you shouldn't be touching virtualisation. Yep, that's seemingly harsh - but it's absolutely fair - the repercussions otherwise can be dire.</span><span style="font-family:sans-serif"><br></span></div><div><div class="h5"><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><br><div class="gmail_quote">On 20 November 2017 at 15:47, Robert Hudson <span dir="ltr"><<a href="mailto:hudrob@gmail.com" target="_blank">hudrob@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">VMs only have a snapshot when something had told VMware (ESXi or vCenter, which tells ESXi) to create a snapshot. They only have large snapshots or chains of snapshots if those snapshots aren't being cleaned up.<div dir="auto"><br></div><div dir="auto">Snapshots in this case are completely independent of the guest OS, and have nothing to do with where files within those guest OSs may be being written to within the guest OS fike system.</div></div><div class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872HOEnZb"><div class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872h5"><div class="gmail_extra"><br><div class="gmail_quote">On 20 Nov. 2017 1:08 pm, "Rory Jones" <<a href="mailto:rory.jones.au@gmail.com" target="_blank">rory.jones.au@gmail.com</a>> wrote:<br type="attribution"><blockquote class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872m_-1702551973454280128quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div id="m_5160377002496267832m_-7222803485135347260m_5367967816000110872m_-1702551973454280128m_3737756761494666065x_compose-container" style="direction:ltr">
<span><span></span></span>
<div>
<div>
<div style="direction:ltr">As others have said, I wouldn’t go messing around with those VMware files. You have a very high chance of borking something. Instead, get to the source of the problem. Find out *why* your VMs have a sudden appetite for data. Unfortunately
I can’t speak for Linux, but on the off chance the suspect VMs are running Windows, there’s an awesome little tool called WinDirStat. For the record, for Mac the equivalent is Disk Inventory X. Run these tools, or the equivalent for your OS. It’ll give you
a pretty picture like you’ve just broken Tetris of the contents of the drive. Go for the big blocks first. Find out what they are. Diagnose from there.</div>
<div><br>
</div>
<div style="direction:ltr">Hope this helps someone in the future.</div>
</div>
<div><br>
</div>
<div class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872m_-1702551973454280128m_3737756761494666065x_acompli_signature">
<div style="direction:ltr">Kind regards,</div>
<div style="direction:ltr">Rory</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_5160377002496267832m_-7222803485135347260m_5367967816000110872m_-1702551973454280128m_3737756761494666065x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> AusNOG <<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.n<wbr>et</a>> on behalf of Daniel Watson <<a href="mailto:dgwatson1988@gmail.com" target="_blank">dgwatson1988@gmail.com</a>><br>
<b>Sent:</b> Monday, November 20, 2017 8:03:19 AM<br>
<b>To:</b> Burt Mascareigne<br>
<b>Cc:</b> <a href="mailto:ausnog@lists.ausnog.net" target="_blank">ausnog@lists.ausnog.net</a><div class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872m_-1702551973454280128elided-text"><br>
<b>Subject:</b> Re: [AusNOG] VMWare Snapshot / Delta VMDK removal</div></font>
<div> </div>
</div>
</div><div class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872m_-1702551973454280128elided-text">
<font size="2"><span style="font-size:10pt">
<div class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872m_-1702551973454280128m_3737756761494666065PlainText">Have a QNAP. So moved the pagefile’s to an ISCIS target. And then removed all the Delta’s<br>
<br>
Freed up 268gb which has allowed the Veeam backups to run<br>
<br>
Daniel<br>
<br>
Sent from my iPhone<br>
<br>
> On 20 Nov 2017, at 8:09 am, Burt Mascareigne <<a href="mailto:Burt@stormnetwork.com.au" target="_blank">Burt@stormnetwork.com.au</a>> wrote:<br>
> <br>
> I feel like you dodged a bullet there.<br>
> <br>
> If you were not able to shut it down, then get a NAS or some device and clone to it at least that will give you a slimmed down clean version.
<br>
> <br>
> <br>
> <br>
> Regards,<br>
> <br>
> <br>
> Burt Mascareigne<br>
> Mobile 0414 450 962 Office (02) 9965 5422<br>
> Address <a href="https://maps.google.com/?q=Level+19,+1+O%E2%80%99Connell+Street,+Sydney+NSW+2000&entry=gmail&source=g" target="_blank">Level 19, 1 O’Connell Street, Sydney NSW 2000</a><br>
> Web <a href="http://www.stormnetwork.com.au" target="_blank">http://www.stormnetwork.com.au</a><br>
> <br>
> <br>
> <br>
> -----Original Message-----<br>
> From: AusNOG [<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">mailto:ausnog-bounces@lists.a<wbr>usnog.net</a>] On Behalf Of Daniel Watson<br>
> Sent: Sunday, 19 November 2017 10:23 PM<br>
> To: <a href="mailto:ausnog@lists.ausnog.net" target="_blank">ausnog@lists.ausnog.net</a><br>
> Subject: Re: [AusNOG] VMWare Snapshot / Delta VMDK removal<br>
> <br>
> Thanks to all those fantastic people who have responded<br>
> <br>
> I have powered off this particular VM for the time being. And have recovered 35gb in swapfile<br>
> <br>
> I have taken a new successful snapshot. Which then allowed me to delete “all” snapshots. Including those old delta files. This will take a few hours to complete. But then should give me back 100gb of space<br>
> <br>
> Appreciate all the assistance. I’ll keep you posted how it goes tomorrow<br>
> <br>
> Daniel<br>
> <br>
> Sent from my iPhone<br>
> <br>
>> On 19 Nov 2017, at 9:47 pm, Daniel Watson <<a href="mailto:dgwatson1988@gmail.com" target="_blank">dgwatson1988@gmail.com</a>> wrote:<br>
>> <br>
>> Sorry to bother the list but this is critical<br>
>> <br>
>> Wanted to know from others. If it is safe to remove a delta VMDK file from a running vm?<br>
>> <br>
>> A server I am now looking after is running out of space like no tomorrow. And there are 3 delta VMDK ‘s taking up 80gb of space<br>
>> <br>
>> Any information would be appreciated<br>
>> <br>
>> Daniel<br>
>> <br>
>> Sent from my iPhone<br>
>> ______________________________<wbr>_________________<br>
>> AusNOG mailing list<br>
>> <a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
>> <a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailma<wbr>n/listinfo/ausnog</a><br>
> ______________________________<wbr>_________________<br>
> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
> <a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailma<wbr>n/listinfo/ausnog</a><br>
______________________________<wbr>_________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailma<wbr>n/listinfo/ausnog</a><br>
</div>
</span></font>
</div></div>
<br>______________________________<wbr>_________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailma<wbr>n/listinfo/ausnog</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailma<wbr>n/listinfo/ausnog</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_5160377002496267832m_-7222803485135347260m_5367967816000110872gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">
<p>Damien Gardner Jnr<br>VK2TDG. Dip EE. GradIEAust<br><a href="mailto:rendrag@rendrag.net" target="_blank">rendrag@rendrag.net</a> - <span><a href="http://www.rendrag.net/" target="_blank">http://www.rendrag.net/</a><u><br></u></span>--<br>We rode on the winds of the rising storm,<br> We ran to the sounds of thunder.<br>We danced among the lightning bolts,<br> and tore the world asunder</p></div></div>
</div>
</blockquote></div></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">
<p>Damien Gardner Jnr<br>VK2TDG. Dip EE. GradIEAust<br><a href="mailto:rendrag@rendrag.net" target="_blank">rendrag@rendrag.net</a> - <span><a href="http://www.rendrag.net/" target="_blank">http://www.rendrag.net/</a><u><br></u></span>--<br>We rode on the winds of the rising storm,<br> We ran to the sounds of thunder.<br>We danced among the lightning bolts,<br> and tore the world asunder</p></div></div>
</div>