When you administrating VMs run on virtualization hosts (VMWare ESXi or Hyper-V) and analyzing performance issues, you quite often face the situations when the size of available memory in the guest OS is much less than the operation system sees (or is assigned to it). For example, 8 GB of memory is allocated to a virtual machine, the Task Manager shows that 1 GB is free though all running processes consume less than 3 GB. Where are the rest 4 GB?
As a rule, this behavior is caused by using the memory overcommit feature in the hypervisor.
Memory overcommit is a hypervisor feature that allows you to allocate to virtual machines more memory than is available on a physical host, but without any guarantees that all requested memory can be allocated at a particular moment. Memory overcommit allows to allocate virtual machines more densely due to dynamic memory distribution between VMs depending on the current host load (the resources of idle VMs may be redistributed among more loaded ones).
One of the mechanisms for implementing memory overcommit in VMWare is Memory Ballooning. In Hyper-V, the same feature is implemented by Dynamic Memory.
Note. By the way, both VMWare and Microsoft widely and effectively use memory compression to save memory resources (this is also one of the implementations of memory overcommit technology).In VMWare ESXi memory overcommitment (virtual memory ballooning) is implemented by vmmemctl.sys driver (included in VMware Tools), which can capture physical memory by creating a fake process (like inflating a balloon) in it if necessary. Thus, the occupied amount of memory becomes unavailable to other applications in the guest OS, and the hypervisor can distribute free memory between other VMs. In Hyper-V Dynamic memory, a dmvsc.sys driver from the integration services (Dynamic Memory VSC component) is used. The memory overcommitment settings are managed by the hypervisor administrator.
How can you detect from within the guest VM that it actually has less physical memory available than what the guest operating system sees?
Let’s consider how to detect if there is a balloon driver in a guest Windows OS and what its size is. So, let’s analyze this example.
8 GB of memory is allocated to a VM running guest Windows Server 2012 R2. The Task Manager shows that 93% of memory is used (7.4 GB of memory is occupied). However, if you sum up the amount of memory used by all running processes, you can come up with an unexpected conclusion that only 2.5 GB of memory is actually used. Where are the rest 5 GB? Neither the Task Manager, nor the Resource Monitor is able to answer this question.
To understand what is going on with the memory, you need to use RamMap tool by Mark Russinovich (in one of the earlier cases, I have shown how to use this tool to diagnose an issue with the high memory usage by metafile on Windows Server). Download the tool from the Microsoft website (https://technet.microsoft.com/en-us/library/ff700229.aspx) and run it with the administrator privileges. After that on the Use Counts tab you can see that most of the memory (5.4 GB) is used by the Driver Locked object.
This is the memory that the hypervisor has occupied and distributed between other virtual machines using the balloon driver in the guest OS. It means that the hypervisor host does not have enough memory or the hypervisor administrator uses the memory limit policies for this VM.
The current VM memory statistics in Hyper-V may be shown by separate performance counters in the Performance Monitor:
- Hyper-V Dynamic Memory –> Guest Visible Memory
- Hyper-V Dynamic Memory –> Physical Memory
To disable this behavior, the hypervisor administrator must disable the Enable Dynamic Memory option in the Hyper-V settings of the VM (or increase the minimum reservation value).
If the VMWare ESXi host is used, you can reserve more memory for this virtual machine in the Resource Settings or reserve all memory – Reserve all guest memory (All locked).
