Mike D’s Blog

Some of my colleagues put together a great post on “debunking the myths of memory overcommit” as I like to call it. These myths come from our competitors trying to spin the fact that they don’t have the ability to oversubscribe memory in their systems. Well, to be fair, Citrix has added a balloon driver to XenServer 5 but more on that later. It’s a very good post and shows the true value of the technology. Go read it now.

Even with great information like this out there I still see a lot of people balking at memory overcommitment like it’s some voodoo witch magic that’s going to do nothing but cause huge headaches. This is a myth primarily produced by the rest of the virtualization industry that is missing this key core feature of virtualization. This leads us to the question – is resource overcommitment bad?

My answer to this question is no. After all, when you’re virtualizing you’re oversubscribing everything. I can count on one hand the number of accounts that I’ve seen that keep the numbers of vCPUs (virtual CPUs) equal to the number of pCPUs (physical CPUs). Generally when you’re virtualizing you’re taking underutilized resources and consolidating them on a host. This means you’re naturally handing out many more vCPUs than pCPUs. The average consolidation ratio in VMware environments is 12:1 on a 2 socket system. Let’s say this 2 socket system is a dual core server (more common right now than quad core although quad core implementations are growing). If we just handed out 1 vCPU to each VM that would give us 12 vCPUs to 4 pCPUs for a 3:1 overcommit on CPUs. The same thing happens on the network and the storage stack. I’ve never seen customers handing out a single network interface to a single VM or a single HBA to a single VM. Overcommit happens all of the time in the network and the storage infrastructure even outside of virtualization. So why then is memory overcommit such a taboo topic? Again, it’s because the competitors don’t do it.

This brings us back to Citrix and their balloon driver. They now claim memory overcommitment since they have a balloon driver. That’s actually true – you can oversubscribe memory with just a balloon driver. Actually, you can oversubscribe memory even without the balloon driver if you do some tricky stuff in the software and if the working set is small enough to fit into physical memory. That last part was the key. As long as the actual working set of virtual memory is small enough to fit into physical memory then you will have very little overhead with memory overcommit. VMware has actually measured the overhead with straight shadow page memory overcommit and it comes in at a whopping 0.012% overhead. Yes, you read that percentage number correct – 12/1000ths of overhead.

This is why VMware has implemented Transparent Page Sharing (TPS). TPS is sort of a de-dupe mechanism for memory. We try to take like pages of memory and just write them to physical memory once. The result is a smaller footprint for the working set of memory. With TPS you can actually have a larger working set of memory than you have room for in physical memory. This is a VMware exclusive and no other vendor has it today (some have something similar on a roadmap that’s 3 years out – namely Microsoft).

After TPS comes the balloon driver. What happens when your working set gets larger than the physical memory? You have contention and all of the VMs start fighting with each other. While I love to watch a good fight it’s no fun when it’s your VMs that are fighting. What happens at this point is the balloon driver kicks in and does some forced garbage collection. It prompts each of the VMs to see if they can free up any of their physical memory. It’s up to the OS inside of the VM to make the decision on what it can free up by moving pages to the operating system’s page file. This impacts performance of the OS since it’s now paging and there’s also more overhead in this process since you must contact each VM and wait for a response on what pages are now zombied. This is important to remember, ballooning ONLY works when there is contention (your VMs are fighting) and it will impact performance inside of the VM. Having ballooning as your first and only defense in helping with memory overcommit is a BAD plan. This is compounded even more in the Citrix case since they have nothing like VMware DRS which will move VMs around if they have memory contention. In a VMware environment if there is ballooning then there is pressure on the memory. This pressure is seen by VMware DRS and the VMs will move over to another host with more room in the working set space. There’s no downtime for this move and it’s transparent to end users since it uses the proven VMware VMotion technology.

The bottom line is memory overcommit is nothing to be scared of as long as it’s implemented correctly with the right underlying technologies. If you’re using VMware then no big deal. If you’re using Citrix then you get some benefits from memory overcommit but you also get the really bad implementation from missing technology underneath. With Microsoft – well, I’m sure their next version in 2010 (that’s 2030 when you do the Microsoft conversion) will have something to address this.

UPDATE (10/10/2008): Well, it turns out that while Citrix XenServer 5 has a CLI for the balloon driver mechanism it’s not being used. At one point I remember them blogging about adding this but can’t seem to find that anymore. I did happen across this recent discussion on the Citrix forums. No, Bruce Williams is not me – I use my real name all of the time. It’s probably someone that read this post and went to Citrix to get some answers. So I guess this brings us full circle back to where VMware is the exclusive provider of memory overcommit technologies. I’m not surprised that Citrix reversed their stance on using just the balloon driver because of performance. In the mean time VMware customers continue to use features that have been available since 2001.

DISCLAIMER: I work for VMware as a Principal Systems Engineer. I have worked in the virtualization industry for over 6 1/2 years. I have personally helped over 4,000 customers go virtual. The views and opinions expressed in this blog are my own and do not represent views of my employer or of anyone else. This blog is written on my own time and with my own energy. I am not commenting in this blog on behalf of my employer or anyone else. If you would like official statements about VMware you can contact my employer’s PR firm. I always welcome and encourage open and constructive conversations and will not block any comments from appearing on this blog other than obvious SPAM or threatening hate mail.

Tags: VMware