Scott Lowe asked a great question on his blog today – Why does XenSource tout having a 64-bit hypervisor? I mean really, what does it give them that VMware doesn’t already have? More host memory support? No. VMware supports 256 GB and XenSource supports 128 GB. More guest memory? No. VMware supports 64 GB and XenSource supports 32 GB. More guest support? No. VMware supports more 64-bit guests than XenSource does. VMware was also the first virtualization vendor with 64-bit guest support. So what are the benefits of the 64-bit hypervisor that XenSource is touting.
I think Keith Adams from VMware told it best on why VMware doesn’t need to go 64-bit in the hypervisor.
I’ve worked in VMware’s virtual machine monitor group since 2000. I’m part of the three-person team who initially brought up 64-bit VMs on AMD hardware before SVM in 2004, and I am the one person team who brought up 64-bit VMs on Intel hardware with VT in 2005.
Our “hypervisor,” the vmkernel, is indeed 32-bit in ESX 3i. However, we make a distinction between our “monitor” and our “hypervisor” that is unfamiliar to Xen folks. Our “monitor” runs at CPL-0 on the bare metal, but is instanced; there is one monitor per VM. The monitor is what actually emulates the CPU, constructs shadow pagetables, programs VT/SVM hardware if appropriate, etc.
Our monitor has been 64-bit (on capable hardware) since 2004. Since the CPU-facing part of the software is 64-bit, it greatly reduces the pressure on us to move the vmkernel to 64-bit. Our customers understand this. When Xen Source sales people try to sell Xen by saying “Xen == 64-bit == new hotness, VMware == 32-bit == old n’ busted”, our customers are right to make fun of them on their blogs, IMHO.
So there you have it. Finally someone called XenSource out on more of their rhetoric. I’m sure I’ll get a bunch of the XenSource fans flaming me on my blog too just like they did to Scott. That’s fine. Just make sure you come with facts this time and not more rants.
DISCLAIMER: These thoughts are my own and are not reflective of my company or anyone or anything else on planet earth.
February 23rd, 2008 at 5:43 pm
You know, as an average joe interested in VM technology, perusing the news feeds for information to improve my skillset, this argument comes off in a way you probably don’t intend.
Whenever I hear anyone in technology say “Well why do you need to do that?” I mentally translate that to “Well, no.. we can’t do that, but we’ll pretend we don’t need to”.
I’ve heard this argument from various vendors in various areas for years.
For example: “What does 64 bit buy you for anything other than huge memory server solutions?” when Windows didn’t have a workable 64 bit OS and Linux did. Or how about “Who really needs more than one core?”
If there is a valid reason to stay 32 bit, the by all means argue that. But this is technology, and if you’re not moving forward you’re falling behind. The “Who really needs” argument comes off as sounding hollow and like a justification (even if it’s true).
February 23rd, 2008 at 10:36 pm
ErikW,
You’re right.
By the same token, when a vendor touts that they have a particular feature when there is no valid reason to have that specific feature or no real business justification for the feature, they are guilty of the same thing.
Also, check out Keith Adams’ comment on my post about VMware’s architecture and its “64-bitness.” You may find that information helpful.
February 24th, 2008 at 12:22 pm
Well, obviously 64 bit has advantages that 32 bit does not. Your argument is that those advantages do not (currently) make any difference, and that may well be right.
Now I don’t really know what a “monitor” versus a “hypervisor” does, but if i’m investing a lot of money in a technology, I want it to be as flexible as possible for future needs, not adequate for todays needs. Not that i’m saying VMWare’s hypervisor won’t be flexible for the future, but in my mind 64 bit now means no rewrite to 64 bit when it’s needed later.
Also, knowing what I do about the X86-64 technology, I know that 64 bit code has access to more registers, can move more data round in single operations, and has better performance in some areas over 32 bit code running on that same CPU, so it makes sense to me that 64 bit might perform better than 32 bit (use fewer CPU cycles), though if and how much performance gain would probably be slim at best (less than 10%) in most cases. Still squeaking out any added performance would be good.
Now, none of what I’ve said above is necessarily true, it’s just what my experience and reasoning tells me.
So, to me, the question is not “what does 64 bit buy you?” but rather “What does staying 32 bit buy you?”, other than a lot of work rewriting and testing currently working code for VMWare.
Also, color me cynical, but I have a strange feeling that if and when VMWare decides to update their hypervisor to 64-bit, they’ll also be trumpeting it as a feature, which will in effect prove your competitor right.
This is why I say that regardless of the realities at work, this argument sounds hollow and a bit too much like a justification.
Technology is about keeping up with your competitors, even if it doesn’t seem to make a difference. It’s all about the perceptions of the consumer.
February 24th, 2008 at 6:58 pm
Very good comment, ErikW. I think Scott is pointing you in the right direction to read Keith’s comments. From a very simplistic view the difference between the monitor and the hypervisor from the point of this conversation can be thought about this way:
- The monitor is a small set of code that controls the access to the registers and memory and bus, etc for the VM. For VMware this is written in 64-bit code and so you still get the performance benefit and large memory benefit and every other benefit of 64-bit.
- The hypervisor for this case just schedules the different monitors (there’s one for each VM) on the different CPUs. So the downside of having a 32-bit hypervisor is really nothing since it’s a glorified scheduler.
I’m downplaying a lot of this and leaving out a lot of other functions but for the purpose of simplifying the conversation you can think about the two this way and see why there probably won’t be a rewrite of the hypervisor to 64-bit anytime soon simply because it’s not all that beneficial.
You can go talk to the Intel and AMD people and get their roadmaps but I think it will be a long time until you see their processors for straight 64-bit (minus the Itanium etc that already are). There are a LOT of legacy 32-bit apps around and some of them still need the 32-bit instruction set. Will this change in the future? I’m sure. Will it happen in the next 2 or 3 years? I doubt it.
So since VMware has a 64-bit VMM today that can access all the 64-bit goodness of a system that’s why I’m saying why does it matter that the scheduler which doesn’t need to access all of the 64-bit goodness need to be 64-bit. There really is no reason.
I was simply trying to clarify this because I’ve been to several customers that ask why VMware doesn’t have a 64-bit hypervisor when Xen does. It’s because for us the hypervisor does not need to be 64-bit today or for the near term future because of the role it plays. Our VMM (virtual machine monitor is what that stands for) does need to be 64-bit but we took care of that a long time ago.
February 26th, 2008 at 5:32 pm
Thanks for your response Mike, that helps clarify a bit more what a Hypervisor (in VMWare terms) is. However, it also raises another question in my mind.
Schedulers are typically time sensitive applications, and many people have spent inordinate amounts of time trying to come up with better ones. For example, the Linux 2.4 and 2.6 kernels have gone through several scheduler rewrites, including features such as O(1) scheduling, fair scheduling, etc…
So wouldn’t it benefit the system to have the scheduler execute as fast, and with as little overhead as possible? Wouldn’t that mean it would benefit the system to have the hypervisor be 64 bit from the perspective of shaving even a few cycles off?
Of course I suppose that means you would have to subscribe to the theory that 64 bit code executes faster than 64 bit code, which is certainly not always the case. I guess i’m just rambling.
February 27th, 2008 at 4:01 pm
Mike,
To those of us customers that are trying to evaluate all of the different flavors out there you really do give your company a bad light. This may be your personal blog but you act very immature on it and personal or not you represent your company poorly with it. Also in my research and visits by your Sales teams VMWare touts quite a few “Unnecessary” features of their own. So as you say to Citrix you should watch what you tout yourself.
And as for the guest post some companies have very strict rules about Blogging and commenting on blogs even though we are not a technology company so not everyone is hiding behind the guest tag but old school thinking requires it.
February 28th, 2008 at 7:00 pm
VMware engineer, here.
“So wouldn’t it benefit the system to have the scheduler execute as fast, and with as little overhead as possible?”"
Yes!
“Wouldn’t that mean it would benefit the system to have the hypervisor be 64 bit from the perspective of shaving even a few cycles off?”
No!
Recompiling for 64 bits doesn’t, in and of itself, make anything faster. In fact, all else being equal, it makes things slower, because the CPU caches are now more poorly utilized; all those 64 bit pointers and integers take up twice as much space in the cache.
All 64-bits gives you is convenient access to more than 4 GB of memory at a time. If your application/kernel/what have you doesn’t need more than 4GB, or if its behavior is phased and its working set tends to be less than 4GB at a time, then 32 bits works just fine. The VMkernel is mostly in the latter category; while the total guest memory on a large box is well over 4GB, that memory is for *guest* use. It’s rare for the VMkernel itself to have to mess around with guest memory, and when it does so it’s fine to use a windowed approach.
Things change, of course, and as we run more and more guests on more and more CPUs, we may move the VMkernel to 64 bits eventually. It’s just a trade-off; you don’t use the hardware just so you can beat your chest about how much of the hardware you use. You use it when and if it benefits your customers to do so.
March 3rd, 2008 at 6:10 pm
Keith,
I think you missed what I said earlier in the thread. Yes, of course access to > 4GB is not an issue with a hypervisor, but what about access to all the extra registers provided by the x64 architecture? What about the added benefit of larger data moves, and operations on larger data values?
And while I realize that code doesn’t have to be 64 b it to access those functions, it still needs to be “64-bit aware” to be able to utilize that. In other words, “No, the code isn’t 64 bit, but it takes advantage of all the features offered by the 64 bit processor”
That seems to me to be the greater benefit of 64 bit, not the amount of memory it accesses.
I mean, we’ve seen very real performance benefits from 64 bit code over 32 bit code in things like SQL Server, Encryption, and Transcoding. Clearly it’s not *just* about memory.