I was at a customer this week that told me Microsoft could migrate live between AMD and Intel processors and asked why we (VMware) couldn’t. I actually get this question a lot and so I’m providing some context to the answer here. Microsoft and XenSource and all of the other competitors out there that do some sort of “live” migration run into the exact same issues that we do. Here’s what’s happening. Basically you’re running an OS on an Intel box and let’s say the processor supports the SSE3 instruction set and your app happens to use that instruction. Now you migrate that to an AMD box that doesn’t support SSE3 but the app is still using it and trying to use it. BAM! Your app and your OS will crash. This can happen with VMotion and Microsoft Quick Migration. Actually anyone that does live migration will get impacted by this. There are several “user mode” instructions like this that we can’t mask out at the virtualization layer.
At VMware we are always conservative so we check for this incompatibility before we let you shoot yourself in the foot potentially. With the Xen based live migration and Microsoft Quick Migration they do not perform the check and so you can actually do the migration but your app and your OS may die as a result.
Actually, in the VC config file you can put in a parameter to prevent VC from doing any checks. With that parameter you can do migrations between anything you want. Just be aware that you may shoot yourself in the foot and it’s not supported as such.
With all of that said, VMware went to the chip manufacturers a while ago to see if we could get a switch where we can turn their advanced features off. They laughed at us originally. Who can blame them. They put the advanced features in there to differentiate themselves and sell processor upgrades. We’re basically telling them not to be competitive. Well, with virtualization showing up everywhere the chip manufacturers have decided to put advanced virtualization capabilities into the chips. It all started with Intel VT-x and AMD SVM. Now both manufacturers are including a switch that will allow us to turn off certain features and make the processors look like they have the same features. Intel and AMD have both added this feature to their recent processor launches. This is really great since now you can buy a new box with a new feature set, add the box to the virtualization cluster, and we’ll go ahead and turn off the advanced features that would normally prevent migration. We can set everything to the lowest common denominator.
I know what you’re thinking. Won’t this be bad because my apps use that feature. Truth be told most of the user mode functions are used on the desktop side primarily. Some server apps still use those advanced features as well but not so many that you need to worry about it. Most of the user mode stuff deals with video and so it’s not terribly useful on the server side. With over 6 million Windows apps and probably twice that on the open source side it would be impossible to create a list of which apps use which features. This is why we do all of these checks in the first place.
Anyhow, that’s why we say you can’t migrate from Intel to AMD just yet and this is why anyone that says they can do it is lying to you or just don’t understand the technology. The later happens to be true with most of VMware’s competitors – especially the field sales. I guess that’s where experience comes in.
I hope that puts some context around this.
UPDATE and CORRECTION:
Finally someone can just post a correction without getting emotional about it. Virtual PC Guy posted a great explanation of why Microsoft checks for the CPU compatibility that falls right in line with the reasons I outlined here. I’m not sure why my original setup did not check for CPU compatibility. I’m also not sure why the guts of Quick Migration and failover – HAVM.vbs found at the end of the instructions for setup – does not clearly show anywhere that a check occurs. Never-the-less, if you happen to try migrating between incompatible processor types then you get the following warning in the interface followed by a VM left in a suspended state and awaiting instructions on where to go and run:
Thank you Virtual PC Guy!!
No on to the XenSource camp. Like I said before I love it when I get Simon excited. To his defense I was using an older version of XenSource and relying on some experience with open source implementations in RHEL5 and SLES10. The RHEL5 and SLES10 implementations still do not perform CPU checks or if they do they certainly don’t tell you about it or warn you when you do a migration. Thankfully my test VM was a simple RHEL5 guest and just migrated between the Intel and AMD systems I had without worries. No warning. No caution. Just migrate. My XenSource 4.1 install did produce a warning when adding the AMD host to the Intel cluster.
I stand corrected on the new processor check feature.
I do question Simon’s memory. In his response back he says:
The total number of patches shipped to date for XenServer: 0. Total hot fixes andpatches shipped last year for VMware VI3/ESX alone: 68*!* of which 17 were critical. Silly me! Those were probably only “feature enhancements”.
Simon, what are all of the hotfixes listed here for XenServer? I guess those are “feature enhancements” as well.
And, oh, you might want to look up what FUD is. This wasn’t FUD. This was just bad blogging on my part and not checking the most recent version of software for one vendor and it failing for another vendor. There’s enough FUD going around from the Citrix and Microsoft field teams telling customers you can migrate between CPU families and VMware can’t. That’s what prompted this post in the first place. Perhaps you can help me out with that while you’re charged up with energy.
UPDATE 2:
For more in-depth information on CPU compatibility with VMotion see the following paper: http://www.vmware.com/resources/techresources/1022.
Tags: live migration, quick migration, vmotion
February 14th, 2008 at 1:49 pm
Migrating from Intel to AMD from Mike D
VMware’s Mike D is back. He talks about how you can easily VMotion from Intel to AMD, but you should understand the risks involved before doing it on production virtual machines. Link: A Little Truth: Mike D’s Virtualization Blog: Migrating
February 19th, 2008 at 3:08 am
Virtualization Clustering and Processor Compatibility
Generally speaking I try to pretend that the rest of the Internet does not exist, and just happily talk
February 19th, 2008 at 3:16 am
Virtualization Clustering and Processor Compatibility
Generally speaking I try to pretend that the rest of the Internet does not exist, and just happily talk
February 19th, 2008 at 3:17 am
Virtualization Clustering and Processor Compatibility
Generally speaking I try to pretend that the rest of the Internet does not exist, and just happily talk
February 19th, 2008 at 4:09 am
XenServer _checks_ processors types before adding host to pool.
February 19th, 2008 at 11:56 am
Perhaps you should read some documentation on a competitive product before slamming it. Or does competitive analysis really mean “spread FUD”?
http://support.citrix.com/article/CTX115813
Summary
This FAQ article includes questions related to XenMotion using XenServer 4.0.1. Refer to CTX115716 – Citrix XenServer 4.0.1H FAQ for the full set of FAQs.
Q: What is XenMotion?
A: XenMotion is a feature that allows you to move a running virtual machine (VM) from one physical XenServer Enterprise server to another without any downtime.
Q: Which of your products support XenMotion?
A: XenMotion is only provided in our XenServer Enterprise product.
Q: What are the requirements to enable XenMotion?
A: You need at least two XenServer Enterprise servers running in a resource pool.
The XenServer Enterprise servers must have similar processor configurations, some type of remote shared storage such as iSCSI or Network File System (NFS), and a gigabit network connecting them.
Q: How similar do the processors need to be on the XenServer Enterprise servers?
A: To use XenMotion, the processors must be the same type, but can have slight differences (such as CPU speed). So, for example, all the systems would need to have Intel Xeon 51xx series processors. They could be different speeds, so you can mix systems with Xeon 5130 and Xeon 5140 processors. The same is true of AMD processors.
Q: Can you XenMotion a VM between an Intel and AMD system?
A: No. You can only XenMotion a VM between systems with the same processor manufacturer and type.
Q: Does XenMotion require you to have the same exact configurations for your server systems?
A: While you do need to have the same type of processor in each system, other configurations can differ. You can have different amounts of memory, different storage controllers, and different network controllers in each system.
Q: What type of storage does a VM need to be stored on to enable XenMotion?
A: A VM must be stored on remote shared storage to allow for XenMotion.
Examples of this are connections to NFS- or iSCSI (through a software iSCSI initiator)-based storage.
Q: What networking speed is required for XenMotion?
A: We recommend that you use Gigabit Ethernet between your physical servers.
Q: How much downtime occurs during a XenMotion?
The actual downtime during a XenMotion is generally 100-150ms. This downtime is so slight that services running in the VM are not interrupted. Most of the 100-150ms downtime is caused by your network switching equipment moving traffic to a new port.
This document applies to:
XenServer 4.0
XenExpress 4.0
XenEnterprise 4.0
February 19th, 2008 at 4:57 pm
Hyper-V also does the check as well – http://blogs.msdn.com/virtual_pc_guy/archive/2008/02/19/virtualization-clustering-and-processor-compatibility.aspx
February 20th, 2008 at 3:35 pm
This is too funny. First you start off with the flimsy evidence of a second hand story from one cusotmer, then claim this happens alot without providing any evidence. Then you make an inflamatory statement based on that flimsy word of mouth evidence -”With the Xen based live migration and Microsoft Quick Migration they do not perform the check and so you can actually do the migration but your app and your OS may die as a result.” Next you go off on a rant based on this flimsy evidence and inflammatory claim. That certainly sounds like spreading fear, uncertainty and doubt to me.
Now that someone else actually did a test and revealed your are full of it, you act like this was a completely innocent mistake. Some how your Microsoft install did not work as designed and you are “not sure why my original setup did not check for CPU compatibility” . Perhaps you should RTFM next time? You admit to using an out of date version of the open source Xen code while calling out “XenSource”, then slip in “My XenSource 4.1 install did produce a warning when adding the AMD host to the Intel cluster. I stand corrected on the new processor check feature.” It is so funny that this is exactly what the documentation says will happen. So you never actually read any docs or tested this on XenServer before writing this post. At least you admit your mistake after the truth is revealed, so maybe this is just uncertainty and doubt on your part.
Now we know you are not very good at testing comeptitive products and your future as a financial analyst isn’t very bright. At least we can all laugh about it now. I cannot wait to read your next post.