Recently there was a well researched and accurate blog on the increase in patches coming out for VMware ESX Server. The post shows that VMware ESX had about 68 patches released this year. Why so many patches? Well, as Nand points out in the corporate VMware blog Vmware has moved to a more traditional release train where we issue patches individually instead of one lump sum. This is the same method used by most larger ISVs (software vendors). For example, you’ll occasionally get patch notifications from Microsoft with several security, functionality, and driver updates. These Microsoft events have become known as “Patch Tuesday” since they come out on the second Tuesday of each month. FYI…This is also known as “Black Tuesday” to many since no one likes to administer a bunch of patches.
Speaking of Microsoft, let’s take a look at where they stand. Microsoft’s new hypervisor based product called “Hyper-V” requires a Windows operating system in the Parent Partition. Given that you’ll need to patch that Windows OS just like any other Windows OS I decided to look at the history of Microsoft patches for Windows Server 2003. I chose 2003 since it’s been around about the same length of time as VMware ESX and it gives us a good history of patching on a Windows server product. I know that Hyper-V will use the upcoming Windows Server 2008 but we don’t have patch counts for that yet and no one can tell the future of how secure or insecure it will be. In order to get a good count I simply loaded up WSUS (Windows Server Update Service) to download all of the patches for Windows Server 2003. I also skipped downloading driver updates even though it would have been a more fair comparison since ESX patches also include driver updates (and a lot of them). Here’s 2 screen shots of the config:
Oh, but you think you’ll just use “Quick Migration” in Hyper-V to migrate your VMs off for patches? Well, I’ll blog more on Quick Migration later but that aside, it’s not integrated into your patching system so you still have a very manual process. VMware has Update Manager which is integrated and will not only notify you of host patches, but let you schedule them and actually do all of the VM migration, patching, and rebooting for you. Did I mention Update Manager is free and included with VMware Infrastructure 3?
Enough about Microsoft. We know their story well. What about the other virtualization vendors? Well, there’s always Virtual Iron. They tell a good story. No install of the hosts since they all boot from the management server. Sounds good, doesn’t it? Well, if you have actually used it and patched it (I have) then you learn the truth. Here’s how patching a Virtual Iron environment goes (steps from FAQ here). Pay close attention to step 10.
10. From the Policy View, run the RebootDataCenter script, selecting “Restart running Virtual Servers” and “Disable VS Tools – required for v3 to v4 upgrade”.
The script will perform the following functions for you (or you may alternatively perform these operations using the Virtualization Manager GUI):
* Stop all Virtual Servers
* Reboot your Nodes
* Turn off (uncheck in the GUI) the VS Tools flag on all Virtual Servers
* Restart all Virtual Servers
So basically you get to reboot your entire datacenter every time Virtual Iron releases a patch. Heck, that’s worse than using Microsoft’s Quick Migration. And they call this an enterprise class virtualization solution? BTW, In the 12 months that Virtual Iron has been out they’ve had 8 releases so it’s basically once a month that you’re shutting down your entire datacenter to patch the hosts. There goes 5 9’s. Heck, there goes 2 9’s.
Virtual Iron actually uses the Xen open-source hypervisor. There are a lot of other vendors out there that use that same hypervisor (Red Hat, SUSE, SUN, Citrix/XenSource, and Oracle to name a few). While the hypervisor itself is pretty good the architecture still requires a general purpose operating system in Domain 0 (the Parent Partition in Microsoft land). What does this mean? Well, you’re back to having to patch a general purpose Linux operating system which introduces downtime for you system. I could go through finding out exactly how many patches Linux has in each distro but someone has already done the work for me. Actually too many to list. Just search for “linux versus microsoft patch” and you’ll find a ton of 3rd party references like this one from eWeek. Needless to say it looks like you’re worse off than Microsoft as far as patches go and you still don’t have an integrated way to patch everything (unless you like writing scripts).
Last, we’ll go into a totally different architecture for the last vendor – Virtuozzo. They tell a really good story on their website and to customers. Since they run one copy of the OS as a host and then basically link to that from the different VMs (VPS as they call them) they tell you to install the patch once on the host and everyone inherits it. Guess what? It works! That’s pretty cool until you start thinking of how this works in a real environment. Let’s say you deploy that patch and it blows up one of the VMs on the host. If you’ve never had a patch blow up something in your environment then I want to meet you. Anyhow, something blew up so we’ll need to back that patch out. Oh wait…all of the other VMs are inheriting that same patch. That means I’ll need to back out the patch for everyone, move the affected VM to another host, and then re-patch the original host. I can’t just revert the patch for the 1 VM. That sucks. What’s worse is let’s say you’re using SMS or another patch system to patch things. Well, as far as it knows you just patched 1 OS – it doesn’t know if it patched all of the other VMs. That sucks too. But wait for it…patches need to first be certified, packaged, and distributed from SWSoft (or Parallels now) before you get them. You can’t just go straight to Microsoft for updates or your existing patch tools to get the updates. What’s even worse than that is SWSoft is a little on the lazy side for getting the patches out. Here’s an email we got for our environment. Notice the email was dated September 20, 2007 and the MS patch it refers to was released on August 14, 2007. A month late for a security patch for the kernel? Yikes!!
From: SWsoft Virtuozzo Announcement [mailto:vz-announce@swsoft.com]
Sent: Thursday, September 20, 2007 2:31 PM
To: OMITTED
Subject: ~*~ [nai-spam] Virtuozzo Windows Update VZU35150
To view this email as a web page, go here.
VIRTUOZZO FOR WINDOWS TECHNICAL UPDATE
SWsoft announces the availability of new updates and enhancements including the following:
Virtuozzo for Windows 3.5.1 SP1: VZU35150
VZU35150 includes the following important update:
· MS KB932596 support: This update enables installation of Microsoft Windows update KB932596 on Virtuozzo servers
For additional information on this update, please visit this site: http://kb.swsoft.com/en/2141.
How to obtain and install
To access the updates, use the Windows Update wizard. For more information on the Update wizard, please visit this site: http://vzwinupdate.swsoft.com
On the Virtuozzo Windows host, run Virtuozzo Update Wizard (vzupdate.exe) and follow the instructions. This wizard will install all the recent Virtuozzo updates and hotfixes in the correct sequence. Otherwise you can install Virtuozzo updates manually.
For instructions on how to check current Virtuozzo version and number of installed Virtuozzo updates, please visit this site: http://kb.swsoft.com/en/1897
You have received this email because you are subscribed to the SWsoft Mailing List.
If you no longer wish to receive announcements, use the following link to unsubscribe.
To learn more about SWsoft and its products, please visit our website at www.swsoft.com.
13755 Sunrise Valley Drive, Suite 600, Herndon, VA 20171 USA
Phone: +1 (703) 815-5670 Fax:+1 (703) 815-5675 Email: info@swsoft.com
So now we know the truth about patching. As you can see, VMware ESX is actually doing pretty good. Sure we release stuff more frequently than in the past and sure there are more of the patches but that’s what’s common in the industry and that’s what customers wanted – patch granularity. We also added Update Manager to the mix to help our customers cope with this and we did it for free. You can thank us later.
Have more topics you’d like to hear debunked? Go ahead and email me or drop in a comment.
February 28th, 2008 at 4:16 pm
Update Manager is a fantastic product, but why oh why can’t we patch 3.0.x with it? If only…