At VMworld this year during my Deploying VMware in a Linux Shop I demonstrated how to PXE boot VMware ESXi and automatically add it to Virtual Center. As promised a long time ago I’m finally documenting everything I did to get it to work. All of the work for this project was really done by an incredible engineer here at VMware named Lance Berc. I’m just putting some more step-by-step to it and making the process easier to find. Thanks, Lance for all of your work on this and the long night just before VMworld trying to troubleshoot issues!
The Tools
Here’s all of the stuff that you’ll need to complete this exercise. Start the downloads now.
- VMware ESXi (free)
- VMware Virtual Center Server
- PXE Boot Midwife Scripts (free)
- Post boot configuration scripts (free)
- VMware Infrastructure Toolkit for Windows (free)
- Virtual Infrastructure Perl Toolkit (free)
- Windows PowerShell (free)
- TFTPd32 for Windows or some other TFTP server (free)
- ISO Buster (shareware)
- WinRAR (free)
- WinImage (free)
- Syslinux 3.72 or later (free)
The Docs
There are two great documents on how to get ESXi to PXE boot and on how to extract the files necessary how to do this. You can get both from the VMware Communities.
The Process
Now that you have all of the necessary documents and software let’s begin. In a nutshell this process works by getting a server to PXE boot in a normal PXE environment, download ESXi bits, and then when the server loads it will run a midwife script that will contact a midwife server that assists in the “birth” of the ESXi host by adding it to Virtual Center. Get it – midwife…birth. Yes, Lance has that kind of humor. Here’s an overview of the setup.
Overall the steps to introduce the new ESXi host to the environment are:
1) Boot ESXi via DHCP/PXE and pass it the address for the midwife server
2) After the initialization the new system sends a birth message to the midwife server
3) The midwife cleans up any previous instances of the server from Virtual Center
4) The midwife uses the Virtual Center API to configure the new system
The midwife script is written in Powershell. The code performs these steps:
1) Set ESX password
2) Adds host to Virtual Center cluster
3) Configures virtual switches
4) Configures port groups
5) Adds NFS partitions
6) Adds iSCSI partitions
7) Configures NTP
Configures VMotion
Obviously the VMware Toolkit for Windows can do a lot more things so feel free to modify the script to do whatever you’d like.
Setup the PXE Environment
I decided to use Windows for my PXE server. I did this because I was creating a demo to run on my laptop and just wanted 1 Windows server to run Virtual Center, the TFTP server, the midwife, the DNS server, and the PXE server. I also used several Windows based tools for the extraction and troubleshooting. Since I’m using a Mac I ran all of this in VMware Fusion.
To keep things a little brief I’m going to assume you can install Windows, configure DNS and Active Directory on it, and install Virtual Center. You’re pretty much choosing the defaults. If you do need help here then just email me and I’ll see if I can walk you through it quickly.
Now that Windows is ready we need a TFTP server to serve up our image. I used TFTPd32. I created a directory for my new build of C:\tftpd32\tftpboot. Below is a screenshot of the general settings for tftpd32.
I ended up using TFTPD32 for my DHCP server in order to simplify things in my lab setup. If you’re using a different DHCP server then you can reference the TFTPd32 documentation on how to configure DHCP to reference the TFTP server and provide the necessary PXE information. Here are the settings for my DHCP server.
The biggest thing you’ll need to make sure of is the boot file references “pxelinux.0″. We’ll put that file into place in just a bit. You can also adjust the IP settings to reflect your environment.
Setting up the Midwife
The midwife is a set of powershell scripts that are launched by a perl process running in the background. In order to get them to run we need to setup 3 things:
- Microsoft Powershell – This is the foundation for the VI Toolkit
- VMware Infrastructure Toolkit (for Windows) – This is what our script uses
- VMware Perl SDK – We use this to launch the process server
To install the Perl SDK simply download the executable and run it. Use all of the defaults and after a few screens it will be fully installed. You can confirm it’s installed by opening a command prompt and running “perl -version”. If you get a response you’re all set.
Next let’s install the Windows Powershell package. If you’ve done a recent, fully patched installation of Windows Server then all you really need to do is download the Powershell package and install it with the defaults. After the install you’ll have an icon on the desktop for Powershell. In order for the scripts to run later we’ll need to relax the security policy for Powershell or sign our scripts. For this lab we’ll just relax the security policy with “Set-ExecutionPolicy Unrestricted” as seen below.
With the Powershell base installed you can now install the VI Toolkit for Windows. Again, all that’s needed is to download and install the tool with the defaults. After the installation completes, launch the VI Toolkit window and issue “Get-VICommand”. If everything is working right then you’ll see a list of commands that you can use with VMware Virtual Infrastructure.
The last step for the midwife server setup is to unzip the files from the midwife into a folder on the Windows server.
That’s it. The midwife server is setup and ready to go. We’ll run it a little later on.
Extracting the Files for PXE
The hardest part of this whole operation is extracting the right files to the right places to the TFTP server so they all get downloaded correctly. You’ll need more tools for this and it’s very important that you put the right files in the right places so pay attention.
In the document from Lance that talks about extraction he uses a tool from Access Data called the Forensic Toolkit. Unfortunately I couldn’t find a version that was free or cheap so I used ISO buster instead. After downloading and installing the tool, launch it and then open the ISO image of ESXi installable. You’ll see 3 different tracks when you open the image. You’ll want to go to the second track (the Rock Ridge track designated by the gold colored RR letters) and extract the install.tgz file to a folder of your choice (I put it on the desktop).
Once you’ve extracted install.tgz use Winrar to extract the files from the archive to a folder. Navigate to /usr/lib/vmware/installer and using Winrar again, extract VMware-VMvisor-big-3.5.0_Update_2-110271.i386.dd.bz2. You’ll now have a dd disk image file. Using WinImage open the dd file that you just extracted. Inside of it you’ll find several partitions. Partition 4 holds the booter files. Partition 5 contains the ESXi files. Partition 8 contains a bunch of tools such as VMTools and the Virtual Infrastructure Client. We’re going to need the ESXi files from Partition 5 (the first partition in the list if opening with WinImage) so go ahead and extract them to a folder on your desktop called “boot”. These are the files that actually run the ESXi server. You can see there aren’t very many of them. It’s a small package.
To finish out our boot folder we’re going to need to get some files from syslinux that will allow our image to actually boot for a PXE install and give us a menu of the different options available to us. Download the zip file for the latest version of syslinux and open the zip archive. Copy the following files from the syslinux distribution to the “boot” folder on your desktop.
- com32\menu\menu.c32
- com32\modules\mboot.c32
- core\pxelinux.0
Next we need to setup the menu that will appear when we PXE boot the host. In your boot folder create another folder named “pxelinux.cfg”. Inside of the pxelinux.cfg folder create a text file named default (do not put the .txt extension on it – you might need to show file extensions to make sure it’s not there). Edit the “default” file to look like the following:
default menu.c32
menu title PXE Boot VMware ESXi
timeout 100label ESXi
menu label Boot VMware ESXi
kernel mboot.c32
append vmkernel.gz PBHOST=172.10.0.120:3333 — binmod.tgz — environ.tgz — cim.tgz — oem.tgz — lance-boot.tgz
ipappend 2label Hard
menu label Boot from local drive
localboot 0
You should edit the PBHOST variable to be the same as the IP address of the host where you just installed the midwife script (usually the Virtual Center server). After you have made the edits to the file copy the entire contents of this boot folder into the folder you created during the TFTP server set (ex. C:\tftpd32\tftpboot).
The last step to prepare the boot folder is to copy lance-boot.tgz (the post boot configuration files) into the boot foler. Do not extract the files. Just copy lance-boot.tgz straight over. This will complete the setup for booting ESXi.
Configure the Midwife
All that’s left is to modify the configuration PowerShell files to match what you’d like in your environment. The first one up is esx-master.ps1. You should edit the top portion of the file to include the Virtual Center server name or IP as well as the administrator account and password.
# To run as current user:
$VC = “localhost”; $VCpassword = ‘vmware’
#$VC = ’172.10.0.120′ ; $VCpassword = ‘vmware’
$defaultESXPassword = ”
#$newESXPassword = $VCpassword
$newESXPassword = ‘vmware’
The last file you’ll need to edit is esx-profile.ps1. This one contains a few other entries that you’ll need to edit such as the NAS or iSCSI settings, the network settings, the datastore names, etc. It shouldn’t take too long to customize this file to your environment. Once you’re done it’s time to try a run-through.
Doing a PXE Boot
The first step is to start the TFTPd server. Simply launch TFTPd32 and you’ll be ready to go. The next step is to launch the midwife server process. Open a command prompt and type in “perl c:\midwife\configsvc.pl”.
Now boot your target server. If it’s connected to the same network as where TFTPd32 is running then you should eventually see the PXE boot menu on your host.
Leave the default choice selected and then you’ll start to see the files being downloaded from the TFTP server.
To finish things off you’ll see ESXi booting.
If everything goes according to plan you should see something very similar to the demo video from the beginning of this blog. I hope you enjoy the tip.
November 6th, 2008 at 8:54 am
That’s great Mike. Thanks!
November 7th, 2008 at 5:31 pm
Mike,
I’ve been looking forward to more information about PXE booting ESXi, however I have a couple reservations before I spend a few hours setting this up…
Isn’t there a lot of stuff that’s missing when you boot ESXi this way? What if you try to install VMware tools on a VM running on this host? The ISO files are not available. What about downloading the VI Client from this box?
I understand that this is meant for fun/test purposes only, but without being able to install VMware tools that’s a pretty big problem. Is there a way to direct the ESXi kernel to mount these off of a network location over NFS or something like that?
November 8th, 2008 at 8:17 am
Not sure if you can point the box to another system for VMTools. However, you could always include the stuff from Partition 8 in the boot image. It will take a lot longer for the PXE boot to happen but at least you’ll have the VI Client and VMTools available.
November 15th, 2008 at 9:33 pm
Hi,
This is critical – my environment is bunch of ESX 3.5i diskless servers, and I need to transfer windows.iso to all of them as part of PXE boot image so I can install vmtools. Any idea how to do this?
Cheers
November 22nd, 2008 at 10:47 pm
I have recently been hacking around with ESXi and PXE. I had earlier found that it is possible to create a custom ESXi installer similar to what is possible with Kickstart and ESX Classical. The problem has been that every time the PXE booted ESXi host reboots, it looses important parts of its identity and configuration. My recent discovery has been a very simple way to maintain state (between reboots) in these PXE booted hosts without having to resort to network transfers or other external boxes. This actually leverages functionality already present in ESXi (that is used when PXE boot is not performed). I have not yet tested this in a VC environment, however I believe strongly that this will work perfectly. At any rate, I will be trying that in the next week or so.
November 26th, 2008 at 8:03 pm
Hi,
I use the method described here to achieve config preservation between reboots:
http://docs.google.com/View?docid=ddcwgcd6_4fs6s7jcf
However I still can’t include vmware tools in any way. I posted on vmware forum but got nowhere. Really strange considering all the buzz around PXE booting and ESXi.
Any advices appreciated.
Cheers
December 1st, 2008 at 11:32 am
The problem with that technique, if I understand it correctly, is that it relies on a remote machine using the remote CLI to do the backup and then get the backup onto the pxe server. My technique relies on no other software packages or network transfers.
As to the vmware tools, that should not be a problem. As I have already accomplished my VC testing, I expect to do Tools testing this week. When this is done I will provide what should be a final update.
December 1st, 2008 at 11:37 am
Just a quick update.
I tested this is a Virtual Center environment last week and it worked perfectly. Indeed what I discovered (and should have known) is that ESXi was designed to handle this scenario out of the box. ESXi seems to be about 99% of the way to complete stateful PXE functionality (for at least some configurations of the ESX host) out of the box. It does raise the question of why this functionality was not included in the shipping product.
To be clear, no modifications to my technique were necesary at all.
March 19th, 2009 at 11:26 am
Mike, nice work! A quesiton around diskless servers?
Can I connect to software connected storage, say iSCSI or NFS, and use this storage to run the ESXI instance, or must I use hardware connected disk (be it DAS or SAN)?
March 19th, 2009 at 9:08 pm
You should be able to use iSCSI or NAS. We're just using the image to boot the box. It's all loaded into memory so you don't need any local storage.
March 23rd, 2009 at 12:14 pm
[...] stateless model drives reminds me of the ESXi automated installs. I think VMware should be doing more in this space as well, as it tackles the scalability issues [...]
March 30th, 2009 at 11:26 pm
[...] Wolf and Mike Dipetrillo both gave us glimpses of a PXE boot future for ESXi in the past. As both bloggers have described, [...]
April 23rd, 2009 at 8:24 pm
Can I run on vSphere environment ( ESXi 4) ? I meet with problem and I have added PBHOST=hostip:port after vmkboot.vgz and vmk.vgz.
<snip>
label visor-161011
kernel ESX/visor-161011/mboot.c32
append ESX/visor-161011/vmkboot.gz — ESX/visor-161011/vmk.gz PBHOST=172.16.22.75:3333 …
….
<snip>
I do not know how it works on ESXi 4.
April 25th, 2009 at 7:28 am
Actually there's a new process integrated with vSphere and is being
documented right now. Once it's done it will be in the docs. It's a
lot nicer of a solution.
Mike
April 30th, 2009 at 2:42 pm
Great post!! A colleague recommended I looked into PXE booting ESXi with a view to learning more about scripting and automation.
Still very new to all this, and probably trying to run before I can walk but I've followed the guide step by step, and I've hit a bit of a jam. ESXi loads fine, but then it doesn't appear to be sending the birth message to the midwife.
I've run network monitor and can't see anything coming back over port 3333. What I have noticed is that after the ESXi server has finished loading it sends another DHCP Discover packet. Is that right?
Totally lost and trying to get my head round this
April 30th, 2009 at 9:35 pm
Usually this is because the PXE boot file doesn't have the right
format for the append string. Check the spaces and the number of
dashes against the blog notes and try again. Hopefully that will fix it.
May 1st, 2009 at 12:12 am
Yeah, I had that issue originally. It would load the vmkernel but then PSOD. It was because I'd C&P'd the script and the dashes weren't being recognised. I'll delete the spaces/dashes and re-add and see what happens. Thanks for the quick response.
May 1st, 2009 at 10:07 am
Nope, no joy
It definitely says it's loading lance-boot.tgz but still no birth message.
May 18th, 2009 at 9:36 am
Thank you all for your excellent ESXi pxeboot instructions. The stateless configuration is great, but I am interested in an ESXi pxeboot setup that would write ESXi (with custom configuration parameters) to disk. Based on my testing and the posts on this site, it doesn't seem like there is a workable *stateful* solution yet. If anyone has this working, please let me know. Thanks!
May 18th, 2009 at 5:15 pm
Most people wanting stateful use a deployment tool such as Altiris.
I'll see what I can dig up for a free(er) way to do this.
May 26th, 2009 at 6:43 pm
[...] http://www.mikedipetrillo.com/mikedvirtualization/2008/11/howto-pxe-boot-esxi.html [...]
July 22nd, 2009 at 6:01 am
Hi Mike,
A couple of quesions:
#1. I am trying to do stateful PXE boot install for ESX 4.0. I am using “append vmkboot.gz — vmkernel.gz —sys.vgz — cim.vgz — ienviron.tgz — image.tgz — install.tgz” in my configuration. However, the image.tgz does not load entirely and hangs at a specific point. I had extracted the ISO as per the VMWare document. Can you point out what can be the probable cause of the image.tgz load failure? I tried giving mem=512M as well and also tried using the SYSLINUX mboot.c32 instead of the default ISO mboot.c32. But invain. Please help.
#2. Is there any official document from VMWare for ESXi 4.0 stateful install via PXE boot?
Thanks and Regards,
Rupali
July 28th, 2009 at 9:26 am
Hey Marc,
I am having the same issue, no birth message. Were you able to get through? I have provided the PBHOST and lance-boot.tgz in my boot file as below:
append vmkboot.gz — vmkernel.gz PBHOST=10.128.132.5:3333 — sys.vgz — cim.vgz — ienviron.tgz — install.tgz — image.tgz — lance-boot.tgz ipappend 2
My ESXi 4.0 installation halts for user inputs after loading all the modules including lance-boot.tgz.
Any idea what can be missing?
July 28th, 2009 at 9:43 am
Lots of things have changed with ESX 4 for doing this. There's a new technote coming out around VMworld timeframe I believe on how to set all of this up as well as an appliance that's already setup with everything to do this. I'm not the one working on that project but I would just wait for that technote if you can. Sorry I can't be any more help right now.
July 30th, 2009 at 4:31 am
Thanks!! Mike for replying though. Just an update so that if someone else out there is trying out ESXi 4.0, one change is to append the PBHOST after vmkboot.gz as below:
append vmkboot.gz PBHOST=10.128.132.5:3333 — vmkernel.gz — sys.vgz — cim.vgz — ienviron.tgz — install.tgz — image.tgz — lance-boot.tgz ipappend 2
That way I did get the birth message atleast. Now either I cn wait for new technote from VMworld or continue with configuring the Midwife scripts as per my environment requirement.
September 2nd, 2009 at 10:47 pm
Thank you. I am learning to hate ESXi because of the booting issues I have been having as of late.
January 13th, 2010 at 6:01 pm
Mike,
Did the ESXi process for this ever come out? All I see is 'experimental' support and a technote that outline the same pxe boot process and no mention of configuration methods or anything. Thanks
January 13th, 2010 at 6:05 pm
Experimental support is all that's there right now. I hear something is targeted for later this year at this point.
January 14th, 2010 at 1:01 am
Mike,
Did the ESXi process for this ever come out? All I see is 'experimental' support and a technote that outline the same pxe boot process and no mention of configuration methods or anything. Thanks
January 14th, 2010 at 1:05 am
Experimental support is all that's there right now. I hear something is targeted for later this year at this point.
June 8th, 2010 at 4:03 pm
[...] UCS platform that can move the identity of one server to another. On the other hand, you can boot ESXi from PXE, stateless. This is a complex system, but could potentially be an answer to boot-from-SAN if you [...]