Nov 06

At VMworld this year during my Deploying VMware in a Linux Shop I demonstrated how to PXE boot VMware ESXi and automatically add it to Virtual Center. As promised a long time ago I’m finally documenting everything I did to get it to work. All of the work for this project was really done by an incredible engineer here at VMware named Lance Berc. I’m just putting some more step-by-step to it and making the process easier to find. Thanks, Lance for all of your work on this and the long night just before VMworld trying to troubleshoot issues!

 

The Tools

 

Here’s all of the stuff that you’ll need to complete this exercise. Start the downloads now.

 

 

 

The Docs

 

There are two great documents on how to get ESXi to PXE boot and on how to extract the files necessary how to do this. You can get both from the VMware Communities.

 

 

 

The Process

 

Now that you have all of the necessary documents and software let’s begin. In a nutshell this process works by getting a server to PXE boot in a normal PXE environment, download ESXi bits, and then when the server loads it will run a midwife script that will contact a midwife server that assists in the “birth” of the ESXi host by adding it to Virtual Center. Get it – midwife…birth. Yes, Lance has that kind of humor. Here’s an overview of the setup.

 

 

Overall the steps to introduce the new ESXi host to the environment are:

1) Boot ESXi via DHCP/PXE and pass it the address for the midwife server
2) After the initialization the new system sends a birth message to the midwife server
3) The midwife cleans up any previous instances of the server from Virtual Center
4) The midwife uses the Virtual Center API to configure the new system

The midwife script is written in Powershell. The code performs these steps:

1) Set ESX password
2) Adds host to Virtual Center cluster
3) Configures virtual switches
4) Configures port groups
5) Adds NFS partitions
6) Adds iSCSI partitions
7) Configures NTP
8) Configures VMotion

Obviously the VMware Toolkit for Windows can do a lot more things so feel free to modify the script to do whatever you’d like.

 

Setup the PXE Environment

 

I decided to use Windows for my PXE server. I did this because I was creating a demo to run on my laptop and just wanted 1 Windows server to run Virtual Center, the TFTP server, the midwife, the DNS server, and the PXE server. I also used several Windows based tools for the extraction and troubleshooting. Since I’m using a Mac I ran all of this in VMware Fusion.

To keep things a little brief I’m going to assume you can install Windows, configure DNS and Active Directory on it, and install Virtual Center. You’re pretty much choosing the defaults. If you do need help here then just email me and I’ll see if I can walk you through it quickly.

Now that Windows is ready we need a TFTP server to serve up our image. I used TFTPd32. I created a directory for my new build of C:\tftpd32\tftpboot. Below is a screenshot of the general settings for tftpd32.

 

 

I ended up using TFTPD32 for my DHCP server in order to simplify things in my lab setup. If you’re using a different DHCP server then you can reference the TFTPd32 documentation on how to configure DHCP to reference the TFTP server and provide the necessary PXE information. Here are the settings for my DHCP server.

 

 

The biggest thing you’ll need to make sure of is the boot file references “pxelinux.0″. We’ll put that file into place in just a bit. You can also adjust the IP settings to reflect your environment.

 

Setting up the Midwife

 

The midwife is a set of powershell scripts that are launched by a perl process running in the background. In order to get them to run we need to setup 3 things:

 

  • Microsoft Powershell – This is the foundation for the VI Toolkit
  • VMware Infrastructure Toolkit (for Windows) – This is what our script uses
  • VMware Perl SDK – We use this to launch the process server

 

To install the Perl SDK simply download the executable and run it. Use all of the defaults and after a few screens it will be fully installed. You can confirm it’s installed by opening a command prompt and running “perl -version”. If you get a response you’re all set.

 

 

Next let’s install the Windows Powershell package. If you’ve done a recent, fully patched installation of Windows Server then all you really need to do is download the Powershell package and install it with the defaults. After the install you’ll have an icon on the desktop for Powershell. In order for the scripts to run later we’ll need to relax the security policy for Powershell or sign our scripts. For this lab we’ll just relax the security policy with “Set-ExecutionPolicy Unrestricted” as seen below.

 

 

With the Powershell base installed you can now install the VI Toolkit for Windows. Again, all that’s needed is to download and install the tool with the defaults. After the installation completes, launch the VI Toolkit window and issue “Get-VICommand”. If everything is working right then you’ll see a list of commands that you can use with VMware Virtual Infrastructure.

 

 

The last step for the midwife server setup is to unzip the files from the midwife into a folder on the Windows server.

 

 

That’s it. The midwife server is setup and ready to go. We’ll run it a little later on.

 

Extracting the Files for PXE

 

The hardest part of this whole operation is extracting the right files to the right places to the TFTP server so they all get downloaded correctly. You’ll need more tools for this and it’s very important that you put the right files in the right places so pay attention.

In the document from Lance that talks about extraction he uses a tool from Access Data called the Forensic Toolkit. Unfortunately I couldn’t find a version that was free or cheap so I used ISO buster instead. After downloading and installing the tool, launch it and then open the ISO image of ESXi installable. You’ll see 3 different tracks when you open the image. You’ll want to go to the second track (the Rock Ridge track designated by the gold colored RR letters) and extract the install.tgz file to a folder of your choice (I put it on the desktop).

 

 

Once you’ve extracted install.tgz use Winrar to extract the files from the archive to a folder. Navigate to /usr/lib/vmware/installer and using Winrar again, extract VMware-VMvisor-big-3.5.0_Update_2-110271.i386.dd.bz2. You’ll now have a dd disk image file. Using WinImage open the dd file that you just extracted. Inside of it you’ll find several partitions. Partition 4 holds the booter files. Partition 5 contains the ESXi files. Partition 8 contains a bunch of tools such as VMTools and the Virtual Infrastructure Client. We’re going to need the ESXi files from Partition 5 (the first partition in the list if opening with WinImage) so go ahead and extract them to a folder on your desktop called “boot”. These are the files that actually run the ESXi server. You can see there aren’t very many of them. It’s a small package.

 

 

To finish out our boot folder we’re going to need to get some files from syslinux that will allow our image to actually boot for a PXE install and give us a menu of the different options available to us. Download the zip file for the latest version of syslinux and open the zip archive. Copy the following files from the syslinux distribution to the “boot” folder on your desktop.

 

  • com32\menu\menu.c32
  • com32\modules\mboot.c32
  • core\pxelinux.0

 

Next we need to setup the menu that will appear when we PXE boot the host. In your boot folder create another folder named “pxelinux.cfg”. Inside of the pxelinux.cfg folder create a text file named default (do not put the .txt extension on it – you might need to show file extensions to make sure it’s not there). Edit the “default” file to look like the following:

 

default menu.c32
menu title PXE Boot VMware ESXi
timeout 100 

label ESXi
menu label Boot VMware ESXi
kernel mboot.c32
append vmkernel.gz PBHOST=172.10.0.120:3333 — binmod.tgz — environ.tgz — cim.tgz — oem.tgz — lance-boot.tgz
ipappend 2

label Hard
menu label Boot from local drive
localboot 0

You should edit the PBHOST variable to be the same as the IP address of the host where you just installed the midwife script (usually the Virtual Center server). After you have made the edits to the file copy the entire contents of this boot folder into the folder you created during the TFTP server set (ex. C:\tftpd32\tftpboot).

The last step to prepare the boot folder is to copy lance-boot.tgz (the post boot configuration files) into the boot foler. Do not extract the files. Just copy lance-boot.tgz straight over. This will complete the setup for booting ESXi.

 

Configure the Midwife

 

All that’s left is to modify the configuration PowerShell files to match what you’d like in your environment. The first one up is esx-master.ps1. You should edit the top portion of the file to include the Virtual Center server name or IP as well as the administrator account and password.

 

# To run as current user:
$VC = “localhost”; $VCpassword = ‘vmware’
#$VC = ’172.10.0.120′ ; $VCpassword = ‘vmware’
$defaultESXPassword = ”
#$newESXPassword = $VCpassword
$newESXPassword = ‘vmware’ 

The last file you’ll need to edit is esx-profile.ps1. This one contains a few other entries that you’ll need to edit such as the NAS or iSCSI settings, the network settings, the datastore names, etc. It shouldn’t take too long to customize this file to your environment. Once you’re done it’s time to try a run-through.

 

Doing a PXE Boot

 

The first step is to start the TFTPd server. Simply launch TFTPd32 and you’ll be ready to go. The next step is to launch the midwife server process. Open a command prompt and type in “perl c:\midwife\configsvc.pl”.

 

 

Now boot your target server. If it’s connected to the same network as where TFTPd32 is running then you should eventually see the PXE boot menu on your host.

 

 

Leave the default choice selected and then you’ll start to see the files being downloaded from the TFTP server.

 

 

To finish things off you’ll see ESXi booting.

 

 

If everything goes according to plan you should see something very similar to the demo video from the beginning of this blog. I hope you enjoy the tip.

VN:F [1.9.3_1094]
Rating: 5.0/5 (4 votes cast)
VN:F [1.9.3_1094]
Rating: +4 (from 4 votes)
HOWTO: PXE Boot ESXi, 5.0 out of 5 based on 4 ratings

View Comments to “HOWTO: PXE Boot ESXi”

  1. Jason Boche Says:

    That’s great Mike. Thanks!

  2. JustinE Says:

    Mike,
    I’ve been looking forward to more information about PXE booting ESXi, however I have a couple reservations before I spend a few hours setting this up…
    Isn’t there a lot of stuff that’s missing when you boot ESXi this way? What if you try to install VMware tools on a VM running on this host? The ISO files are not available. What about downloading the VI Client from this box?
    I understand that this is meant for fun/test purposes only, but without being able to install VMware tools that’s a pretty big problem. Is there a way to direct the ESXi kernel to mount these off of a network location over NFS or something like that?

  3. Mike DiPetrillo Says:

    Not sure if you can point the box to another system for VMTools. However, you could always include the stuff from Partition 8 in the boot image. It will take a lot longer for the PXE boot to happen but at least you’ll have the VI Client and VMTools available.

  4. HarisB Says:

    Hi,

    This is critical – my environment is bunch of ESX 3.5i diskless servers, and I need to transfer windows.iso to all of them as part of PXE boot image so I can install vmtools. Any idea how to do this?

    Cheers

  5. TSoP Says:

    I have recently been hacking around with ESXi and PXE. I had earlier found that it is possible to create a custom ESXi installer similar to what is possible with Kickstart and ESX Classical. The problem has been that every time the PXE booted ESXi host reboots, it looses important parts of its identity and configuration. My recent discovery has been a very simple way to maintain state (between reboots) in these PXE booted hosts without having to resort to network transfers or other external boxes. This actually leverages functionality already present in ESXi (that is used when PXE boot is not performed). I have not yet tested this in a VC environment, however I believe strongly that this will work perfectly. At any rate, I will be trying that in the next week or so.

  6. HarisB Says:

    Hi,

    I use the method described here to achieve config preservation between reboots:
    http://docs.google.com/View?docid=ddcwgcd6_4fs6s7jcf

    However I still can’t include vmware tools in any way. I posted on vmware forum but got nowhere. Really strange considering all the buzz around PXE booting and ESXi.

    Any advices appreciated.

    Cheers

  7. TSoP Says:

    The problem with that technique, if I understand it correctly, is that it relies on a remote machine using the remote CLI to do the backup and then get the backup onto the pxe server. My technique relies on no other software packages or network transfers.

    As to the vmware tools, that should not be a problem. As I have already accomplished my VC testing, I expect to do Tools testing this week. When this is done I will provide what should be a final update.

  8. TSoP Says:

    Just a quick update.
    I tested this is a Virtual Center environment last week and it worked perfectly. Indeed what I discovered (and should have known) is that ESXi was designed to handle this scenario out of the box. ESXi seems to be about 99% of the way to complete stateful PXE functionality (for at least some configurations of the ESX host) out of the box. It does raise the question of why this functionality was not included in the shipping product.

    To be clear, no modifications to my technique were necesary at all.

  9. Vaughn Says:

    Mike, nice work! A quesiton around diskless servers?

    Can I connect to software connected storage, say iSCSI or NFS, and use this storage to run the ESXI instance, or must I use hardware connected disk (be it DAS or SAN)?

  10. Mike DiPetrillo Says:

    You should be able to use iSCSI or NAS. We're just using the image to boot the box. It's all loaded into memory so you don't need any local storage.

  11. Conrey.org » redhat buzztacular! Says:

    [...] stateless model drives reminds me of the ESXi automated installs. I think VMware should be doing more in this space as well, as it tackles the scalability issues [...]

  12. ESX/ESXi 3.5 Update 4 Released - PXE Boot ESXi Experimentally Supported | VM /ETC Says:

    [...] Wolf and Mike Dipetrillo both gave us glimpses of a PXE boot future for ESXi in the past. As both bloggers have described, [...]

  13. huiming Says:

    Can I run on vSphere environment ( ESXi 4) ? I meet with problem and I have added PBHOST=hostip:port after vmkboot.vgz and vmk.vgz.
    <snip>
    label visor-161011
    kernel ESX/visor-161011/mboot.c32
    append ESX/visor-161011/vmkboot.gz — ESX/visor-161011/vmk.gz PBHOST=172.16.22.75:3333 …
    ….
    <snip>
    I do not know how it works on ESXi 4.

  14. Mike DiPetrillo Says:

    Actually there's a new process integrated with vSphere and is being
    documented right now. Once it's done it will be in the docs. It's a
    lot nicer of a solution.

    Mike

  15. Marc Bertasius Says:

    Great post!! A colleague recommended I looked into PXE booting ESXi with a view to learning more about scripting and automation.

    Still very new to all this, and probably trying to run before I can walk but I've followed the guide step by step, and I've hit a bit of a jam. ESXi loads fine, but then it doesn't appear to be sending the birth message to the midwife.

    I've run network monitor and can't see anything coming back over port 3333. What I have noticed is that after the ESXi server has finished loading it sends another DHCP Discover packet. Is that right?

    Totally lost and trying to get my head round this :)

  16. Mike DiPetrillo Says:

    Usually this is because the PXE boot file doesn't have the right
    format for the append string. Check the spaces and the number of
    dashes against the blog notes and try again. Hopefully that will fix it.

  17. Marc Bertasius Says:

    Yeah, I had that issue originally. It would load the vmkernel but then PSOD. It was because I'd C&P'd the script and the dashes weren't being recognised. I'll delete the spaces/dashes and re-add and see what happens. Thanks for the quick response.

  18. Marc Bertasius Says:

    Nope, no joy :( It definitely says it's loading lance-boot.tgz but still no birth message.

  19. BJones Says:

    Thank you all for your excellent ESXi pxeboot instructions. The stateless configuration is great, but I am interested in an ESXi pxeboot setup that would write ESXi (with custom configuration parameters) to disk. Based on my testing and the posts on this site, it doesn't seem like there is a workable *stateful* solution yet. If anyone has this working, please let me know. Thanks!

  20. Mike DiPetrillo Says:

    Most people wanting stateful use a deployment tool such as Altiris.
    I'll see what I can dig up for a free(er) way to do this.

  21. Automating ESXi Builds - PXE and Custom CD - InterVirt VMware Says:

    [...] http://www.mikedipetrillo.com/mikedvirtualization/2008/11/howto-pxe-boot-esxi.html [...]

  22. Rupali Says:

    Hi Mike,
    A couple of quesions:
    #1. I am trying to do stateful PXE boot install for ESX 4.0. I am using “append vmkboot.gz — vmkernel.gz —sys.vgz — cim.vgz — ienviron.tgz — image.tgz — install.tgz” in my configuration. However, the image.tgz does not load entirely and hangs at a specific point. I had extracted the ISO as per the VMWare document. Can you point out what can be the probable cause of the image.tgz load failure? I tried giving mem=512M as well and also tried using the SYSLINUX mboot.c32 instead of the default ISO mboot.c32. But invain. Please help.
    #2. Is there any official document from VMWare for ESXi 4.0 stateful install via PXE boot?

    Thanks and Regards,
    Rupali

  23. Parul Says:

    Hey Marc,

    I am having the same issue, no birth message. Were you able to get through? I have provided the PBHOST and lance-boot.tgz in my boot file as below:
    append vmkboot.gz — vmkernel.gz PBHOST=10.128.132.5:3333 — sys.vgz — cim.vgz — ienviron.tgz — install.tgz — image.tgz — lance-boot.tgz ipappend 2

    My ESXi 4.0 installation halts for user inputs after loading all the modules including lance-boot.tgz.

    Any idea what can be missing?

  24. Mike DiPetrillo Says:

    Lots of things have changed with ESX 4 for doing this. There's a new technote coming out around VMworld timeframe I believe on how to set all of this up as well as an appliance that's already setup with everything to do this. I'm not the one working on that project but I would just wait for that technote if you can. Sorry I can't be any more help right now.

  25. Parul Says:

    Thanks!! Mike for replying though. Just an update so that if someone else out there is trying out ESXi 4.0, one change is to append the PBHOST after vmkboot.gz as below:
    append vmkboot.gz PBHOST=10.128.132.5:3333 — vmkernel.gz — sys.vgz — cim.vgz — ienviron.tgz — install.tgz — image.tgz — lance-boot.tgz ipappend 2

    That way I did get the birth message atleast. Now either I cn wait for new technote from VMworld or continue with configuring the Midwife scripts as per my environment requirement. :-)

  26. Dale S Says:

    Thank you. I am learning to hate ESXi because of the booting issues I have been having as of late.

  27. bwcook0 Says:

    Mike,
    Did the ESXi process for this ever come out? All I see is 'experimental' support and a technote that outline the same pxe boot process and no mention of configuration methods or anything. Thanks

  28. Mike DiPetrillo Says:

    Experimental support is all that's there right now. I hear something is targeted for later this year at this point. :(

  29. bwcook0 Says:

    Mike,
    Did the ESXi process for this ever come out? All I see is 'experimental' support and a technote that outline the same pxe boot process and no mention of configuration methods or anything. Thanks

  30. Mike DiPetrillo Says:

    Experimental support is all that's there right now. I hear something is targeted for later this year at this point. :(

  31. Bringing ESXi Back [Take 'em to the chorus] « Tony Pittman's Blog Says:

    [...] UCS platform that can move the identity of one server to another.  On the other hand, you can boot ESXi from PXE, stateless.  This is a complex system, but could potentially be an answer to boot-from-SAN if you [...]

Leave a Reply

blog comments powered by Disqus