After a long struggle of 48 hours, I could finally able to install the Apple's latest Mac OS X El Capitan v10.11 on a Virtual Machine (VM). The experiment was conducted on Vmware ESXi 6.0 virtualization platform running over physical IBM Server X3650 M3. Following are the installation steps:
Vmware ESXi 6.0 doesn’t support OS X out of the box.
The maximum version of Mac OS X, OS X, or macOS supported by each G3 and later Mac follows. For complete specs on a particular system, click the name of the Mac. For all Macs that are compatible with a specifc maximum supported version of Mac OS X - courtesy of EveryMac.com's Ultimate Mac Sort - click the OS of interest.
Grab the El Capitan installer app from the Apple Appstore. Note, you need a real Mac in order to download and convert the El Capitan installation disk. Once downloaded minimize or close the installer.
Open the terminal and change directory to /tmp using command cd /tmp
Create a new script file: nano ElCapitan.sh
Paste the following into the script file and save:
#!/bin/bash
# Mount the installer image
hdiutil attach /Applications/Install OS X El Capitan.app/Contents/SharedSupport/InstallESD.dmg -noverify -nobrowse -mountpoint /Volumes/install_app
# Create the ElCapitan Blank ISO Image of 7316mb with a Single Partition - Apple Partition Map
hdiutil create -o /tmp/ElCapitan.cdr -size 7316m -layout SPUD -fs HFS+J
# Mount the ElCapitan Blank ISO Image
hdiutil attach /tmp/ElCapitan.cdr.dmg -noverify -nobrowse -mountpoint /Volumes/install_build
# Restore the Base System into the ElCapitan Blank ISO Image
asr restore -source /Volumes/install_app/BaseSystem.dmg -target /Volumes/install_build -noprompt -noverify -erase
# Remove Package link and replace with actual files
rm /Volumes/OS X Base System/System/Installation/Packages
cp -rp /Volumes/install_app/Packages /Volumes/OS X Base System/System/Installation/
# Copy El Capitan installer dependencies
cp -rp /Volumes/install_app/BaseSystem.chunklist /Volumes/OS X Base System/BaseSystem.chunklist
cp -rp /Volumes/install_app/BaseSystem.dmg /Volumes/OS X Base System/BaseSystem.dmg
# Unmount the installer image
hdiutil detach /Volumes/install_app
# Unmount the ElCapitan ISO Image
hdiutil detach /Volumes/OS X Base System/
# Convert the ElCapitan ISO Image to ISO/CD master (Optional)
hdiutil convert /tmp/ElCapitan.cdr.dmg -format UDTO -o /tmp/ElCapitan.iso
# Rename the ElCapitan ISO Image and move it to the desktop
mv /tmp/ElCapitan.iso.cdr ~/Desktop/ElCapitan.iso
Now ensure that the script file has execute permissions by running chmod +x ElCapitan.sh and run with sh ./ElCapitan.sh. This will take some time, so just be patient. Once done it should have created a file called ElCapitan.iso on your desktop.
The next step should be fairly simple if you have used ESXi before. As usual open your datastore and transfer the iso to wherever you normally store your installation files. Then create a new vm, select Typical and you should be able to select Other and then Apple Mac OS X 10.10 (64-bit). Complete the wizard, mount the ISO and install Mac OS X as usual. Please note that you must format the hard drive using the disk utility before it will be visible in the install wizard.
It’s very important to install the VMware Tools as soon as possible after the install has finished. Without it Mac OS X might not awake from sleep and in general be very unreliable. I also recommend to turn off sleep and the screensaver in settings as these may cause issues.
Finally, reboot and start enjoying your virtual Mac on non-Apple hardware.