Objective 9.2 – Install ESXi Hosts Using Auto Deploy

Knowledge - Identify Auto Deploy Requirements
Auto Deploy Environment Requirements

• Do not use VLAN tagged networks at the boot NIC
• 2GB of disk space (minimum) for Auto Deploy repository
• DHCP server in the environment
• TFTP server in the environment
• Set up a remote Syslog server (optional). Leverage vSphere Syslog Collector
• Set up ESXi Dump Collector and configure hosts to leverage

Software Requirements

• Microsoft .NET 2.0
• Microsoft Powershell 2.0
• VMware vSphere PowerCLI
• Set up a remote Syslog server (optional but recommended). Leverage vSphere Syslog Collector
• Set up ESXi Dump Collector (optional but recommended) and configure hosts to leverage

Skills and Abilities - Install The Auto Deploy Server

 

Skills and Abilities - Utilize Auto Deploy Cmdlets to deploy ESXi hosts
Form your image using image builder,  then create a new deploy rule using the image when you do this the contents of the image are compared to the contents of the Autodepoly cache and uploads any required packages which are not already in the Autodepoly cache.

Skills and Abilities - Configure Bulk Licensing

# Load the vSphere License snapin
Add-PSSnapin VMware.VimAutomation.License

# Connect to the vCenter Server system you want to use and bind the associated license manager to a variable.
Connect-VIServer 192.168.1.89

# Retrieve the datacenter in which the hosts for which you want to use the bulk licensing feature are located.
$HostContainer = Get-Datacenter -Name VCAP

# Bind the associated license manager to a variable
$LicenseDataManager = Get-LicenseDataManager

# Create a new LicenseData object and a LicenseKeyEntry object with associated type ID and license key.
$LicenseData = New-Object VMware.VimAutomation.License.Types.LicenseData
$LicenseKeyEntry = New-Object Vmware.VimAutomation.License.Types.LicenseKeyEntry
$LicenseKeyEntry.TypeId = "vmware-vsphere”
$LicenseKeyEntry.LicenseKey = "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX"

# Associate the LicenseKeys attribute of the LicenseData object you created in step 3 with the LicenseKeyEntry object.
$LicenseData.LicenseKeys += $LicenseKeyEntry

# Update the license data for the data center with the LicenseData object and verify that the license is associated with the host container.
$LicenseDataManager.UpdateAssociatedLicenseData($hostContainer.Uid, $licenseData)
$LicenseDataManager.QueryAssociatedLicenseData($hostContainer.Uid)

# Provision one or more hosts with Auto Deploy and assign them to the data center or to the cluster that you assigned the license data to.

Skills and Abilities - Provision/Re-provision ESXi Hosts Using Auto Deploy

1. Install AutoDeploy.
2. Configure the TFTP server.
3. In a vSphere Client connected to the vCenter Server system that Auto Deploy is registered with, click Home in the navigation bar and select Auto Deploy in the Administration tab to display the Auto Deploy page.
4. Click Download TFTP ZIP to download the TFTP configuration file and unzip the file to the directory in which your TFTP server stores files.
5. Set up your DHCP server to point to the TFTP server on which the TFTP ZIP file is located.
6. Specify the TFTP Server's IP address in DHCP option 66 (frequently called next-server).
7. Specify the boot file name, which is undionly.kpxe.vmw-hardwired in the DHCP option 67 (frequently called boot-filename).
8. Set each host you want to provision with Auto Deploy to network boot or PXE boot, following the manufacturer's instructions.
9. Create custom image if required.
10. Write a rule that assigns an image profile to hosts.

Skills and Abilities - Configure An Auto Deploy Reference Host
In an environment where no state is stored on the host, a reference host helps you set up multiple hosts with the same configuration. You configure the reference host with the logging, coredump, and other settings that you want, save the host profile, and write a rule that applies the host profile to other hosts as needed. You can configure the storage, networking, and security settings on the reference host and set up services such as syslog and NTP.

This is done by associating a Host Profile to a deploy rule
     -Item my_host_profile

Objective 9.1 – Install ESXi hosts with custom settings

Knowledge - Identify Custom Installation Options
Custom installation options allow you to create standard installation ISO files including any required drivers,  and or custom kernel settings.  There are other advanced boot loader options to facilitate custom kickstart scripted lights out installations.

Knowledge - Identify ESXi Image Builder Requirements

•     Microsoft .NET 2.0
•     Microsoft PowerShell
•     vSphere PowerCLI

Skills and Abilities - Create/Edit Image Profiles
The flow of what you are trying to create

A short video of adding FusionIO device drivers to ESXi 5 and forming out to iso

Commands pasted as test as used during the video which add FusionIO drivers to ESXi 5 base.

Skills and Abilities - Install/Uninstall Custom Drivers
# Import VMware and VMware Image Builder SnapIn's

Add-PSSnapin VMware.VimAutomation.Core
Add-PSSnapin VMware.ImageBuilder

# Download the offline bundle version of ESXi from vmware.com
# import the VMware image into image builder builder
Add-EsxSoftwareDepot "C:\VMs\Home Lab Setup\CustomDepot\VMware-ESXi-5.0.0-469512-depot.zip"

# Download the offline driver bundle you want to add from vendor example HP IO Accelerator
# and import the VMware image into image builder builder
Add-EsxSoftwareDepot "C:\VMs\Home Lab Setup\CustomDepot\ioSphere3.6.1_ESXi-5.x\Utilities\fusionio-cimprovider-esxi5-bundle-3.6.1-114.zip"

# View contents of depot
Get-EsxSoftwareDepot

# View all images in depot
Get-EsxImageProfile

# Clone an existing image to form a new custom image
New-EsxImageProfile -CloneProfile ESXi-5.0.0-469512-standard -Name ESXi-5.0.0-469512.FusionIO -AcceptanceLevel PartnerSupported -Vendor HP

# Find name of packages
Get-EsxSoftwarePackage | where {$_.Vendor -eq "Fusion-io"} | Format-Table -AutoSize

# Add additional packages into new custom image
Add-EsxSoftwarePackage -ImageProfile ESXi-5.0.0-469512.FusionIO -SoftwarePackage fio

# Export image as ISO
Export-EsxImageProfile -ImageProfile ESXi-5.0.0-469512.FusionIO -ExportToIso -FilePath "C:\VMs\Home Lab Setup\CustomDepot\ESXi-5.0.0-469512.FusionIO.iso"

Skills and Abilities - Configure Advanced Boot Loader Options
When booting ESXi you can enter boot options by pressing tab,  this allows you to modify from default mboot.c32 -c boot.cfg.

 

 A little later you can also interrupt the boot loader by pressing Shift+O

At the runweasel command prompt,

Point to any bespoke kickstart file you have created, this can be hosted for access via http \ ftp,  and to used by specifying using ks= option to specify the location of the installation script.
    ks=protocol://<serverpath>
    ks=file://etc/vmware/weasel/ks.cfg  (default)

Skills and Abilities - Configure Kernel Options


ESXi uses the common mboot.c32 boot loader during installation. The boot.cfg file contains the Kernel path, Kernel options and boot modules used by mboot.c32.

As above you can interupt the boot loader and point to an alternate boot.cfg.

Skills and Abilities - Given A Scenario, Determine When To Customize A Configuration

Many deployment tools allow PXE boot and deploy OS by way of custom kickstart files if you use such a tool to deploy other OS you can easily start deploying ESXi via kickstart method.

A common requirement for building custom images, is when you are using newly released hardware, or if you are building a home lab on some unusual hardware drivers.

Objective 8.2 – Administer vSphere Using the vSphere Management Assistant

Knowledge - Identify vMA prerequisites
vMA is a 64bit virtual appliance, to run this you require an ESXi host able to run 64bit VM, 3GB of disk and 1vCPU and 600MB of vRAM

Knowledge - Identify vMA specific commands
While vMA can be used to issue commands to run in the context of each ESXi and vCenter host,  it has specific commands for managing itself,  for example adding \ removing hosts.

Knowledge - Determine when vMA is needed

vMA is a Linux-based virtual machine that is pre-installed with a command-line interface and select third-party agents needed to manage your vSphere infrastructure. vMA includes the vSphere SDK for Perl and the vSphere Command-Line Interface (vSphere CLI). vMA also includes an authentication component named vi-fastpass. vi-fastpass allows direct connection to established target servers without user intervention.

It would be used for running centrally PowerCLI scripts within your estate, it is useful in a *nix only environment where Powershell cannot be used.

 
Skills and Abilities - Install and configure vMA

 

Skills and Abilities - Add/Remove target servers

Connect to vMA using using SSH client and login as vi-admin

To add server
sudo vifp addserver <servername>

To remove server
sudo vifp removeserver <servername>

It is the same commands to add and remove vCenter server as ESXi host

 

 
Skills and Abilities - Perform updates to the vMA
Logon to vMA via web browser change to Update tab

Skills and Abilities - Use vmkfstools to manage VMFS datastores
To issue commands to remote ESXi servers you first connect within vMA
     vifptarget --set 192.168.1.84

You can then issue commands as though you were using PowerCLI interface for managing datastores with vmkfstools or indeed anything else.

To disconect
     vifptarget --clear

Skills and Abilities - Use vmware-cmd to manage VMs
You can then issue commands as though you were using PowerCLI interface for managing VMs with vmware-cmd command set.

Skills and Abilities - Use esxcli to manage ESXi Host configurations
You can then issue commands as though you were using PowerCLI interface for managing hosts with esxcli.

Skills and Abilities - Troubleshoot common vMA errors and conditions
 vMA provides the vma-support script that enables you to collect various system configuration information and other logs. You can run this script by issuing the following command:

     sudo vma-support

The script generates the information and log bundle and appends it to the vmware.log file on the ESXi host on which vMA is deployed.

Objective 7.2 – Configure and Maintain the ESXi Firewall

Knowledge - Identify esxcli Firewall Configuration Commands

Knowledge - Explain the three firewall security levels

• High Security (Default) – Firewall is configured to block all incoming and outgoing traffic, except for ports 22,123,427,443,902,5989, and 5988. These are ports used for basic ESXi communication
• Medium Security – All incoming traffic is blocked, except on the default ports and any ports you specifically open. Outgoing traffic is not blocked
• Low Security – There are no ports blocked on either incoming or outgoing traffic. This setting is equivalent to removing the fireall

Skills and Abilities - Enable/Disable Pre-Configured Services
Navigate GUI to ESX Host -> Configuration -> Security Profile -> Firewall properties then check \ uncheck any pre-configured as required.

Skills and Abilities - Configure Service Behavior Automation

Skills and Abilities - Open/Close Ports In The Firewall
To open and close ports,  check and uncheck in the GUI,  apply configuration across hosts using Host Profiles.

Skills and Abilities - Create A Custom Service
To create a custom service is not intuitive,  you need to create a xml file within /etc/vmware/firewall folder and format the file contents like copied from services.xml file for other service.

For my test I created, to create a rule for port 999 both ways, TCP and UDP.

To add the rule I can run
 esxcli network firewall refresh

To note this creates a rule which is not reboot persistent so you can test,  if you want to add persistent you can update the services.xml to do this you need to modify its security attributes like.


Skills and Abilities - Set Firewall Security Level

Objective 6.5 – Troubleshoot vCenter Server and ESXi Host Management

Knowledge - Identify CLI Commands And Tools Used To Troubleshoot Management Issues
CLI tools you can use are those to manage network switch configuration and VMkernel IP settings as well as service restart tools.

The ESXi host console can also be used to manage network connectivity issues.

Skills and Abilities - Troubleshoot vCenter Server service and database connection issues
Manage Virtual Center Appliance Services

• View status of all services service –-status-all
  
• View status of VPX daemon service vmware-vpxd status
• Restart service vmware-vpxd restart

For Windows Virtual Center,  manage services using services.msc

In the event of a database issue preventing the Virtual Center Service starting or error events are logged to

%ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\vpxd.log
 
Skills and Abilities - Troubleshoot the ESXi firewall

The firewall ruleset can be viewed and managed via the GUI or via CLI

esxcli network firewall

To diagnose firewall related issues it is important to understand what port specific traffic is communicating via

Skills and Abilities - Troubleshoot ESXi host management and connectivity issues

To restart host management agents
/etc/init.d/hostd restart
/etc/init.d/vpxa restart

If you have multiple VMkernel connections and want to reset only a specific
esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0

To restart all management agents
services.sh restart

Skills and Abilities - Determine The Root Cause Of a vSphere Management Or Connectivity Issue

ESX host and vCenter communicate over TCP,  to test base connectivity you can use vmkping from host to the IP of vCenter.

Communication often requires A and PTR records to be correctly reflecting name and IP combination. If you have multiple VMkernel ports for things like network attached storage you will need multiple DNS records with varying names to reflect each IP DNS record.

Skills and Abilities - Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor an environment
From ESX console you can adjust network uplinks, VLAN info,  set IP address and test connectivity to the local gateway.

Objective 6.4 – Troubleshoot Storage Performance and Connectivity

Knowledge - Identify logs used to troubleshoot storage issues

• /var/log/vmkernel.log - Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
• /var/log/storagerm.log - Logs StorageIO events
• /var/log/vobd.log- VMkernel Observation events

Knowledge - Describe the attributes of the VMFS-5 file system

VMFS5 has the following improvements over VMFS3:

• Support of greater than 2TB storage devices for each VMFS extent.
• Increased resource limits such as file descriptors.
• Standard 1MB file system block size with support of 2TB virtual disks.
• Support of greater than 2TB disk size for RDMs in physical compatibility mode.
• Scalability improvements on storage devices that support hardware acceleration.
• Default use of hardware assisted locking, also called atomic test and set (ATS) locking, on storage devices that support hardware acceleration.
• Ability to reclaim physical storage space on thin provisioned storage devices.
• Online in-place upgrade process that upgrades existing datastores without disrupting hosts or virtual machines that are currently running.

Skills and Abilities - Use Esxcli To Troubleshoot Multipathing And PSA-Related Issues

All storage commands reside and can be built out from

   esxcli storage

To view and modify claimrules

   esxcli storage core claimrule

   esxcli storage core claimrule list

To view all paths for a LUN

   esxcli storage core path list --device=<device id>

(this example is a single path iSCSI LUN,  for multipath this entry would repeat for each path)

Skills and Abilities - Use esxcli To Troubleshoot VMkernel Storage Module Configurations

To view loaded plugins and register missing

   esxcli storage core plugin

   esxcli storage core plugin list

Skills and Abilities - Use esxcli to troubleshoot iSCSI related issues

All storage commands reside and can be built out from

   esxcli iscsi

Skills and Abilities - Troubleshoot NFS mounting and permission issues
To mount a valid NFS exported volume
    esxcfg-nas --add <datastore name> --host <NFS server IP\name> --share <NFS export path>

If the NFS fails to mount some things to troubleshoot might include,

• Can the NFS server name\IP be connected to from the client,  you can use vmkping <name\ip>
• Are any firewall ports blocking NFS traffic
• Is the export details correct
• Is NFS presenting as NFS v3
• Is NFS allowing root access
• Is root squash disabled
• NFS servers can be configured to allow access to specific IPs or netmasks, ensure your host IP is added to the security export for NFS to mount

Skills and Abilities - Use esxtop/resxtop and vscsiStats to identify storage performance issues
Within resxtop\esxtop
d: disk adapter

KAVG - latency generated by the ESXi kernel
DAVG - latency generated by the device driver
QAVG - latency generated from the queue
GAVG - latency as it appears to the guest VM (KAVG + DAVG)

u: disk device

QUED – LUN queue depth (amount of I/Os the LUN can queue)
%USD – percentage of the queue depth being actively used by the ESXi kernel

Using vscsiStats we can view the MS response time in histogram format to see how much latency is occurring to the VM.

To start gathering vcscsiStats data
   vscsiStats -s

To stop gathering vcscsiStats data
   vscsiStats -x

To view data
   vscsiStats -p

Skills and Abilities - Configure And Troubleshoot VMFS Datastores Using vmkfstools
vmkfstools can be used to manage datastores and VM disk files,  some options for datastores management we can Create, Grow, Span, and upgrade VMFS3-5
 

Albeit not for VCAP as that is 5.0 but worth mentioning as well in 5.1 vmkfstools -y was introduced to free unused high watermark SAN blocks,  in 5.5 this was replaced with esxcli storage vmfs unmap. This is essential to recover SAN space if you use LUNs which are Thin provisioned at the SAN tier.

Skills and Abilities - Troubleshoot snapshot and re-signaturing issues
 All snapshot commands are under the esxcli storage vmfs snapshot namespace,  from here you can list,  resignature and mount any LUNs in this state.

 
Skills and Abilities - Analyze log files to identify storage and multipathing problems

/var/log/vmkernel.log - Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
/var/log/storagerm.log - Logs StorageIO events
/var/log/vobd.log- VMkernel Observation events

Objective 6.3 – Troubleshoot Network Performance and Connectivity

Knowledge - Identify vCLI commands and tools used to troubleshoot vSphere networking configurations

• esxcfg-vswitch
• esxcfg-nics
• esxcfg-vmknic
• esxcfg-route
• esxcli network

Knowledge - Identify logs used to troubleshoot network issues

• DHCP issues – /var/log/dhclient.log
• Networking driver and device issues – /var/log/vmkernel.log
• vCenter issues – /var/log/vpxa.log

Skills and Abilities - Utilize net-dvs To Troubleshoot vSphere Distributed Switch Configurations
./usr/lib/vmware/bin/net-dvs

Lists a very verbose level of detail about packets passing through each vDS

Skills and Abilities - Utilize vSphere CLI Commands To Troubleshoot ESXi Network Configurations
Real world usages I have come across for the four primary vCLI network configuration tools, to note these tools can be used for many more things in addition to the below.

On newly built servers if NICs cabled incorrectly default vSwitch links to invalid uplink for this esxcfg-vswitch is useful to add and remove uplinks,  similarly if no native VLAN is set within a trunk esxcfg-vswitch can be used to set a portgroup VLAN to allow access.  If a network port is failing to negotiate correctly esxcfg-nics can be used to force various speeds and settings until successful negotiation occurs. If you want to add or manage VMkernel ports, esxcfg-vmknic can be useful.  To view or set a default gateway or static routing esxcfg-route I had occasion to use static routing once as my iSCSI \ NFS SAN was located on remote subnets (720 Mpps link but still routed) to ensure these routed correctly I defined static routes between end the networks.

Skills and Abilities - Troubleshoot Private VLANs
 
Skills and Abilities - Troubleshoot vmkernel related network configuration issues
/var/log/vmkernel.log

vmkping can be used to test the vmkernel networking

Skills and Abilities - Troubleshoot DNS and routing related issues
vSphere 5.0 and later rely heavily on both A and PTR DNS records being in place and correct. Prior to starting any troubleshooting ensure nslookup can resolve correct name to IP (A record) and IP to name (PTR record) nslookup is included as command on ESXi CLI.

Unlikely but possible,  DNS can be disabled via the ESXi Firewall.

 

For troubleshooting routing, esxcfg-route can be used to view the routing table also traceroute and ping can be used to generate traffic to establish the issue.

If you need to capture the traffic packets this can be done by following this kb

Skills and Abilities - Use esxtop/resxtop To Identify Network Performance Problems
The best reference for esxtop threshold Yellow Bricks summarized, to note network performance can manifest as CPU performance issue if incorrect MTU, speed or driver incorrect.  So to check CPU and network counters.

Skills and Abilities - Analyze Troubleshooting Data To Determine If The Root Cause For A Given Network Problem Originates In The Physical Infrastructure or vSphere Environment
 
Skills and Abilities - Configure and administer Port Mirroring

Skills and Abilities - Utilize Direct Console User Interface (DCUI) and ESXi Shell to troubleshoot, configure, and monitor ESXi networking
If your ESXi host networking has failed,  connecting to DCUI is useful way to get the configuration working you can

• Add \ remove uplinks from vSwitches
• Change IP settings
• Revert your from broken vDS to vSS
• Test management network connectivity
• Look at log files