Objective 6.4 – Troubleshoot Storage Performance and Connectivity

Knowledge - Identify logs used to troubleshoot storage issues

• /var/log/vmkernel.log - Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
• /var/log/storagerm.log - Logs StorageIO events
• /var/log/vobd.log- VMkernel Observation events

Knowledge - Describe the attributes of the VMFS-5 file system

VMFS5 has the following improvements over VMFS3:

• Support of greater than 2TB storage devices for each VMFS extent.
• Increased resource limits such as file descriptors.
• Standard 1MB file system block size with support of 2TB virtual disks.
• Support of greater than 2TB disk size for RDMs in physical compatibility mode.
• Scalability improvements on storage devices that support hardware acceleration.
• Default use of hardware assisted locking, also called atomic test and set (ATS) locking, on storage devices that support hardware acceleration.
• Ability to reclaim physical storage space on thin provisioned storage devices.
• Online in-place upgrade process that upgrades existing datastores without disrupting hosts or virtual machines that are currently running.

Skills and Abilities - Use Esxcli To Troubleshoot Multipathing And PSA-Related Issues

All storage commands reside and can be built out from

   esxcli storage

To view and modify claimrules

   esxcli storage core claimrule

   esxcli storage core claimrule list

To view all paths for a LUN

   esxcli storage core path list --device=<device id>

(this example is a single path iSCSI LUN,  for multipath this entry would repeat for each path)

Skills and Abilities - Use esxcli To Troubleshoot VMkernel Storage Module Configurations

To view loaded plugins and register missing

   esxcli storage core plugin

   esxcli storage core plugin list

Skills and Abilities - Use esxcli to troubleshoot iSCSI related issues

All storage commands reside and can be built out from

   esxcli iscsi

Skills and Abilities - Troubleshoot NFS mounting and permission issues
To mount a valid NFS exported volume
    esxcfg-nas --add <datastore name> --host <NFS server IP\name> --share <NFS export path>

If the NFS fails to mount some things to troubleshoot might include,

• Can the NFS server name\IP be connected to from the client,  you can use vmkping <name\ip>
• Are any firewall ports blocking NFS traffic
• Is the export details correct
• Is NFS presenting as NFS v3
• Is NFS allowing root access
• Is root squash disabled
• NFS servers can be configured to allow access to specific IPs or netmasks, ensure your host IP is added to the security export for NFS to mount

Skills and Abilities - Use esxtop/resxtop and vscsiStats to identify storage performance issues
Within resxtop\esxtop
d: disk adapter

KAVG - latency generated by the ESXi kernel
DAVG - latency generated by the device driver
QAVG - latency generated from the queue
GAVG - latency as it appears to the guest VM (KAVG + DAVG)

u: disk device

QUED – LUN queue depth (amount of I/Os the LUN can queue)
%USD – percentage of the queue depth being actively used by the ESXi kernel

Using vscsiStats we can view the MS response time in histogram format to see how much latency is occurring to the VM.

To start gathering vcscsiStats data
   vscsiStats -s

To stop gathering vcscsiStats data
   vscsiStats -x

To view data
   vscsiStats -p

Skills and Abilities - Configure And Troubleshoot VMFS Datastores Using vmkfstools
vmkfstools can be used to manage datastores and VM disk files,  some options for datastores management we can Create, Grow, Span, and upgrade VMFS3-5
 

Albeit not for VCAP as that is 5.0 but worth mentioning as well in 5.1 vmkfstools -y was introduced to free unused high watermark SAN blocks,  in 5.5 this was replaced with esxcli storage vmfs unmap. This is essential to recover SAN space if you use LUNs which are Thin provisioned at the SAN tier.

Skills and Abilities - Troubleshoot snapshot and re-signaturing issues
 All snapshot commands are under the esxcli storage vmfs snapshot namespace,  from here you can list,  resignature and mount any LUNs in this state.

 
Skills and Abilities - Analyze log files to identify storage and multipathing problems

/var/log/vmkernel.log - Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
/var/log/storagerm.log - Logs StorageIO events
/var/log/vobd.log- VMkernel Observation events