Close

17th September 2017

Objective 7.3 – Troubleshoot vSphere Upgrades and Migrations

Onwards through section 7 moving to, Objective 7.3 – Troubleshoot vSphere Upgrades and Migrations.

As always, this article is linked to from the main VCP6.5-DCV Blueprint.

Happy Revision

Simon

Objective 7.3 – Troubleshoot vSphere Upgrades and Migrations

Collect upgrade diagnostic information

Collect Installation Logs for vCenter Server Appliance

  • Access the appliance shell.

If you have direct access to the appliance

  • Press Alt+F1.

To connect remotely

  • Use SSH or another remote console connection to start a session to the appliance.
  • Enter a user name and password that the appliance recognizes.
  • In the appliance shell , run the pi shell command to access the Bash shell.
  • In the Bash shell, run the vc-support.sh script to generate a support bundle.

This command generates a .tgz file in /var/tmp.

  • Export the generated support bundle to the user@x.x.x.x:/tmp folder.

scp /var/tmp/vc-etco-vm-vlan11-dhcp-63-151.eng.vmware.com-2014-02-28–21.11.tgz user@x.x.x.x:/tmp

  • Determine which firstboot script failed.

cat /var/log/firstboot/firstbootStatus.json

Collect Installation Logs by Using the Installation Wizard

You can use the Setup Interrupted page of the installation wizard to browse to the generated .zip file of the vCenter Server for Windows installation log files.

If the installation fails, the Setup Interrupted page appears with the log collection check boxes selected by default.

  • Leave the check boxes selected and click Finish.

The installation files are collected in a .zip file on your desktop, for example, VMware-VCS-logs-time-of-installation-attempt.zip, where time-of-installation-attempt displays the year, month, date, hour, minutes, and seconds of the installation attempt.

  • Retrieve the log files from the .zip file on your desktop.

Retrieve Installation Logs Manually

  • Navigate to the installation log file locations.

%PROGRAMDATA%\VMware\vCenterServer\logs directory, usually C:\ProgramData\VMware\vCenterServer\logs

%TEMP% directory, usually C:\Users\username\AppData\Local\Temp

The files in the %TEMP% directory include vc-install.txt, vminst.log, pkgmgr.log, pkgmgr-comp-msi.log, and vim-vcs-msi.log.

  • Open the installation log files in a text editor for examination.

Collect Database Upgrade Logs

You can retrieve the database upgrade logs after you complete the vCenter Server upgrade process.

  • Navigate to the database upgrade log locations.
  • Open the database upgrade logs in a text editor for examination.

Database Upgrade Locations

  • For pre-upgrade checks, review the %TEMP%\..\vcsUpgrade\vcdb_req.out file.

The vcdb_req.err file tracks any errors that were identified during the pre-upgrade phase.

  • For export details, review the %TEMP%\..\vcsUpgrade\vcdb_export.out file.

The vcdb_export.err file contains errors that were identified during the export phase of the upgrade.

  • For import details, review the ProgramData\Vmware\CIS\logs\vmware\vpx\vcdb_import.out file.

The vcdb_import.err file contains errors that were identified during the import phase of the upgrade process.

  • For in-place upgrade log details, review the ProgramData\Vmware\CIS\logs\vmware\vpx\vcdb_inplace.out file.

The vcdb_inplace.err file contains in-place upgrade errors.

Collect Logs to Troubleshoot ESXi Hosts

  • Enter the vm-support command in the ESXi Shell or through SSH.
  • Navigate to the /var/tmp/ directory.
  • Retrieve the log files from the .tgz file.

Recognize common upgrade and migration issues with vCenter Server and vCenter Server Appliance

The installation and upgrade precheck script runs tests to identify problems on the host machine that can cause an installation, upgrade, or migration to fail.

For interactive installations, upgrades, and migrations, the errors or warnings are displayed on the final panel of the installer, where you are asked to confirm or cancel the installation or upgrade. For scripted installations, upgrades, or migrations, the errors or warnings are written to the installation log file.

vSphere Update Manager provides custom messages for these errors or warnings. To see the original errors and warnings returned by the precheck script during an Update Manager host upgrade scan, review the Update Manager log file vmware-vum-server-log4cpp.log.

Error and Warning Codes That Are Returned by the Installation and Upgrade Precheck Script

64BIT_LONGMODESTATUS

The host processor must be 64-bit.

COS_NETWORKING

Warning. An IPv4 address was found on an enabled service console virtual NIC that has no corresponding address in the same subnet in the vmkernel. A separate warning appears for each such occurrence.

CPU_CORES

The host must have at least two cores.

DISTRIBUTED_VIRTUAL_SWITCH

If the Cisco Virtual Ethernet Module (VEM) software is found on the host, the test checks that the upgrade also contains the VEM software. The test also determines whether the upgrade supports the same version of the Cisco Virtual Supervisor Module (VSM) as the existing version on the host. If the software is missing or is compatible with a different version of the VSM, the test returns a warning. The result indicates which version of the VEM software was expected on the upgrade ISO and which versions, if any, were found. You can use ESXi Image Builder CLI to create a custom installation ISO that includes the appropriate version of the VEM software.

HARDWARE_VIRTUALIZATION

Warning. If the host processor doesn’t have hardware virtualization or if hardware virtualization is not turned on in the host BIOS, host performance suffers. Enable hardware virtualization in the host machine boot options. See your hardware vendor’s documentation.

MD5_ROOT_PASSWORD

This test checks that the root password is encoded in MD5 format. If a password is not encoded in MD5 format, it might be significant only to eight characters. In this case, any characters after the first eight are no longer authenticated after the upgrade, which can create a security issue.

MEMORY_SIZE

The host requires the specified amount of memory to upgrade.

PACKAGE_COMPLIANCE

vSphere Update Manager only. This test checks the existing software on the host against the software contained on the upgrade ISO to determine whether the host has been successfully upgraded. If any of the packages are missing or are an older version than the package on the upgrade ISO, the test returns an error and indicates which software was found on the host and which software was found on the upgrade ISO.

PARTITION_LAYOUT

You can upgrade or migrate software only if at most one VMFS partition on the disk is being upgraded and the VMFS partition must start after sector 1843200.

POWERPATH

This test checks for installation of EMC PowerPath software, consisting of a CIM module and a kernel module. If either of these components is found on the host, the test checks that matching components,such as CIM, vmkernel and module, also exist in the upgrade. If they do not exist, the test returns a warning that indicates which PowerPath components were expected on the upgrade ISO and which, if any, were found.

PRECHECK_INITIALIZE

This test checks that the precheck script can be run.

SANE_ESX_CONF

The /etc/vmware/esx.conf file must exist on the host.

SPACE_AVAIL_ISO

vSphere Update Manager only. The host disk must have enough free space to store the contents of the installer CD or DVD.

SPACE_AVAIL_CONFIG

vSphere Update Manager only. The host disk must have enough free space to store the legacy configuration between reboots.

SUPPORTED_ESX_VERSION

You can upgrade or migrate to ESXi 6.5 only from version 5.5 or 6.0 ESXi hosts.

TBOOT_REQUIRED

This message applies only to vSphere Update Manager upgrades. The upgrade fails with this error when the host system is running in trusted boot mode (tboot), but the ESXi upgrade ISO does not contain any tboot VIBs. This test prevents an upgrade that can make the host less secure.

UNSUPPORTED_DEVICES

Warning. This test checks for unsupported devices. Some PCI devices are not supported in ESXi 6.5.

UPDATE_PENDING

This test checks the host for VIB installations that require a reboot. This test fails if one or more such VIBs is installed, but the host has not yet been rebooted. In these conditions, the precheck script is unable to reliably determine which packages are currently installed on the host, so it might not be safe to rely on the rest of the precheck tests to determine whether an upgrade is safe.

If you encounter this error, restart the host and retry the upgrade.

Microsoft SQL Database Set to Unsupported Compatibility Mode Causes vCenter Server Installation or Upgrade to Fail

vCenter Server installation with a Microsoft SQL database fails when the database is set to compatibility mode with an unsupported version.

The following error message appears: The DB User entered does not have the required permissions needed to install and configure vCenter Server with the selected DB. Please correct the following error(s): %s

The database version must be supported for vCenter Server. For SQL, even if the database is a supported version, if it is set to run in compatibility mode with an unsupported version, this error occurs. For example, if SQL 2008 is set to run in SQL 2000 compatibility mode, this error occurs.

Make sure the vCenter Server database is a supported version and is not set to compatibility mode with an unsupported version.

Create/Locate VMware log bundles

When the vSphere Web Client is connected to the vCenter Server system, you can select hosts from which to generate and download system log files and the option to include vCenter Server and vSphere Web client logs.

To collect ESX/ESXi and vCenter Server diagnostic data:

  • Start the vSphere Web Client and log in to the vCenter Server system.
  • Under Inventory Lists, select vCenter Servers.
  • Click the vCenter Server that contains the ESX/ESXi hosts from which you want to export logs.
  • Click the Monitor tab and click System Logs.
  • Click Export System Logs.
  • Select the ESX/ESXi hosts from which you want to export logs.
  • Select the Include vCenter Server and vSphere Web Client logs option. This step is optional.
  • Click Next.
  • Select the system logs that are to be exported.
  • Select Gather performance data to include performance data information in the log files.
  • Click Next.
  • Click Generate Log Bundle. The Download Log Bundles dialog appears when the Generating Diagnostic Bundle task completes.
  • Click Download Log Bundle to save it to your local computer.
  • After the download completes, click Finish or generate another log bundle.

Using the Log Browser to view, search, and export Logs for troubleshooting

You can view, search, and export one or more vCenter Server and ESXi log files at a time using the log browser.

You can also export, manage, and view different log types.

Log in to vSphere Web Client and provide the credentials. When you are inside the Web Client, you have two ways to navigate to the Log Browser: Log Browser for a specific host or vCenter Server, Browse Log Files from Different Objects.

Log Browser for a specific host or vCenter Server

  • Navigate to the host or vCenter Server that contain the logs you want to retrieve.
  • Click the Monitor tab.
  • Click Log Browser.
  • If no logs for the host or vCenter Server are available, click Retrieve now to retrieve the logs for that object.

    The retrieved logs are based on a current snapshot of the system. Retrieving logs can take a few minutes. You can perform other tasks while the logs are being retrieved.

  • Click Refresh to retrieve newer logs.
  • Select the type of log you want to browse.

    The log displays in the browser.

Browse Log Files from Different Objects

You can browse multiple log files coming from different objects within the Log Browser at the same time. This is helpful if you want to simultaneously compare log files.

  • From the vSphere Web Client home, click Log Browser.
  • Click the View tab.
  • Select an object (ESXi host or vCenter Server) to view its logs.
  • Open a new browser window by selecting Actions > New Browser Window and select another object to view its logs.

Determine alternative methods to upgrade ESXi hosts in event of failure

Upgrade Hosts Interactively

To upgrade ESXi 5.5 hosts or ESXi 6.0 hosts to ESXi 6.5, you can boot the ESXi installer from a CD, DVD, or USB flash drive.

Installing or Upgrading Hosts by Using a Script

You can quickly deploy ESXi hosts by using scripted, unattended installations or upgrades. Scripted installations or upgrades provide an efficient way to deploy multiple hosts.

PXE Booting the ESXi Installer

You can use the preboot execution environment (PXE) to boot a host. Starting with vSphere 6.0, you can PXE boot the ESXi installer from a network interface on hosts with legacy BIOS or using UEFI.

Upgrading Hosts by Using esxcli Commands

By using vSphere CLI, you can upgrade a ESXi 5.5 host or ESXi 6.0 host to version 6.5 and update or patch ESXi 5.5, ESXi 6.0, and ESXi 6.5 hosts.

Configure vCenter Server logging options

  • Log into the vSphere Web Client with administrative privileges
  • Under Resources, select vCenter Servers
  • Click the vCenter Server to update the level of logging
  • Select the Settings tab in right hand navigation screen and choose General
  • From the General  tab click Edit
  • The Edit vCenter Server Settings dialog will be displayed. Select Logging Settings
  • Select the level of logging from the Logging Options dropdown.
  • Click OK when finished

The available options are:

None (Disable Logging)

Turns off logging

Error (Errors Only)

Displays only error log entries

Warning (Errors and Warnings)

Displays warning and error log entries

Info (Normal Logging – Default)

Displays information, error, and warning log entries

Verbose (Verbose)

Displays information, error, warning, and verbose log entries

Trivia (Extended Verbose)

Displays information, error, warning, verbose, and trivia log entries