Batching mfit guest data collection

Stay organized with collections Save and categorize content based on your preferences.

This tutorial walks you through the process of discovering and collecting data on a batch of VMware VMs. Large batches of VMs may be difficult to monitor and troubleshoot. We recommend batching into smaller groups of less than 1,000.

Objectives

  • Install the mfit CLI
  • Run a VMWare inventory discovery
  • Generate a CSV file with all the discovered VMs
  • Split the CSV file into multiple CSV files containing a subset of the VMs
  • Run guest data collection on all the VMs in a CSV file
  • Generate a fit assessment report

Costs

This tutorial uses on-prem resources and incurs no Google Cloud costs.

Before you begin

  1. Prepare a Linux workstation to install and run the mfit CLI. The Linux workstation should have the following minimum requirements:
    • Linux kernel versions 2.6.23 or later
    • 4 GB RAM and 10 GB disk space
    • Network connectivity to vCenter and ESX hosts
  2. Ensure that you have a vSphere user with the necessary permissions:
    • Read privileges to VMs
    • Read privileges to all the ESX hosts
    • Guest operation modifications
    • Guest operation program execution
    • Guest operation queries

Install the mfit CLI

  1. SSH to the mfit Linux workstation:

    ssh WORKSTATION_IP

  2. Create a directory for the mfit tool on a Linux machine:

    mkdir mfit
cd mfit

  3. Download the assessment tool and make it executable:

    curl -O "https://mfit-release.storage.googleapis.com/$(curl -s https://mfit-release.storage.googleapis.com/latest)/mfit"
chmod +x mfit

  4. Add the assessment tool to your shell path:

    PATH=$PATH:`pwd`

Perform a VMware discovery

  1. To perform the discovery, run the following command:

    ./mfit discover vsphere -u USERNAME --url https://VSPHERE_URL

The output of the discover command should look like the following:

[+] Found 528 VMs
Collecting data...
528 / 528 [---------------------------------------------------] 100.00% 5 p/s
[✓] Collection completed.

Generate a CSV file with all the discovered VMs

To generate a CSV file, use the export script from the Migrate to Containers repo on GitHub.

scripts/mFit/mfit_discovery_export_guest_tools.sh 

vms=$(mfit report --format csv | tail -n +2 | awk -F ',' '{; print $2";"$3";"$4";;;"}')
printf "%s\n" "Name;Platform VM ID;OS Family;IP;USERNAME;PASSWORD"
printf "%s\n" "$vms"
The script uses the mfit report command to generate a CSV file containing all the VMs that were discovered in the previous step.

  1. Download the script by running the commands:

    curl -O https://raw.githubusercontent.com/GoogleCloudPlatform/migrate-to-containers/main/scripts/mFit/mfit_discovery_export_guest_tools.sh
chmod +x mfit_discovery_export_guest_tools.sh

  2. Execute the script and save the output to a file by running the command:

    ./mfit_discovery_export_guest_tools.sh > VMS_CSV_FILE

Split the CSV file into multiple CSV files with a subset of the VMs

To split the large CSV file into smaller CSV files, you will use the splitter script from the Migrate to Containers repo on GitHub.

scripts/mFit/csv_splitter.sh 
tail -n +2 $1 | split -d -l $num_of_lines - $2
for file in $2*
do
    head -n 1 $1 > tmp_file
    cat "$file" >> tmp_file
    mv -f tmp_file "$file"
done

  1. Download the script by running the commands:

    curl -O https://raw.githubusercontent.com/GoogleCloudPlatform/migrate-to-containers/main/scripts/mFit/csv_splitter.sh
chmod +x csv_splitter.sh

  2. Execute the script by running the command:

    ./csv_splitter.sh VMS_CSV_FILE CSV_FILES_PREFIX LINES_PER_FILE VMS_CSV_FILE CSV_FILES_PREFIX

The arguments for the csv_splitter script are as followed:

VMS_CSV_FILE - The input CSV file to split

CSV_FILES_PREFIX - The prefix for the generated CSV files

LINES_PER_FILE - The number of lines per generated CSV file. Defaults to 10 if not supplied

The script will generate various CSV files, you can list the files by running the command:

   ls -lrt CSV_FILES_PREFIX*

Run guest data collection on all the VMs in a CSV files

This step is done repeatedly for each of the CSV files that were generated in the previous step. To run the collection for the VMs in a CSV file, use the collection script from the Migrate to Containers GitHub repo.

scripts/mFit/mfit_collect_vms_guest_tools.sh 

url_regex='^https?://[-A-Za-z0-9\+&@#/%?=~_|!:,.;]*[-A-Za-z0-9\+&@#/%=~_|]\.[-A-Za-z0-9\+&@#/%?=~_|!:,.;]*[-A-Za-z0-9\+&@#/%=~_|]$'

read -r -p "CSV file name: " CSV_FILE
if [[ ! -r "$CSV_FILE" ]]; then
    echo "The file $CSV_FILE does not exist or can not be accessed."
    exit 1
fi

read -r -p "vSphere URL: " VSPHERE_URL
if ! [[ $VSPHERE_URL =~ $url_regex ]]; then
    echo "vSphere URL $VSPHERE_URL IS not a valid URL"
    exit 1
fi
read -r -p "vSphere username: " VSPHERE_USER
read -r -s -p "vSphere password: " VSPHERE_PASSWORD
echo ""
read -r -p "Default username: " DEFAULT_USER
read -r -s -p "Default password: " DEFAULT_PASSWORD

echo ""

# CSV fields
#NAME;PLATFORM VM ID;OS;IP;USERNAME;PASSWORD
while IFS=";" read -r vm_name vm_id os ip username password
do
   if [ -z "$username" ]
    then
      username=$DEFAULT_USER
    fi
    if [ -z "$password" ]
    then
          password=$DEFAULT_PASSWORD
    fi

    if [ ! -z "$vm_id" ]
    then
      # Note that the below only works for VMWare VMs
      # vm_id after the last slash if prefixed by vSphere IP
      mod_vm_id=$(echo "$vm_id" | sed 's/.*\///')
      mfit discover vsphere guest --url $VSPHERE_URL -u $VSPHERE_USER -p $VSPHERE_PASSWORD --vm-user $username --vm-password $password $mod_vm_id
    else
      echo "Skipping VM: $vm_name because it has no VM ID"
    fi
 done < <(tail -n +1 $CSV_FILE)
The script uses the mfit discover vsphere guest command to upload and execute the collection script on the VM. It's best to execute this script in the background and direct the output to a log file. To do so, you will need to create a txt file which will be passed as input to the script.

  1. Create the text file by running the command:

    cat <<EOF >> INPUT_FILE
CSV_FILES_PREFIXCSV_FILE_SUFFIX
VSPHERE_URL
VSPHERE_USER
VSPHERE_PASSWORD
VM_USER
VM_PASSWORD
EOF

  2. Execute the collection script by running the command:

    nohup ./mfit_collect_vms_guest_tools.sh <INPUT_FILE > /tmp/mfit.out 2>&1 &

  3. Repeat steps 1 and 2 for each of the CSV files.

Generate a fit assessment report

Once all the data is collected, you can produce a detailed HTML report. You do so by running the command:

   ./mfit report --format html --full > REPORT_NAME.html

What's next