Troubleshooting VM Manager

This document describes how to troubleshoot issues with VM Manager.

For more information about VM Manager, see VM Manager.

Before you begin

  • If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

    Select the tab for how you plan to use the samples on this page:

    Console

    When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

    gcloud

    1. Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init
    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init

Overview

To troubleshoot issues, first verify that VM Manager is set up properly. If the setup is correct and you have still have issues, you can then review logs. When you review logs, you can identify issues with your VM or in your VM Manager workflow, such as errors in commands or scripts, that you can isolate and resolve.

You can collect the following information from logs:

  • Any error message or warning logged by the VM. This is ideal for identifying VM-level errors, or errors from other services running on your VM. To review these logs, see Inspect Cloud Logging.
  • Verbose debug information logged by the OS Config agent. This is useful for identifying issues with any of the operations run by VM Manager. To inspect debug logs for the OS Config agent, see Inspect debug logs.

After identifying the issues or errors, you can also review the common errors section to see potential fixes.

Inspect Cloud Logging

You can use the Google Cloud console quick links for each feature to view logs for a specific VM.

OS patch

  1. In the Google Cloud console, go to the Patch Jobs tab on the OS patch management page.

    Go to Patch Jobs

  2. Click the name of the patch job that you want to debug.
  3. Scroll-down to Updated VM instances.
  4. For a specific VM, under Logs, click View.

OS configuration

This procedure is supported for OS configuration management (preview). For OS configuration management (beta), use the debug logs option, in the following section.

  1. In the Google Cloud console, go to the VM instances tab on the OS configuration management page.

    Go to VM instances

  2. Click the name of the VM that you want to debug.
  3. Scroll-down to Policies.
  4. Under Logs, click View.

Inspect debug logs

You can identify issues with any VM Manager feature by enabling debugging for the OS Config agent and viewing debug log.

Enable debug logging for the OS Config agent

You can enable debug logging by setting the osconfig-log-level=debug metadata on the VM instance or project.

To enable debug logging on your VM, complete the following steps:

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Click the name of the VM for which you want to set the metadata value.

  3. On the Instance details page, click Edit to edit the settings.

  4. Under Custom metadata, add the following metadata entries:

    Key: osconfig-log-level
    Value: debug

  5. Click Save to apply your changes to the VM.

gcloud

Use the instances add-metadata command with the --metadata=osconfig-log-level=debug flag.

gcloud compute instances add-metadata VM_NAME \
    --metadata=osconfig-log-level=debug

Replace VM_NAME with the name of your VM.

REST

For instructions on setting instance metadata, follow the API instructions for setting instance metadata.

The following key-value pair is required as part of the metadata property:

Key: osconfig-log-level
Value: debug

View debug logs

When debug logging is enabled, the OS Config agent writes log entries to Cloud Logging and the serial port console.

After you enable debug logging on your VM, it takes the OS Config agent approximately ten minutes to start writing debug messages in Cloud Logging. You can reduce this wait time by restarting the agent or rebooting your VM. For more information about Cloud Logging, see Viewing Cloud Logging logs.

To view debug logs you can use the following options:

  • Cloud Logging: use the Google Cloud console or Google Cloud CLI

  • Serial port console

Console

  1. Go to the Logging > Logs Explorer (Logs Explorer) page in the Google Cloud console:

    Go to Logs Explorer

  2. If needed, select an existing Google Cloud project at the top of the page, or create a new project.

  3. In the Resource drop-down list, select VM Instance. A list of available VMs (instance_id) displays.

  4. Click on the VM that you want to view.

  5. Click Add.

  6. In the Log name drop-down list, select OSConfigAgent.

  7. Click Add.

  8. Your query should be similar to the following:

    resource.type="gce_instance" resource.labels.instance_id="136126869923081757"
logName="projects/my-vm-manager-project/logs/OSConfigAgent"

  9. Click Run query.

gcloud

Run the gcloud logging read command.

 gcloud logging read "resource.type=gce_instance AND logName=projects/PROJECT_ID/logs/OSConfigAgent"

Replace PROJECT_ID with your project ID.

Serial port

To view debug log information from the serial port console, see Viewing serial port output.

Common errors

Authentication issues

For VM Manager to work, you must have the following:

  • An attached service account. VM Manager uses this service account to sign requests to the API service.
  • Make sure that the attached service account has the roles/logging.logWriter role to write logs to the Logging API.
  • Google Cloud OS Config Service Agent. VM Manager creates this service agent when starting patch jobs and gives it the Cloud OS Config Service Agent role. To create OS policies, you need not configure this service agent.

If you use VM Manager and you don't have an attached service account, or a Google Cloud OS Config Service Agent, you might see the following errors while working with patch jobs:

Service account permissions are missing. Verify that the service account has the correct permissions and try again.
OSConfigAgent Error main.go:88: error getting token from metadata: metadata: GCE metadata "instance/service-accounts/default/identity?audience=osconfig.googleapis.com&format=full" not defined
message: "Error running OPERATION_NAME: error calling OPERATION_NAME: code: "PermissionDenied", message: "The caller does not have permission", details: []"

Authentication issues might also prevent VM instances from showing up on the Patch dashboard.

To resolve these issues, try one or all of the following:

  • Verify that all VMs have an attached service account.

  • Ensure that the Cloud OS Config Service Agent role (roles/osconfig.serviceAgent) is set on the Google Cloud OS Config Service Agent.

    gcloud projects add-iam-policy-binding PROJECT_ID \
  --member='serviceAccount:service-PROJECT_NUMBER@gcp-sa-osconfig.iam.gserviceaccount.com' \
  --role='roles/osconfig.serviceAgent'

    Replace the following:

Error when excluding packages for patch updates

If you specify wildcards or special characters for the name of packages when excluding packages in a patch job, OS patch management might ignore the list and update all packages.

To resolve this issue, update the OS Config agent to version 20220829.00 and use forward slashes (/) to wrap the package name.

In the following example, OS patch management excludes yum packages with prefix google- in the package name.

      gcloud compute os-config patch-jobs execute --instance-filter-all

       --yum-excludes=/google-.*/

What's next?