Google Cloud's Agent for SAP troubleshooting guide

This guide shows you how to resolve issues with version 3.6 (latest) of Google Cloud's Agent for SAP.

For troubleshooting and support information from SAP, see the following SAP Notes:

Logging

Check the logs in the directory that is specific to your operating system.

To view the logs for Google Cloud's Agent for SAP, navigate to the following paths:

Linux

/var/log/google-cloud-sap-agent.log

Windows

C:\Program Files\Google\google-cloud-sap-agent\logs\google-cloud-sap-agent.log

If you've enabled the Backint feature of Google Cloud's Agent for SAP, then review the following logs:

  • The SAP HANA backup.log file, which contains information about the SAP HANA backup and recovery operations.
  • The SAP HANA backint.log file, which contains the calls to the backup interface that is used by the Backint feature of the agent and other external backup tools.
  • The Backint log file of Google Cloud's Agent for SAP: /var/log/google-cloud-sap-agent/backint.log. This file includes information about when the Backint feature was enabled, related operational events, and indications of communication errors with Cloud Storage and related configuration or permissions issues.
  • The Google Cloud logs in Cloud Logging, which can contain errors related to the service account used by Google Cloud's Agent for SAP. To view these logs, in the Google Cloud console, go to the Logs explorer page.

Common issues

The following sections provide information about common issues related to using Google Cloud's Agent for SAP, their causes and resolution.

Issue: Insufficient IAM permissions

Issue: Google Cloud's Agent for SAP logs show insufficient IAM permissions error.

Cause: The service account used by the agent does not have the required IAM permissions to access the Cloud Monitoring API, or if you're using the Backint feature of the agent, then insufficient permissions to access the Cloud Storage bucket.

Resolution: To resolve this issue, perform the following steps:

  1. In Google Cloud console, on the VM instance details page, note the name of the VM service account. For example: sap-example@example-project-123456.iam.gserviceaccount.com.

  2. Go to the IAM & Admin page, and for the noted service account, ensure that it includes the following IAM roles:

    Feature Required IAM roles
    SAP Host Agent metrics collection
    Process Monitoring metrics collection
    Workload Manager evaluation metrics collection
    SAP HANA monitoring metrics collection
    Backint based backup and recovery for SAP HANA
    • Storage Object Admin (roles/storage.objectAdmin)
    • Cloud KMS CryptoKey Encrypter/Decrypter (roles/cloudkms.cryptoKeyEncrypterDecrypter), if you have used a customer-managed encryption key to encrypt your SAP HANA backups in Cloud Storage. For information about this feature, see Encryption options for backups.

    For more information, see Required Cloud Storage permissions.

For more information about the authentication required for Google Cloud's Agent for SAP, see Authentication and access.

To confirm the permissions that the Cloud Monitoring agent requires, see the following Monitoring documentation:

Issue: Incorrect access scopes for the VM service account

Issue: If you limit the access scopes on your host VM instance, then Google Cloud's Agent for SAP logs might show insufficient IAM permissions error.

Cause: Google Cloud's Agent for SAP requires minimum Cloud API access scopes on the host VM instance. This error occurs when the service account does not have the required access scopes.

Resolution: Access scopes are the legacy method of specifying permissions for your VM instance. Compute Engine recommends configuring your VM instances to allow all access scopes to all Cloud APIs and using only the IAM permissions of the VM service account to control access to Google Cloud resources.

To resolve this issue, as a best practice, set the all cloud-platform access scope on the VM instance, then securely limit the service account's API access with IAM roles. For example:

  • https://www.googleapis.com/auth/cloud-platform

If you do limit the access scopes of your VM instance, then you must ensure that the host VM instance has the following access scopes:

  • https://www.googleapis.com/auth/source.read_write
  • https://www.googleapis.com/auth/compute
  • https://www.googleapis.com/auth/servicecontrol
  • https://www.googleapis.com/auth/service.management.readonly
  • https://www.googleapis.com/auth/logging.admin
  • https://www.googleapis.com/auth/monitoring
  • https://www.googleapis.com/auth/trace.append
  • https://www.googleapis.com/auth/devstorage.full_control

If you have enabled the collection of the Process Monitoring metrics or the SAP HANA monitoring metrics, then the access scopes of the host VM instance must also have write access to publish metric data to your Google Cloud project:

  • https://www.googleapis.com/auth/monitoring.write

To change the access scopes, you need to stop your VM instance, make the changes, and then restart the VM instance. For instructions, see the Compute Engine documentation. You don't need to make any changes to permissions for IAM roles for this issue.

Issue: Missing or incorrect SAP Host Agent

Issue: Google Cloud's Agent for SAP logs show missing or incorrect SAP Host Agent error.

Cause: SAP Host Agent or the required minimum patch level for the SAP Host Agent is not installed. For Google Cloud's Agent for SAP to work, your SAP system must have the SAP Host Agent installed and the required minimum patch level for the Host Agent is maintained.

Resolution: To resolve this issue, install the required version of the SAP Host Agent. For instructions to install the SAP Host Agent, see the SAP documentation.

For version requirements for the SAP Host Agent, see the following SAP Notes:

Issue: Installation of Google Cloud's Agent for SAP failed

Issue: Installation of the agent fails when you run the package manager install command such as yum, zypper, or googet, or you see the following error message: Package matching google-cloud-sap-agent-VERSION_NUMBER already installed. Checking for update..

Cause: Installation of the agent fails for one of the following reasons:

  • You're downgrading the agent to a non-latest version and your compute instance runs on an older version of RHEL for SAP, such as 7.9.
  • The compute instance that hosts the agent has been created without an external IP address.

Resolution: To resolve this issue, perform one of the following, as appropriate to your scenario:

  • If you want to downgrade your agent instance to a non-latest version and your compute instance runs on an older version of RHEL for SAP, such as 7.9, then run the following command:

    RHEL

    sudo tee /etc/yum.repos.d/google-cloud-sap-agent.repo << EOM
[google-cloud-sap-agent]
name=Google Cloud Agent for SAP
baseurl=https://packages.cloud.google.com/yum/repos/google-cloud-sap-agent-el$(cat /etc/redhat-release | cut -d . -f 1 | tr -d -c 0-9)-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOM
sudo yum downgrade google-cloud-sap-agent-VERSION_NUMBER.x86_64

    SLES15

    sudo zypper addrepo --refresh https://packages.cloud.google.com/yum/repos/google-cloud-sap-agent-sles15-x86_64 google-cloud-sap-agent
sudo zypper install google-cloud-sap-agent-VERSION_NUMBER.x86_64

    SLES 12

    sudo zypper addrepo --refresh https://packages.cloud.google.com/yum/repos/google-cloud-sap-agent-sles12-x86_64 google-cloud-sap-agent
sudo zypper install google-cloud-sap-agent-VERSION_NUMBER.x86_64

    Replace VERSION_NUMBER with the agent's version number that you want to install, such as 3.1-606637668. For information about the agent versions that you can install, see List all available versions of the agent.

  • If the compute instance hosting the agent doesn't have an external IP address, then set up a NAT gateway that gives the host server outbound access to the internet. For information about how to set up a NAT gateway, see the deployment guide for your SAP system. For example, for SAP NetWeaver, see:

Issue: Collection of the SAP HANA monitoring metrics failed

Issue: While upgrading from the monitoring agent for SAP HANA, after you install Google Cloud's Agent for SAP, you see an error message similar to the following:

tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead

Cause: Google Cloud's Agent for SAP cannot start the collection of the SAP HANA monitoring metrics because the target SAP HANA instances use SSL certificates that are specified with Common Name (CN).

Resolution: To resolve this issue, complete the following steps:

  1. For the SAP HANA instances that you want to monitor using Google Cloud's Agent for SAP, you must switch to using a Subject Alternate Name (SAN) SSL certificate instead of SSL certificates that are specified with Common Name (CN).

  2. Establish an SSH connection with your host VM instance or Bare Metal Solution server.

  3. Open the configuration file of Google Cloud's Agent for SAP:

    /etc/google-cloud-sap-agent/configuration.json

  4. In the hana_monitoring_configuration section, set the parameter enabled to true.

  5. In the hana_monitoring_configuration.hana_instances section, perform the following for each SAP HANA instance that uses the TLS/SSL protocol for secure communication:

    1. Specify the parameter enable_ssl and set its value to true.

    2. Specify the parameter host_name_in_certificate and set the SAP HANA hostname, as specified in the TLS/SSL certificate, as its value.

    3. Specify the parameter tls_root_ca_file and set the path, where the TLS/SSL certificate is stored, as its value.

  6. Save the configuration file.

  7. Restart Google Cloud's Agent for SAP for the new settings to take effect:

    sudo systemctl restart google-cloud-sap-agent

  8. Verify that the agent is collecting the SAP HANA monitoring metrics. For instructions, see View the other metrics.

  9. Uninstall the monitoring agent for SAP HANA.

Issue: Connection refused error

Issue: SAP Host Agent logs show the connection refused error.

Cause: Google Cloud's Agent for SAP cannot start up because the port 18181 is not available. Google Cloud's Agent for SAP listens for requests on port 18181. This port must be available for the agent to start up.

Resolution: To resolve this issue, make sure that the port 18181 is available for Google Cloud's Agent for SAP. If another service is using port 18181, then you might need to restart that other service or otherwise reconfigure it to use another port.

Issue: For the OS images SLES 15 SP4 for SAP and later, Google Cloud's Agent for SAP is not running

Issue: When you use the SLES "for SAP" OS images, Google Cloud's Agent for SAP is preinstalled for you. But, for the OS images SLES 15 SP4 for SAP and later, the preinstalled Agent for SAP does not start on its own.

To verify that the agent is running or not, perform the following steps:

  1. Connect to your host VM instance or Bare Metal Solution server.

  2. Run the following command: 

    systemctl status google-cloud-sap-agent

    If the agent is not running, then the output contains inactive (dead). For example: 

    google-cloud-sap-agent.service - Google Cloud Agent for SAP
 Loaded: loaded (/usr/lib/systemd/system/google-cloud-sap-agent.service; disabled; vendor preset: disabled)
 Active: inactive (dead)

Cause: The Agent for SAP doesn't start on its own because of a problem with the OS packaging.

Resolution: To resolve the issue, perform the following steps:

  1. Connect to your host VM instance or Bare Metal Solution server.

  2. Run the following commands:

    sudo sed -i 's~ /usr/sap~ -/usr/sap~g' /usr/lib/systemd/system/google-cloud-sap-agent.service
sudo systemctl restart google-cloud-sap-agent

  3. Verify that the agent is running: 

    systemctl status google-cloud-sap-agent

    You should see an output similar to the following:

    google-cloud-sap-agent.service - Google Cloud Agent for SAP
  Loaded: loaded (/usr/lib/systemd/system/google-cloud-sap-agent.service; disabled; vendor preset: disabled)
  Active: active (running) since Wed 2023-07-12 03:07:23 UTC; 7s ago
Main PID: 6117 (google_cloud_sa)
   Tasks: 6
  Memory: 8.8M (max: 1.0G limit: 1.0G available: 1015.1M)
  CGroup: /system.slice/google-cloud-sap-agent.service
           └─ 6117 /usr/bin/google_cloud_sap_agent startdaemon

Issue: System discovery fails due to lack of execute permission for the /usr/sap directory

Issue: In the Google Cloud's Agent for SAP logs, you see a warning No execute permission for /usr/sap directory, some of the discovery operations will fail. Please ensure that the root user has execute permission for /usr/sap directory.

Cause: The root user does not have the execute permission for the /usr/sap/ directory. This stops the root user from accessing the SAP system profile and SAP tools located in the /usr/sap directory. This can restrict the agent from discovering the SAP systems running on the host compute instance, and consequently restrict the expected behavior of the agent's features.

Resolution: To resolve this issue, ensure that the root user has execute permission for the /usr/sap/ directory. You can do this by running the following command:

chmod +x /usr/sap

Issue: Low throughput

Issue: You experience a lower than expected throughput.

Cause: For the Backint feature of Google Cloud's Agent for SAP, you might have enabled compression using the compress configuration parameter.

Resolution: To resolve this issue, verify that compression is not enabled. To do so, perform the following steps:

  1. Connect to your SAP HANA host by using SSH.

  2. Open your Backint configuration file:

    /usr/sap/SID/SYS/global/hdb/opt/backint/backint-gcs/parameters.json

    Replace SID with the SID of the system where you installed the Backint related files.

  3. Verify that the compress parameter is set to false.

  4. If you make any change, then save the configuration file.

Issue: Multistreaming errors

Issue: You get errors while multistreaming data backups.

Cause: For the Backint feature of Google Cloud's Agent for SAP, you might have specified a higher number of parallel streams.

Resolution: To resolve this issue, perform the following steps:

  1. Connect to your SAP HANA host by using SSH.

  2. Open your Backint configuration file:

    /usr/sap/SID/SYS/global/hdb/opt/backint/backint-gcs/parameters.json

    Replace SID with the SID of the system where you installed the Backint related files.

  3. If you've set a value for the parallel_streams parameter, then lower it to 1.

    If this doesn't resolve the error, then contact Customer Care.

Issue with connecting to Cloud Storage

Issue: You're facing a connectivity issue with your Cloud Storage bucket.

Cause: This can be caused by the following:

  • A temporary connectivity issue.
  • The service account used by Google Cloud's Agent for SAP doesn't have the permissions required to access your Cloud Storage.
  • There's an issue with your configuration.

Resolution: To troubleshoot a connectivity issue with Cloud Storage bucket, check the following:

  • Retry the failed backup or recovery to make sure that the error wasn't just a momentary connectivity issue.
  • If Google Cloud's Agent for SAP is using your VM's service account, then make sure that the VM has access to all cloud scopes, or if you're limiting access, then make sure that the VM has access to storage.googleapis.com. For more information, see Enable access to Cloud APIs and metadata servers.
  • If you created a service account for use with Google Cloud's Agent for SAP, then ensure the following:
    • The service account is assigned the Storage Object Admin role.
    • The service account is assigned the Cloud KMS CryptoKey Encrypter/Decrypter role if you're using a customer-managed encryption key to encrypt backups in Cloud Storage. For more information, see Encryption for backups.
  • Make sure that you've correctly specified the parameters in the Backint configuration file, PARAMETERS.json, and the SAP HANA global.ini file.

You can test access to your Cloud Storage bucket using the self-diagnosis feature built into Google Cloud's Agent for SAP. For more information, see Self diagnostics.

Issue with a disk snapshot based backup or recovery operation

Issue: A backup or recovery operation performed for SAP HANA by using the agent's hanadiskbackup or hanadiskrestore command is not successful.

Cause: This issue can occur due to any of the following reasons:

  • The agent experiences an error and exits the backup or recovery operation.
  • The agent takes more time than expected to create the disk snapshot or upload it to Cloud Storage due to an issue with an underlying Google Cloud service.
  • The agent crashes with no clear error or success message.
  • The backup or recovery operation fails because of an issue at the operating system level.
  • The agent is made to wait longer than expected by your SAP HANA database, which might be because your database is experiencing a very heavy workload.

Resolution: To resolve this issue, use the following information:

  • If it's a backup operation that is not successful, then do the following:

    1. Verify that the agent is running. If it's not, then restart the agent, and then retry the backup operation.

    2. If the agent is running as expected, then check the status of the snapshot creation:

      gcloud compute snapshots list
  --filter="sourceDisk:projects/PROJECT_ID/zones/ZONE_ID/disks/DISK_NAME"

      Replace the following:

      • PROJECT_ID: the Google Cloud project where the source disk is deployed
      • ZONE_ID: the Compute Engine zone where the source disk is deployed, for example, us-central1-a
      • DISK_NAME: the name of the source disk

      The output status can be one of the following: CREATING, UPLOADING, or READY.

    3. If the status of the snapshot creation is CREATING or UPLOADING, then we recommend that you wait for this operation to complete, especially if this is the first snapshot that you're creating for the disk hosting your /hana/data volume.

      If this operation fails, then the agent automatically exits the procedure and updates logs. You can retry creating the backup by running the hanadiskbackup command. The agent takes care of finishing the unsuccessful backup operation in the SAP HANA database and it creates a new snapshot based backup.

      If you used the -confirm-data-snapshot-after-create argument with the hanadiskbackup command, and the snapshot upload operation fails, then you need to manually delete the snapshot's entry in your SAP HANA backup catalog and retry snapshot creation. To delete the entry in the SAP HANA backup catalog, you can use the BACKUP CATALOG DELETE statement. For information about the -confirm-data-snapshot-after-create argument, see its description in Create a disk snapshot based backup.

    4. If the status of the snapshot creation is READY but the hanadiskbackup command still appears to be running, then this indicates that the agent is waiting on SAP HANA to do its part of the backup procedure.

    5. If the hanadiskbackup command appears to be running longer than expected, then do the following to resume normal operations for your SAP HANA database:

      1. Unfreeze the XFS file system that contains the /hana/data volume:

        xfs_freeze -u PATH_TO_HANA_DATA_VOLUME

      2. From your SAP HANA database, get the backup ID of the failed storage snapshot:

        SELECT BACKUP_ID, STATE_NAME, COMMENT FROM M_BACKUP_CATALOG WHERE ENTRY_TYPE_NAME = 'data snapshot' and STATE_NAME = 'prepared'

      3. In SAP HANA, update the entry for the failed snapshot as unsuccessful:

        BACKUP DATA FOR FULL SYSTEM CLOSE SNAPSHOT BACKUP_ID UNSUCCESSFUL 'Do not use - manually terminated';

      Replace BACKUP_ID with the BACKUP_ID that you fetched in the preceding step.

    6. To resolve the issue of the unsuccessful backup operation, contact Customer Care with the agent's support bundle.

  • If it's a recovery operation that is unsuccessful, then complete the following steps:

    1. Verify that the original disk hosting the /hana/data volume is attached to your host VM. If it's not, then attach it by using the gcloud compute instances attach-disk command.

    2. Rescan your volume groups and logical volumes:

      sudo /sbin/dmsetup remove_all
sudo /sbin/vgscan -v --mknodes
sudo /sbin/vgchange -ay
sudo /sbin/lvscan
sudo mount -av

    3. If the rescan is unsuccessful, then restart your host VM. This makes the operating system rescan the volume groups.

    4. In SAP HANA, verify that the /hana/data volume is live.

    5. If the /hana/data volume is still not live, then contact Customer Care with the agent's support bundle to resolve the issue.

Metrics based on SAP HANA monitoring views are missing

Issue: SAP HANA monitoring metrics that are based on the monitoring views in your SAP HANA database are missing, and you can see the following error in the logs of Google Cloud's Agent for SAP:

{sql: Scan error on column index 0, name \"MEM_USED\": converting NULL to int64 is unsupported"}

Cause: The SAP HANA database user, which you've specified in the agent's configuration file for the hana_monitoring_configuration.hana_instances.user parameter, doesn't have the permission to read the monitoring views in your SAP HANA database. Consequently, the queries that the agent sends to the monitoring views are returned with no data.

Resolution: To resolve this issue, grant the specified database user the CATALOG READ system privilege or the MONITORING role in your SAP HANA database. To grant this privilege, you run the GRANT statement using an administrative user, such as SYSTEM. For example:

  • To grant the MONITORING role, run the following:

    GRANT MONITORING TO DB_USERNAME

  • To grant the CATALOG READ system privilege, run the following:

    GRANT CATALOG READ TO DB_USERNAME

    Replace DB_USERNAME with the SAP HANA database username that you've specified in the agent configuration file for the hana_monitoring_configuration.hana_instances.user parameter. For information about this configuration parameter, see Configuration parameters.

Getting support for Google Cloud's Agent for SAP

If you need help resolving a problem with Google Cloud's Agent for SAP, then gather the required diagnostic information and contact Cloud Customer Care. For more information, see Google Cloud's Agent for SAP diagnostic information.