Troubleshooting SLES pay-as-you-go registration

Linux

---

This document describes how to resolve issues you might encounter when you connect Compute Engine virtual machine (VM) instances running pay-as-you-go (PAYG) SUSE Linux Enterprise Server (SLES) to the SUSE Subscription Management Tool (SMT) repository.

Before you begin

• Ensure that the VM has an associated service account.
• Ensure that the Service Metadata API is accessible from the VM.
• Use the sc-repocheck tool to automatically troubleshoot the issues.
• Check the steps described in the SUSE PAYG troubleshooting guide.
• If you haven't already, then set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:
  

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI, then initialize it by running the following command:

     gcloud init

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.
  2. Set a default region and zone.

Network issues

Unresolvable domain name

You might encounter the following issues if the VM can't connect to the smt-gce.susecloud.net SMT server:

SUSEConnect error: SocketError: getaddrinfo: Name or service not known

ping: unknown host smt-gce.susecloud.net

These issues are likely caused by an incorrect resolution of the SMT server domain name smt-gce.susecloud.net. This domain is not globally resolvable, so you must set its IP address according to the VM region, by doing the following:

Check the /etc/hosts file to make sure it contains an entry with the smt-gce.susecloud.net domain.

cat /etc/hosts | grep -i smt

The output looks similar to the following, but the IP address might be different:

# Added by SMT registration do not remove, retain comment as well
108.59.80.221   smt-gce.susecloud.net   smt-gce

If the /etc/hosts file doesn't contain the same lines as the preceding example, do the following:

1. Find an IP address that corresponds with your VM's region from the list of SUSE SMT IP addresses.

2. Edit the file to add the SUSE SMT IP address and any other information that is missing.

Note: While you can modify the /etc/hosts file for troubleshooting purposes, it is important to know that using the registercloudguest tool might overwrite your changes.

Network unavailability

You may encounter the following errors due to network unavailability, even if the VM is able to resolve Compute Engine Update Server domain name:

Unexpected exception.
Not ready to read within timeout.

Repository 'SLE-Module-Adv-Systems-Management12-Pool' is invalid.
Repository 'SLE-Module-Adv-Systems-Management12-Updates' is invalid.

The following are some examples of errors in the /var/log/cloudregister log file, yo may find during the investigation:

WARNING:Unable to remove client registration from server
WARNING:HTTPSConnectionPool(host='smt-gce.susecloud.net', port=443): Max retries exceeded with url: /connect/systems (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',))

INFO:Region server arguments: ?regionHint=europe-central2
ERROR:No response from: [('34.118.112.80', None), ('34.116.251.218', None), ('34.116.224.144', None)]

To find out more about the cause of the issue, perform a network connectivity test. The following example shows how to test an HTTPS connection using cURL:

curl -sSI -m 5 -o /dev/null \
  -w 'Response code (>0 is OK): %{http_code}\n' \
  'https://smt-gce.susecloud.net'

The output of the command contains an HTTP response code or an error message. The following are common responses and errors:

• Successful response:

  Response code (>0 is OK): 200
  

• Request timeout error:

  Response code (>0 is OK): 000
  curl: (28) Connection timed out after 5001 milliseconds
  

• Unresolvable domain error:

  Response code (>0 is OK): 000
  curl: (6) Could not resolve host: smt-gce.susecloud.net
  

Note: Any response code that is greater than 0 is considered a positive response because a connection with the server was established, even if the response contains a client or server error response code.

In certain scenarios, such as strict host firewall rules, the default IP address associated with the smt-gce.susecloud.net domain might not be available. To ensure that the issue is not only related to the current IP address, perform a network connectivity tests for alternate regional servers. Retrieve the list of regional servers by doing the following:

WebUI

Go to SUSE WebUI to obtain the list of regional update servers.

CLI

Use pint tool to obtain the list of regional update servers via CLI.

1. Install required package

   sudo zypper install python3-susepubliccloudinfo
   

2. Use the following command with specific region

   pint google servers --region us-central1
   

3. The successful output contains a list of entries in XML format

   <?xml version='1.0' encoding='UTF-8'?>
   <servers>
     <server ip="146.148.73.14" name="" region="us-central1" type="regionserver-sles"/>
     <server ip="162.222.182.90" name="" region="us-central1" type="regionserver-sap"/>
     <server ip="108.59.80.221" name="smt-gce.susecloud.net" region="us-central1" type="smt"/>
     <server ip="108.59.85.41" name="smt-gce.susecloud.net" region="us-central1" type="smt"/>
     <server ip="108.59.80.58" name="smt-gce.susecloud.net" region="us-central1" type="smt"/>
   </servers>
   

To find the full list of SUSE server IPs for Google Cloud, view the following documents:

• Region Servers
• SMT Servers

The network unavailability may be due to VM misconfiguration. In case of issues it is necessary to perform network diagnostics to identify the root cause.

Registration failed

You might encounter the following error if you have VMs that have a private IP address in Cloud NAT:

ERROR:  Registration failed: Registering system to registration proxy https://smt-gce.susecloud.net
command '/usr/bin/zypper --non-interactive refs Python_3_Module_x86_64' failed
Error: zypper returned 4 with 'Problem retrieving the repository index file for service 'Python_3_Module_x86_64':
Timeout exceeded when accessing 'https://smt-gce.susecloud.net/services/2045/repo/repoindex.xml?credentials=Python_3_Module_x86_64'.

To resolve this issue, review the Cloud NAT configuration to verify that the minimum ports per VM instance parameter is set to at least 256.

For more information, check the Registration and zypper failed for Compute Engine instances behind Cloud NAT SUSE support bulletin.

No response

If your VM experiences problems communicating with update and region servers, you may observe the following errors:

• SUSEConnect error:

  SUSEConnect error: Errno::ETIMEDOUT: Connection timed out - connect(2) for "smt-gce.susecloud.net" port 443
  

• zypper error:

  Error retrieving metadata for 'SLE-Module-Adv-Systems-Management12-Pool':
  Not ready to read within timeout.
  ...
  

These errors can be caused by the absence of a response from update and region servers. To verify if this is the case, check the /var/log/cloudregister logs for similar content:

INFO:Region server arguments: ?regionHint=europe-central2
INFO:Using API: regionInfo
INFO:Region server arguments: ?regionHint=europe-central2
INFO:Getting update server information, attempt 1
INFO:   Using region server: 130.211.242.136
ERROR:  No response from: 130.211.242.136
INFO:   Using region server: 35.187.193.56
ERROR:  No response from: 35.187.193.56
INFO:   Using region server: 162.222.182.90
ERROR:  No response from: 162.222.182.90
INFO:   Using region server: 130.211.88.88
ERROR:  No response from: 130.211.88.88
ERROR:  None of the servers responded
ERROR:  Attempted: [IPv4Address('130.211.242.136'), IPv4Address('35.187.193.56'), IPv4Address('162.222.182.90'), IPv4Address('130.211.88.88')]
...
...
...
ERROR:Request not answered by any server after 3 attempts
ERROR:Exiting without registration

To resolve this issue, try one or more of the following:

• Confirm that the VM has an external IP address or that the Virtual Private Cloud subnet uses a NAT (either Cloud NAT or custom solution).

• If you modified the default network routing rules, such as limiting public Internet access or routing traffic through an on-premises network, add routes manually for SMT IPs through the default gateway of Compute Engine, by doing the following:

  1. Go to the Routes page in the Google Cloud console.

     Go to the Routes page

  2. Under the Route Management tab look for a route that includes the SUSE SMT IP addresses and verify that it has the Compute Engine default gateway set as the next hop.

  3. If the route is missing, you can add it by clicking on Create Route and entering the necessary information.

• If you're using an internal passthrough Network Load Balancer, for example with additional intermediary network software (such as firewalls, custom NATs, etc.), make sure that the load balancer is being used as the next hop for VM traffic, by doing the following:

  1. Go to the VM instances page in the Google Cloud console.

     Go to the VM instances page

  2. Click the name of the VM you want to check. The VM details page opens.

  3. In the Network interfaces section, click View details.

  4. In the Firewall and routes details section locate the route that defines the path to the desired IP address range.

  5. Click the name of the route and confirm that internal passthrough Network Load Balancer or its IP address is set as the next hop.

  If there is no route that defines the path to the desired IP address range, or if the next hop of the route is different from internal passthrough Network Load Balancer, then set up internal passthrough Network Load Balancer as the next hop.

• If you're using an internal passthrough Network Load Balancer, confirm that it's located in the same region as the VM.

  1. Go to the VM instances page in the Google Cloud console.

     Go to the VM instances page

  2. Locate the VM you want to check and note down its region.

  3. Go to the Load balancing page in the Google Cloud console.

     Go to the Load balancing page

  4. Locate the internal passthrough Network Load Balancer used and check if it is in the same region as the VM.

  5. If the VM and the internal passthrough Network Load Balancer aren't in the same region, enable global access.

OS configuration issues

Unknown registration status

If you don't know whether or not your pay-as-you-go (PAYG) SUSE Linux Enterprise Server (SLES) is registered, run the following command:

sudo SUSEConnect --status-text

The output contains the version and registration status of the SUSE products, including SUSE Linux Enterprise Server.

Installed Products:
------------------------------------------

  SUSE Linux Enterprise Server 12 SP5
  (SLES/12.5/x86_64)

  Registered

------------------------------------------
...

If the status is Not Registered, start from the re-registration process to fix the issue.

Incorrect base product symlink

You may encounter the following errors if the base product link points to an incorrect product file:

ERROR:Unable to obtain product information from server "108.59.85.41,None"
        Unprocessable Entity
        {"type":"error","error":"Unmet product dependencies, activate one of these products first: SUSE Linux Enterprise Server 12 x86_64...
        ...
Unable to register modules, exiting.

This error is caused by an incorrect product file (i.e. sle-module-toolchain.prod) being pointed to by the /etc/products.d/baseproduct symbolic link.

To resolve this issue, update the symlink at /etc/products.d/baseproduct to point to the appropriate base product file, by doing the following:

1. Navigate to the /etc/products.d directory

     cd /etc/products.d
   

2. Run the following command replacing SLES.prod with SLES_SAP.prod if SLES for SAP is installed:

     sudo ln -sf SLES.prod baseproduct
   

Note: If the SLES.prod or SLES_SAP.prod product file is missing, you can create a new VM instance with the same boot disk image and copy the correct file from there to the /etc/products.d directory.

Instance identity information unavailability

You may encounter the following errors if the instance identity information is not available for the VM:

ERROR:Data collected from stderr for instance data collection "b'Unable to access instance identity information\n'"

To access the instance metadata for identity tokens all VMs must be associated with a service account.

For more information, read the Public Cloud Infrastructure Update.

To check that the VM is relevant to this situation, run the following command on the VM:

curl -s -H 'Metadata-Flavor: Google' \
  'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=test'

Example of a successful response with an identity token:

eyJhbGciOiJSUzI1NiIsImtpZCI6IjkzOTd0MDQxSHQ2NDNxNzkzUjY1MDIwNzEyMjZPNnppaTdqNTl3eTciLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJ0ZXN0IiwiYXpwIjoiMjY1MDIwMDUyMzgzMjYyNTk0ODU2IiwiZXhwIjoxNjgzNzEyNTQzLCJpYXQiOjE2ODM3MTI4NjQsImlzcyI6Imh0dHBzOi8vYWNjb3VudHMuZ29vZ2xlLmNvbSIsInN1YiI6IjQ1NjA2MzQ5MDg5Mzc0Njg3ODI5NyJ9.EpzQ3NZ8mKStdpH10fL34qsKG0rjQEflzvLJLm2tVNX4xBJAkMhi8lcs5InUEY-QMK3njgbzdzNtD1fXoIfKoeWsqkA8vG3NkBz5zqRrtaB2STcO14H5tjIdTBsrCtET447tRXlGG5cvgMcWnRDZG92-jUZEpWki_Ri4T69X5-bBWkfE2Thm3oSUW4fScdeVOEmOgWnzD2jeVqQ_2YniywvpkT-rLzKfN-5AgN66zgBfXqJVTC90KFMebfiaOoL7z6ZSM9AjZGf45QEMZjxjd-Xzyee6ZWK8s0RE3hJlytb3zYcLt3tJwQ1WhnrC2ToJ-ZmKxxK3xKDLCvCQ6Ny5to

If the metadata returned is not a token but an error message such as the following, the VM is affected:

{
  "error": "invalid_request",
  "error_description": "Service account not enabled on this instance"
}

To remediate this issue, perform the following steps:

1. Stop the VM:

   gcloud compute instances stop VM_NAME
   

2. Add a service account to the VM:

   gcloud compute instances set-service-account VM_NAME \
     --service account SERVICE_ACCOUNT \
     --no-scopes
   

   Note: Flag --no-scopes removes all scopes from the VM. If no flag is specified for access scope, default scopes will be assigned to the compute instance's service account, or the service account will retain its current scopes.

3. Start the VM:

   gcloud compute instances start VM_NAME
   

4. After adding the missing service account, run the following command from the VM to re-register the SLES:

   sudo registercloudguest --force-new
   

   Check details in the re-registration section.

Registration behind proxies

You might encounter an issue if your VMs configured to utilize any kind of proxying software. The following example demonstrates an attempt to register SLES via an HTTP proxy.

ERROR: Baseproduct registration failed
ERROR: Registering system to registration proxy https://smt-gce.susecloud.net

Announcing system to https://smt-gce.susecloud.net ...
SUSEConnect error: Net::HTTPFatalError: 503 "Service Unavailable"

SUSE on Compute Engine does not provide official support for operating system registration when performed through intermediaries that modify the original communication, such as proxies of man-in-the-middle (MITM) or non-transparent types.

Note: It is important to note that Google Cloud does not accept any responsibility for the registration process using 3rd-party proxying software.

The official solution to resolve this issue is to Set up Cloud NAT and route VM traffic through it.

Common workarounds

Re-registration

In some cases, a re-registration approach can be used to work around registration issues.

To force a new registration use the following command:

sudo registercloudguest --force-new

If successful, the following line will be output.

Registration succeeded

Details of re-registration process can be found in the /var/log/cloudregister.

Successful example

INFO:Forced new registration
INFO:Clean current registration server: ('108.59.80.221', None)
...
INFO:Starting new HTTP connection (1): 169.254.169.254
INFO:Region server arguments: ?regionHint=us-central1
INFO:Using region server: 130.211.242.136
INFO:Starting new HTTPS connection (1): 130.211.242.136
INFO:Starting new HTTPS connection (1): 108.59.80.58
INFO:Modified /etc/hosts, added: 108.59.80.58   smt-gce.susecloud.net   smt-gce
...
INFO:Starting new HTTPS connection (1): 108.59.80.58
DEBUG:"GET /api/health/status HTTP/1.1" 200 None
INFO:Current update server will be used: "('108.59.80.58', None)"
INFO:Starting new HTTPS connection (1): smt-gce.susecloud.net
DEBUG:"POST /connect/systems/products/migrations HTTP/1.1" 422 None
INFO:Registration: /usr/sbin/SUSEConnect --url https://smt-gce.susecloud.net --product sle-module-containers/12/x86_64 --instance-data /var/lib/cloudregister/9c982106-78de-48fe-a662-20383da4c760

Failed example

INFO:Forced new registration
INFO:Using API: regionInfo
INFO:Starting new HTTP connection (1): 169.254.169.254
INFO:Region server arguments: ?regionHint=us-central1
INFO:Using region server: 130.211.242.136
INFO:Starting new HTTPS connection (1): 130.211.242.136
ERROR:No response from: 130.211.242.136
INFO:Using region server: 130.211.88.88
INFO:Starting new HTTPS connection (1): 130.211.88.88
ERROR:No response from: 130.211.88.88
INFO:Using region server: 146.148.73.14
INFO:Starting new HTTPS connection (1): 146.148.73.14
ERROR:No response from: 146.148.73.14
ERROR:None of the servers responded
ERROR:  Attempted: ['130.211.242.136', '130.211.88.88', '146.148.73.14']
ERROR:Exiting without registration

Note: If the re-registration process failed, this may be due to the Incorrect base product symlink.

Deregistration

In some cases, such as major release upgrade, you may encounter the following errors because the system is already registered to SUMA:

Can't get available migrations from server: SUSE::Connect::ApiError: The requested products 'SUSE Manager Client Tools for SLE 12 x86_64' are not activated on the system.

This system is managed by SUSE manager.

Resolve the issue, by doing the following:

1. Remove the SUSE Manager Client Tools module as described in the Deleting modules and extensions guide.

2. Deregister from SUMA, by following the How to deregister a SUSE Manager Client guide.

3. Run the following commands from VM to cleanup old registration:

     sudo SUSEConnect --cleanup && \
       sudo registercloudguest --clean && \
       sudo rm -f /etc/SUSEConnect && \
       sudo rm -f /etc/zypp/{repos,services,credentials}.d/* && \
       sudo rm -f /var/lib/cloudregister/* && \
       sudo rm -rf /var/cache/zypp/* && \
       sudo rm -rf /var/cache/cloudregister/* && \
       sudo sed -i '/^# Added by SMT reg/,+1d' /etc/hosts
   

4. Run the following command to register the system again:

     sudo registercloudguest --force-new
   

   Check details in the re-registration section.

5. When the registration process is done, refresh the services and repositories, and check if all the expected repositories for the system provided by the SMT server are present:

     sudo zypper ref -s && \
       sudo zypper ls && \
       sudo zypper lr -U