Troubleshooting

Learn about troubleshooting steps that you might find helpful if you run into problems using Migrate for Anthos 1.7.

Executing shell commands on your container

You can access a container through a bash shell or through PowerShell using the kubectl exec command.

Note: In the examples below, for PowerShell replace /bin/bash with cmd.exe, powershell.exe, or pwsh.exe depending on your version.

Use kubectl describe pods to find the name of the Pod in your cluster that you want to connect to.
In the following example, the command lists the suitecrm-0 pod.
```
kubectl describe pods | grep Name

Name:               suitecrm-0
```
Execute shell commands using one of the following methods:
- Use kubectl exec to open a bash command shell where you can execute commands.
```
kubectl exec -it pod-name -- /bin/bash
```
  The following example gets a shell to the suitecrm-0 pod:
```
kubectl exec -it suitecrm-0 -- /bin/bash
```
- Use kubectl exec to execute commands directly.
```
kubectl exec -it pod-name -- /bin/bash -c "command(s)"
```
  The following example lists the root directory of the suitecrm-0 pod:
```
kubectl exec -it suitecrm-0 -- /bin/bash -c "ls /"
```

For more information, see the Kubernetes documentation.

Debugging Kubernetes resources

More help is available at the following pages:

If you are experiencing a problem with Google Kubernetes Engine, see the GKE troubleshooting page.
If you are experiencing an issue related to your cluster, refer to Troubleshooting Clusters in the Kubernetes documentation.
If you are having an issue with your application, its pods, or its controller object, refer to Troubleshooting Applications.

RESOURCE_OPERATION_RATE_EXCEEDED error appears in migration status or logs

The RESOURCE_OPERATION_RATE_EXCEEDED error occurs when the source platform of a migration is Compute Engine and you exceed the limit on disk snapshots when migrating a Compute Engine VM.

As part of the migration, Migrate for Anthos creates a snapshot of the disk images from a VM running on the source platform. You can only create a disk snapshot at most once every 10 minutes, or 6 times an hour, for a Compute Engine VM. This error indicates that you have exceeded that limit.

To avoid hitting this limit, we recommend that you create the migration cluster in the same zone as the Compute Engine VM. When the cluster is in the same zone as the VM, Migrate for Anthos can clone the disk, rather than create a snapshot, which is a more efficient process and bypasses the snapshot limit.

See:

Configuring a processing cluster on Google Cloud for Windows workloads for more on creating a cluster.
Creating frequent snapshots efficiently for more on the limit.

A device listed in /etc/fstab fails to mount

By default, the system parses /etc/fstab and mounts all of the listed devices to the required mount points. If a device is not recognized or not mounted, the workload pod will not get to a ready state.

For example, consider a source Linux VM in Amazon EC2 that has an ephemeral disk where persistence is not guaranteed. These disks are not streamed to the target, causing the container to fail on mounting it.

If this happens, you might see messages such as the following:

Unable to locate resolve [X] fstab entries: …
Error: Failed mount -o defaults /dev/mapper/mpathe-part /rootfs/mnt/ephemeral

You can work around this by either:

Editing /etc/fstab on the Linux VM to remove the device mount command.
Setting the HC_INIT_SKIP_MOUNT_FAILURES environment variable to configure the system to skip mount failures and continue.

To set the HC_INIT_SKIP_MOUNT_FAILURES environment variable:

Create a configmap in the migration namespace, v2k-system, on the migration processing cluster. For example, define the configmap in a file named jobs-config.yaml:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: jobs-config
  namespace: v2k-system
data:
  environment: |
    HC_INIT_SKIP_MOUNT_FAILURES = true
```
Apply the configmap to the cluster:
```
kubectl apply -f jobs-config.yaml
```

To view the config map, use the command:

kubectl describe configmaps -n v2k-system

Edit the migration plan to add the configmap. The migration plan is the yaml file that you generated when you created the migration. See Customizing a migration plan for more on editing the migration plan.

In the migration plan, edit the configs section to add the configmap:
```
configs:
  jobsConfig:
    name: jobs-config
```
Save your edits and then execute the migration as described in Executing a migration.

Failure in capabilities of deployed container because of AppArmor

AppArmor lets a system administrator restrict capabilities of a deployed container by using custom profiles. In some cases, you might have to apply a profile to your deployed container to customize its capabilities.

To customize the AppArmor profile:

Create the profile on the cluster where you are deploying your migrated container. See the AppArmor documentation for more information.

Edit the deployment_spec.yaml file to add the HC_APPARMOR_PROFILE environment variable with the name of the AppArmor profile:

spec:
  containers:
  - image: gcr.io/my-project/my-container:v1.0.0
    name: my-container
    env:
    - name: HC_APPARMOR_PROFILE
      value: "apparmor-profile-name"
    securityContext:
      privileged: true
...

See Reviewing generated deployment files for more on editing deployment_spec.yaml.

FailedAttachVolume error appears in migration status or logs (preview)

If you are performing a migration using Anthos clusters on AWS to perform the migration, you might see the FailedAttachVolume error in the migration status or logs with the accompanying message of:

"The encrypted volume was unable to access the KMS master key"

This error means that your AWS VM uses an encrypted disk and you have not configured access correctly. See Prerequisites for migrating Linux VMs using AWS processing clusters for more.

ProvisioningFailed error appears in migration status or logs (preview)

If you are performing a migration using Anthos clusters on AWS to perform the migration, you might see the ProvisioningFailed error in the migration status or logs with the accompanying message of:

"ProvisioningFailed - "Could not create volume V_ID"

This error can occur when you try to export data into a new encrypted volume, but the cluster has no access to the encryption key, and therefore fails to create the EBS disk. See Prerequisites for migrating Linux VMs using AWS processing clusters for more.

I would like personalized support

Paid support is available for customers migrating with Migrate for Anthos. Reach out so we can help.

Providing information to Google Cloud support

The Sysreport provides Migrate for Anthos support with information about your cluster's configuration for faster time to problem resolution.

You can access the script from Cloud Shell.

Open Cloud Shell

Next, run the collect_sysreport.sh script.

/google/migrate/anthos/collect_sysreport.sh [-n NAMESPACE] [-o OUTPUT_DIRECTORY] [-m MIGRATION]

Where:

[NAMESPACE]: (Optional) The namespace in which Migrate for Anthos was deployed (default is v2k-system) .
[OUTPUT_DIRECTORY]: (Optional) Path to the directory in which the Sysreport will be saved (default is $HOME).
[MIGRATION]: (Optional) Additional information collected for a migration.

The script creates anthos-migrate-logs.TIMESTAMP.tar.xz, which you provide to Google Cloud support.

By default, the script collects:

Logs from the Migrate for Anthos CSI controller and CSI nodes.
Syslog from the Migrate for Anthos CSI node hosts.
Logs from Migrate for Anthos controller.
All Migrate for Anthos entities from the cluster.
If you specify a migration, and the migration uses the default artifacts repository, collect the migration artifacts from the Cloud Storage or S3 bucket.
The output of:
- kubectl cluster-info
- kubectl get nodes; kubectl describe node
- kubectl version
- kubectl top node
The workload's logs.
The output of:
- ps aux
- netstat -tlnp
- iptables -t nat -L
- fstab
- kubectl get pod
- kubectl describe pod
- kubectl top pod --all-namespaces --containers
- kubectl cluster-info dump
- kubectl api-resources -o wide
- kubectl top pod --all-namespaces --containers
- kubectl api-resources -o wide
- kubectl get componentstatuses --all-namespaces
- kubectl get endpoints --all-namespaces
- kubectl get events --all-namespaces
- kubectl describe limits --all-namespaces
- kubectl get namespaces
- kubectl describe pvc --all-namespaces
- kubectl describe pv --all-namespaces
- kubectl describe quota --all-namespaces
- kubectl describe sa --all-namespaces
- kubectl describe services --all-namespaces
- kubectl describe services --all-namespaces
- kubectl get ingresses --all-namespaces
- kubectl describe networkpolicies --all-namespaces
- kubectl get podsecuritypolicies --all-namespaces
- kubectl get clusterrolebindings --all-namespaces
- kubectl describe storageclasses --all-namespaces
- kubectl describe volumeattachments --all-namespaces