Troubleshooting

Learn about troubleshooting steps that you might find helpful if you run into problems using Migrate for Anthos 1.5.

Executing shell commands on your container

You can access a container through a bash shell or through PowerShell using the kubectl exec command.

Note: In the examples below, for PowerShell replace /bin/bash with cmd.exe, powershell.exe, or pwsh.exe depending on your version.

Use kubectl describe pods to find the name of the Pod in your cluster that you want to connect to.
In the following example, the command lists the suitecrm-0 pod.
```
kubectl describe pods | grep Name

Name:               suitecrm-0
```
Execute shell commands using one of the following methods:
- Use kubectl exec to open a bash command shell where you can execute commands.
```
kubectl exec -it pod-name -- /bin/bash
```
  The following example gets a shell to the suitecrm-0 pod:
```
kubectl exec -it suitecrm-0 -- /bin/bash
```
- Use kubectl exec to execute commands directly.
```
kubectl exec -it pod-name -- /bin/bash -c "command(s)"
```
  The following example lists the root directory of the suitecrm-0 pod:
```
kubectl exec -it suitecrm-0 -- /bin/bash -c "ls /"
```

For more information, see the Kubernetes documentation.

Debugging Kubernetes resources

More help is available at the following pages:

If you are experiencing a problem with Google Kubernetes Engine, see the GKE troubleshooting page.
If you are experiencing an issue related to your cluster, refer to Troubleshooting Clusters in the Kubernetes documentation.
If you are having an issue with your application, its pods, or its controller object, refer to Troubleshooting Applications.

RESOURCE_OPERATION_RATE_EXCEEDED error appears in migration status or logs

The RESOURCE_OPERATION_RATE_EXCEEDED error occurs when the source platform of a migration is Compute Engine and you exceed the limit on disk snapshots when migrating a Compute Engine VM.

As part of the migration, Migrate for Anthos creates a snapshot of the disk images from a VM running on the source platform. You can only create a disk snapshot at most once every 10 minutes, or 6 times an hour, for a Compute Engine VM. This error indicates that you have exceeded that limit.

To avoid hitting this limit, we recommend that you create the migration cluster in the same zone as the Compute Engine VM. When the cluster is in the same zone as the VM, Migrate for Anthos can clone the disk, rather than create a snapshot, which is a more efficient process and bypasses the snapshot limit.

See:

Configuring a Cloud cluster for Linux VMs for more on creating a cluster.
Creating frequent snapshots efficiently for more on the limit.

A device listed in /etc/fstab fails to mount

By default, the system parses /etc/fstab and mounts all of the listed devices to the required mount points. If a device is not recognized or not mounted, the workload pod will not get to a ready state.

For example, consider a source Linux VM in Amazon EC2 that has an ephemeral disk where persistence is not guaranteed. These disks are not streamed to the target, causing the container to fail on mounting it.

If this happens, you might see messages such as the following:

Unable to locate resolve [X] fstab entries: …
Error: Failed mount -o defaults /dev/mapper/mpathe-part /rootfs/mnt/ephemeral

You can work around this by either:

Editing /etc/fstab on the Linux VM to remove the device mount command.
Setting the HC_INIT_SKIP_MOUNT_FAILURES environment variable to configure the system to skip mount failures and continue.

To set the HC_INIT_SKIP_MOUNT_FAILURES environment variable:

Create a configmap in the migration namespace, v2k-system, on the migration processing cluster. For example, define the configmap in a file named jobs-config.yaml:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: jobs-config
  namespace: v2k-system
data:
  environment: |
    HC_INIT_SKIP_MOUNT_FAILURES = true
```
Apply the configmap to the cluster:
```
kubectl apply -f jobs-config.yaml
```

To view the config map, use the command:

kubectl describe configmaps -n v2k-system

Edit the migration plan to add the configmap. The migration plan is the yaml file that you generated when you created the migration. See Customizing a migration plan for more on editing the migration plan.

In the migration plan, edit the configs section to add the configmap:
```
configs:
  jobsConfig:
    name: jobs-config
```
Save your edits and then execute the migration as described in Executing a migration.

Failure in capabilities of deployed container because of AppArmor

AppArmor lets a system administrator restrict capabilities of a deployed container by using custom profiles. In some cases, you might have to apply a profile to your deployed container to customize its capabilities.

To customize the AppArmor profile:

Create the profile on the cluster where you are deploying your migrated container. See the AppArmor documentation for more information.

Edit the deployment_spec.yaml file to add the HC_APPARMOR_PROFILE environment variable with the name of the AppArmor profile:

spec:
  containers:
  - image: gcr.io/my-project/my-container:v1.0.0
    name: my-container
    env:
    - name: HC_APPARMOR_PROFILE
      value: "apparmor-profile-name"
    securityContext:
      privileged: true
...

See Reviewing generated deployment files for more on editing deployment_spec.yaml.

I would like personalized support

Paid support is available for customers migrating with Migrate for Anthos. Reach out so we can help.

Providing information to Google Cloud support

The Sysreport provides Migrate for Anthos support with information about your cluster's configuration for faster time to problem resolution.

You can access the script from Cloud Shell.

Open Cloud Shell

Next, run the collect_sysreport.sh script.

/google/migrate/anthos/collect_sysreport.sh [NAMESPACE] [DEPLOYMENT_NAME] [--workloads]



Where:


[NAMESPACE] is the namespace where your Migrate for Anthos
components were installed.
[DEPLOYMENT_NAME] is name given when you created your
Migrate for Anthos deployment in the GKE marketplace.
--workloads collects additional data from your migrated workloads.
See below for more information.


The script creates anthos-migrate-logs.TIMESTAMP.tar.xz, which you provide
to Google Cloud support.

By default, the script collects:


Logs from the Migrate for Anthos CSI controller and CSI nodes.
Syslog from the Migrate for Anthos CSI node hosts.
The output of:

kubectl cluster-info
kubectl get nodes; kubectl describe node
kubectl version
kubectl top node



With the --workloads flag enabled, for every workload the script collects:


The workload's logs.
The output of:

ps aux
netstat -tlnp
iptables -t nat -L
fstab
kubectl get pod
kubectl describe pod
kubectl top pod --all-namespaces --containers
kubectl cluster-info dump
kubectl api-resources -o wide
kubectl top pod --all-namespaces --containers
kubectl api-resources -o wide
kubectl get componentstatuses --all-namespaces
kubectl get endpoints --all-namespaces
kubectl get events --all-namespaces
kubectl describe limits --all-namespaces
kubectl get namespaces
kubectl describe pvc --all-namespaces
kubectl describe pv --all-namespaces
kubectl describe quota --all-namespaces
kubectl describe sa --all-namespaces
kubectl describe services --all-namespaces
kubectl describe services --all-namespaces
kubectl get ingresses --all-namespaces
kubectl describe networkpolicies --all-namespaces
kubectl get podsecuritypolicies --all-namespaces
kubectl get clusterrolebindings --all-namespaces
kubectl describe storageclasses --all-namespaces
kubectl describe volumeattachments --all-namespaces