Learn about troubleshooting steps that you might find helpful if you run into problems using Migrate for Anthos 1.7.
Executing shell commands on your container
You can access a container through a bash
shell or through PowerShell using the
kubectl exec
command.
- Use
kubectl describe pods
to find the name of the Pod in your cluster that you want to connect to.In the following example, the command lists the suitecrm-0 pod.
kubectl describe pods | grep Name Name: suitecrm-0
- Execute shell commands using one of the following methods:
- Use
kubectl exec
to open a bash command shell where you can execute commands.kubectl exec -it pod-name -- /bin/bash
The following example gets a shell to the suitecrm-0 pod:
kubectl exec -it suitecrm-0 -- /bin/bash
- Use
kubectl exec
to execute commands directly.kubectl exec -it pod-name -- /bin/bash -c "command(s)"
The following example lists the root directory of the suitecrm-0 pod:
kubectl exec -it suitecrm-0 -- /bin/bash -c "ls /"
- Use
For more information, see the Kubernetes documentation.
Debugging Kubernetes resources
More help is available at the following pages:
If you are experiencing a problem with Google Kubernetes Engine, see the GKE troubleshooting page.
If you are experiencing an issue related to your cluster, refer to Troubleshooting Clusters in the Kubernetes documentation.
If you are having an issue with your application, its pods, or its controller object, refer to Troubleshooting Applications.
RESOURCE_OPERATION_RATE_EXCEEDED error appears in migration status or logs
The RESOURCE_OPERATION_RATE_EXCEEDED
error occurs when the source platform of a migration
is Compute Engine and you exceed the limit on disk snapshots when migrating a
Compute Engine VM.
As part of the migration, Migrate for Anthos creates a snapshot of the disk images from a VM running on the source platform. You can only create a disk snapshot at most once every 10 minutes, or 6 times an hour, for a Compute Engine VM. This error indicates that you have exceeded that limit.
To avoid hitting this limit, we recommend that you create the migration cluster in the same zone as the Compute Engine VM. When the cluster is in the same zone as the VM, Migrate for Anthos can clone the disk, rather than create a snapshot, which is a more efficient process and bypasses the snapshot limit.
See:
Configuring a processing cluster on Google Cloud for Windows workloads for more on creating a cluster.
Creating frequent snapshots efficiently for more on the limit.
A device listed in /etc/fstab fails to mount
By default, the system parses /etc/fstab
and mounts all of the listed devices
to the required mount points. If a device is not recognized or not mounted,
the workload pod will not get to a ready state.
For example, consider a source Linux VM in Amazon EC2 that has an ephemeral disk where persistence is not guaranteed. These disks are not streamed to the target, causing the container to fail on mounting it.
If this happens, you might see messages such as the following:
Unable to locate resolve [X] fstab entries: …
Error: Failed mount -o defaults /dev/mapper/mpathe-part /rootfs/mnt/ephemeral
You can work around this by either:
- Editing
/etc/fstab
on the Linux VM to remove the device mount command. - Setting the
HC_INIT_SKIP_MOUNT_FAILURES
environment variable to configure the system to skip mount failures and continue.
To set the HC_INIT_SKIP_MOUNT_FAILURES
environment variable:
Create a configmap in the migration namespace,
v2k-system
, on the migration processing cluster. For example, define the configmap in a file namedjobs-config.yaml
:apiVersion: v1 kind: ConfigMap metadata: name: jobs-config namespace: v2k-system data: environment: | HC_INIT_SKIP_MOUNT_FAILURES = true
Apply the configmap to the cluster:
kubectl apply -f jobs-config.yaml
To view the config map, use the command:
kubectl describe configmaps -n v2k-system
Edit the migration plan to add the configmap. The migration plan is the yaml file that you generated when you created the migration. See Customizing a migration plan for more on editing the migration plan.
In the migration plan, edit the
configs
section to add the configmap:configs: jobsConfig: name: jobs-config
Save your edits and then execute the migration as described in Executing a migration.
Failure in capabilities of deployed container because of AppArmor
AppArmor lets a system administrator restrict capabilities of a deployed container by using custom profiles. In some cases, you might have to apply a profile to your deployed container to customize its capabilities.
To customize the AppArmor profile:
Create the profile on the cluster where you are deploying your migrated container. See the AppArmor documentation for more information.
Edit the
deployment_spec.yaml
file to add theHC_APPARMOR_PROFILE
environment variable with the name of the AppArmor profile:spec: containers: - image: gcr.io/my-project/my-container:v1.0.0 name: my-container env: - name: HC_APPARMOR_PROFILE value: "apparmor-profile-name" securityContext: privileged: true ...
See Reviewing generated deployment files for more on editing
deployment_spec.yaml
.
FailedAttachVolume error appears in migration status or logs (preview)
If you are performing a migration using Anthos clusters on AWS
to perform the migration,
you might see the FailedAttachVolume
error in the migration status or logs
with the accompanying message of:
"The encrypted volume was unable to access the KMS master key"
This error means that your AWS VM uses an encrypted disk and you have not configured access correctly. See Prerequisites for migrating Linux VMs using AWS processing clusters for more.
ProvisioningFailed error appears in migration status or logs (preview)
If you are performing a migration using Anthos clusters on AWS
to perform the migration,
you might see the ProvisioningFailed
error in the migration status or logs
with the accompanying message of:
"ProvisioningFailed - "Could not create volume V_ID"
This error can occur when you try to export data into a new encrypted volume, but the cluster has no access to the encryption key, and therefore fails to create the EBS disk. See Prerequisites for migrating Linux VMs using AWS processing clusters for more.
I would like personalized support
Paid support is available for customers migrating with Migrate for Anthos. Reach out so we can help.
Providing information to Google Cloud support
The Sysreport provides Migrate for Anthos support with information about your cluster's configuration for faster time to problem resolution.
You can access the script from Cloud Shell.
- Open Cloud Shell
-
Next, run the
collect_sysreport.sh
script./google/migrate/anthos/collect_sysreport.sh [-n NAMESPACE] [-o OUTPUT_DIRECTORY] [-m MIGRATION]
Where:
- [NAMESPACE]: (Optional) The namespace in which Migrate for Anthos
was deployed (default is
v2k-system
) . - [OUTPUT_DIRECTORY]: (Optional) Path to the directory
in which the Sysreport will be saved (default is
$HOME
). - [MIGRATION]: (Optional) Additional information collected for a migration.
The script creates anthos-migrate-logs.TIMESTAMP.tar.xz
,
which you provide to Google Cloud support.
By default, the script collects:
- Logs from the Migrate for Anthos CSI controller and CSI nodes.
- Syslog from the Migrate for Anthos CSI node hosts.
- Logs from Migrate for Anthos controller.
- All Migrate for Anthos entities from the cluster.
- If you specify a migration, and the migration uses the default artifacts repository, collect the migration artifacts from the Cloud Storage or S3 bucket.
- The output of:
kubectl cluster-info
kubectl get nodes; kubectl describe node
kubectl version
kubectl top node
- The workload's logs.
- The output of:
ps aux
netstat -tlnp
iptables -t nat -L
fstab
kubectl get pod
kubectl describe pod
kubectl top pod --all-namespaces --containers
kubectl cluster-info dump
kubectl api-resources -o wide
kubectl top pod --all-namespaces --containers
kubectl api-resources -o wide
kubectl get componentstatuses --all-namespaces
kubectl get endpoints --all-namespaces
kubectl get events --all-namespaces
kubectl describe limits --all-namespaces
kubectl get namespaces
kubectl describe pvc --all-namespaces
kubectl describe pv --all-namespaces
kubectl describe quota --all-namespaces
kubectl describe sa --all-namespaces
kubectl describe services --all-namespaces
kubectl describe services --all-namespaces
kubectl get ingresses --all-namespaces
kubectl describe networkpolicies --all-namespaces
kubectl get podsecuritypolicies --all-namespaces
kubectl get clusterrolebindings --all-namespaces
kubectl describe storageclasses --all-namespaces
kubectl describe volumeattachments --all-namespaces