Troubleshooting

Learn about troubleshooting steps that you might find helpful if you run into problems using Migrate for Anthos 1.5.

Executing shell commands on your container

You can access a container through a bash shell or through PowerShell using the kubectl exec command.

  1. Use kubectl describe pods to find the name of the Pod in your cluster that you want to connect to.

    In the following example, the command lists the suitecrm-0 pod.

    kubectl describe pods | grep Name
    
    Name:               suitecrm-0
  2. Execute shell commands using one of the following methods:
    • Use kubectl exec to open a bash command shell where you can execute commands.
      kubectl exec -it pod-name -- /bin/bash

      The following example gets a shell to the suitecrm-0 pod:

      kubectl exec -it suitecrm-0 -- /bin/bash
    • Use kubectl exec to execute commands directly.
      kubectl exec -it pod-name -- /bin/bash -c "command(s)"

      The following example lists the root directory of the suitecrm-0 pod:

      kubectl exec -it suitecrm-0 -- /bin/bash -c "ls /"

For more information, see the Kubernetes documentation.

Debugging Kubernetes resources

More help is available at the following pages:

RESOURCE_OPERATION_RATE_EXCEEDED error appears in migration status or logs

The RESOURCE_OPERATION_RATE_EXCEEDED error occurs when the source platform of a migration is Compute Engine and you exceed the limit on disk snapshots when migrating a Compute Engine VM.

As part of the migration, Migrate for Anthos creates a snapshot of the disk images from a VM running on the source platform. You can only create a disk snapshot at most once every 10 minutes, or 6 times an hour, for a Compute Engine VM. This error indicates that you have exceeded that limit.

To avoid hitting this limit, we recommend that you create the migration cluster in the same zone as the Compute Engine VM. When the cluster is in the same zone as the VM, Migrate for Anthos can clone the disk, rather than create a snapshot, which is a more efficient process and bypasses the snapshot limit.

See:

A device listed in /etc/fstab fails to mount

By default, the system parses /etc/fstab and mounts all of the listed devices to the required mount points. If a device is not recognized or not mounted, the workload pod will not get to a ready state.

For example, consider a source Linux VM in Amazon EC2 that has an ephemeral disk where persistence is not guaranteed. These disks are not streamed to the target, causing the container to fail on mounting it.

If this happens, you might see messages such as the following:

Unable to locate resolve [X] fstab entries: …
Error: Failed mount -o defaults /dev/mapper/mpathe-part /rootfs/mnt/ephemeral

You can work around this by either:

  • Editing /etc/fstab on the Linux VM to remove the device mount command.
  • Setting the HC_INIT_SKIP_MOUNT_FAILURES environment variable to configure the system to skip mount failures and continue.

To set the HC_INIT_SKIP_MOUNT_FAILURES environment variable:

  1. Create a configmap in the migration namespace, v2k-system, on the migration processing cluster. For example, define the configmap in a file named jobs-config.yaml:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: jobs-config
      namespace: v2k-system
    data:
      environment: |
        HC_INIT_SKIP_MOUNT_FAILURES = true
    
  2. Apply the configmap to the cluster:

    kubectl apply -f jobs-config.yaml
  3. To view the config map, use the command:

    kubectl describe configmaps -n v2k-system
  4. Edit the migration plan to add the configmap. The migration plan is the yaml file that you generated when you created the migration. See Customizing a migration plan for more on editing the migration plan.

    In the migration plan, edit the configs section to add the configmap:

    configs:
      jobsConfig:
        name: jobs-config
    
  5. Save your edits and then execute the migration as described in Executing a migration.

Failure in capabilities of deployed container because of AppArmor

AppArmor lets a system administrator restrict capabilities of a deployed container by using custom profiles. In some cases, you might have to apply a profile to your deployed container to customize its capabilities.

To customize the AppArmor profile:

  1. Create the profile on the cluster where you are deploying your migrated container. See the AppArmor documentation for more information.

  2. Edit the deployment_spec.yaml file to add the HC_APPARMOR_PROFILE environment variable with the name of the AppArmor profile:

    spec:
      containers:
      - image: gcr.io/my-project/my-container:v1.0.0
        name: my-container
        env:
        - name: HC_APPARMOR_PROFILE
          value: "apparmor-profile-name"
        securityContext:
          privileged: true
    ...
    

    See Reviewing generated deployment files for more on editing deployment_spec.yaml.

I would like personalized support

Paid support is available for customers migrating with Migrate for Anthos. Reach out so we can help.

Providing information to Google Cloud support

The Sysreport provides Migrate for Anthos support with information about your cluster's configuration for faster time to problem resolution.

You can access the script from Cloud Shell.

  1. Open Cloud Shell

Next, run the collect_sysreport.sh script.

/google/migrate/anthos/collect_sysreport.sh [NAMESPACE] [DEPLOYMENT_NAME] [--workloads]

Where:

  • [NAMESPACE] is the namespace where your Migrate for Anthos components were installed.
  • [DEPLOYMENT_NAME] is name given when you created your Migrate for Anthos deployment in the GKE marketplace.
  • --workloads collects additional data from your migrated workloads. See below for more information.

The script creates anthos-migrate-logs.TIMESTAMP.tar.xz, which you provide to Google Cloud support.

By default, the script collects:

  • Logs from the Migrate for Anthos CSI controller and CSI nodes.
  • Syslog from the Migrate for Anthos CSI node hosts.
  • The output of:
    • kubectl cluster-info
    • kubectl get nodes; kubectl describe node
    • kubectl version
    • kubectl top node

With the --workloads flag enabled, for every workload the script collects:

  • The workload's logs.
  • The output of:
    • ps aux
    • netstat -tlnp
    • iptables -t nat -L
    • fstab
    • kubectl get pod
    • kubectl describe pod
    • kubectl top pod --all-namespaces --containers
    • kubectl cluster-info dump
    • kubectl api-resources -o wide
    • kubectl top pod --all-namespaces --containers
    • kubectl api-resources -o wide
    • kubectl get componentstatuses --all-namespaces
    • kubectl get endpoints --all-namespaces
    • kubectl get events --all-namespaces
    • kubectl describe limits --all-namespaces
    • kubectl get namespaces
    • kubectl describe pvc --all-namespaces
    • kubectl describe pv --all-namespaces
    • kubectl describe quota --all-namespaces
    • kubectl describe sa --all-namespaces
    • kubectl describe services --all-namespaces
    • kubectl describe services --all-namespaces
    • kubectl get ingresses --all-namespaces
    • kubectl describe networkpolicies --all-namespaces
    • kubectl get podsecuritypolicies --all-namespaces
    • kubectl get clusterrolebindings --all-namespaces
    • kubectl describe storageclasses --all-namespaces
    • kubectl describe volumeattachments --all-namespaces