Migrated workload fails on 1.20 and later with Ubuntu nodes
AppArmor lets a system administrator restrict capabilities of a deployed container. To deploy a migrated container workload, Migrate to Containers requires the AppArmor profile provided by the Linux LXC package.
However, as part of the update to Google Kubernetes Engine (GKE) 1.20, Ubuntu node images no longer include the LXC package by default. That means new deployment clusters that use Ubuntu nodes, or Ubuntu nodes for existing deployment clusters that are upgraded to GKE 1.20 and later, cannot run your migrated workloads.
When deploying container workloads on Ubuntu nodes for GKE versions 1.20 and later, the workload can enter a crash-loop and the logs contain error messages in the form:
D0806 01:59:15.000000 8 hcutil.py:136] SHELL CMD: aa-exec -p lxc-container-default echo 123 D0806 01:59:15.000000 8 hcutil.py:168] SHELL STDERR: b"aa-exec: ERROR: profile 'lxc-container-default' does not exist\n" D0806 01:59:15.000000 8 hcutil.py:168] SHELL STDERR: b'\n' D0806 01:59:15.000000 8 hcutil.py:210] SHELL COMPLETED: 1 (aa-exec -p lxc-container-default echo 123) required AppArmor profile 'lxc-container-default' does not exist, if this pod runs on a GKE node of type 'Container Optimized OS' (COS) please run `migctl setup install --cos-runtime` to install it E0806 01:59:15.000000 8 hcmain.py:22] Traceback (most recent call last): File "./hcmain.py", line 19, in safe_action File "./hcmain.py", line 74, in run_action File "./hcrunner.py", line 392, in validateSystem Exception: Invalid system or AppArmor profile E0806 01:59:15.000000 8 hcmain.py:23] Invalid system or AppArmor profile I0806 01:59:15.000000 8 termination_log.py:4] writing error to termination log at /dev/termination-log
To resolve this issue:
Connect to the deployment cluster using a command in the form:
gcloud container clusters get-credentials CLUSTER --zone ZONE --project PROJECT
Manually install the Linux LXC package on the deployment cluster by using the command:
migctl setup install --cos-runtime