Release Notes

This page documents production updates to Migrate for Anthos. You can periodically check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

You can see the latest product updates for all of Google Cloud on the Google Cloud release notes page.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/migrateanthos-release-notes.xml

November 17, 2020

On November 17, 2020 we released Migrate for Anthos 1.5.1.

You must upgrade your installation to install 1.5.1, even if you are currently running version 1.5. We strongly recommend that you always upgrade to the latest release.

170604382: Running migctl when not connected to a cluster no longer results in a panic error, but instead returns an error message describing the issue.

169919740: When using a custom services blocklist to disable a service in a workload, if the service was already disabled by default, the migrated container no longer can crash when deployed.

171173082: Mistakenly creating a local VMware source on a Cloud-based cluster, normally used only in an on-prem migration, no longer results in the source being in PROCESSING state forever but instead returns an error.

170566991: For Windows migrations, only HTTP and HTTPS site bindings are supported. See WindowsGenerateArtifacts CRD.

170618192: Similarly to Linux migrations, Windows migrations now add to the generate artifacts object an annotation containing the migration spec and comments.

When creating a migration source for Compute Engine workload, Migrate for Anthos now tests that the GCP specified project exists. Source creation fails when either the project does not exist or the user has no read permissions to the project. An error message is shown indicating that the required project could not be found.

October 21, 2020

On October 21, 2020 we released Migrate for Anthos 1.5.

Support for migrating Windows VM workloads has moved from the Beta stage to general availability. This release also extends the Google Cloud Console to support migrations of Windows workloads using Migrate for Anthos. See Migrating a Windows VM for more.

Migrate for Anthos offers new tools that you run on a Linux or Windows VMs to determine the workload's fit for migration to a container. See Using the Linux discovery tool and Using the Windows discovery tool for more.

Custom Services Blocklist is a new feature that lets you modify the default set of services to disable in a migrated container. See Custom Services Blocklist for more.

The image field value of the GenerateArtifactsFlow CRD defines the names and locations of two images created from a migrated VM. In previous releases, the names contained a predefined tag.

To ensure that the tag value is unique, the format of the tag has changed for this release to specify the timestamp of the migration.

You can also explicitly set the tag if you prefer to another value. See Setting the name of the container image for more.

When you deploy your migrated Windows containers to a cluster, you can now use a Group Managed Service Account (gMSA) to execute the container under a specific service account identity. See Configuring gMSA for more.

Changed the default settings on the Cloud processing cluster for migrating Linux workloads:

  • You no longer have to specify the --scopes "cloud-platform" option when creating Cloud processing clusters for migrating Linux workloads.

  • You now must create a service account, with the necessary permissions, to:

    • Accessing Container Registry and Cloud Storage

    • Use Compute Engine as the migration source

If you currently have a processing cluster that uses the --scopes "cloud-platform" option, your cluster will continue to work. However, for new processing clusters, you should use the new procedure. See Enabling Google services and configuring service accounts for more.

171123825: In some cases, migration process might fail, and Cloud Logging indicate errors such as:

"failed to load map, error 6"

or:

"failed in domap for addition of new path sdd"

Workaround: Delete the migration and restart it. In rare cases, a re-installation of the product is required.

170706786: The Linux Discovery Tool might return exit code 0 even when not all information was collected successfully.

Workaround: Make sure you run the tool as a 'root' user or as a user with full sudo access.

167656057: Installation on a GKE cluster with ACM might fail. Indication of the error can be seen in the Migrate for Anthos upgrade job, in the v2k-system namespace.

For example:

kubectl logs -n v2k-system controllers-upgrade-fzlmz

Shows this error:

failed to validate admission controller - admission webhook "validation.gatekeeper.sh" does not support dry run

Workaround: gatekeeper is an ACM component. Manually deleting the upgrader job fixes the issue.

For example:

kubectl delete job -n v2k-system controllers-upgrade

157062328: In some cases, adding a service to the blocklist using a configmap will not actually stop that service from running on the deployed workload.

Workaround: Disable the service using the Dockerfile (rather than a config-map), and rebuild the image.

163800225: kubectl port-forward might not work properly for a deployed workload.

Please contact support for more information.

171173082: Mistakenly creating a local VMware source on a Cloud-based cluster, normally used only in an on-prem migration, results in the source being in PROCESSING state forever.

For example, you use migctl to check the source status:

migctl source status local-vmw-src

The State displays as:

PROCESSING Message: Post "https://1.2.3.4/sdk": context deadline exceeded

Workaround: Delete the local VMware source, and create a remote/streaming VMware source.

170604382: Running migctl when not connected to a cluster results in a panic error such as the one below, followed by a stack-trace:

migctl setup install panic: Cannot create kubernetes client

Workaround: Connect a cluster, and re-run migctl.

171714535: In a GKE on-prem environment configured to use an egress HTTP/HTTPS proxy, the migration process might get stuck.

Workaround: Please contact support for more information.

170566991: For Windows migrations, only HTTP and HTTPS site bindings are supported.

Example of unsupported bindings:

<site name="Default Web Site" id="1">
  <application path="/">
    <virtualDirectory path="/" physicalPath="%SystemDrive%\inetpub\wwwroot" />
  </application>
  <bindings>
    <binding protocol="http" bindingInformation="*:80:" />
    <binding protocol="net.tcp" bindingInformation="808:*" />
    <binding protocol="net.pipe" bindingInformation="*" />
    <binding protocol="net.msmq" bindingInformation="localhost" />
    <binding protocol="msmq.formatname" bindingInformation="localhost" />
  </bindings>
</site>

Workaround: Edit the migration-plan to remove the unsupported binding.

169919740: When using a custom services blocklist to disable a service in a workload, ensure that the service is not already disabled by default. See Services disabled by Migrate for Anthos for a list of services disabled by default. If the service was already disabled by default, the migrated container might crash when deployed. Error information about the crash is written to the logs.

Workaround: Remove the already disabled service from your custom services blocklist.

170627229: Migrated workload of a JBoss application might fail at startup. Cloud Logging indicates an error in the form:

ERROR [org.jboss.as.server] (Controller Boot Thread) ...:
Caught exception during boot: java.lang.IllegalStateException: ...:
Could not rename /opt/jboss-7.1.1/standalone/configuration/.../standalone_xml_history/current to
/opt/jboss-7.1.1/standalone/configuration/.../standalone_xml_history/...

Workaround: Update your Dockerfile to remove the directory by adding a line in the form:

RUN rm -rf jboss-path/standalone/configuration/standalone_xml_history/current

Where jboss-path is a path such as /usr/local/share/jboss or /opt/jboss-7.1.1.

September 24, 2020

On September 24, 2020 we updated Migrate for Anthos 1.4.

Changed the default settings on the Cloud processing cluster for migrating Linux workloads:

  • You no longer have to specify the --scopes "cloud-platform" option when creating Cloud processing clusters for migrating Linux workloads.

  • You now must create a service account, with the necessary permissions, to:

    • Accessing Container Registry and Cloud Storage

    • Use Compute Engine as the migration source

If you currently have a processing cluster that uses the --scopes "cloud-platform" option, your cluster will continue to work. However, for new processing clusters, you should use the new procedure. See Enabling Google services and configuring service accounts for more.

You can now use the Google Cloud Console to:

  • Install Migrate for Anthos on a processing cluster
  • Create a migration source for a Compute Engine VM

See Installing Migrate for Anthos and Adding a migration source for more.

July 28, 2020

On July 28, 2020 we released Migrate for Anthos 1.4.

For instructions on upgrading from version 1.3, see Upgrading Migrate for Anthos to 1.4.

Added support for Anthos GKE on-prem clusters running on VMware. On-prem support lets you migrate source VM workloads in a vCenter/vSphere environment to a GKE on-prem cluster running in the same vCenter/vSphere environment. See Prerequisites for migrating Linux VMs on-prem for the requirements for on-prem migration.

The Google Cloud Console provides a web-based, graphical user interface that you can use to manage your Google Cloud Console(GCP) projects and resources. Migrate for Anthos now supports the migration of workloads by using the Google Cloud Console. See Migrate for Anthos management interfaces.

In this release, the Migrate for Anthos Google Cloud Console does not support migrations for Windows or for on-prem, including monitoring Windows or on-prem migrations.

You can use Migrate for Anthos to migrate Windows VMs to workloads on GKE. This process clones your Compute Engine VM disks and uses the clone to generate artifacts (including a Dockerfile and a zip archive with extracted workload files and settings) you can use to build a deployable container image. This feature is currently in Beta. See Adding a Windows migration source.

Migrate for Anthos now includes Custom Resource Definitions (CRDs) that enable you to easily create and manage migrations using an API automation solution or code. For example, you can use these CRDs to build your own automated tools.

For more on the Migrate for Anthos CRDs, see CRD overview.

Added the new --json-key sa.json option to the migctl source create ce command to create a migration for Compute Engine, where sa.json specifies a service account. See Optionally creating a service account when using Compute Engine as a migration source for more.

To edit the migration plan, you must now use the migctl migration get my-migration command to download the plan. After you are done editing the plan, you have to upload it by using the migctl migration update my-migration command. See Customizing a migration plan for more.

Added the node-selectors and tolerations options to the migctl setup install installation command that lets you install Migrate for Anthos on a specific set of nodes or node pools in a cluster. See Installing Migrate for Anthos.

The migctl migration cleanup command has been removed and is no longer necessary.

In previous releases, you used a command in the form: migctl source create ce my-ce-src --project my-project --zone zone to create a migration for Compute Engine. The --zone option has been removed when creating a Compute Engine migration. Using the --zone option in this release causes an error.

The migctl migration logs command has been removed. You now use the Google Console to view logs.

By default Migrate for Anthos installs to and performs migrations in the v2k-system namespace. In previous releases, you could specify the namespace. The option to specify a namespace has been removed.

GKE on-prem preview: If a source was created with migctl source create using the wrong credentials, you could not delete the migration with migctl migration delete. This issue has been fixed in the GA release of on-prem support.

160309992: Editing a migration plan from the GUI console might fail if it was also edited using migctl.

161135630: Attempting multiple migrations of the same remote VM (from VMware, AWS or Azure) simultaneously, might result in a stuck migration process.

Workaround: Delete the stuck migration.

161214397: In case of a missing service-account to upload container images to the Container Registry, the migration might get stuck.

Workaround: Add the service-account. If you are using the Migrate for Anthos CRD API, delete the GenerateArtifactsTask object using kubectl and recreate it either using the CRD or re-running migctl migration generate-artifacts. Alternatively, you can use migctl to delete the migration and recreate it. You can first download the migration YAML using migctl migration get to backup any customizations you have made.

161110816: migctl migration create with a source that doesn't exist fails with a non-informative error message: request was denied.

161104564: Creating a Linux migration with wrong os-type specification causes the migration process to get stuck until deleted.

160858543, 160836394, 160844377, 154430477, 154403665, 153241390,153239696, 152408818, 151516642, 132002453: Unstable network in Migrate for Compute Engine infrastructure, or a GKE node restart, might cause migration to get stuck.

Workaround: Delete the migration and re-create it. If recreating the migration does not solve the issue, please contact support.

161787358: In some cases, upgrading from version v1.3 to v1.4 might fail with Failed to convert source message.

Workaround: Re-run the upgrade command.

153811691, 153439420: Migrate for Anthos support for older Java does not handle OpenJDK 7 and 8 CPU resource calculations.

152974631: Using GKE nodes with CPU and Memory configurations below the recommended values might cause migrations to get stuck.

157890913, 160082702, 161125635, 159693579:A migration might continue to indicate that it is running, while an issue encountered prevents further processing.

Workaround: Check event messages on the migration object using the verbose migctl status command: migctl migration status migration_name -v. You might be able to correct the issue to allow the migration to continue or the migration should be deleted and recreated if an Error event is listed without further retries.

An example is when creating a Windows migration on a cluster with no Windows nodes. In this case the event message will show:

Warning FailedScheduling 10s Pod discover-xyz 0/1 nodes are available: 1 node(s) didn't match node selector.

March 30, 2020

v1.3.0

New migctl CLI for deploying Migrate for Anthos, creating and operating migrations using a structured workflow and a migration processing cluster.

Introducing a unified migration workflow across all supported VM sources -- VMware, AWS EC2, Azure VMs and Compute Engine VMs.

Migrations are defined and operated using a Kubernetes CRD.

Automated generation of a suggested migration plan (specified in a CRD), CI/CD artifacts and deployment specs. The migration process now results in extracting and generating container and deployment artifacts, including a container image and a Dockerfile, extracted data in a persistent volume, deployment/statefulSet, PVC and PV specs in an auto-generated YAML file for easy workload deployment.

The Migrate for Anthos software runtime layer now offers a compatibility feature for older Java versions that are not container aware by reflecting the correct resource allocations in the container's /proc file system.

Migrate for Anthos v1.0 Marketplace deployment is now removed. Migrate for Anthos v1.3 allows installation in v1.0 compatibility mode where needed.

Preview features -- contact your Google Sales representative to enroll.

  • Migrating Windows VMs with IIS ASP.NET web applications to Windows 2019 containers on GKE.
  • Processing migrations in Anthos on-prem.

147211918: In some cases, migration from AWS or Azure as a source can be stuck with no progress. If this happens, run kubectl describe storageclass to view related events. You can also check the status of the matching Cloud Details in Migrate for Compute Engine.

146699220: When the source VM has a systemd service with a NICE config property, the service might not start when running in a container.

Workaround: Remove the NICE property in the source VM before the migration.

144896313: Migration of Security-Enhanced Linux (SELinux) is not supported.

149900626: Some file systems not listed in Compatible VM operating systems may fail to migrate. When running migctl migration logs migration-name, the logs in Cloud Logging may show a message such as:

failed to mount - exit status 32 - mount: /tmp/bootdir: unknown filesystem type 'LVM2\_member'.

152194161: Your migrated workload container fails when running a cluster with GKE nodes of type "COS". When you run the command kubectl logs [podname], you might see the following:

apparmor.go:385] Couldn't set label to lxc-container-default - write /proc/1/attr/current: no such file or directory

This is an indication that the required AppArmor profiles are not installed on the GKE nodes. To solve this, run migctl setup install --cos-runtime.

148334068: When Migrating a physical VM from on-premises connected via Migrate for Compute Engine, Migrate for Anthos attempts to optimize network utilization and discards (rather than stream) blocks that are not in use on the source VM file system. In some cases, this might cause VMware storage sessions to time out. For assistance, please contact support.

GKE on-prem preview: If a source was created with migctl source create using the wrong credentials, migrations will fail. Attempts to delete the migration with migctl migration delete may hang in a "Terminating" state, as in the following example:

$ migctl migration list
NAME       PROCESS              STATE                   STATUS        PROGRESS   AGE
my-vm-01   generate-artifacts   createSourceSnapshots   TERMINATING   [2/13]

December 19, 2019

v1.0.1

When specifying a mixed-case value for the clone_vm_disks script's -A <var>app-name</var> argument, the YAML file generated by the script would include a workload name that could not be deployed.

The command now checks for a valid input value.

The Migrate for Compute Engine password could be inadvertently logged in Stackdriver Logging.

Migrate for Anthos failed to recognize a reference to a disk specified as a PARTUUID in the fstab file. PARTUUID is now supported.

Deleting a StatefulSet attached to a persistent volume would leave the volume in an attached state./p>

Using configuration YAML from a version prior to 1.0 caused the pod to enter a crashloop stage.

An error message now is now displayed to request that you update to the latest definition.

When resolving block devices in a multipath device, the operation appeared to succeed even if there was an error with one of the block devices.

Resolving source storage devices would sometimes fail without error if one of the devices has no partitions.

Using kubectl exec on a migration pod would sometimes display superfluous bash warnings about LC_ALL.

Attempting to switch to a non-root user with the su command after connecting to the machine with ssh would fail when you had previously used su to switch to another user.

Migrate for Anthos CSI drive would sometimes fail connecting to the migrated VM.

The kubectl cp command would fail when copying files to the migrated pod.

November 13, 2019

v1.0.0

Migrate for Anthos supports migrating existing VMware, Amazon EC2, Azure, and Compute Engine VMs to containers on Google Kubernetes Engine. For more information, see Benefits of Migrate for Anthos.

You can monitor export of short-term storage to a persistent volume using kubectl. For more information, see Exporting streaming PVs to permanent storage.

Using a ConfigMap, you can have content from application log files you specify written to Stackdriver Logging (a default list is included). For more, see Configuring logging to Stackdriver Logging.

For information on operating systems supported by Migrate for Anthos, see Compatible VM operating systems.

On the Migrate for Compute Engine portlet in VMWare vCenter, VMs will be shown as Managed by Migrate for Compute Engine during migration process. Only the cache and storage migration status are updated in this view. Other functionality, such as Migrate for Compute Engine actions, may not be functional.

For known issues and workarounds, see Troubleshooting Migrate for Anthos.

VMs using EFI configurations are not compatible for migration with this release.

Operating systems running systemd versions lower than 234 are limited to 65536 open files.

When using a private GKE cluster, the GKE master might be unable to reach Migrate for Anthos infrastructure (specifically, the admission-controller) by default. This is because the admission-controller pod listens on port 9443.

To work around this issue, add port 9443 to the firewall rules of the master node. For more, see Creating a private cluster.

When specifying a mixed-case value for the clone_vm_disks script's -A <var>app-name</var> argument, the YAML file generated by the script includes a workload name that can not be deployed.

To work around this issue, specify the argument's value in lowercase only.

The Migrate for Compute Engine password can be inadvertently logged in Stackdriver Logging.

Migrate for Anthos fails to recognize a reference to a disk specified as a PARTUUID in the fstab file.

Deleting a StatefulSet attached to a persistent volume will leave the volume in an attached state.

Using configuration YAML from a version prior to 1.0 causes the pod to enter a crashloop stage.

To work around this, update your YAML file to conform to the latest definition.

When resolving block devices in a multipath device, the operation appears to succeed even if there was an error with one of the block devices.

Resolving source storage devices would sometimes fail without error if one of the devices has no partitions.

Using kubectl exec on a migration pod sometimes displays superfluous bash warnings about LC_ALL. These are only cosmetic.

Attempting to switch to a non-root user with the su command after connecting to the machine with ssh fails when you have previously used su to switch to another user.

To work around this issue, use kubectl exec instead of ssh to get a shell to the container.

Migrate for Anthos CSI drive may sometimes fail connecting to the migrated VM.

The kubectl cp command fails when copying files to the migrated pod.