Cloud NAT prevents Composer environment deletion

Problem

When you try to delete a Composer environment, it times out and fails with the following error message:

Google Compute Engine: The subnetwork resource 'projects/<project-id>/regions/<region>/subnetworks/<subnet-name>' is already being used by 'projects/<project-id>/regions/<region>/nats/<nat-gateway-name>'.

You have a private IP Composer environment and configured Cloud NAT for it (therefore having a Cloud NAT gateway for its pods and services secondary IP ranges).

Environment

  • All Cloud Composer environments

Solution

  1. Remove the pods and services' secondary IP ranges being used by the environment's Google Kubernetes Engine cluster from the Cloud NAT gateway configuration before deleting the environment.
Note: It is not necessary to remove the primary address range from the NAT gateway configuration.

Cause

Having Composer automatically create pods and services' secondary IP ranges and then configuring Cloud NAT for the subnet and these ranges makes it so the environment deletion will fail. This is because Composer will remove the secondary IP ranges when the environment is deleted.

Since these ranges are being used by Cloud NAT, Composer is not allowed to delete them and thus the overall operation fails. If the pods and services' secondary IP ranges were to be created manually and indicated in the environment creation request, the issue does not present itself because Composer will not try to delete the secondary IP ranges (given that they were not created by Composer).