Troubleshoot namespace stuck in the Terminating state

Autopilot Standard

---

This page shows you how to resolve issues with a namespace stuck in the Terminating state.

Namespaces use Kubernetes finalizers to prevent deletion if one or more resources within a namespace still exist. When you delete a namespace using the kubectl delete command, the namespace enters the Terminating state. The namespace stays in the Terminating state until Kubernetes deletes its dependent resources and clears all finalizers. The namespace lifecycle controller first lists all resources in the namespace that GKE needs to delete.

If GKE can't delete a dependent resource, or if the namespace lifecycle controller can't verify that the namespace is empty, then the namespace remains in the Terminating state until you resolve the issue.

To resolve a namespace stuck in the Terminating state, identify and remove any unhealthy components blocking the deletion. Try these solutions in order:

1. Find and remove unavailable API services.
2. Find and remove remaining resources.
3. Force delete the namespace.

Find and remove unavailable API services

1. List unavailable API services:

   kubectl get apiservice | grep False
   

2. Troubleshoot any unresponsive services:

   kubectl describe apiservice API_SERVICE
   

   Replace API_SERVICE with the name of the unresponsive service.

3. Check if the status of the namespace is still Terminating:

   kubectl get ns | grep Terminating
   

Find and remove remaining resources

1. List all the resources remaining in the terminating namespace:

   kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n NAMESPACE
   

   Replace NAMESPACE with the name of the namespace you want to delete.

2. Remove any resources displayed in the output.

3. Check if the status of the namespace is still Terminating:

   kubectl get ns | grep Terminating
   

Force delete the namespace

Caution: Use the solution in this section only as a last resort if the previous solutions don't work. Force deleting a namespace could result in resources remaining in your cluster that you can't access.

Remove the finalizers blocking namespace deletion to force the namespace to terminate:

1. Save the namespace manifest as a YAML file:

   kubectl get ns NAMESPACE -o yaml > ns-terminating.yml
   

2. Open the manifest in a text editor and remove all values in the spec.finalizers field:

   vi ns-terminating.yml
   

3. Verify that the finalizers field is empty:

   cat ns-terminating.yml
   

   The output should look similar to the following:

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
     name: NAMESPACE
   spec:
     finalizers:
   status:
     phase: Terminating
   

4. Start an HTTP proxy to access the Kubernetes API:

   kubectl proxy
   

5. Replace the namespace manifest using curl:

   curl -H "Content-Type: application/yaml" -X PUT --data-binary @ns-terminating.yml http://127.0.0.1:8001/api/v1/namespaces/NAMESPACE/finalize
   

6. Check if the status of the namespace is still Terminating:

   kubectl get ns | grep Terminating
   

What's next

If you need additional assistance, reach out to Cloud Customer Care.