Troubleshoot namespace stuck in the Terminating state

This page shows you how to resolve issues with a namespace stuck in the Terminating state.

Namespaces use Kubernetes finalizers to prevent deletion if one or more resources within a namespace still exist. When you delete a namespace using the kubectl delete command, the namespace enters the Terminating state. The namespace stays in the Terminating state until Kubernetes deletes its dependent resources and clears all finalizers. The namespace lifecycle controller first lists all resources in the namespace that GKE needs to delete.

If GKE can't delete a dependent resource, or if the namespace lifecycle controller can't verify that the namespace is empty, then the namespace remains in the Terminating state until you resolve the issue.

To resolve a namespace stuck in the Terminating state, identify and remove any unhealthy components blocking the deletion. Try these solutions in order:

  1. Find and remove unavailable API services.
  2. Find and remove remaining resources.
  3. Force delete the namespace.

Find and remove unavailable API services

  1. List unavailable API services:

    kubectl get apiservice | grep False

  2. Troubleshoot any unresponsive services:

    kubectl describe apiservice API_SERVICE

    Replace API_SERVICE with the name of the unresponsive service.

  3. Check if the status of the namespace is still Terminating:

    kubectl get ns | grep Terminating

Find and remove remaining resources

  1. List all the resources remaining in the terminating namespace:

    kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n NAMESPACE

    Replace NAMESPACE with the name of the namespace you want to delete.

  2. Remove any resources displayed in the output.

  3. Check if the status of the namespace is still Terminating:

    kubectl get ns | grep Terminating

Force delete the namespace

Remove the finalizers blocking namespace deletion to force the namespace to terminate:

  1. Save the namespace manifest as a YAML file:

    kubectl get ns NAMESPACE -o yaml > ns-terminating.yml

  2. Open the manifest in a text editor and remove all values in the spec.finalizers field:

    vi ns-terminating.yml

  3. Verify that the finalizers field is empty:

    cat ns-terminating.yml

    The output should look similar to the following:

    apiVersion: v1
kind: Namespace
metadata:
  annotations:
  name: NAMESPACE
spec:
  finalizers:
status:
  phase: Terminating

  4. Start an HTTP proxy to access the Kubernetes API:

    kubectl proxy

  5. Replace the namespace manifest using curl:

    curl -H "Content-Type: application/yaml" -X PUT --data-binary @ns-terminating.yml http://127.0.0.1:8001/api/v1/namespaces/NAMESPACE/finalize

  6. Check if the status of the namespace is still Terminating:

    kubectl get ns | grep Terminating

What's next