Troubleshoot Kf Spaces

Use these steps to troubleshoot various issues that can occur when using Kf Spaces.

Object is stuck deleting

Run the following command to get the resource information, then check for the causes listed below:

kubectl get spaces.kf.dev SPACE_NAME -o yaml

The kf CLI can help check for some of the issues:

kf doctor space/SPACE_NAME
Possible CauseSolution
Deletion timestamp is in the future.

With clock skew the metadata.deletionTimestamp may still be in the future. Wait a few minutes to see if the object is deleted.

Finalizers exist on the object.

Finalizers are present on the object, they must be removed by the controller that set them before the object is deleted.

If you want to force a deletion without waiting for the finalizers, edit the object to remove them from the metadata.finalizers array.

Warning: Removing finalizers without allowing the controllers to complete may cause errors, security issues, data loss, or orphaned resources.

Dependent objects may exist.

The object may be waiting on dependents to be deleted before it is deleted. See the Kubernetes garbage collection guide to learn more. Have an administrator check all objects in the namespace and cluster to see if one of them is blocking deletion.

If you need to remove the object without waiting for dependents, use kubectl delete with the cascade flag set to: --cascade=orphan.

Object generation state drift

Run the following command to get the resource information, then check for the causes listed below:

kubectl get spaces.kf.dev SPACE_NAME -o yaml

The kf CLI can help check for some of the issues:

kf doctor space/SPACE_NAME
Possible CauseSolution
Object has generation version drift.

This error usually occurs Kf controller did not read the latest version of the object, this error is usually self-recovered once Kubernetes replicas reach eventual consistency, and it usually does not require action from users.

Object reconciliation failed

Run the following command to get the resource information, then check for the causes listed below:

kubectl get spaces.kf.dev SPACE_NAME -o yaml

The kf CLI can help check for some of the issues:

kf doctor space/SPACE_NAME
Possible CauseSolution
Object has TemplateError

This error usually occurs if user has entered an invalid property in the custom resource Spec, or the configuration on the Space/Cluster is bad.

To understand the root cause, user can read the longer error message in the object’s status.conditions using the command:kubectl describe RESOURCE_TYPE RESOURCE_NAME -n space. For example: kubectl describe serviceinstance my-service -n my-space.

Object has ChildNotOwned error (Name conflicts)

This error usually means that the object(s) the controller is trying to create already exists. This happens if the user created a K8s resource that has the same name as what the controller is trying to create; but more often it happens if user deletes a resource then Kf controller tries to re-create it. If a child resource is still hanging around, its owner will be the old resource that no longer exists.

To recover from the error, it is recommended that user deletes the impacted resource and then recreates it. To delete the object, use a Kf deletion command or use the kubectl delete RESOURCE_TYPE RESOURCE_NAME -n SPACEcommand. For example, kf delete-space my-space or kubectl delete space my-space.

To recreate a resource, use a Kf command. For example: kf create-space my-space.

Object has ReconciliationError

This error usually means that something has gone wrong with the HTTP call made (by Kf controller) to the Kubernetes API servier to create/update resource.

To understand the root cause, user can read the longer error message in the object’s status.conditions using the command:kubectl describe RESOURCE_TYPE RESOURCE_NAME -n space. For example: kubectl describe serviceinstance my-service -n my-space.