vSphere Kubernetes – Force deleting stuck ‘terminating’ namespaces and contexts

Like many vSphere administrators and architects following the release of vSphere 7 and VMware Cloud Foundation 4 I’ve been upgrading my lab and starting to play with the VMware Tanzu Portfolio.  I’m not going to detail how to get that working, Cormac Hogan and David Stamen have put together great guides for that.  This post is just posting a fix for a problem that I’ve encountered in the lab

Namespaces and Contexts stuck ‘terminating’

Something that I’ve come across has been namespaces stuck in a terminating state.  For this example I’d deleted the namespace within vCentre, but for whatever reason the action was getting stuck before finalising on the kubernetes cluster.  The namespace and context below was created for a TanzuKubernetesCluster. In the yaml I’d made an error and tried to deploy the cluster with 100 worker virtualmachines using a virtualmachineclass of guaranteed-medium – with maps to 2 vCPUs each and 8Gb RAM.  It’s no surprise that this caused me problems in the lab!

To resolve the issue and to force the deletion of the namespace that we want to remove we need to perform a couple of actions;

Firstly we’re going to proxy in one terminal session

kubectl proxy

From another session we’re going to run

kubectl get ns tkg-guest-02 -o json | jq '.spec.finalizers=[]' | curl -X PUT http://localhost:8001/api/v1/namespaces/tkg-guest-02/finalize -H "Content-Type: application/json" --data @-

Sure enough the namespace has been removed.

We’re not quite done, because the context was still there;

Removing the context is completing by the following command;

kubectl config delete-context tkg-guest-02

And we’re done.

The Namespace and the Context that was stuck terminating have been removed, can be recreated and used as needed.

Hopefully this is useful for someone, thanks

Simon