NSX-T password expired SDDC Manager rotation fails

NSX-T password expired SDDC Manager rotation fails

With VMware Cloud Foundation password management of SDDC components is handed off to the SDDC manager.  The SDDC manager can rotate the passwords for VMware components such as ESXi, Virtual Centre and NSX-T.  A very useful feature and a great way. of keeping admin and root accounts secure in the infrastructure.  So what happens when the rotation is perhaps missed and the NSX-T passwords expire?

This is exactly the situation a friend found themselves in recently.  The NSX-T manager policies caused an expiration of the passwords for the root and admin accounts.  This in turn meant that password rotation scripts from the SDDC manager would not run.  Now by logging onto the managers via SSH the accounts prompt for password resets, but wouldn’t that break the SDDC manager? How do I avoid breaking the SDDC manager? Why can’t I find any documentation about this on the VMware website, surely someone has come across this before!

Well I couldn’t find anything either, so it might be that folk have not experienced this before.  Logically we should be able to reset the passwords via SSH and then set them back to what the SDDC manage expects for password rotation.  Sometime logic doesn’t work so does this actually work if put to the test.

So I’m going to break a lab environment and then try and fix it.

What does SDDC manager think my passwords are?

The quickest way to do this is to use the SDDC manager API to send a get credential request across the environment, from the SDDC manager itself.

Running this API call will provide you with JSON output that contains the passwords for the environment, or as we say above what SDDC manager thinks your passwords are.  The output is JSON, so to make it easier to consume using Notepad++ or similar to read it will make it much easier to find the information that you want.  And what you want is the passwords that SDDC manager thinks are in use, so you can set them back.

I’ve blanked out the Passwords, names and IPs.

Resetting passwords via SSH

If the passwords have expired when you try and log on from an SSH session the first action you’ll be asked is to reset the passwords.

Admin

In my lab the passwords were not expired so I changed them from the command line.  Logging in as admin you’ll be prompted with NSX CLI so the command is;

set user admin password passwordstring

Root

Logging in as root, the CLI is bash so the command to change the password is;

 

passwd

Failed process

So running the password rotation from within SDDC manager, we expect it to fail as the passwords are now out of sync.  Sure enough as you can see below the process fails.

In the details of the task we can see that the process fails when trying to rotate the admin password and so stops the process before going onto the root account.

Setting passwords back

For my lab environment for this test the SDDC manager thinks the passwords are “VMware123!VMware123!” for both root and admin.  We’re going to set those back.

Admin

Root

Working process

So running the password rotation from within SDDC manager, we now expect it to work, and sure enough the password rotation works!

Gotchas!

NSX CLI

set user admin password myPaswordContainsa?

Passing a “?” in the password sting of the command to change the admin password doesn’t work.  Annoyingly, what happens is that a password string with a “?” the NSX CLI takes that as a request to provide help on the command.  Which is not particularly helpful.

Timeouts

Having reset passwords a couple of times, the first time I tried to rotate the passwords I got a lockout error.  So I left it 20 minutes and it started to work.  I’m not sure if that was linked to me resetting stuff to quickly or causing the failure initially. However, it is something to consider if you do hit a failure first up might be worth leaving it for a spell and picking it up in 20 minutes.

So to wrap up, if the NSX-T passwords expire use the available CLI tools to set them to what the SDDC manager expects and the password rotation will start to work again.

Hopefully this is useful to someone!

Thanks

Simon