PowerShell CloudConfig and vRealize Cloud Assembly

Adventures with PowerShell, CloudConfig and vRealize Cloud Assembly

This week I had a customer with some errors trying to use the following componenets together: PowerShell, CloudConfig, and cloudbase-init within a Cloud Assembly blueprint.  To put it bluntly, the code was not running correctly.  The customer could run the code locally without any issue.  However, as soon as the same PowerShell was included within a CloudConfig element it did not work and they could not understand why.

I don’t often get to be hands on and troubleshoot these things nowadays, so with thanks to my collegues Chris and Dean, I was able to get access to the environment components I needed to test and troubleshoot.

The two code elements that the customer team had difficulty with were:

• Formatting and adding labels to disks for a deployed windows server VM
• Setting the AD description after a server had been added to active directory

Original PowerShell Disk Commands

# Get the disk and format
Get-Disk $number | Initialize-Disk -PartitionStyle GPT -PassThru | New-Partition -DriveLetter $letter -UseMaximumSize | Format-Volume -FileSystem NTFS -AllocationUnitSize 4096 -Force -Confirm:$false

# Set the label on the new volume
Set-Volume -DriveLetter $letter -NewFileSystemLabel $label

Original PowerShell Domain Commands

# Domain join
Add-Computer -domainName $domain -Credential $creds -Restart:$false

# Set Domain description
Get-adcomputer -hostName $hostname | Set-ADComputer -description $description -Credential $creds

There is nothing particularly complicated in either of the scripts, although it is worth noting that the two elemements that are consistantly erroring in the code are being processed via a command pipeline.

PowerShell within CloudConfig

The code that was created contained multiple elements, and therefore followed the cloudbase-init multi-part content formatting, details for this can be found here.  The code below is taking the locally working code and transposing that into CloudConfig elements for inclusion in the cloud assembly blueprint.

N.B.  it is worth pausing here for a second and noting that the code uses the properties of the deployed disk resource to pass drive letters, labels and disk numbers. Properties under a resource appear to be treated as an array, and therefore require the square brackets. for example ${resource.Cloud_vSphere_Disk_1.unitNumber} will pull the value back but enclose it within square brackets, which will break the PowerShell element. Whereas ${resource.Cloud_vSphere_Disk_1[0].unitNumber}, will correctly pass the variable.  Thanks to my collegues Dean (who provided advice whilst travelling on the train no less) and Katherine for helping me work this out.

CloudConfig Disk-Commands Element

It might be a spoiler to say, but this syntax won’t work, so please don’t copy it.

#ps1_sysnative
Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -DiskNumber ${resource.Cloud_vSphere_Disk_1[0].unitNumber} -DriveLetter ${resource.Cloud_vSphere_Disk_1[0].driveLetter} -UseMaximumSize | Format-Volume -DriveLetter ${resource.Cloud_vSphere_Disk_1[0].driveLetter} -FileSystem NTFS -AllocationUnitSize 4096 -Force -Confirm:$false

Set-Volume -DriveLetter ${resource.Cloud_vSphere_Disk_1[0].driveLetter} -NewFileSystemLabel "${resource.Cloud_vSphere_Disk_1[0].name}"

CloudConfig Domain Command Element

It might be a spoiler to say, but this syntax won’t work, so please don’t copy it.

#ps1_sysnative

if ( ("${input.domain}".ToLower() -ne "workgroup") -and ( $(Get-ComputerInfo -Property CsDomain).CsDomain.ToLower() -ne "${input.domain}".ToLower()) ) {
$creds = New-Object System.Management.Automation.PSCredential ${input.ADUsername} ,(ConvertTo-SecureString -String "${secret.ADSecret}" -AsPlainText -Force)
Add-Computer -domainName ${input.domain} -Credential $creds -Restart:$false

Get-adcomputer -hostName ${input.Hostname} | Set-ADComputer -description "${input.ADDescription}" -Credential $creds

}

Disk Element Deployment Troubleshooting

Troubleshooting the CloudConfig elements provided by cloudbase-init within deployments via cloud assembly can be a little bit frustrating.  This is because of the way the two systems interact. Cloudbase-init is sent asynchronously to the deployed objects, that is to say it is a fire and forget operation.  Cloud Assembly will not report on the success or failure of the cloudbase-init elements, just that they were sent at the time of deployment.  To troubleshoot the effectiveness of the cloudbase-init deployments we need to look within the deployed object itself.

Outside of the deployment we can inspect the CloudConfig code, this allows confirmation that the assigned variable values within the blueprint are being passed to the CloudConfig element.

This allows confirmation that the code looks as if it should indeed run within the host. The next step is to look for errors within the the cloudbase-init.log file within the deployed object (the location is set at the time that cloudbase-init is installed and by default will be “C:\Program Files\Cloudbase Solutions\Cloudbase-init\log”.  Searching the logs for the commands that have not successfully implemented is a shortcut for finding the part of the log that contains any error messages.

Searching for the New-Partition cmdlet provides an error that is worth exploring further. The error is stating that the command will not take an input via a command pipeline.  Thefore the most suitable next step is to break down the command pipeline and run the cmdlets individually within the script.

Working CloudConfig Disk-Commands Element

#ps1_sysnative
Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru

New-Partition -DiskNumber ${resource.Cloud_vSphere_Disk_1[0].unitNumber} -DriveLetter ${resource.Cloud_vSphere_Disk_1[0].driveLetter} -UseMaximumSize

Format-Volume -DriveLetter ${resource.Cloud_vSphere_Disk_1[0].driveLetter} -FileSystem NTFS -AllocationUnitSize 4096 -Force -Confirm:$false

Set-Volume -DriveLetter ${resource.Cloud_vSphere_Disk_1[0].driveLetter} -NewFileSystemLabel "${resource.Cloud_vSphere_Disk_1[0].name}"

The above takes the four elements of the initial script (initialise, partition, format and label) and runs them individually, calling the variables directly rather than via a command pipeline.

The logs are not easy to digest in a screenshot, so instead here is the successful output from this element of the script, now that the command pipeline method has been removed.

Domain Element Deployment Troubleshooting

Whilst the active directory PowerShell script was also working via a command pipeline, there was a fundemental problem with running this locally from cloudbase-init. The ActiveDirectory PowerShell modules are not available by default on deployed windows servers, and therefore will not be available on the deployed object, nor should they be available on the object as it is published into production.  The Active Directory PowerShell modules are installed as a windows feature, not an import-module/install-module cmdlet.  Thankfully, there is a way to add windows features via PowerShell.

Adding the Active Directory PowerShell Modules

Place this element appropriately in the CloudConfig to make the Active Directory PowerShell modules available to cloudbase-init on the deployment object.

#ps1_sysnative
Add-WindowsFeature -Name RSAT-AD-PowerShell

The above screen capture highlights the output from a get-module command after the RSAT-AD-PowerShell module is installed.

Removing the Active Directory PowerShell Modules

After the Active Directory elements have been executed, use this element, appropriately placed, to remove the Active Directory PowerShell modules.

#ps1_sysnative
Uninstall-WindowsFeature -Name RSAT-AD-PowerShell

Working CloudConfig Domain-Commands Element

#ps1_sysnative

if ( ("${input.domain}".ToLower() -ne "workgroup") -and ( $(Get-ComputerInfo -Property CsDomain).CsDomain.ToLower() -ne "${input.domain}".ToLower()) ) {
$creds = New-Object System.Management.Automation.PSCredential ${input.ADUsername} ,(ConvertTo-SecureString -String "${secret.ADSecret}" -AsPlainText -Force)
Add-Computer -domainName ${input.domain} -Credential $creds -Restart:$false

Set-ADComputer -identity ${input.hostname} -description "${input.ADDescription}" -Credential $creds

}

To build the working element, the command pipelines have been removed and the target for the Set-ADComputer cmdlet has been defined by passing the variable ${input.hostname} to the -identity switch.  The values being passed into the CloudConfig AD PowerShell element can be seen below.

The CloudConfig AD PowerShell element running successfully means that these values are passed onto the required componenets as expected, as can be seen below.

This is a busy image so I’ve highlighted the pertintent elements. The server “tmm-ad5” has joined domain “simon.local” and had the computer object description updated to the text passed in the CloudConfig.  In addition the uninstall for the RSAT-AD-PowerShell module has been completed as evidenced by the fact it is no longer listed in the output from get-module.

Conclusion

When working with CloudConfig, cloudbase-init and PowerShell the following seems to be resonable advice:

• Know where the cloudbase-init logs are located – for a default windows installation this is “C:\Program Files\Cloudbase Solutions\Cloudbase-init\log”
• If you are using PowerShell command pipelines, be prepared to go looking through the aformentioned log
• If you have the Advanced or Enterprise edition of the vRealize Suite, then all of this is almost certainly much easier to accomplish in SaltStackConfig

Special thanks to my collegue Chris who spent part of his Friday afternoon helping me piece all of this together.

Hopefully this is useful to someone else.

Thanks

Simon