Wednesday, March 07, 2018

How to use vROps to find a VM causing a broadcast storm

A customer recently reached out to me with the question if vROps could help him find a particular VM obviously causing a broadcast storm.
The only information he got from the networking department was the host name and NIC of one of his ESXi hosts. The vCenter metrics did not reveal any valuable information and he had to check every single VM (more than 30 per ESXi host) one by one. At the end he asked if vROps is capable of finding that bad guy.

First step was to check if we have such a metric for VMs and whether it is active (collecting data) or not.
As you can see in the following figures, vROps knows that metric but it is disabled by default.

Fig. 1: disabled metrics in the default policy

Now, you could go and just activate these metrics for all of your VMs and check the values. But, in an environment with several thousands of VMs it will add additional load and you will need these metrics only for some few VMs and for a limited period of time.

Let us make it more dynamic and configurable for future use, just in case your NOC may come to you with another ESXi host you have to check.

The idea is pretty simple, we need a policy which enables the needed metrics and we need a group of VMs we would like this policy to be applied to.

Step 1: create a new policy and activate the broadcast metrics for Virtual Machine object type.

The following figure shows you the filters and settings to activate the right things.

Fig. 2: enable metrics in a new policy

We want to get this policy applied only to a dynamic group of VMs we would like to investigate. This is where the concept of Custom Groups comes into play.

Custom Groups work as a container for any objects you may have and the settings of a Custom Group allow the membership to by dynamic based on a wide range of properties, relations etc.

Now we could go to vROps, create a new custom group and define that group to contain all VMs which are children of a particular predefined ESXi host. This would be semi-dynamic.

Let's re-think this strategy.

I many cases the admin dealing with a broadcast storm in a vSphere environment do not have to be the vROps admin in his org.
Wouldn't it be better if the vSphere admin set "something" in vCenter and at the end he will see a dashboard or receive a report in/from vROps?

Exactly, we go for the vSphere Tags.

Our new tag will designate a host as being "under investigation", time for the next step.

Step 2: create a vSphere Tag

Fig. 3: create a vSphere Tag
As we have our tag we continue with a "two-staged-custom-group".
The first group will dynamically contain ESXi hosts under investigation, and the second group will contain Virtual Machines which run on those hosts.
This will give us the freedom of creating multiple "second-stage-groups" which may have different policies assigned, in case we would like to investigate another behaviour which requires another metrics etc.

Step 3: create a new custom group for the ESXi host(s)

Fig. 4: "first-stage-custom-group" - Host System

Anytime we assign our new vSphere Tag to a ESXi host, this host will become member of this group.

Step 4: assign the tag to a ESXi host and wait a collection cycle to get the custom group populated:

Fig. 5: vSphere tag assigned to a host

Fig. 6: "first-stage-custom-group" - dynamically populated

Time for the "second-stage", the custom group containing the VMs.

Step 5: create a new custom group for the VMs:

Fig. 7: "second-stage-custom-group" - dynamically populated

Once we created the custom group for the VMs, this group gets populated with VMs which run on tagged ESXi hosts.
We see that the metrics we need for our investigation get collected:

Fig. 8: Collecting metrics

At this point we have everything to create a dashboard for our vSphere admin to quickly help him find the bad guy:

Fig. 9: Dashboard with the results of the investigation

Hope, this post will help others during their RCAs.


Saturday, March 03, 2018

vROps SuperMetric using logical expressions

vRealie Operations Super Metrics are a very flexible and powerful way to extend the capabilities of the product way beyond the OOB content.
There are many blog articles out there explaining how to basically use super metrics but only very few sources gives some examples how to put logical expressions into your formulas. So the question is, how dos this work?

Using some simple examples I am going to explain how the magic of logical expressions work in vROps Super Metrics.
First of all some fundamentals:
  • Super Metric working on a selected object itself, like ESXi cluster in this example, which is just showing the actual metric (we will need soon):
avg(${this, metric=summary|total_number_hosts})
  • Super Metric working on direct descendants of a selected object, in this case ESXi hosts in a cluster, which is counting the powered on hosts:
count(${adaptertype=VMWARE, objecttype=HostSystem, metric=sys|poweredOn, depth=1})
Now, let's put the pieces together and build a super metric which follows this pattern:
condition ? inCaseOfTrue : elseCase;

Even if this doesn't mean anything in terms of semantics, the syntax of such an expression might look like this one:

count(${adaptertype=VMWARE, objecttype=HostSystem, metric=sys|poweredOn, depth=1})
== avg(${this, metric=summary|total_number_hosts})
&& avg(${adaptertype=VMWARE, objecttype=HostSystem,
metric=cpu|usage_average, depth=1} as cpuUsage) > 40 ? avg(cpuUsage ):5

One could translate this formula into that statement:

If all ESXi hosts in a given cluster are powered on AND
clusters average CPU usage is greater than 40
THEN show me the average CPU usage,
ELSE show me 5

Friday, February 02, 2018

Horizon View Automation with PowerShell and VMware Orchestrator

For a customer PoT, I had the challenge to automate the rollout of a new Horizon View environment with an initial set of functionality like adding a vCenter, and provision an initial Pool and additional tasks like:
·      Scale-Up: Change the Pool size
·      Scale-Out: Add an additional vCenter and create a new Pool on the second vCenter
I want to share the scripts I’ve created to do so, but with the note, that they are not bullet proof and in an absolute PoT character J

For the installation, the following binaries have to be stored in the C:\TEMP directory of the VM where you want to install Horizon:
·      VMware-viewconnectionserver-x86_64-7.3.2-7161118.exe
·      VMware-PowerCLI-6.5.0-4624819.exe
· (download from here:

The first thing we need to do, is to enable interactive sessions on the VM, unfortunately the PowerCLI installation will only work interactively because we cannot get rid of an annoying window which opens during installation. So run the following script on the VM to set the VMware Tools service to allow interactive sessions:
REG ADD HKLM\SYSTEM\CurrentControlSet\Control\Windows /v NoInteractiveServices /t REG_DWORD /d 0x0 /f
SC CONFIG VMTools type=own type=interact

Now we need to restart the VMware Tools or what I did, restart the whole server and wait until the VMware Tools are up and running again.

Next step is to run the installation of Horizon View. We run now the installation of our two binaries with the following script interactively (you need to replace the admin SID with an SID valid for your environment. I’ve used an domain account that’s why I specified the admin SID here):
Start-Process C:\TEMP\VMware-viewconnectionserver-x86_64-7.3.2-7161118.exe -ArgumentList '/s /v"/qn VDM_SERVER_INSTANCE_TYPE=1 FWCHOICE=2 VDM_INITIAL_ADMIN_SID=S-1-5-xx-xxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx-xxxx VDM_SERVER_RECOVERY_PWD=vmware"' -PassThru -Wait
Start-Sleep -Seconds 5
Start-Process C:\TEMP\VMware-PowerCLI-6.5.0-4624819.exe -ArgumentList '/s /qn /w /V"/qr /L*v PowerCLI.log"' -PassThru -Wait
cd "C:\Program Files\VMware\VMware View\Server\extras\PowerShell"

The next step is to install the VMware HV Helper PowerShell module. Therefore we run the following script (from now on we don’t need to run our scripts interactively any more):
Set-ExecutionPolicy unrestricted -force

Add-Type -AssemblyName System.IO.Compression.FileSystem

function Unzip
    param([string]$zipfile, [string]$outpath)

    [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
Unzip "C:\TEMP\" "C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Modules\"
dir "C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Modules\VMware.Hv.Helper\" | Unblock-File

Again we need to restart the VM, because otherwise when starting PowerShell we cannot access the fresh installed PowerShell modules.

Now we have an installed Horizon View environment, but nothing configured so far. We start now with the configuration. In my case I’ve started adding an AD user group which will get entitled to the new Pool we create. I’ve used the build-in vRO workflow: “Create a user group in an organizational unit”. Next step was to entitle the AD administrative account locally on the VM, so we need to run this bash script to entitle the user to the local Administrators:
NET LOCALGROUP Administrators [email protected] /ADD

Next step is to install the Horizon View license. Therefore we run the following script:
Import-Module VMware.VimAutomation.HorizonView
Import-Module VMware.VimAutomation.Core
Add-PSSnapin VMware.View.Broker
Connect-HVServer -server localhost -user hv-admin -password Password -domain
Disconnect-HVServer -server localhost -Force -Confirm:$false

Now we go and install the vCenter certificate to the VM, so we can add the vCenter to Horizon View using the Horizon View Administrator user account. To do so, run the following script (I’m sure there is a better way to do so but I was not sure where to import the certificate so I was importing it in nearly every certificate store J):
$secpasswd = ConvertTo-SecureString "Password" -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential ("[email protected]", $secpasswd)
Invoke-Command -Computer $env:COMPUTERNAME -ScriptBlock {

$url = "https://<URL>"
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[System.Uri] $u = New-Object System.Uri($url)
[Net.ServicePoint] $sp = [Net.ServicePointManager]::FindServicePoint($u);

[System.Guid] $groupName = [System.Guid]::NewGuid()

[Net.HttpWebRequest] $req = [Net.WebRequest]::create($url)
$req.Method = "GET"
$req.Timeout = 600000 # = 10 minutes
$req.ConnectionGroupName = $groupName

[Net.HttpWebResponse] $result = $req.GetResponse()


$outfilename = "C:\TEMP\Export.cer"

[System.Byte[]] $data = $sp.Certificate.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert)
[System.IO.File]::WriteAllBytes($outfilename, $data)
Write-Host $outfilename

Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\LocalMachine\Root
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\LocalMachine\CA
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\LocalMachine\My
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\LocalMachine\AuthRoot
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\LocalMachine\TrustedDevices
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\CurrentUser\Trust
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\CurrentUser\CA
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\CurrentUser\AuthRoot
Import-Certificate -FilePath "C:\TEMP\Export.cer" -CertStoreLocation Cert:\CurrentUser\Myn
} -Credential $cred

Now we can add the vCenter to our Horizon View environment using the following script, again executed using the HV administrator account:
Import-Module VMware.VimAutomation.HorizonView
Import-Module VMware.VimAutomation.Core
Import-Module VMware.Hv.Helper
Add-PSSnapin VMware.View.Broker

$hvServer = Connect-HVServer -server 'localhost' -user 'hv-admin' -password 'Password' -domain ''
$Global:hvServices = $hvServer.ExtensionData

$vcService = New-Object VMware.Hv.VirtualCenterService
$certService = New-Object VMware.Hv.CertificateService
$vcSpecHelper = $vcService.getVirtualCenterSpecHelper()

$vcPassword = New-Object VMware.Hv.SecureString
$enc = [system.Text.Encoding]::UTF8
$vcPassword.Utf8String = $enc.GetBytes('Password')

$serverSpec = $vcSpecHelper.getDataObject().serverSpec
$serverSpec.serverName = '<URL>'
$serverSpec.port = 443
$serverSpec.useSSL = $true
$serverSpec.userName = 'administrator[email protected]'
$serverSpec.password = $vcPassword
$serverSpec.serverType = $certService.getServerSpecHelper().SERVER_TYPE_VIRTUAL_CENTER

$certData = $certService.Certificate_Validate($hvServices, $serverSpec)
$certificateOverride = New-Object VMware.Hv.CertificateThumbprint
$certificateOverride.sslCertThumbprint = $certData.thumbprint.sslCertThumbprint
$certificateOverride.sslCertThumbprintAlgorithm = $certData.thumbprint.sslCertThumbprintAlgorithm

$vcSpecHelper.getDataObject().CertificateOverride = $certificateOverride

$vcId = $vcService.VirtualCenter_Create($hvServices, $vcSpecHelper.getDataObject())

Disconnect-HVServer -server localhost -Force -Confirm:$false

Now we have our vCenter added to Horizon View, we can create a Pool, here a script to create an instant-clone Pool:
Import-Module VMware.VimAutomation.HorizonView
Import-Module VMware.VimAutomation.Core
Import-Module VMware.Hv.Helper
Add-PSSnapin VMware.View.Broker

$hvServer = Connect-HVServer -server 'localhost' -user 'hv-admin' -password 'Password' -domain ''
$Global:hvServices = $hvServer.ExtensionData

New-HVPool -InstantClone -PoolName '<poolName>' -PoolDisplayName '<poolDisplayName>' -Description '"<poolDescription>"' -UserAssignment FLOATING -ParentVM '<ParentVMName>' -SnapshotVM '<SnapshotName>' -VmFolder 'InstantClones' -HostOrCluster '<ClusterName>' -NamingMethod PATTERN -NamingPattern "<poolName>-{n:fixed=3}" -Usevsan $True -Datastores 'vsanDatastore' -NetBiosName '<Domain>' -EnableProvisioning $TRUE -StopOnProvisioningError $TRUE -MaximumCount 5 -SpareCount 2 -MinimumCount 2 -ProvisioningTime ON_DEMAND -AdContainer 'OU=Clients' -DomainAdmin hv-admin -Vcenter <vCenter> -ResourcePool ‘<ResourcePoolName>’

Set-HVPool -PoolName <poolName> -enableHTMLAccess $true

Disconnect-HVServer -server localhost -Force -Confirm:$false

The last missing piece is to entitle our created AD user group to access the Pool:
Import-Module VMware.VimAutomation.HorizonView
Import-Module VMware.VimAutomation.Core
Import-Module VMware.Hv.Helper
Add-PSSnapin VMware.View.Broker

$hvServer = Connect-HVServer -server 'localhost' -user 'hv-admin' -password 'Password' -domain ''
$Global:hvServices = $hvServer.ExtensionData

$queryService = New-Object VMware.Hv.QueryServiceService
$defn = New-Object VMware.Hv.QueryDefinition
$defn.queryEntityType = 'ADUserOrGroupSummaryView'
$queryResults = $queryService.QueryService_Create($hvServices, $defn)

try {
   while ($queryResults.results -ne $null) {
      foreach ($result in $queryResults.Results) {
         [VMware.Hv.ADUserOrGroupSummaryView]$adUserOrGroupSummaryViewResult = $result
         if ($adUserOrGroupSummaryViewResult.Base.Name -eq "<poolName>_User") {
                     $adUserOrGroupSummaryView = $adUserOrGroupSummaryViewResult

      # Fetch next page
      if ($ -eq $null) {
      $queryResults = $queryService.QueryService_GetNext($hvServices, $
} finally {
   if ($ -ne $null) {
      $queryService.QueryService_Delete($hvServices, $

$hvPool = Get-HVPool -PoolName "<poolName>"

$userEntitlementBase = New-Object -Type VMware.Hv.UserEntitlementBase
$userEntitlementBase.UserOrGroup = $adUserOrGroupSummaryView.Id
$userEntitlementBase.Resource = $hvPool.Id

$userEntitlementService = New-Object -TypeName VMware.Hv.UserEntitlementService
$userEntitlementService.UserEntitlement_Create($hvServices, $userEntitlementBase)

Disconnect-HVServer -server localhost -Force -Confirm:$false

That’s it. Now we have a running and configured Horizon View environment with an initial Pool and an entitled AD user group to access the Pool.

To scale a Pool, you can use the following two scripts. The first, to get the current values of the pool:
Import-Module VMware.VimAutomation.HorizonView
Import-Module VMware.VimAutomation.Core
Import-Module VMware.Hv.Helper
Add-PSSnapin VMware.View.Broker

$hvServer = Connect-HVServer -server 'localhost' -user 'hv-admin' -password 'Password' -domain ''
$Global:hvServices = $hvServer.ExtensionData

$HVPool = Get-HVPool -PoolName <poolName>

Disconnect-HVServer -server localhost -Force -Confirm:$false
And the following javascript part to parse the output:
var maxNumberOfMachines = scriptOutputTextOut.match(/MaxNumberOfMachines\s*:\s*[0-9]+/).shift().match(/[0-9]+/).shift();
var numberOfSpareMachines = scriptOutputTextOut.match(/NumberOfSpareMachines\s*:\s*[0-9]+/).shift().match(/[0-9]+/).shift();
var minNumberOfMachines = scriptOutputTextOut.match(/MinNumberOfMachines\s*:\s*[0-9]+/).shift().match(/[0-9]+/).shift();

And second, to set the provisioning values on the Pool:
Import-Module VMware.VimAutomation.HorizonView
Import-Module VMware.VimAutomation.Core
Import-Module VMware.Hv.Helper
Add-PSSnapin VMware.View.Broker

$hvServer = Connect-HVServer -server 'localhost' -user 'hv-admin' -password 'Password' -domain ''
$Global:hvServices = $hvServer.ExtensionData

Set-HVPool -PoolName <poolName> -key 'automatedDesktopData.vmNamingSettings.patternNamingSettings.maxNumberOfMachines' -value <maxNumberOfMachines>
Set-HVPool -PoolName <poolName> -key 'automatedDesktopData.vmNamingSettings.patternNamingSettings.numberOfSpareMachines' -value <numberOfSpareMachines>
Set-HVPool -PoolName <poolName> -key 'automatedDesktopData.vmNamingSettings.patternNamingSettings.minNumberOfMachines' -value <minNumberOfMachines>

Disconnect-HVServer -server localhost -Force -Confirm:$false

To scale out the Horizon View environment to use a second vCenter and create a second Pool. All scripts are already provided, you just need to combine them.

Have fun with Horizon View automation. For me it was pain in the a** to get so far ;-)