Go into your vCenter (if you have more than one, you will need to do this for each.)  Go into the “Hosts and Clusters” view.  On the left pane, select the vCenter Server itself.  On the right pane, select the “Virtual Machines” tab.  You can optionally click the “State” field title to sort by state.  You may also click the host field to sort VMs by the host names (I would recommend this if you have multiple clusters with multiple editions of ESXi).  You can then right-click the virtual machine titles and add the field for “Memory Size” as shown below.  Right-click right on the word “State” in the title of the column.

Once the Memory field has been added (it will probably be far on the right), drag the filed so it’s just to the right of the “State” field.  Now go to the “File” Menu at the top of the vSphere client and select “File” then “Export” and then “Export List”.  Export the file selecting “Excel Workbook” as the file type.  Once exported, open the list in Excel.  In Excel, add a column to the right of the memory column like so:

Edit Cell D2 and put in this formula:   =IF(B2=”Powered On”,VALUE(LEFT(C2,(LEN(C2)-3))),0)

Copy this formula down the entire column.  This checks to see if the VM state is powered on (you do not draw from your vRAM license if it is not).  It then removes the “MB” and converts the value to a numeric so you can sum them up.

Scroll down to the last row and edit the cell in the next empty row in the memory column to something like this:  =ROUNDUP(SUM(D2:D65)/1024,0)

Where D65 is actually the last cell with the memory data in it, your row number will vary depending on how many VMs you have.

The ROUNDUP will round up the memory allocation (in case you have some VM’s with 4000MB allocated to them instead of 4096MB) and you need to divide the sum by 1024 to convert to GB of vRAM.

If you would rather run a PowerCLI script to gather the info, you can find a great article on how to do it here.

I hope your number turns out ok.  If it does not, all is not lost.  There are ways to shrink your vRAM footprint so the impact of the licensing is not as bad.  If you would like me to have a look, email me, I can provide a service to see how much vRAM you have allocated but never use.  That may prolong the next license purchase a bit or perhaps soften the expense.  The new licensing does not need to always be negative, maybe we just need to learn how to size our VMs with the licensing in mind.

UPDATE: If you copy and paste these formulas into Excel, the ASCII is different for some reason.  Just backspace over the quotes (“) and readd them.  The quotes are what Excel has an issue with.

Here are some of the features:

• Deploys as an appliance, very easy to install
• Must be deployed on a new ESXi 5.0 installation
• Deploys a VSA Cluster Service on the vCenter server
• The VSA Cluster Service can deploy the VSA “Agent VMs” to each of the ESXi 5.0 hosts
• The appliance will use the local space available and present the storage on the network as an NFS datastore
• Replicates the local storage to the local storage on another host in the cluster for redundancy.
• If a host fails, the appliance storing the replica will immediately take over the failed “Agent VM’s” IP address and share the storage from the replica
• v1.0 supports 2 or 3 ESXi hosts in a cluster (Typically for the essentials kits)
• Sold as a separate SKU with one price with no license capacity restrictions (no technical size limits that I could find)
• Supports 25 VMs (configured on 2 ESXi hosts) or 35 VMs (configured on 3 ESXi hosts)
• It is the only scenario where VMware recommends running vCenter on a physical or standalone ESXi hypervisor (To protect you from running into a Catch-22 as vCenter is managing the VSAs
• Recommended to use RAID10 on the hardware RAID controllers in the hosts (to protect from a single drive failure)
• Uses RAID 1 (Mirroring) between hosts for redundancy
• Supports Storage vMotion for when you are ready to migrate to hardware shared storage
• Can put the whole VSA cluster in maintenance mode or just a single node.  Can also replace a node and have the VSA rebuild onto it for redundancy or for rolling upgrades.

Here’s how it works: Imagine I have 3 hosts numbered 1,2 and 3.  Once the VSA gets installed, it creates two volumes on the available local storage on each host.  So host 1 will have volumes 1A and 1B, host 2 has 2A and 2B, host 3 has 3A and 3B.  Once the VSAs are configured, they will be redundant so that 1A (which stores VMs) mirrors to 2B, 2A mirrors to 3B and 3A mirrors to 1B.  If any VSA get’s dropped, the VSA running the mirror copy takes the IP address of the failed VSA and keeps right on chugging.

My Take

The Pros: Great solution for SMBs without shared storage to take advantage of HA, vMotion, etc.  I also think this is an outstanding solution for companies with remote offices who want to have redundancy in 2 or 3 ESXi hosts but don’t want to put shared storage in each site.

The Cons:  Way too much overhead.  VMware is recommending hardware RAID10 from the local drives if possible.  If I have 4 x 1TB drives in a server (4TB RAW disk capacity).  I use RAID10 as per VMware’s recommendation, this means 2TB gets presented to the ESXi host.  Now the VSA uses half of that storage for VMs and half as a target to mirror the VSA from one of the other hosts.  So out of 4TB of RAW disk, I get <1TB of capacity to store VMs on (don’t forget, I need room to store ESXi itself).  Thats a 75% reduction from RAW capacity = too much overhead.

Overall I still think it’s worth it.  It’s still going to be less expensive that a shared storage frame (even with the overhead loss).  I think for remote sites, you can’t beat it.  I can’t wait to see what they add to it in v2.0.

VMware vSphere 5.0

• ESXi Convergence – No more ESX, only ESXi (they said they would do it, they meant it)
• New VM Hardware:  Version 8 – New Hardware support (VS5 still supports VM Hardware 4 & 7 as well if you still want to migrate to the old hosts)
  • 3D graphics Support for Windows Aero
  • Support for USB 3.0 devices

• Platform Enhancements (Blue Requires Hardware v8)
  • 32 vCPUs per VM
  • 1TB of RAM per VM
  • 3D Graphics Support
  • Client-connected USB devices
  • USB 3.0 Devices
  • Smart-card Readers for VM Console Access
  • EFI BIOS
  • UI for Multi-core vCPUs
  • VM BIOS boot order config API and PowerCLI Interface
• vSphere Auto Deploy – mechanism for having hosts deploy quickly when needed ( I’m going to wait and see how customers use this one.)
• Support for Apple Products – Support for running OSX 10.6 Server (Snow Leopard) on Apple Xserve hardware. (although I betting technically, you can get it to run on any hardware, you will just not be compliant in your license)
• Storage DRS – Just like DRS does for CPU and Memory, now for storage
  • Initial Placement – Places new VMs on the storage with the most space and least latency
  • Load Balancing – migrates VMs if the storage cluster (group of datastores) gets too full or the latency goes too high
  • Datastore Maintenance Mode  - allow you to evacuate VMs from a datastore to work on it (does not support Templates or non-registered VMs yet…)
  • Affinity & Anti-Affinity – Allows you to make sure a group of VMs do not end up on the same datastore (for performance or Business Continuity reasons) or VMs that should always be on the same datastore.  Can be at the VM or down to the individual VMDK level.
  • Support for scheduled disabling of Storage DRS – perhaps during backups for instance.
• Profile-Driven Storage – Creating pools of storage in Tiers and selecting the correct tier for a given VM.  vSphere will make sure the VM stays on the correct tier(pool) of storage.  (Not a fan of this just yet.  What if just 1GB of the VM needs high-tier storage? This makes you put the whole VM there.)
• vSphere File System – VMFS5 is now available.  (Yes, This is a non-disruptive upgrade, however I would still create new and SVmotion)
  • Support for a single extent datastore up to 64TB
  • Support for >2TB Physical Raw Disk Mappings
  • Better VAAI (vStorage APIs for Array Integration) Locking with more tasks
  • Space reclamation on thin provisioned LUNs
  • Unified block size (1MB) (no more choosing between 1,2,4 or 8)
  • Sub-blocks for space efficiency (8KB vs. 64KB in VS4)
• VAAI now a T10 standard – All 3 primitives (Write Same, ATS and Full Copy) are now T10 standard compliant.
  • Also now added support for VAAI NAS Primitives including Full File Clone (to have the nas do the copy of the vmdk files for vSphere) and Reserve Space (to have the NAS create thick vmdk files on NAS storage)
• VAAI Thin Provisioning – Having the storage do the thin provisioning and then vSphere telling the storage which blocks can be reclaimed to shrink the space used on the storage
• Storage vMotion Enhancements
  • Now supports storage vMotion with VMs that have snapshots
  • Now supports moving linked clones
  • Now supports Storage DRS (mentioned above)
  • Now uses mirroring to migrate vs change block tracking in VS4.  Results in faster migration time and greater migration success.
• Storage IO Control for NAS – allows you to throttle the storage performance against “badly-behaving” VMs also prevents them from stealing storage bandwidth from high-priority VMs.  (Support for iSCSI and FC was added in VS4.)
• Support for VASA (vStorage APIs for Storage Awareness) – Allows storage to integrate tighter with vcenter for management.  Provides a mechanism for storage arrays to report their capabilities, topology and current state.  Also helps Storage DRS make more educated decisions when moving VMs.
• Support for Software FCoE Adapters – Requires a compatible NIC and allows you to run FCoE over that NIC without the need for a CNA Adapter.
• vMotion Enhancements
  • Support for multiple NICs.  Up to 4 x 10GbE or 16 x 1GbE NICs
  • Single vMotion can span multiple NICs (this is huge for 1GbE shops)
  • Allows for higher number of concurrent vMotions
  • SDPS Support (Slow Down During Page Send) – throttles busy VMs to reduce timeouts and improve success.
  • Ensures less than 1 second switchover in almost all cases
  • Support for higher latency networks (up to ~10ms)
  • Improved error reporting – better, more detailed logging (thank you vmware!)
  • Improved Resource Pool Integration – now puts VMs in the proper resource pool
• Distributed Resource Scheduling/Dynamic Power Management Enhancements
  • Support for “Agent VMs” – These are VMs that work per host (currently mostly vmware services – vshield, edge, app, endpoint, etc)  DRS will not migrate these VMs
  • “Agents” do not need to be migrated for maintenance mode
• Resource pool enhancements – now more consistent for clustered vs. non-clustered hosts.  No longer can modify resource pool settings on the host itself when it is managed by vcenter.  It does allow for making changes if the host gets disconnected from vCenter
• Support for LLDP Network Protocol – Standards based vendor-neutral discovery protocol
• Support for NetFlow – Allows collection of IP traffic information to send to collectors (CA, NetScout, etc) to provide bandwidth statistics, irregularities, etc.  Provides complete visibility to traffic between VMs or VM to outside.
• Network I/O Control (NETIOC) – allows creation of network resource pools, QoS Tagging, Shares and Limits to traffic types, Guaranteed Service Levels for certain traffic types
• Support for QoS (802.1p) tagging – provides the ability to Q0S tag any traffic flowing out of the vSphere infrastructure.
• Network Performance Improvements
  • Multiple VMs receiving multicast traffic from the same source will see improved throughput and CPU efficiency
  • VMkernel NICs will see higher throughput with small messages and better IOPs scaling for iSCSI traffic
• Command Line Enhancements
  • Remote commands and local commands will now be the same (new esxcli commands are not backwards compatible)
  • Output from commands can now be formatted automatically (xml, CSV, etc)
• ESXi 5.0 Firewall Enhancements
  • New engine not based on iptables
  • New engine is service-oriented and is a stateless firewall
  • Users can restrict specific services based on IP address and Subnet Mask
  • Firewall has host-profile support
• Support for Image Builder – can now create customized ESXi CDs with the drivers and OEM add-ins that you need.  (Like slip-streaming for Windows CDs) Can also be used for PXE installs.
• Host Profiles Enhancements
  • Allows use of an answer file to complete the profile for an automated deployment
  • Greatly expands the config options including: iSCSI, FCoE, Native Multipathing, Device Claming, Kernel Module Settings & more)  (I don’t think Nexus is supported yet)
• Update Manager Enhancements
  • Can now patch multiple hosts in a cluster at a time.  Will analyze and see how many hosts can be patched at the same time and patch groups in the cluster instead of one at a time.  Can still do one at a time if you prefer.
  • VMTools can now be scheduled at next VM reboot
  • Can now configure multiple download URLs and restrict downloads to only the specific versions of ESX you are running
  • More management capabilities: update certificates, change DB password, proxy authentication, reconfigure setup, etc.
• High Availability Enhancements
  • No more Primary/Secondary concept, one host is elected master and all others are slaves
  • Can now use storage-level communications – hosts can use “heartbeat datastores” in the event that network communication is lost between the hosts.
  • HA Protected state is now reported on a per/VM basis.  Certain operations no longer wait for confirmation of protection to run for instance power on.  The result is that VMs power on faster.
  • HA Logging has been consolidated into one log file
  • HA now pushes the HA Agent to all hosts in a cluster instead of one at a time.  Result:  reduces config time for HA to ~1 minute instead of ~1 minute per host in the cluster.
  • HA User Interface now shows who the Master is, VMs Protected and Un-protected, any configuration issues, datastore heartbeat configuration and better controls on failover hosts.
• vCenter Web Interface – Admins can now use a robust web interface to control the infrastructure instead of the GUI client.
  • Includes VM Management functions (Provisioning, Edit VM, Poer Controls, Snaps, Migrations)
  • Can view all objects (hosts clusters, datastores, folders, etc)
  • Basic Health Monitoring
  • View the VM Console
  • Search Capabilities
  • vApp Management functions (Provisioning, editing, power operations)
• vCenter Server Appliance – Customers no longer need a Windows license to run vCenter.  vCenter can come as a self-contained appliance (This has been a major request in the community for years)
  • 64-bit appliance running SLES 11
  • Distributed as 3.6GB, Deployment range is 5GB to 80GB of storage
  • Included database for 5 Hosts or 50 VMs (same as SQL Express in VS4)
  • Support for Oracle as the full DB (twitter said that DB2 was also supported but I cannot confirm in my materials)
  • Authentication thru AD and NIS
  • Web-based configuration
  • Supports the vSphere Web Client
  • It does not support:  Linked Mode vCenters, IPv6, SQL, or vCenter heartbeat (HA is provided thru vSphere HA)
• vCenter Heartbeat 6.4 Enhancements
  • Allows the active and standby nodes to be reachable at the same time, so both can be patched and managed
  • Now has a plug-in to the vSphere client to manage and monitor Heartbeat
  • Events will register in the vSphere Recent Tasks and Events
  • Alerts will register in the alarms and display in the client
  • Supports vCenter 5.0 and SQL 2008 R2

That’s what I have on vSphere 5, next up is SRM5, vShield5, Storage Appliance, and vCloud Director 1.5.

There are 3 scenarios of MSCS clusters and ESX: Cluster-in-a-box (both MSCS nodes are on the same physical host – great for testing), cross-host (where each of the MSCS node VMs resides on different ESX hosts), and physical-virtual (where one MSCS node is physical, one is virtual).  The requirements for MSCS can change, even in the minor updates, so check the documentation often.  Here’s my compiled list of requirements/tips for MSCS on ESX 4.0:

• You are still limited to two-node clusters with MSCS on ESX 4.
• From a storage perspective, you can use local storage (for cluster-in-a-box) or Fiber Channel (for cross-host or physical-virtual clusters).  There is still no support for NFS or iSCSI (I personally think this is because FC and local storage have more predictable performance – although iSCSI is improving on this).
• If you are doing cross-host, both hosts must be running the same version of ESX (this just makes sense really).
• The MSCS node VMs cannot move as part of HA or DRS.  (HA is being a little redundant for MSCS, DRS is because MSCS is so hyper-sensitive to network connectivity that even a ping loss could failover the MSCS cluster).
• You cannot use MSCS with Fault Tolerance  (i.e. FT VM’s can reside on the same physical ESX hosts, but MSCS node VMs cannot run as FT pairs)
• You cannot vMotion MSCS node VM’s.  (Same reason as DRS).
• You cannot use N-Port ID Virtualization (NPIV)
• If you are using FC and using the native multipathing in ESX, you cannot use round robin as a path policy.
• You must use VM hardware version 7 with ESX/ESXi 4.0 (if you migrated the VMs from ESX 3.5 or before, make sure to upgrade your VM hardware version)
• Failover clustering with Windows Server 2008 is not supported with virtual compatibility mode RDM’s, for Win2008 use physical compatibility mode RDMs.
• You cannot use thin-provisioned disks for the Windows OS vmdk’s, they have to be thick.
• For Win2000 and Win2003 use LSI Logic Parallel as the controller type for the shared storage.  For Win2008 use LSI Logic SAS.
• For physical-virtual MSCS clusters, use RDMs in physical compatibility mode (this just makes sense if you think about it)
• You cannot run storage multipathing software in the VMs or on ESX (i.e. no PowerPath VE).
• You cannot over-commit memory for the MSCS node VMs, set the Memory Reservation option for each of the nodes to the amount of memory assigned to the virtual machine.
• Set the disk I/O timeout to 60 sec. or more (HKLM\System\CurrentControlSet\Services\Disk\TimeOutValue) in the registry.

You can find all the details and steps walking you thru the setup of MSCS on ESX in this article .  If you’re not on vSphere yet but you want to run MSCS nodes as VMs, you can find the proper docs for your version of ESX in a freshly updated KB article located here .

What does Fault Tolerance do to prevent a split brain if both Primary and Secondary VMs become isolated?

Fault Tolerance (FT) uses an on-disk generation number file.  When FT is enabled the primary VM creates a file on shared storage called generation.N where N is a counter number.  The secondary VM is started and when it connects to the primary, the primary tells the secondary what the generation number is.  Once the Primary or secondary detects that there is a failure in the other half of the VM pair, it will try to rename the generation.N file to generation.N+1.  If the rename succeeds, the VM takes over as being the Primary (or remains the primary if it already was) and takes corrective action to rebuild a secondary and become protected again.  If the rename of the generation.N file fails, that means that the other VM in the pair already renamed the file and took over and the current VM shuts down.

There you have it, the disk subsystem prevents both VM’s from becoming the primary at the same time and creating a split brain.

vCenter 4.0 build: 162902

The wait is over!  vSphere is out and available!  Go get it here !

You can read all of the details in the white paper here .

vCenter 4.0 (vCenter is now 4.0 to be on par with ESX) includes linked mode in the standard edition of vCenter.  Those users who have current support on vCenter standard edition will receive linked mode as part of their included subscription.  Linked mode is not included in the foundation edition of vCenter or in the vSphere essentials editions .  Linked mode uses a ADAM (Active Directory Application Mode) database to replicate it’s configuration between multiple vCenter servers.  Some of the things stored in the ADAM database include:

• Connection information (IP addresses and ports)
• Certificates and thumbprints
• License Information
• User Roles

When vCenter servers are "linked," connecting to any one will allow an administrator to see and manage any of the vCenters in the link group thru one vSphere Client assuming they have rights to do so (as seen below).

Permissions can be configured so that an admin may only have rights to certain vCenter servers in the link group and not others.  The scalability has also been increased dramatically.  A single vCenter instance can manage 300 ESX hosts and 3000 VMs.  A linked mode group can manage 1000 ESX hosts and 10,000 VM’s (now you know why a search field was added to the client.)

You can read about linked mode and all of the new features in vCenter here .

1. Receive activation code in email
2. Go to license portal
3. Generate a new license file
4. Choose server or host based file
5. Select license quantities to activate
6. Download or email license file
7. Install license server if needed
8. Upload license file to license server
9. Re-read license file to server
10. Configure licensing in VC/ESX UI
11. Product activated.

The great news is that this has been drastically simplified in vSphere 4.0.  Licenses are now delivered as 25-character keys.  The procedure now looks like this:

1. Receive license keys in email
2. Enter license key into vSphere Client GUI
3. Assign licenses to ESX hosts
4. Product activated.

Here’s a shot of what the new licensing config looks like in the new vSphere client:

This definitely got the biggest "Thank You" from the audience.  The license server will still be available in vSphere 4.0 so that you have backward compatibility.  For instance, if you wanted to run some ESX 3.5 and some 4.0 together.  vCenter 4.0 will be required to manage both versions of ESX together (vCenter 2.5 cannot manage ESX 4.0 hosts however vCenter 4.0 is backward compatible).

Not a ground-breaking feature by any means, but one solving some pain that customers have felt for a long time.  Enjoy, reports about more of the new features coming soon.  Next up: vCenter Linked Mode

Stephen talked about the main focus of VMware.  That is, to enable customers to deliver IT as a service, at the same time remembering 3 key factors:  Efficency, Control, and Choice .  He went on to discuss vSphere.  He gave some awesome statistics on vSphere, particularly: over 3 million Engineering hours have gone into vSphere .  For the Efficiency section, he discussed the scalability and performance.  He talked about power savings and storage savings .  For Control , he talked about Config-Control and how to maintain consistency in host configuration.    He then discussed distributed switches and how they add control to the networking layer.  vApp and how it will let admins control their apps better as a single unit.  He then went on to demo a very polished Fault Tolerant configuration.  This was received very well and with a resounding applause from the crowd.  He talked about vShield Zones and how they protect applications from talking to the infrastructure that they shouldn’t for customers without advanced firewall experience.  Lastly, Choice .  The goal is to allow any server, any storage, any OS, and any app to run local in the private cloud or a cloud in a different location, public or private.  He showed a list of the supported OSes for vSphere compared to those of Hyper-V and it was not even close.  I guess it is an advantage to not have an OS business.

After the functionality section Stephen introduced Bogomil Balansky (below), VP of Product Marketing, VMware.

Bogomil introduced the Partners to the licensing and packaging of vSphere.  I apologize, due to legal restrictions I cannot list those here.  Hopefully we will find out more information from the announcement coming April 21st .

Stephen returned to the stage and discussed the API and plug in capability of vSphere which was a great segway to the next presenters: Adam Famularo (below on left) SVP and GM, Recovery Management and Data Modeling Business Unit, CA, and the second presenter was Roger Pilc (below, seated), Corporate SVP and GM, CA.

Adam showed a great video on the University of Texas, Brownsville, and how they used CA XOsoft to recover their MS Exchange environment on VMware at their disaster recovery site during Katrina.  He went on to discuss CA’s organization and how they are all aligned.  He discussed many of the channel programs that are available (this is a Partner conference after all).  He talked about upcoming Arcserve and XOsoft releases.  I apologize, I don’t know what is public or not from the information that was presented so I’m going to leave the details out on that one.  Roger took the lead at this point.  He discussed the cloud and how it is transforming business.  That management is one of the key pieces needed to take it to the next level.  He closed with an explanation of the VMware – CA relationship and how that joint effort can truly improve overall service quality from the cloud.

Now onto my labs today, I’ll check back in soon.