To enable Network I/O Control we need to have a DvS in place.  If we select the distributed switch and then select the Resource Application tab on the right, this gives us the “properties” option on the far right.  By selecting the Properties option, you can enable Network I/O Control on the DvS.  Once enabled you can see all of the System network resource pools.  There is one at the bottom of the list labeled “vSphere Replication (VR) Traffic”.  Selecting it and then clicking the “Edit Settings” link just below it opens up the settings window.

From here, you can edit the adapter shares.  The shares are to balance the bandwidth so that network flows can use the bandwidth thats available from a given dvuplink.  The shares are for a given dvUplink.

Alternatively, you can uncheck the Unlimited checkbox and set a host limit.  Keep in mind that this is Megabits per sec, not MegaBytes.  This is also the limit of the combined set of dvUplinks on a given host.

Lastly, a QOS priority tag can be used.  The traffic will have a 802.1p tag applied to it.  The IEEE does not standardize or mandate the use of the priority tag applied to the packets but the switches should treat higher tags with higher priority.  The choices are None, 1-7.

While not the granular controls that we may wish for, say individual bandwidth controls on a per VM or per-site replication limits, these settings and options are a start.  Hopefully in the future in vSphere Replication v2 we will have more granular controls for bandwidth throttling but until then, these are what we can use.  Happy computing.

Here are some of the features:

• Deploys as an appliance, very easy to install
• Must be deployed on a new ESXi 5.0 installation
• Deploys a VSA Cluster Service on the vCenter server
• The VSA Cluster Service can deploy the VSA “Agent VMs” to each of the ESXi 5.0 hosts
• The appliance will use the local space available and present the storage on the network as an NFS datastore
• Replicates the local storage to the local storage on another host in the cluster for redundancy.
• If a host fails, the appliance storing the replica will immediately take over the failed “Agent VM’s” IP address and share the storage from the replica
• v1.0 supports 2 or 3 ESXi hosts in a cluster (Typically for the essentials kits)
• Sold as a separate SKU with one price with no license capacity restrictions (no technical size limits that I could find)
• Supports 25 VMs (configured on 2 ESXi hosts) or 35 VMs (configured on 3 ESXi hosts)
• It is the only scenario where VMware recommends running vCenter on a physical or standalone ESXi hypervisor (To protect you from running into a Catch-22 as vCenter is managing the VSAs
• Recommended to use RAID10 on the hardware RAID controllers in the hosts (to protect from a single drive failure)
• Uses RAID 1 (Mirroring) between hosts for redundancy
• Supports Storage vMotion for when you are ready to migrate to hardware shared storage
• Can put the whole VSA cluster in maintenance mode or just a single node.  Can also replace a node and have the VSA rebuild onto it for redundancy or for rolling upgrades.

Here’s how it works: Imagine I have 3 hosts numbered 1,2 and 3.  Once the VSA gets installed, it creates two volumes on the available local storage on each host.  So host 1 will have volumes 1A and 1B, host 2 has 2A and 2B, host 3 has 3A and 3B.  Once the VSAs are configured, they will be redundant so that 1A (which stores VMs) mirrors to 2B, 2A mirrors to 3B and 3A mirrors to 1B.  If any VSA get’s dropped, the VSA running the mirror copy takes the IP address of the failed VSA and keeps right on chugging.

My Take

The Pros: Great solution for SMBs without shared storage to take advantage of HA, vMotion, etc.  I also think this is an outstanding solution for companies with remote offices who want to have redundancy in 2 or 3 ESXi hosts but don’t want to put shared storage in each site.

The Cons:  Way too much overhead.  VMware is recommending hardware RAID10 from the local drives if possible.  If I have 4 x 1TB drives in a server (4TB RAW disk capacity).  I use RAID10 as per VMware’s recommendation, this means 2TB gets presented to the ESXi host.  Now the VSA uses half of that storage for VMs and half as a target to mirror the VSA from one of the other hosts.  So out of 4TB of RAW disk, I get <1TB of capacity to store VMs on (don’t forget, I need room to store ESXi itself).  Thats a 75% reduction from RAW capacity = too much overhead.

Overall I still think it’s worth it.  It’s still going to be less expensive that a shared storage frame (even with the overhead loss).  I think for remote sites, you can’t beat it.  I can’t wait to see what they add to it in v2.0.

VMware vSphere 5.0

• ESXi Convergence – No more ESX, only ESXi (they said they would do it, they meant it)
• New VM Hardware:  Version 8 – New Hardware support (VS5 still supports VM Hardware 4 & 7 as well if you still want to migrate to the old hosts)
  • 3D graphics Support for Windows Aero
  • Support for USB 3.0 devices

• Platform Enhancements (Blue Requires Hardware v8)
  • 32 vCPUs per VM
  • 1TB of RAM per VM
  • 3D Graphics Support
  • Client-connected USB devices
  • USB 3.0 Devices
  • Smart-card Readers for VM Console Access
  • EFI BIOS
  • UI for Multi-core vCPUs
  • VM BIOS boot order config API and PowerCLI Interface
• vSphere Auto Deploy – mechanism for having hosts deploy quickly when needed ( I’m going to wait and see how customers use this one.)
• Support for Apple Products – Support for running OSX 10.6 Server (Snow Leopard) on Apple Xserve hardware. (although I betting technically, you can get it to run on any hardware, you will just not be compliant in your license)
• Storage DRS – Just like DRS does for CPU and Memory, now for storage
  • Initial Placement – Places new VMs on the storage with the most space and least latency
  • Load Balancing – migrates VMs if the storage cluster (group of datastores) gets too full or the latency goes too high
  • Datastore Maintenance Mode  - allow you to evacuate VMs from a datastore to work on it (does not support Templates or non-registered VMs yet…)
  • Affinity & Anti-Affinity – Allows you to make sure a group of VMs do not end up on the same datastore (for performance or Business Continuity reasons) or VMs that should always be on the same datastore.  Can be at the VM or down to the individual VMDK level.
  • Support for scheduled disabling of Storage DRS – perhaps during backups for instance.
• Profile-Driven Storage – Creating pools of storage in Tiers and selecting the correct tier for a given VM.  vSphere will make sure the VM stays on the correct tier(pool) of storage.  (Not a fan of this just yet.  What if just 1GB of the VM needs high-tier storage? This makes you put the whole VM there.)
• vSphere File System – VMFS5 is now available.  (Yes, This is a non-disruptive upgrade, however I would still create new and SVmotion)
  • Support for a single extent datastore up to 64TB
  • Support for >2TB Physical Raw Disk Mappings
  • Better VAAI (vStorage APIs for Array Integration) Locking with more tasks
  • Space reclamation on thin provisioned LUNs
  • Unified block size (1MB) (no more choosing between 1,2,4 or 8)
  • Sub-blocks for space efficiency (8KB vs. 64KB in VS4)
• VAAI now a T10 standard – All 3 primitives (Write Same, ATS and Full Copy) are now T10 standard compliant.
  • Also now added support for VAAI NAS Primitives including Full File Clone (to have the nas do the copy of the vmdk files for vSphere) and Reserve Space (to have the NAS create thick vmdk files on NAS storage)
• VAAI Thin Provisioning – Having the storage do the thin provisioning and then vSphere telling the storage which blocks can be reclaimed to shrink the space used on the storage
• Storage vMotion Enhancements
  • Now supports storage vMotion with VMs that have snapshots
  • Now supports moving linked clones
  • Now supports Storage DRS (mentioned above)
  • Now uses mirroring to migrate vs change block tracking in VS4.  Results in faster migration time and greater migration success.
• Storage IO Control for NAS – allows you to throttle the storage performance against “badly-behaving” VMs also prevents them from stealing storage bandwidth from high-priority VMs.  (Support for iSCSI and FC was added in VS4.)
• Support for VASA (vStorage APIs for Storage Awareness) – Allows storage to integrate tighter with vcenter for management.  Provides a mechanism for storage arrays to report their capabilities, topology and current state.  Also helps Storage DRS make more educated decisions when moving VMs.
• Support for Software FCoE Adapters – Requires a compatible NIC and allows you to run FCoE over that NIC without the need for a CNA Adapter.
• vMotion Enhancements
  • Support for multiple NICs.  Up to 4 x 10GbE or 16 x 1GbE NICs
  • Single vMotion can span multiple NICs (this is huge for 1GbE shops)
  • Allows for higher number of concurrent vMotions
  • SDPS Support (Slow Down During Page Send) – throttles busy VMs to reduce timeouts and improve success.
  • Ensures less than 1 second switchover in almost all cases
  • Support for higher latency networks (up to ~10ms)
  • Improved error reporting – better, more detailed logging (thank you vmware!)
  • Improved Resource Pool Integration – now puts VMs in the proper resource pool
• Distributed Resource Scheduling/Dynamic Power Management Enhancements
  • Support for “Agent VMs” – These are VMs that work per host (currently mostly vmware services – vshield, edge, app, endpoint, etc)  DRS will not migrate these VMs
  • “Agents” do not need to be migrated for maintenance mode
• Resource pool enhancements – now more consistent for clustered vs. non-clustered hosts.  No longer can modify resource pool settings on the host itself when it is managed by vcenter.  It does allow for making changes if the host gets disconnected from vCenter
• Support for LLDP Network Protocol – Standards based vendor-neutral discovery protocol
• Support for NetFlow – Allows collection of IP traffic information to send to collectors (CA, NetScout, etc) to provide bandwidth statistics, irregularities, etc.  Provides complete visibility to traffic between VMs or VM to outside.
• Network I/O Control (NETIOC) – allows creation of network resource pools, QoS Tagging, Shares and Limits to traffic types, Guaranteed Service Levels for certain traffic types
• Support for QoS (802.1p) tagging – provides the ability to Q0S tag any traffic flowing out of the vSphere infrastructure.
• Network Performance Improvements
  • Multiple VMs receiving multicast traffic from the same source will see improved throughput and CPU efficiency
  • VMkernel NICs will see higher throughput with small messages and better IOPs scaling for iSCSI traffic
• Command Line Enhancements
  • Remote commands and local commands will now be the same (new esxcli commands are not backwards compatible)
  • Output from commands can now be formatted automatically (xml, CSV, etc)
• ESXi 5.0 Firewall Enhancements
  • New engine not based on iptables
  • New engine is service-oriented and is a stateless firewall
  • Users can restrict specific services based on IP address and Subnet Mask
  • Firewall has host-profile support
• Support for Image Builder – can now create customized ESXi CDs with the drivers and OEM add-ins that you need.  (Like slip-streaming for Windows CDs) Can also be used for PXE installs.
• Host Profiles Enhancements
  • Allows use of an answer file to complete the profile for an automated deployment
  • Greatly expands the config options including: iSCSI, FCoE, Native Multipathing, Device Claming, Kernel Module Settings & more)  (I don’t think Nexus is supported yet)
• Update Manager Enhancements
  • Can now patch multiple hosts in a cluster at a time.  Will analyze and see how many hosts can be patched at the same time and patch groups in the cluster instead of one at a time.  Can still do one at a time if you prefer.
  • VMTools can now be scheduled at next VM reboot
  • Can now configure multiple download URLs and restrict downloads to only the specific versions of ESX you are running
  • More management capabilities: update certificates, change DB password, proxy authentication, reconfigure setup, etc.
• High Availability Enhancements
  • No more Primary/Secondary concept, one host is elected master and all others are slaves
  • Can now use storage-level communications – hosts can use “heartbeat datastores” in the event that network communication is lost between the hosts.
  • HA Protected state is now reported on a per/VM basis.  Certain operations no longer wait for confirmation of protection to run for instance power on.  The result is that VMs power on faster.
  • HA Logging has been consolidated into one log file
  • HA now pushes the HA Agent to all hosts in a cluster instead of one at a time.  Result:  reduces config time for HA to ~1 minute instead of ~1 minute per host in the cluster.
  • HA User Interface now shows who the Master is, VMs Protected and Un-protected, any configuration issues, datastore heartbeat configuration and better controls on failover hosts.
• vCenter Web Interface – Admins can now use a robust web interface to control the infrastructure instead of the GUI client.
  • Includes VM Management functions (Provisioning, Edit VM, Poer Controls, Snaps, Migrations)
  • Can view all objects (hosts clusters, datastores, folders, etc)
  • Basic Health Monitoring
  • View the VM Console
  • Search Capabilities
  • vApp Management functions (Provisioning, editing, power operations)
• vCenter Server Appliance – Customers no longer need a Windows license to run vCenter.  vCenter can come as a self-contained appliance (This has been a major request in the community for years)
  • 64-bit appliance running SLES 11
  • Distributed as 3.6GB, Deployment range is 5GB to 80GB of storage
  • Included database for 5 Hosts or 50 VMs (same as SQL Express in VS4)
  • Support for Oracle as the full DB (twitter said that DB2 was also supported but I cannot confirm in my materials)
  • Authentication thru AD and NIS
  • Web-based configuration
  • Supports the vSphere Web Client
  • It does not support:  Linked Mode vCenters, IPv6, SQL, or vCenter heartbeat (HA is provided thru vSphere HA)
• vCenter Heartbeat 6.4 Enhancements
  • Allows the active and standby nodes to be reachable at the same time, so both can be patched and managed
  • Now has a plug-in to the vSphere client to manage and monitor Heartbeat
  • Events will register in the vSphere Recent Tasks and Events
  • Alerts will register in the alarms and display in the client
  • Supports vCenter 5.0 and SQL 2008 R2

That’s what I have on vSphere 5, next up is SRM5, vShield5, Storage Appliance, and vCloud Director 1.5.

Let’s go thru an example.  First things first, before you export, make sure you have all the information you need.  In the vSphere client, pick your level to look at the Storage View (the vCenter Server, the datacenter, a cluster, a host, etc) and select the Storage Views tab on the right.  Right-click the title bar in the right pane and make sure you have all of the fields you need.  The menu will look something like this one here:

Make sure you select all of the columns of data that you want (or don’t want).  Once you have that cleaned up, we go for the export.  Your instinct would be to go to File and Export like this:

Notice how the Export List option is greyed out?  That’s where you would expect to find the export function for the Storage Views.  I think there is a bug in the interface as that is not really where it is located in this case.  For Storage Views, move the cursor to some white space on the right or the bottom of the right pane and right-click.  You should see a popup menu that looks like something like this:

Select “Export List” from the menu and you can save the storage view as Excel, HTML, CSV, etc.  There, now you have some great data for graphs or whatever you like.

One more tip:  If you export the fields to Excel, the cells will all be text and the data values will have “GB” in the cells with them.  If you want to remove the “GB” and convert to numeric so you can work with the values here’s how:

Let’s say Cell C3  has a value of “7.77 GB”.  Create a new column and for the value put in  =VALUE(LEFT(C3,LEN(C3)-3))

The LEFT() function will cutoff the “GB” and the VALUE() function will convert the TEXT to a numeric value that you can add, subtract, etc.

Now you can add a cell at the bottom and use the AVERAGE() function and find out what your average VM size really is!  Or SUM() the Snapshot column and find out how much space your VM snaps are taking up on storage.  There are all kinds of options, have fun and enjoy!

Duncan wrote an explanation in the community thread for what is going on:

“It most definitely has to do with the type of datamover used. When a different blocksize is used for the destination the legacy datamover is used which is the FSDM. When the blocksize is equal the new datamover is used which is FS3DM. FS3DM decides if is will use VAAI or just the software component, in either case unfortunately the zeroes will not be gobbled. I have validated it and reported it to engineering that this is desireable. The team will look into it but unfortunately I cannot make any promises if or when this feature would be added.”

It sounds to me like if your storage supports VAAI and ESXi offloads the svmotion to it, there’s no hope of shrinking because VAAI is not that intelligent.  When you svmotion between two different LUN block sizes (or between iSCSI, NFS or FC) it uses old-school svmotion (probably because the VMFS block layout changes) which will actually convert the VM to thin when it moves.

Hopefully VMware engineering will change svmotion to automatically use old-school svmotion when you select to change from thick or thin and only use VAAI when you choose to keep the same format during the move.  We’ll have to wait and see….

You may know a little about Avamar.  It’s a backup solution that uses source-based deduplication to perform backups.  It basically always takes full backups and it only stores pieces of files that it has not seen before in the entire environment.  Every thing that it has seen across your organization is tracked.  This includes which client it was seen on and when, but only one copy of the file piece is stored on disk.  This creates extremely efficient and rapid backup.  For VMware enviornments, Avamar can take file level backups by running a client in the GuestOS or a VM image level backup by running a proxy VM in the infrastructure.

When you combine these two technologies together, the result is the best of both worlds.  Specifically referring to the image level backups with CBT enabled.  This means that Avamar only backs up the pieces of the vmdk files that it has not seen before and with CBT, it only scans the blocks that have changed from the last backup when looking for pieces to deduplicate.  Very efficient and very optimized – we’re talking hundreds of GB in just minutes.

Here the issue I ran into: I added a client to Avamar and setup a policy to do Image Level backups of the VM.  I kicked one off and the Avamar starts by creating a snapshot of the VM and mounting the snap to the Avamar Proxy VM.  Avamar then queries CBT on vSphere and gets the list of blocks that changed since the last backup.  The proxy then scan thru only the blocks that changed and only send the file segments within those blocks that it has not seen before to the actual disks for backup.  When finished, Avamar unmounts the snap from the proxy and deletes the snap.  When I ran thru this procedure at the customer site, the first backup took about 15 minutes for 100GB on their system.  This is expected as there is no CBT information yet so the proxy must read thru the entire 100GB to determine what file pieces it has and has-not seen before and that takes the majority of the 15 minutes.  On the second backup however, I expect that CBT will only show the proxy the blocks that changed and then it will dedupe only those and store all of the other blocks in Avamar from the inventory of blocks it already has (as CBT said those blocks have not changed).  When I did go and run the second backup it took 15 minutes. It should have taken only a minute or two.  What’s the deal?

The solution:  I did some hard digging on the net for a solution. I was sent this article on the EMC support site from one of our other Engineers (thanks Judson!).  Basically it said that VMware has an issue (documented here) with CBT and VMware snapshots.  In a very specific scenario, a customer could restore a snapshot of a VM from vCenter and it’s CBT information would be inaccurate.  When the backup or replication was looking to CBT for the blocks that changed, it could provide incorrect information.  This would backup or replicate incomplete information without showing an error of any kind.  That’s bad.

Avamar knows about this issue and protects people.  It does this by looking to see if CBT is in use and if there are any VMware snapshots older than the last retained backup of the data in Avamar.  If there is an older snap, Avamar assumes that a customer could revert to it any time (or already did) and that the CBT data could be invalid – so it ignores the CBT info and reads thru the entire VM.  This is why my backup above took 15 minutes each time.  I had snapshots on that VM older than the oldest Avamar backup retained.  When I removed the snaps the next backup took 15 minutes (I later found that this was to reset the CBT information).  The next backup after that took 47 seconds.  Now we’re in business.

If you see these kinds of performance issues on Image Level backups in Avamar, try cleaning out the VMware Snapshots.  This issue does not affect file level backups only Image-level. I hope this helps out the users who are trying to run Image level with Avamar.  Now you’ll know what to try when performance for the backups slows down for no apparent reason.

Thanks and good computing.

I found a few references here and there.  Most customers keep the paging file inside the standard VM disks to avoid making the environment too complex.  I was about to give up and suggest he not separate the paging file, until I came across this discussion in the VMware communities.  I was stunned to see one customer report that they were seeing a reduction of 60% of the replication traffic on certain VMs.  They even saw a 80% reduction of replicated traffic from one of their Citrix servers.  Now I was cooking.  This was amazing and made perfect sense to me how some servers could be more defined by this than others.  I was off to find a documented procedure from VMware to do so.  Unfortunately I came up empty in the Knowledge Base, white papers and best practices.  If you know of such a VMware documented procedure, please post in the comments with a reference link and I will update my article.

After reviewing the communities discussion line-by-line, I came across this NetApp SRM technical paper written by Jeremy Merill and Larry Touchette (whom I was fortunate enough to work with while I was at NetApp).  It contains some great procedures for optimizing NetApp storage and Site Recovery Manager.  Beginning on page 41 however, is a great vendor-agnostic procedure for separating out the Windows Paging file to reduce it’s impact on storage replication.  Jackpot.  Basically there are a few points to consider.  First the windows swap file is configured for a specific drive letter and path.  How do you make sure the recovered machine gets the same drive letter when it comes up in the recovery plan?  The answer is in the procedure.  Jeremy and Larry describe how to configure the source VM with it’s Windows Paging file seperated.  Then they clone the VMDK where the paging file resides and copy it to the destination side.  The reason they do this is because the disk has a signature on it that tells Windows what drive letter it is.  Then they altered the recovery plan for SRM to mount that destination disk to the VM before it powers on.  So that when the recovery VM does come online, it has it’s page file disk as the same letter with the page file intact.  Because the Windows paging file gets wiped on every boot, the data in it is not important, only the location where it’s stored.  They repeat this procedure for every VM in the recovery plan.  I realize that this could be a large undertaking to separate all these page files but the more VMs, the more replication data saved.

I’m very curious if some have tried separating out the page file and what replication improvements you have seen.  Please report those in the comments if you have tried it.  I’d love to hear the results.

Before today, I did not know that.  What this means is that if you have an existing View 4.0 pool of desktops that use user data disks, 4.5 will not recognize them as persistent disks when you upgrade. Here’s how I found out:  View 4.5 has a new feature called “Disposable File Redirection” where it can put the windows swap file and default temp directory on it’s own virtual disk.  When creating a pool with the wizard it looks like this:

This is a great feature because when you create a linked clone pool the windows swap could increase the size of the linked clone by 2-4GB on first boot of the OS when the guest OS creates the swap file.  By seperating the temp and swap file to another virtual disk, you can keep the linked clone with the Guest OS small and more efficient.  So I wanted to take my user data disk from my desktop in the old pool (Created on View 4.0) and move it over to a new desktop pool built under View 4.5.  I selected my desktop in the old pool and clicked “Remove…”  This presented me the option to save my persistant disk to another datastore and destroy the rest of the desktop files.

Perfect, I thought, move my user data and then pick it up later.  The wizard allowed me to select the option to move persistent disks to another datastore, I choose it and click to continue without an error or warning of any kind.  As I waited, watching the destination datastore I had selected, nothing appeared.  Looking back thru the vCenter logs, View had deleted my desktop but did not do anything with the user data disk, my user data disk was deleted with the rest of the desktop.  It seems that upgrading to 4.5 does not convert user data disks to persistent disks (as mentioned in the doc above).  I believe this is a problem when the wizards do not warn you at all that they are not one in the same.

Some good news however, I did find a procedure that works to migrate my users.  Please note that I am guessing that this is not supported by VMware due to the note from the docs listed above.  Hopefully we will get a KB article with a supported procedure soon.

1. Create a new pool of desktops and create it from the same snap as your old pool (View 4.5 will use the existing replica if you do that – very smart).  Select user assignment for desktops (not floating) and make sure to use a seperate disk for temp and swap files like the first screenshot above.
2. Kick your user out of their old VM if they are in it and put that desktop in Maintenance Mode. (Under inventory, select desktops, select their desktop / more commands/ enter maintenance mode)  This is so they can’t sneak in on you while you are working on it.
3. Go into vCenter and shut down their old VM.  If the VM is in a suspended state, power on the VM and then shutdown the guest OS.  You want their desktop VM in a completely powered off state.
4. Get a copy of the User Data disk.  I did this easily by cloning the VM to another datastore.  Recognize that you’ll need to know the size or name of the user data disk vmdk file.  The name of the vmdk has “user-disk” in it.  The easiest way is to look it up in the datastore browser and write/remember it’s size.
5. In View Manager, go to the persistent section and click the detached tab at the top of the right pane
6. Click the “Import from vCenter” button
7. Select your vCenter server, the datastore containing your copy of your VM.  The new linked clone pool that you are going to add it to, the view folder to put it in, the file itself (here select the user data disk where you copied it to.  Mine was only 4GB in size so it was easy to determine the user data disk vmdk.)  Lastly select the user that owns this persistent disk  i.e. Who’s was it in the old pool?
8. Now that you have the detached persistent disk imported, you can select the persistent disk and click the “Recreate Desktop” button.  Note: the user associated with the persistent disk cannot have an asigned desktop in the new pool already.
9. Once the new desktop is created with the persistent disk, select pools on the left pane and double-click the new pool you created.  Go to the inventory tab on the right and select the desktop that you just created.  When it is highlighted, select the “View Composer” drop down and select rebalance.  This will rebalance the desktop and move the persistent disk file to the correct datastore and directory that it is supposed to be in.
10. Now make sure your user is entitled to the new pool (View 4.5 will automatically assign the VM to them) and you may optionally remove their entitlement to their old pool (so they are not confused by two similar desktops when they login).
11. Once all of this is completed, you can allow your user to connect in and test.  They should have their desktop in a just-refreshed condition.
12. When everything checks out ok, you can go remove their desktop from the old pool.

I can’t stress enough that this procedure may not be supported by VMware.  If/when they post a KB article on this topic, I will be sure to link it here.  I hope this helps out anyone that may have 4.0 Desktops with User-Data disks and upgrade.  Moving the user-data to the 4.5 friendly persistent disks adds a lot of flexibility for your users in the future.

PCoIP Display Protocol: Information and Scenario-Based Network Sizing Guide – (12 pages) A good paper with very good insight on the PCoIP protocol used in VMware View.  It gives some good suggestions and the required bandwidths needed to satisfy the end users on their desktop experience.  A must have for view deployments.  (****, 4 of 5 stars)

Application Presentation to VMware View Desktops with Citrix XenApp – (3 pages) This is a whitepaper to show how to deploy applications in VMware View desktops from XenApp.  While I can see this being useful for View admins who use XenApp, the description and instructions are very minimal.  Probably something better suited for a KB article. (**, 2 of 5 stars)

Timekeeping in VMware Virtual Machines – (26 pages) This is a very important topic for all VM Admins to know.  Time is relevant to everything in a VM, whether you are trying to authenticate to Active Directory or troubleshooting using event logs, accurate time is very important.  This paper goes into some really great detail on how VMware maintains accurate time in VMs.  If you are a VMware admin, this should be a standard read.   (*****, 5 of 5 stars, vKeeper)

SAN System Design and Deployment Guide – (244 pages of storage goodness)  I have a storage background so I specifically enjoy this one.  If you are running ESX on SAN shared storage (you should be on some type of shared storage) then this is a must read.  This whitepaper is also very helpful if you are studying for the VCP or one of the new VCAP exams.  This is another paper I keep local and definitely one all VM admins with SAN should review.  (*****, 5 of 5 stars, vKeeper)

Best Practices for Running vSphere on NFS Storage – (14 pages) On the heels of the SAN design and deployment guide, this paper describes the best practices for running NFS on vSphere.  I like the fact that this article references outdated best practices that have changed and why they have changed.  This is a HUGE help to admins who google a topic only to find conflicting information.  My only regret on this paper is that I would like to see more detail on the advanced options and how they affect the performance of NFS.  Still a important doc for VM Admins using NFS storage.  Should be reviewed by all of them to make sure they are current in their deployment of NFS best practices.  (****, 4 of 5 stars)

Location Awareness in VMware View 4 – (8 pages) Good information for View Admins to know where to find out where their clients are connecting from.  This is a common request from hospitals to have printers “follow the user” as they float from terminal to terminal.  There are some advanced topics in this article and some Active Directory knowledge is definitely required especially when using loopback mode in group policy processing.  Good info and hopefully View will include some GUI-based  native features in the future to assist with this.  (***, 3 of 5 stars)

VMware vSphere 4.0 Security Hardening Guide – (70 pages) This is a outstanding reference for any VM Admin.  Security affects everyone’s environment, from the 3-man shop to the largest infrastructure.  Setting the precedence of a solid, secure enviornment from the ground up will provide you with a infrastructure that is solid as a rock. I recommend reviewing this paper often and keeping this one handy   (*****, 5 of 5 stars, vKeeper)

VMware vStorage Virtual Machine File System – Technical Overview and Best Practices – (13 pages) This is a entry level paper on some of the very basics of VMFS and how they relate to RDMs.  This should be a good introduction to VMFS to new VM Admins.  I hoped with “Best Practices” in the title that there would be more technical references (advanced options for VMFS and how tweaking them affects the storage performance for instance).  I was also disappointed to see the LUN size question answered vaguely, suggesting to refer to the storage vendor to size your LUNs appropriately.  I prefer Duncan’s approach to LUN sizing and it’s what I recommend to all of my customers.  (***, 3 of 5 stars)

Look for the vPaper Report again next quarter (hopefully with some new releases in between). Until then, happy reading!

When I suggested CBT to the customer they asked, “Why that one?” and how it worked.  So I explained:  When we replicate from source to target, the first copy is a full copy of the data (the “seed” it is often called).  When we go to replicate the next time, we don’t want to replicate the whole thing again, just what has changed since the last time we replicated (often called a “differential”).  The replication software needs to determine what’s changed.  Prior to ESX 4, there was not a built in method to do this.  The software would have to find another method, such as compare snapshot information and determine which blocks are new.  That uses CPU cycles on the ESX hosts and takes time (differential mode in vReplicator takes  roughly 1 minute per GB of VM data).  On the other hand, CBT is a feature in ESX4 that tracks the block changes that have occurred since a point in time.  It does not keep a copy of the changed data in a separate location, just a log that the blocks in question have changed.  This is a huge help to backup and replication technologies who typically have to determine what has changed on the disks via their own methods.  Now, ESX can tell them directly what has changed and they can get right to copying those changed blocks.  This makes the overall replication and backup jobs much quicker.

Now for a few lessons learned in using it.  First, it requires hardware version 7 VM’s (HW7) and ESX4.  VM’s need to have their VMtools upgraded to the latest version and then you can upgrade the VMs to HW7 when they are powered off via right clicking them (this updates the virtual hardware presented to the VMs and will require another reboot in Windows after powering it on when the OS discovers the new virtual HW and loads the drivers – thanks Microsoft!).  Second, CBT it is not on by default.  It is set per VM and is an advanced option you can set in the VM’s config.  Some software have the capability to change the CBT setting for you.  In our case, vReplicator has this option on the CBT options page.  On that page, it will check every VM that it can see and if they are HW7.  If they are HW7, they will show as supported.  On that screen, you will also see a checkbox for the “enabled” field.  When you click the enabled box on your HW7 VMs, vReplicator makes the change for you in the VM’s configuration.  However, as mentioned earlier, you must completely power down that VM and power it back on.  The reason for this is that, to start using it, ESX needs to create the tracking log for each disk (the log is about .5MB for ever GB of VMDK or Virtual Mapped RDM and it’s stored with the VM) and ESX only does this setup process at VM boot time.  So make note, a restart won’t work.  It has to be a VM power down and VM power back on.  There is a great article that taught me a few things on CBT by Eric Siebert that goes into a little more technical detail and you can find it here.

Once we got this process completed, my customer’s replication jobs ran MUCH faster.  The data being copied from the source to the target was the same, but the time it took vReplicator to determine what to replicate went from minutes to seconds.  Great news too was that we were able to change the replication method on the fly (from Differential to CBT, if you’re using hybrid, I think you need to re-seed).

My final advice, is make sure you understand if your backup/replication software can use CBT and what you need to enable it.  It does take a bit of work to upgrade the tools and virtual hardware (use Update Manager!).  However it’s well worth it in the long run.

Then it hit me, a single ESX host currently can see up to 256 LUNs.  Initially I thought, “but they’re never going to run more than 256 VMs on a host.”  No, but they did want to start using vMotion.  Now the LUNs will need to be presented to all hosts.  This 256 LUN limit no longer relates to the single host but to the cluster as a whole.  With all LUNs presented to all hosts, as long as they keep provisioning one-LUN-per-VM, they will be limited to 255 VM’s for the cluster (one of the LUNs is for booting ESX).  This was a limit they were most certainly going to hit (and at an accelerated pace, now that they have vMotion).

This made sense quickly to the customer.  The story has a happy ending: next week we’re upgrading them to vSphere and going to storage vMotion those VMs to a place with a better design.  There’s one thing I’ve learned about storage and virtualization is that there are no wrong designs.  However, there are ones that limit functionality.

The moral of the story is to know thy vmware maximums!  Make sure to check if a single host’s limitation could affect the design of an entire cloud.

Happy Earth Day!

Dynamic Storage Provisioning – A very nice introduction to Thin Provisioning in vSphere.  The concept of Thin Provisioning is explained as well as when you want to use it.  Some interesting points I noted in the article: Thin disks expand in chunks the size of the vmdk’s block size (1MB by default) and the only way to defragment a vmdk currently? – Storage vMotion.

Performance Study of VMware Thin Provisioning – Very good read on the performance impact of thin provisioned disks vs. thick disks.  I was somewhat surprised by the results.  There’s not nearly as much of a performance impact as I thought there would be.  A very good read if you are looking to justify Thin Disks for some applications within your organization.

VMCI Socket Performance – This was a very interesting paper that I’m still scratching my head on.  VMCI is the interface that programmers can use to communicate between VM’s on a given host.  So if a programmer is writing an app and it requires 2 VMs that do a lot of communication to each other, they can communicate on the VMCI interface instead of traversing the TCP/IP network and going thru all of the networking stack.  This paper shows the performance of using VMCI instead of TCP/IP for Windows and Linux boxes.  I’m scratching my head because the results are not as linear as I would have expected and there are scenarios that perform much better or worse than others.  Take a read and make your own conclusions.

VMware vCenter Site Recovery Manager 4.0 Performance and Best Practices for Performance – I’m probably the #1 fan of SRM.  I think what SRM does for DR is like what a conductor does for an orchestra.  As you may know, SRM 4.0 now scales to 1,000 VMs.  It can take a while to optimize that number of VMs for recovery.  This paper is an excellent resource for optimizing the setup and config of SRM to scale effectively to a very large number of VMs.

There are 3 scenarios of MSCS clusters and ESX: Cluster-in-a-box (both MSCS nodes are on the same physical host – great for testing), cross-host (where each of the MSCS node VMs resides on different ESX hosts), and physical-virtual (where one MSCS node is physical, one is virtual).  The requirements for MSCS can change, even in the minor updates, so check the documentation often.  Here’s my compiled list of requirements/tips for MSCS on ESX 4.0:

• You are still limited to two-node clusters with MSCS on ESX 4.
• From a storage perspective, you can use local storage (for cluster-in-a-box) or Fiber Channel (for cross-host or physical-virtual clusters).  There is still no support for NFS or iSCSI (I personally think this is because FC and local storage have more predictable performance – although iSCSI is improving on this).
• If you are doing cross-host, both hosts must be running the same version of ESX (this just makes sense really).
• The MSCS node VMs cannot move as part of HA or DRS.  (HA is being a little redundant for MSCS, DRS is because MSCS is so hyper-sensitive to network connectivity that even a ping loss could failover the MSCS cluster).
• You cannot use MSCS with Fault Tolerance  (i.e. FT VM’s can reside on the same physical ESX hosts, but MSCS node VMs cannot run as FT pairs)
• You cannot vMotion MSCS node VM’s.  (Same reason as DRS).
• You cannot use N-Port ID Virtualization (NPIV)
• If you are using FC and using the native multipathing in ESX, you cannot use round robin as a path policy.
• You must use VM hardware version 7 with ESX/ESXi 4.0 (if you migrated the VMs from ESX 3.5 or before, make sure to upgrade your VM hardware version)
• Failover clustering with Windows Server 2008 is not supported with virtual compatibility mode RDM’s, for Win2008 use physical compatibility mode RDMs.
• You cannot use thin-provisioned disks for the Windows OS vmdk’s, they have to be thick.
• For Win2000 and Win2003 use LSI Logic Parallel as the controller type for the shared storage.  For Win2008 use LSI Logic SAS.
• For physical-virtual MSCS clusters, use RDMs in physical compatibility mode (this just makes sense if you think about it)
• You cannot run storage multipathing software in the VMs or on ESX (i.e. no PowerPath VE).
• You cannot over-commit memory for the MSCS node VMs, set the Memory Reservation option for each of the nodes to the amount of memory assigned to the virtual machine.
• Set the disk I/O timeout to 60 sec. or more (HKLM\System\CurrentControlSet\Services\Disk\TimeOutValue) in the registry.

You can find all the details and steps walking you thru the setup of MSCS on ESX in this article .  If you’re not on vSphere yet but you want to run MSCS nodes as VMs, you can find the proper docs for your version of ESX in a freshly updated KB article located here .

• VMware vSphere 4: Exchange Server on NFS, iSCSI, and Fibre Channel – a very nice paper on how NFS, iSCSI, and Fiber Channel performed running a very large number of mailboxes on Exchange.
• Performance Best Practices for VMware vSphere 4.0 – Absolutely a must-read for new administrators and a new addition for my Admin Bookmarks page .  This is also a great review for those looking to upgrade to vSphere.
• Using Local Disks in VMware View Deployment – A very interesting idea: in a remote office running local ESX servers, use a Storage Appliance to gain HA with local disks only for desktops running in View.
• VMware Fault Tolerance Recommendations and Considerations on VMware vSphere 4 – The defacto-standard read if you are planning to use FT in your enviornment.  This article is a keeper of all of the current ins-and-outs of FT.
• What’s New in VMware vSphere 4: Virtual Networking – a great reference for all of the new capabilities in networking in 4.0.  Very detailed and technical.
• What Is New in VMware vSphere 4: Storage – The complete list of everything that’s new from a storage perspective.  From thin-provisioning to Pluggable Storage Architecture, it’s all there.

For instance, this utility only works on XP, Vista, Server 2003 and Server 2008.  It allows you to run a command from the command line pointing it at you Data Recovery appliance.  The utility then responds with a list of all of the restore points the machine you are running on has available.  You can then select a restore point from the list and the utility mounts that restore point as additional drive(s) on the current VM.  You can then use explorer to grab the single files you need out of the backup image.  When completed, you type the "unmount" command on the command line which will unmount all of the disks mounted by the utility.

It is still in the Experimental stage but looks very promising for admins just needing to grab a few files out of Data Recovery.

You can grab it and the docs for it in the Data Recovery download section here .

• VMware VDI Storage Considerations <-A great paper which discusses which protocol to use, how to lay out datastores, etc for View implementations.
• Running Microsoft Office SharePoint Server 2007 on VMware Infrastructure <-A great paper for those wanting to virtualize Sharepoint.
• Automating SharePoint Provisioning and Deployment on VMware vSphere <- This paper describes how you can setup sharepoint sites in templates for rapid deployment of new sharepoint instances.
• Virtualizing a Windows Active Directory Domain Infrastructure <- I know many have already done this  but it is still a great reference and includes best practices.  Definately want to confirm your settings against this white paper
• SAP Solutions on VMware vSphere: High Availability <- A good white paper on the availability options for SAP implementations and the benefits of each.

Performance is definitely the harder of the two to determine.  Not every storage frame is created equal.  I’ve held Engineering positions at EMC and NetApp.  I can tell you that there’s a lot of marketing you are going to have to weed through to find out what you need from a performance standpoint.  In addition, no two customer’s environments require the same storage performance.  What I often suggest is to track the disk I/O on your physical machines.  You can use the standard OS Performance counters to do this.  Take a note on the physical hosts and gather the average MB/second of storage reads and writes and the average numbers of transactions per second.  You will also want the peak of these counters as well.  You can provide an aggregate of these to your storage vendor’s engineer to confirm their frame can deliver your minimum performance requirements.  Clearly you would want to pad these numbers to anticipate future growth.  If you do not have the expertise (or time) in-house, check with your storage vendor or Partner to see if they offer any virtualization/storage assessments.  Some offer free for basic results and some have paid assessments which can get extremely detailed and leave "no stone unturned".  To answer the performance question, the more detail you have on your own performance requirements, the better off you will be.  VMware has a very high-level protocol performance whitepaper which describes how the protocols perform from ESX server.  Remember that there are different price points for the protocols with Fiber Channel typically being the most expensive.  Price is always another factor to consider but every customer weighs price differently in the list of priorities.  Look for a balance of performance, price and functionality.  Know the priority of each factor for your company and review it often during the evaluation process, it may change.

The second factor in the protocol decision is functionality.  While I know that performance can be a difficult factor to clarify, functionality is pretty cut and dry. These are listed by protocols with the most functionality to the least.

Fiber Channel

Fiber Channel (FC) has most complete set of functionality.  If you go with a FC solution, there are no suprises with functionality limitations.  All of product functionality is supported (vMotion, HA, DRS, Lifecycle Manager, Lab Manager, Boot from SAN, etc.).  The other protocols run into a few limitations however.

iSCSI

From a VMware perspective booting the ESX hypervisor from SAN is only supported on iSCSI if you use hardware-based iSCSI initators.  The included software-based initatior does not support ESX boot-from-SAN.  Please remember, I’m referring to booting the ESX hypervisor itself from iSCSI is not supported with the software initiator, booting the VM’s from iSCSI is fully supported. Second, Microsoft Cluster Services is not supported on iSCSI, only Fiber Channel.  If you wanted to run MSCS nodes inside virtual machines, iSCSI will not work for you.

NFS

You cannot boot ESX from NFS.  That seems obvious but I’ll list it for the newcomers.  Second, you cannot do Raw Device Mappings on NFS.  Raw Device Mappings are presenting a LUN directly to a VM’s operating system.  No LUNs here on NFS.  Third, Storage vMotion is not supported going to or from NFS.  Storage vMotion is migrating where the VMs are stored while they are running.  If you want to migrate any VM’s to or from NFS today, you have to shut down the VMs and do a cold migration to move them.  Fourth, Site Recovery Manager currently does not support NFS.  If you want to use SRM to manage your DR plan today, NFS will not get you there.

One additional point I want to make is two pieces of functionality that NFS does support that all the other protocols do not.  They are thin provisioning and dynamic expandability.  NFS will allow you to thin provision VM’s and it’s actually the default behavior.  Thin provisioning is creating a virtual disk for a virtual machine and the only space taken on physical disk is the amount of data stored in that virtual disk.  For example, if I create a VM with a virtual disk that is 10GB.  I then load an OS in that VM that takes up 5GB of that virtual disk.  On local storage, iSCSI, and Fiber Channel this scenario will use 10GB of physical storage (the full size of the virtual disk, even if I’ve stored anything in it or not).  In the same scenario, NFS only takes up 5GB and it’s virtual disk file would grow dynamically as I put more in the VM.  VMware has announced that thin provisioning is coming to VMFS datastores in 2009 , NFS does it today.  Additionally, some storage frames do this behind the scenes at a storage frame level.  Check with your vendor’s capabilities.

The last benefit that NFS has over the others is dynamic expandibility.  ESX uses datastores to store it’s VMs.  Datastores can be comprised of one or more LUNs, or in the case of NFS, a mountpoint.  With an NFS datastore, the filesystem is maintained by the storage frame.  You can dynamically expand the NFS datastore by increasing it’s size on the storage frame and the ESX servers will recognize the increased storage with no additional effort.  With the VMFS datastores, you have to add extents to increase the datastore’s size.  In the case of VMFS, there’s a bit of administrative work to make this happen.

Local Storage

My least favorite and only recommended to customers just getting into virtualization who cannot afford a SAN or for small Remote Offices with no shared storage available.  Local does allow you to boot ESX from it (obviously).  However vMotion, DRS and HA are not permitted.  You cannot use it for a MSCS cluster.  It is not supported with SRM or Storage vMotion.

Here’s a quick summary chart of the functionality differences:

In summary, know what you require for performance and make sure your storage vendor can meet your demands.  Then make sure the protocol has the functionality that you need at a price you are willing to accept.  You’ll be off and running in no time, with a infrastructure that has the elasticity and versatility to respond to whatever your business can ask of it.