HiperLogic

Virtualization, High Performance Computing, Enterprise Computing

A customer recently asked me how often they needed to do a “full” backup with Veeam so they would have a full backup in the event there was corruption of one of their differentials. With Veeam Backup’s synthetic backup, this scenario doesn’t exist!!

Veeam uses a synthetic backup which means EVERY backup gives you a full backup, at incremental speed, in a file called jobname.vbk. This .vbk file does NOT depend on any past differentials ( which have the extension .vrb).

The fantastic thing about synthetic backup is never taking the time and space penalty after your 1st backup for a full backup, with all the comfort of a traditional full backup.

The veeammeup blog has a good technical description of this wizardry of how synthetic backup works, which I duplicate here.

The way Veeam’s Synthetic Backup works is that after the first full backup, all subsequent backups are incremental, meaning just the changes from the last backup run, forever. Veeam Backup “injects” the changes into the full recovery file (.VBK) and then also saves any data replaced during this process into the reversed incremental changes file (.VRB). The .VBK file is always a full recovery file and the largest file in the directory. The .VBK file also has the most current modified date as it gets updated after each backup cycle.

The .VRB files do not change, as they contain the .VBK data blocks which were replaced by incremental data for that particular incremental backup run. To restore or roll-back to a particular date/time, all related .VRB files are applied to the .VBK file in the required order to get you back to that point-in-time.

The retention policy specifies how many of the .VRB files you want to keep, this also then corresponds to how “far” you can roll-back. If the retention policy is set to 14, then the 15th time the backup job runs it will delete the oldest .VRB file.

If you archive the .VBK and associated .VRB files as a set to tape, you can recover them back to disk if needed and import them into Veeam Backup to roll-back or recover to any point in time in that set.

So to be clear, if you accidently delete the .VRB file(s) you can get back to your full, as the .VBK is the “full” file. (This should never happen, but I had a customer who had an admin delete these files on their filer to free up some space before they knew the implications).

We love PowerShell at HiperLogic, because we use it to automate just about everything.

Veeam Backup has PowerShell integration, and this post is about creating a small script to meet a user requirement to get a list of VM’s in a backup job called “mytestjob”. The customer wanted to integrate this information into another program via powershell glue.

To run this code, run the following from the powershell console:

add-pssnapin "VeeamPSSnapIn"
$job = Get-VBRJob -name "mytestjob"
$job.GetObjectsInJob() | foreach { $_.Location }

One other interesting easter egg of things to come, run this powershell line

 Get-VBRJob |get-member |select-string "HyperV"

Veeam has announced the next release of Veeam Reporter Enterprise, now known as Veeam Reporter 4.0. This release is currently in Beta.

This new release builds on Microsoft SQL Server Reporting Services (SSRS), which will benefit customers by removing the need to have Microsoft office applications installed for reports, allows access from any client, and allows Veeam to ship reporting packs faster and outside of major releases. Note there is no charge for SSRS with SQL Server Express (free.)

There is a new web dashboard for 4.0, this new dashboard can be customized and embedded into other websites like Sharepoint.

Veem Reporter 4.0 now includes historical performance data which allows for trouble shooting and capacity planning.

Finally new functionality has been added to support new vSphere technologies such as host profiles and distributed switches, integration with Veeam Business View, and Visio diagram support for up to 6000 VM’s.

Veeam has announced that the next release of Veeam Backup and Replication will have some exciting new features,  including the new  SureBackup technology we have covered in this blog before.

Another major change in Veeam Backup and Replication 5.0 ( Scheduled for Q3 2010) is that there will now be TWO editions, Standard Edition 5.0, and Enterprise Edition 5.0

Note that ALL THE CURRENT FEATURES of Veeam Backup and Replication 4.1.1 are in Standard Edition, plus all the new 5.0 features. NO current features have been removed and moved to the Enterprise edition, which is very good for existing customers. All existing Veeam Backup and Replication Customers ( on maintenance) will get 5.0 Standard Edition at no charge.  Veeam is creating an Enterprise Edition to target new high end features that not all customers may want to pay a premium for.

Currently, there are differences in three areas between Standard and Enterprise Edition:

Recovery verification: Standard Edition will provide basic recovery verification (verifying the latest backup
of a VM by manually mounting and testing the backup file), while Enterprise Edition will offer automated
and more advanced recovery verification (including testing any available restore point, not just the most
recent one).

 Windows guest file system indexing: Standard Edition will maintain an index of Windows guest files in
current backups, while Enterprise Edition will also include files in archived backups, for using google like search functionality.

Universal application‐item recovery: Enterprise Edition will include universal application‐item recovery
(Standard Edition will not). This is the ability to restore individual application objects, such as exchange email
messages, directory objects, or database records, from a regular backup.  Note that individual file level recovery is STILL available in both editions, just like it always has been.

Customers who are current on maintenance will receive Standard Edition v5.0 as a regular product update at no additional charge. Standard Edition will include all the functionality in the current v4.1 software plus the major new features described in this blog, and the usual fixes and improvements.

For customers that want the enhanced functionality of Enterprise Edition,  the Enterprise upgrade fee will be waived on licenses purchased on or before August 31st, 2010. You must have maintenance on all sockets purchased all the way up to the time you decide to upgrade, no later than 3 months after the 5.0 Enterprise release. ( We recommend customers always stay on maintenance, so this should be a non-issue).

Many customers want to know if they should upgrade to Enterprise, or stick with Standard?  For customers that want the enterprise features mentioned, you should upgrade. But Note that all FUTURE purchases of Enterprise will be $330 per socket MORE expensive than Standard edition.  (MSRP is 659 on Standard, 989 on Enterprise.) With this in mind, you may want to buy any more sockets you anticipate you will need this year BEFORE June 18th to save money.

If you do NOT need the new Enterprise features, then you may not want to take advantage of the free upgrade to Enterprise because all future Enterprise sockets will be $330 more expensive than Standard sockets. In the case you don’t need the new Enterprise features now or in the future, you may want to stick with Standard Edition.

If there are any questions on the new Veeam Backup and Replication 5.0, please contact HiperLogic and we will help you determine what is best for your particular situation

Pano Logic was featured on CNN Money Tech Talk. Very nice video here.

Microsoft launched a few virtualization grenades today, here is a quick summary of all the interesting news:

1. Vista Enterprise Computing Desktop (VECD) licenses are now toast as of July 1st, now for PC’s that are covered by Software Assurance you will no longer have to purchase an additional VECD license just to access virtual Windows Desktops. For companies not under software assurance you will need to pay about $100 per device a year to access virtual desktops, called a “Virtual Desktop Access” license.

2. Windows Server 2008 R2 SP1 ( no ship date known yet ) will have new enhancements to the RDP protocol ( called RemoteFX ) that will enable better graphics performance when accessing Windows 7 virtual desktops on a LAN. The reported caveat here is RemoteFX is supported only on Virtual Desktops running on Hyper-V, not VMware. Of course there are already plenty of excellent protocols already that replace RDP on VMware, like Console Direct from Pano, PCoIP, HP RGS, and a host of others.

3. Hyper-V R2 Dynamic Memory. This new feature allows all the memory available on a physical host to be dynamically distributed to virtual machines running on that host as necessary. That means based on changes in workload, your VMs will be able to receive new memory allocations without a service interruption. As I understand it, they will not be doing over-commit, so densities will still be higher on VMware, but still a nice addition for Hyper-V when it releases, currently there is no beta it was just an announcement.

3. XP Mode no longer requires Intel-VT or AMD-V on Windows 7, so that is good news, get the bits/news from here.

Microsoft also announced the so called “Rescue for VMware VDI promotion” which allows VMware View customers that weren’t happy with View to trade in up to 500 licenses at no additional cost, while the VDI Kick Start promotion offers new customers a 50 percent discount off the estimated retail price in year one.

The one thing I can say here is if you weren’t happy with VDI on VMware, either with Pano, View, or other stack, you are very likely to be equally unhappy with VDI on Hyper-V. It isn’t like Microsoft has the magic blue pill, issues with VDI at large scales are more or less the same for all the major players. HiperLogic personally likes an agnostic approach like Pano Logic, they currently leverage VMware on the backend, but will soon support Hyper-V which helps avoid vendor lock-in.

SureBackup is a major new announcement from Veeam that was revealed on March 22nd ( See http://www.veeam.com/surebackup/) that allows **automated** validation of your backup jobs, a first in the virtualization ( or physical ) world.

SureBackup does this magic by booting all of the VM’s in your backup image in a sandbox environment ( Called the Virtual Lab ) to ensure your VM’s are 100% consistent and restorable.

SureBackup at a high level works by dynamically mounting a on-the fly NFS datastore that Veeam creates that presents VM’s directly from the compressed de-duped backup images. ( This is the Veeam Patent-Pending technology). This Virtual Lab is setup on an isolated network (customizable) and dedicated resource pool (customizable) on your existing ESX server farm. *NO* extra storage is required since it runs directly from the backup file.

A beneficial side-effect of this is universal item level recovery, for example you can pull an individual Exchange mailbox, SQL Server table, or any other “application” specific item that is not just a plain file ( Veeam already has file level recovery ) through the SureBackup Virtual Lab. You can run these VM’s directly from the compressed de-duplicated files, you aren’t required to extract or pull out the VMDK’s and therefore don’t require any additional storage space.

The SureBackup feature will release with the next version of Veeam Backup and Replication 5 ( sometime late summer). This is a simple upgrade, and you will be able to use this feature with your old Veeam Backup 4.0 backups and your new 5.0 backups.

It has been reported that at the Microsoft Virtual Desktop Infrastructure (VDI) webcast there will be a major announcement ( for the better) on how virtual desktops are currently licensed, e.g. VECD licensing. Also there will be some information on improvments to RDP to catch up to competing protocols ala PCoIP, Pano Console Direct, Citrix HDX etc it has been reported.

Sign-up to get the info live at http://desktopvirtualizationhour.com/, we will update the HiperLogic blog on the 18th with any breaking news from the webcast.

If you haven’t heard, the “legacy” backup method of using VMware VCB will be removed from the next release of vSphere, and the “modern” method will be to use the VMware Storage APIs for Data Protection (VADP). From the letter available here at VMware’s site:

The purpose of this letter is to inform you of our vSphere backup product strategy, ongoing enhancements, and end of availability plans for VMware Consolidated Backup.

VMware Backup Product Strategy
VMware released vStorage APIs for Data Protection (VADP) with the vSphere 4.0 release in May, 2009. VADP is the next generation of VMware’s backup framework. We have also been working with several backup partners to integrate VADP into their solutions to make backup of vSphere Virtual Machines fast, efficient and easy to deploy compared to VCB and other backup solutions. Several of our major backup partners have already released VADP integrated backup products and we expect most of the major backup partners to have VADP integrated backup software by the upcoming feature release of the vSphere platform in 2010.

Future Product Licensing
Given the strong interest and adoption of VADP by our backup eco-system and the benefits offered by VADP compared to VMware Consolidated Backup (VCB), we are announcing the End of Availability for VCB starting with next vSphere feature release in 2010. Starting with the next vSphere platform feature release, VCB will be removed from vSphere platform. VADP integrated backup products (including VMware Data Recovery) will be the recommended option for efficient backup and restoration of vSphere Virtual Machines. This will allow us to focus new value added feature development on VADP instead of two backup frameworks (VCB and VADP).”

This means if you plan on upgrading to vSphere and use VCB, you will want to have in your planning and budgeting a way to migrate to a backup vendor that supports VADP. Contact your preferred backup vendor today to see what their plans are for VADP support.

I personally recommend Veeam Backup and Replication ( HiperLogic is a Veeam Gold Partner, so maybe I am a bit biased) for the simple matter of fact that ESX 3.5 users can use it today with their 3.5 installation, and migrate seamlessly to current and future vSphere releases. There is some peace of mind that the mechanism you back up with today will let you restore tomorrow.

VMware has a backup appliance bundled with vSphere that uses VADP known as vDR (VMware Data Recovery), but as of the current vSphere release I typically don’t recommend it, see the post on this subject vDR vs Veeam Backup. It is quite possible that VMware’s vDR product will get some more development pumped into it once VCB is removed from vSphere this year, by then though Veeam SureBackup will be out.

Tags: , ,

There are variety of factors that influence the speed of cloning from a template, this post is to highlight some of the big wins if you are having issues.

Read the VMware kb article 1004002 and 1004028

First, you should start with this VMware kb article 1004002 which addresses some of the basics, if you have only one template that is creating a problem see kb article 1004028

Templates should be in their own datastore. – A Best Practice

By storing templates in their own datastore the  read intensive I/O ( reading template ) is separated from the write intensive I/O ( writing new VM from template ). With NFS in particular, ESX only uses one TCP stream per datastore, so by using different datastores ( a best practice) the time to create a VM from a template may be reduced.

Thick and eager zero – The slow boat to China

Don’t deploy your VM to be thick and eager zero, deploying thick and eager zero will be MUCH slower than thin or zero thick.

Simply not enough spindles. – The immutable laws of Physics

Creating a VM from template is intense, you are reading the template and writing a new VM both of which are single large block sequential I/O stream operations. If you simply don’t have enough spindles, the amount of I/O you can push will be limited. Going forward Solid State Drives (SSD’s) may eliminate the issue of “spindle count”.

Utilize Your Storage Array

Cloning at the storage array level can be near instantaneous, for example NetApp has their Rapid Cloning Utility RCU. If you don’t have array support, there are multiple other ways to tune your storage assuming you have sufficient spindles, like write caching, stripe size, RAID choice, and so on. This is vendor specific to some degree and beyond a 5 tips article. :-) If you are on ESX 3.5 ( not vSphere) see VMware kb article 1003469 for a I/O tunable that can help speed up large block disk I/O.

To configure syslog on ESXi to forward to a central logger, you can just use the VIC and go to Configuration->Advanced Setting->Syslog ->Remote.

To do this on a bunch of ESXi servers, you will want to do this programatically using either the vSphere vCLI (Perl) or PowerCLI (Power Shell). Get these tools free at http://www.vmware.com/go/sysadmintools.

In the vCLI :

vicfg-syslog.pl –p 514 –s < IP of your syslog server > –server < IP of your ESXi host>

Then enter root username, and root password of your ESXi host.

In Power Shell:

Set-VMHostSysLogServer -SysLogServerPort 514 –SysLogServer < IP of your syslog sever > -VMHost < IP of your ESXi host >

Update 6/23/2010 : See this website for caveat with syslog on ESXi http://www.virtuallyghetto.com/2010/06/esxi-syslog-caveat.html

Tags:

Interesting video showing how the massive online game EVE is using HPC Server 2008, part of the game is written in MPI according to the video:

Tags:

VMware has extended the deadline for VCP3 holders to qualify for VCP4 certification without having to attend an official vSphere training course, you now have until January 31st 2010.

The free 2nd chance re-take exam offer is also extended until the end of the month, note that January 22nd 2010 would be the last day for 1st attempt at VCP4 exam if the free re-take is required since there is a cooling off period before you can take the 2nd exam. See this VMware Blog for information on how to register for the free re-take.

Tags: ,

If you  have not yet updated to vSphere Update One, this post is for you to avoid the purple diagnostic screen of death with the error “COS Panic: Int3 @ mp_register_ioapic ” when upgrading from ESX 4.0 to 4.0 U1 after a reboot. ( Specifically if you are using HP’s insight management agents in the COS. )

 See KB Article 1016070 for the latest.

Basically SHUT DOWN all HP COS agents before upgrading. This is only for ESX and not ESXi since there is no COS in ESXi. You should also examine 3rd party agents and plugins for compatibility in the lab before upgrading in production. For example, EMC storage viewer is broken with vSphere Update One, slated to be fixed in the EMC 2.1 release.

Make sure to read all the release notes and manual before attempting the upgrade, and have backups of course:

The vSphere Update One release notes:
http://www.vmware.com/support/vsphere4/doc/vsp_vc40_u1_rel_notes.html

The vSphere Upgrade Checklist here:
http://www.vmware.com/files/pdf/vsphere-migration-prerequisites-checklist.pdf

The vSphere Upgrade guide here:
http://www.vmware.com/pdf/vsphere4/r40/vsp_40_upgrade_guide.pdf

As any new items pop up, we will update this post.

Tags:

The following information provides highlights of some of the enhancements available in this release of VMware ESX:

VMware View 4.0 support – This release adds support for VMware View 4.0, a solution built specifically for delivering desktops as a managed service from the protocol to the platform.

Windows 7 and Windows 2008 R2 support –This release adds support for 32-bit and 64-bit versions of Windows 7 as well as 64-bit Windows 2008 R2 as guest OS platforms. In addition, the vSphere Client is now supported and can be installed on a Windows 7 platform. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Enhanced Clustering Support for Microsoft Windows – Microsoft Cluster Server (MSCS) for Windows 2000 and 2003 and Windows Server 2008 Failover Clustering is now supported on an VMware High Availability (HA) and Dynamic Resource Scheduler (DRS) cluster in a limited configuration. HA and DRS functionality can be effectively disabled for individual MSCS virtual machines as opposed to disabling HA and DRS on the entire ESX/ESXi host. Refer to the Setup for Failover Clustering and Microsoft Cluster Service guide for additional configuration guidelines.

Enhanced VMware Paravirtualized SCSI Support – Support for boot disk devices attached to a Paravirtualized SCSI ( PVSCSI) adapter has been added for Windows 2003 and 2008 guest operating systems. Floppy disk images are also available containing the driver for use during the Windows installation by selecting F6 to install additional drivers during setup. Floppy images can be found in the /vmimages/floppies/ folder.

Improved vNetwork Distributed Switch Performance – Several performance and usability issues have been resolved resulting in the following:

•Improved performance when making configuration changes to a vNetwork Distributed Switch (vDS) instance when the ESX/ESXi host is under a heavy load
•Improved performance when adding or removing an ESX/ESXi host to or from a vDS instance

Increase in vCPU per Core Limit – The limit on vCPUs per core has been increased from 20 to 25. This change raises the supported limit only. It does not include any additional performance optimizations. Raising the limit allows users more flexibility to configure systems based on specific workloads and to get the most advantage from increasingly faster processors. The achievable number of vCPUs per core depends on the workload and specifics of the hardware. For more information see the Performance Best Practices for VMware vSphere 4.0 guide.

Enablement of Intel Xeon Processor 3400 Series – Support for the Xeon processor 3400 series has been added. For a complete list of supported third party hardware and devices, see the VMware Compatibility Guide.

Tags:

VMware has a great new performance whitepaper on thin vs thick disk performance: 

http://www.vmware.com/pdf/vsp_4_thinprov_perf.pdf 

The summary is there is no performance impact when using thin provisioned disks compared to thick disks with VMware.

NetApp has a great resource for Thin vs Thick Basics:

http://blogs.netapp.com/virtualstorageguy/2009/10/vce-101-thin-provisioning-part-1-the-basics.html )

and a followup article looking at the advantages of de-dup and thin provisioning at the array level: 

 http://blogs.netapp.com/virtualstorageguy/2009/10/vce-101-thin-provisioning-part-2-going-beyond.html )

During a recent customer VMware healthcheck, we noted a customer did perform the VMware NFS timeout tunings as recommended by NetApp TR-3749. However none of the guests had the correct guest level timeout tuning.

TR-3749 doesn’t explicitely spell out that you should make sure and update your guest level disk timeouts to match the NFS timeout tunings.

Also note that these disk timeout settings if done once in the guest, may be reset through a VMware Tools upgrade. (See http://communities.vmware.com/thread/212235) so watch out.

See NetApp article kb41511 “VMware ESX Guest OS I/O Timeout Settings for NetApp Storage Systems”  for more details.

Tags: ,

We frequently get questions from customers with NetApp if they should store the swap with the VM or on a separate datastore, and how big should they make their swap datastore.

NetApp recommends in NetApp TR-3749 to use a separate datastore, as to avoid storing “transient” data like swap when using array based snapshots or disk-to-disk replication with SnapMirror and/or SnapVault. (Also See TR-3428).

For sizing, you can thin provision and use autogrow on the NetApp side to avoid having to worry about an “exact” space figure as this space is dynamic in nature depending on the number of VM’s, the memory in those VM’s, and memory reservations, and is used only when there is memory pressure.

If you want an absolute worse case add up all the VM’s you would create with the memory you anticipate allocating to them, and use that result if you need an absolute worse case number.

Tags: ,

PowerShell is a fantastic way to automate just about everything in a Windows and VMware environment, enabling your IT department to do with more with less, which is the must-have skill of today.

Unix aficionados switching to PowerShell may at first struggle with how to recreate their favorite functionality from Unix land, like grep and awk. PowerShell can do all the powerful string manipulations that awk/sed/perl can do, just in a different manner.

For example, a typical Linux scenario using awk and grep might be:

grep “somestring” foo.txt | awk ‘{print $2}’

The powershell equivalent is below:

select-string -pattern “somestring” foo.txt | foreach { $_.ToString().split(” “)[2] }

The very excellent PowerShell Community Extensions also provide some nice items you may desire from Unix/Linux toolbox.

Tags: ,

Now that VMware Data Recovery 1.0 (vDR)  is released and  included with many VMware  bundles ( Other than VMware Essentials and VMware Standard Edition),  customers ask us questions on when to use vDR  1.0 vs. an enterprise solution like Veeam Backup and Replication 4.1. ( HiperLogic sells/supports both products.)

Our general opinion is that vDR 1.0 is not a production level backup tool yet , and everything vDR 1.0 does, Veeam Backup and Replication does ( e.g. data deduplication,  VSS support, full VM and file level recovery, etc). 

Note that vDR is a 1.0.x release. VMware rolls out great features on an aggressive schedule. However,  if 1.0, 1.01, 1.0.2, 1.2  bugs/issues with your backups gives you trouble sleeping at night, stick with Veeam Backup and Replication or similar enterprise level tool that is time tested and customer weathered.

The following table highlights some of the major differentiators that you will want to consider when trying to decide if vDR 1.0 offers sufficient protection for your environment.  

VMware vDR vs Veeam Backup and Replication
Feature VMware vDR Veeam Backup and Replication 4.1
Windows/Linux Image and File Level Recovery Yes Yes
De-duplication of Backups Yes Yes
Suppports more than 100 VMs per appliance No Yes
Full GUI and Command Line/PowerShell support No Yes
Email reporting of backup job status No Yes
Built-in Replication for Disaster Recovery No Yes

This is not an exhaustive list of course, this is just some of the “biggies”. The next release of Veeam Backup and Replication 5.0 will really set a new bar.

A study of companies that lost data in a disaster conducted by the U.S. Small Business Administration revealed 50% never reopened, and 90% went out of business within two years.  With a list price cost of Veeam Backup coming in at less than $1,000 per dual socket ESX server, Veeam Backup continues to be an excellent insurance policy against disaster for SMB and the Enterprise.

Updates ( Last Update 7/13/2010 for vSphere 4.1 )

vDR doesn’t support linked Virtual Center’s.

vDR appliance has a limit of 1TB per data store, max 2 data stores per appliance.

vDR 1.2 Now has File Level Restore (FLR) for use with Linux, though from the 1.2 release notes there are changes to the guests needed to work in many cases 6/11/2010

Tags: ,

« Older entries § Newer entries »

© 2006-2010 HiperLogic, LLC.  |  Serving the Ann Arbor, Southeast Michigan, and Ohio region  |  (888)-268-3930  |