HiperLogic

Virtualization, High Performance Computing, Healthcare IT, Enterprise Computing

VMware

You are currently browsing the archive for the VMware category.

We had a customer using the free Microsoft SQL Server database with vCenter 5.0 that had surpassed the 4GB limit, at which point vCenter started crashing.

To fix the issue, stop the vCenter service, and connect to the vCenter database with SQL Server express. Make a backup of the database before you start.

Next issue the following commands:

truncate table VPX_HIST_STAT1;
truncate table VPX_SAMPLE_TIME1;

truncate table VPX_HIST_STAT2;
truncate table VPX_SAMPLE_TIME2;

truncate table VPX_HIST_STAT3;
truncate table VPX_SAMPLE_TIME3;

truncate table VPX_HIST_STAT4;
truncate table VPX_SAMPLE_TIME4;

Open the vCenter database dbo.VPX_PARAMETER. Adjust the following parameters in the table:

event.maxAge   30
event.maxAgeEnabled  true
task.maxAge 30
task.maxAgeEnabled true

After making these changes go to the programmability folder in Sql Server Management Studio, click stored procedures, execute dbo.cleanup_event_tasks_proc. This might take a while.

After that completes, right click on the databases and go to shrink database to get it under 4GB. You can restart vCenter now.

Tags:

Recently a customer was receiving some unusual errors with their VMware vCenter Appliance (VCVA) including:

  1. “An unknown connection error occurred. (The request failed because of a connection failure. (Unable to connect to the remote server))”.
    Trying to connect over vSphere Web Client presented us with a different but more detailed error
  2. “The authentication server returned an unexpected error: ns0:RequestFailed: STS token validators are not initialized. The error may be caused by a malfunctioning identity source.”

If you see this, Login to the VCVA as root, and issue the command df –h and check if the database partition /dev/sdb3 is 100% full.

	vcenter01:~ # df -h
	Filesystem      Size  Used Avail Use% Mounted on
	*cut*
	/dev/sdb3        20G   19G     0 100% /storage/db

If you find that this partition is out of space, you can make a new larger partition and migrate the existing data over.

The basic process is to attach a new virtual disk of greater size, partition it with fdisk, format as ext3, mount it in a temporary location, and transfer the existing data from /dev/sdb3 over to the new disk using tar or other tool.

Make sure that ALL processes are quiesced by doing fuser /storage/db before the data copy, and stopping any processes that are actively accessing /storage/db/

After the migration is complete, change /etc/fstab so that /storage/db is pointing to your new partition ( like /dev/sdc1) and reboot, and you are back in business.

Tags:

vSphere 5 is Out!

vSphere 5 is out! Rodney Mach has a presentation on What’s New available Here [pdf]

Tags:

Veeam announced today that their #1 VMware Backup for VMware is coming to Hyper-V in Veeam Backup and Replication version 6.

While not all the Veeam Backup functionality present in the VMware version will be present for Hyper-V, Veeam is bringing these important backup technologies to the Hyper-V platform:

– Both Backup and Replication in One Product.
- File Level Recovery (FLR). The ability to pull individual files from an image based backup to avoid the need for agents.
– A Hyper-V implementation of Change Block Tracking (CBT) to allow Veeam to only backup only the blocks that have changed in the VM. Veeam had CBT even before VMware had an official API for it, so I assume they are leveraging that investment to bring that to HYPER-V.
- Compression and De-Duplication.
- Support for the Free Hyper-V (System Center Virtual Machine Manager (SCVMM) is not required.)

Veeam Backup & Replication v6 is slated to be released in Q4 2011. Note that the v6 release will have a lot of new features for VMware customers as well, Veeam is supporting multiple hypervisors like many virtualization ISV’s, and I anticipate they will continue to innovate on both hypervisor platforms.

I was asked today how to perform an application consistent backup of MySQL in a Linux Virtual Machine on VMware for a customer evaluating Veeam Backup.

On Windows, Veeam can do application consistent backups through the Veeam Microsoft Volume Shadow Copy Services (VSS) integration for VSS aware applications. Unfortunately Linux does not have a system like Microsoft Volume Shadow Copy Services (VSS) that allows applications to be made application consistent by backup applications like VMware vDR or Veeam Backup.

A simple workaround is to leverage VMware tools ability to call custom freeze/thaw scripts during a quiesced snapshot to enable customers to do the required application quiescence.

The example below is for MySQL, for other applications you need to know the application specific command to quiesce the application and modify the scripts below appropriately.

Step 1. Make sure VMware Tools are installed in the Linux VM
VMware Tools MUST be installed and should be up to date in your Linux Virtual Machine.
Step 2. Determine how to quiesce the application
To quiesce MySQL we will do a mysqldump to create a transactionally consistent backup of mysql without downtime.

Note the disadvantage of this method is that it takes more space since it creates a copy of the database. Alternatively, If you can stop/start mysql during an off hour backup that is a simple way to quiesce ( no transactions can occur if the database is stopped ) as well. You would stop mysql in the pre-freeze, and start mysql in the post-thaw in that case.

Step 3. Create /usr/sbin/pre-freeze-script
Below are the commands to type at the Linux command shell to create the pre-freeze script which will create a transactionally consistent backup of MYSQL and store the resulting statements in /var/mysqlbackup.todaysdate.sql

The resulting .sql file produced by mysqldump contains a set of SQL INSERT statements that can be used to reload the dumped tables at a later time

cat <<EOF > /usr/sbin/pre-freeze-script
mysqldump --single-transaction --all-databases > /var/mysqlbackup.\`date +"%m%d%y"\`.sql
EOF

Check the above command worked properly by examining the created script.

cat /usr/sbin/pre-freeze-script

The result of the cat command above should look like below

mysqldump --single-transaction --all-databases > /var/mysqlbackup.`date +"%m%d%y"`.sql

Make the script executable

chmod 755 /usr/sbin/pre-freeze-script

Step 4. Create the /usr/sbin/post-thaw-script
In this case we don’t have to call anything in the post-thaw so create an empty script :

  touch /usr/sbin/post-thaw-script
  chmod 755 /usr/sbin/post-thaw-script

Step 5. Enable VMware Tools Quiescence
Now check “Use VMware Tools Quiescence” in Veeam Backup, and Veeam will ask VMware tools to quiesce the VM, and VMware Tools will call /usr/sbin/pre-freeze-script and /usr/sbin/post-thaw-script to create the application consistent snapshot.

After the Veeam backup, login to the VM and validate the backup file with the date of the backup exists:

root@vCentos ~# ls -la /var/mysqlbackup.032411.sql
-rw-r--r-- 1 root root 422120 Mar 24 11:59 /var/mysqlbackup.032411.sql
root@vCentos ~#

You can customize the above procedure for any Linux application that provides a method to quiesce itself.

Tags: , , , ,

VMware announced today a new product called vCenter Operations. The one takeaway about this new solution is that it enables the VMware admin to quickly find out what is abnormal in the environment through self-learning technology.

For example, an application owner will come to the VMware team and say “something is not normal with my application, it is slower than normal, what is going on?”.

The VMware admin now has to determine what “normal” is for the application is, determine if the problem is the application or the infrastructure, and package all of this information up to report back to the application owner.

With vCenter Operations a skilled VMware admin can quickly understand what is abnormal to narrow the scope of the issue, quickly identify what the problem and remediate the issue.

vCenter Operations has a very different UI than you may be accustomed to. In the screenshot below, vCenter Operations is reporting on three metrics: Workload, Health, and Capacity:

  1. workload score (0-100) – The amount of resources the objects wants to use compared to the resources the object has.
  2. health score (0-100). How normal is the object behaving based on past historical behavior of the object.
  3. capacity score (0-100). How much capacity do you have on a object, less than 30 days capacity left would be red.

Note I use the word “object”, and object can be many things like a virtual machine etc.

vCenter Operations

Another detail view:

vCenter Operations Detail

vCenter Operations Detail

vCenter Operations is packaged as a virtual appliance and comes in three editions, see below for some the basic differences. For the standard edition pricing starts at $50 per VM (Less than 500 VMs). With standard edition you don’t get access to 3rd party integration, customized dashboards, or AD integration.

vCenter Operations Edition Comparison

Note vCenter Operations still requires the VMware admin to interpret the abnormal metrics and create a plan of action, it does not offer a prescriptive recommendation (yet).

Tags: ,

There was a recent discussion online on the merits of using VMware High Availability (HA) and Fault Tolerance (FT) to protect an application that had HA and FT baked in. I believe this discussion will start popping up more and more in the near future, many new “cloud scale” applications are coded in such a way to be highly available, scalable, and fault tolerant which alleviates the need for features like Hypervisor level HA and FT.

For example, Google uses a software stack that has allowed them to withstand 1,600 server failures in a 1,800-unit cluster with uninterrupted service. This same distributed model that assumes failure of the underlying resource is how cloud applications are commonly developed today. VMware with their investment in Spring Source has recognized the importance of supporting the next generation of applications that will not be depend on keeping a single OS instance up and running at all costs. The future is going to revolve around applications built and designed to run on unreliable, cheap, expendable resources.

Tags:

I recommend the following reading list for Java on VMware vSphere:

Enterprise Java Applications on VMware – Best Practices Guide [PDF]

Java in Virtual Machines on VMware [PDF].

Check out this blogger on Java and VMware as well who also wrote a whitepaper on the topic.

Tags: ,

I was recently asked what was required to enable vSphere to take advantage of Intels hardware memory management unit (MMU) virtualization technology, called Extended Page Tables™ (EPT). If you aren’t aware EPT improve performance for MMU heavy workloads.

While the setting varies from vendor to vendor, all you do is enable Intel Virtualization in the ESX BIOS, and vSphere will do what it thinks is best with regards to hardware vs software MMU with no intervention required.

If you want to dive deeper, See page 21 of The vSphere Performance Best Practices Guide. under Configuring ESX for Hardware-Assisted Virtualization.

Especially note that vSphere will use the hardware MMU sometimes, and software MMU sometimes, depending on a mix of factors. See:

http://communities.vmware.com/docs/DOC-9882

for the default settings matrix when the VM is set to Automatic for CPU/MMU Virtualization (the default) . See the best practices guide mentioned above for how to change from Auto if you want to override the VMware recommendation like in the screenshot below:

Change MMU Setting in vSphere
The only tradeoff for EPT is you give up the memory savings of VMware TPS (Transparent Page Sharing). This VMware KB article sums it up very nicely:

http://kb.vmware.com/kb/1021095

In a nutshell, you give up TPS for EPT because for EPT to be really effective large memory pages are used, but using large memory pages really reduces the odds ( i.e. makes it impossible) to find two pages of identical memory to be shared. Under heavy memory pressure, vSphere will break down the large pages into smaller pages in order for TPS to kick in before it has to resort to swapping.

Incidentally Large Memory Pages is one reason Microsoft decided against implementing TPS in Hyper-V. . Microsoft has enabled large pages by default in 2008/2008 R2/Windows 7. While it may be in the future TPS will have diminished importance, it is a nice feature to get ( for free ) today on vSphere.

Tags: , ,

Many legacy programs still use USB license dongles to lock an application to a specific host. This kind of locking makes it difficult to virtualize the server hosting the locked application.

With vSphere 4.1, it is now fully supported to pass through USB to the ESX host, enabling you to plug the license dongle into your ESX host, and attached that to a virtual machine. The USB pass through even works when you vMotion the VM to another ESX host. The vSphere guide has a list of supported USB license key style dongles that have been validated.

However, there is one large caveat with this solution, and that is that if you need to take the ESX host down that is hosting the USB key, for example to do routine maintenance, or during an unplanned HA event, the VM will no longer be able to access the USB key.

This often means customer prefer to stick with the pre-vSphere 4.1 “tried and true” method of using a USB over IP adapter. This method while costing a couple hundred bucks has the advantage the VM still has access to the USB device at all times, assuming the network stays up.

USB over IP

USB over IP

Recently a customer needed to do some maintenance on the filer backing their centralized VMware swap datastore without taking downtime, this post will help other folks who face a similar issue.

As a refresher, the VMware .vswp file is used by ESX/ESXi to allow for memory overcommit, each running virtual machine (VM) has a .vswp file whose size is the virtual machine memory MINUS the memory reservation for the VM. The .vswp file is typically deleted when the VM is powered off, and re-created when the VM is powered on. The .vswp files can be configured to be in a centralized datastore, or to be stored alongside the associated VM (the default).

With that background, the issue for this customer was that with a centralized swap datastore, it wasn’t clear to them how to take the filer backing the swap datastore down without incurring downtime of a virtual machine to re-create the .vswp in the new location via a power VM power off/power on operation.

Luckily, the vswp file is also re-created when you do a VMotion to another host, or a Storage vMotion to another LUN. The simple solution was after changing the swap data store location from the current centralized location, to VMotion all VMs off of host A to other hosts, then back. Repeate for each host in the cluster. A quick PowerCLI script took care of the whole process.

When you are done, all .vswp files for running VMs will have been re-created at the new location. You can validate no VMs are using the old swap datastore, then remove it.

I had a customer that needed to uninstall a broken vCenter plugin, uninstalling the offending application was not unregistering the plugin and it showed up with errors.

The easy way to do this is to login to vCenter with the following URL:

  https://<vCenter>/mob/?moid=ExtensionManager

You will see a list of extensions below, click on the image to expand it:

Now type the name of the extension between the double quotes, and click invoke method to unregister like the screenshot below.

You may need to re-login to the VIC client to see the update.

vSphere 4.1 is now released! A slew of new features. Join the webinar on Friday at 3:00 EST that goes into a deep dive on the new features, sign-up at A list of documents related to this release:

VMware vSphere 4.1 Presentation – Whats New Technical

VMware vSphere 4.1 Brochure – EN

VMware vSphere4.1 DataSheet – Standard Edition EN

VMware vSphere 4.1 Whats New Flyer – Enterprise EN

VMware vSphere 4.1 Whats New Flyer – SMB – Now with VMotion!!!

VMware vSphere 4.1 DataSheet – Essentials Editions EN

Download links vSphere 4.1:
vCenter: http://bit.ly/9M6Rcz
ESX: http://bit.ly/9ypsFj
ESXi: http://bit.ly/arqXl

Tags:

NetApp has released a fantastic PowerShell kit to automate operations with their filers.

To get started, download the DataOntap.zip and install.ps1 from the NetApp NOW site, and place them in c:\temp\ on a Windows box with PowerShell ( I am using 2008 R2 with PowerShell Version 2 )

Start PowerShell, and type the following:

set-executionpolicy remotesigned
cd c:\temp\
 ./install.ps1

after that, start PowerShell again and type

     Get-Module -listavailable

to make sure you see DataONTAP installed.

Now you can type

Import-Module DataONTAP

To access the DataONTAP cmdlets in your session. You will need to do this each time you start PowerShell, or you can add the above command to your PowerShell profile to automatically load them.

To see help and examples, you can use the following command:

Show-NaHelp

For detailed help on a cmdlet, use get-help as you normally would with PowerShell:

Get-Help Connect-NaController –full

There are MANY great cmdlets available to automate common NetApp administration tasks.

A great example is a script mixing both NetApp and VMware PowerShell automation by Jase McCarty here which automatically provisions and mounts storage on all ESX hosts.

I modified Jase’s script slightly to work with PowerShell 2.0 and to use root access instead of AD credentials. This script must be ran from VMware PowerCLI to work:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# Add the  DATA ONTAP Module
Import-module DataONTAP
# Set my variables. Change for your site
##########################
$vCenter = "192.168.15.70"
$Filer = "192.168.15.3"
$aggr = "aggrx"
$newvol = "volx"
$narootpasswd = "netapprootpasshere"
#########################

#Connect to NetApp as root
$password = ConvertTo-SecureString $narootpasswd -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList "root",$password

Connect-NaController $Filer -Credential $cred

# Create a new 6GB volume
New-NaVol $newvol $aggr 6g

# Set some options for the new volume
Set-NaVolOption $newvol no_atime_update yes
Set-NaVolOption $newvol fractional_reserve 0

# Set the SnapShot Reserve to 0
Set-NaSnapshotreserve $newvol 0
Set-NaSnapshotschedule $newvol -Weeks 0 -Days 0 -Hours 0

# Add an NFS export
Add-NaNfsExport /vol/$newvol -Persistent -ReadWrite all-hosts -NoSuid -SecurityFlavors sys,krb5

# Get all the vSphere Hosts and add the NFS export. This requires VMware PowerCLI

# Connect to vCenter
Connect-VIServer $vCenter

$Hosts = Get-VMHost
ForEach ($h in $Hosts)
{
New-Datastore -Nfs -VMHost $h.Name -NAME $newvol -Path /vol/$newvol -NfsHost $Filer;
}

While vSphere 4.1 is not out yet, there are numerous leaks and announcements on the forums and other blogs on what to expect in 4.1

A partial list of features expected include:

1. ESX and ESXi direct support for Microsoft Active Directory through Likewise technology
2. Memory Compression for even higher VM density per host.
3. Scripted installations for ESXi making large rapid deployments supported and quick.
4. Storage I/O Control – Think of this is shares and limits for storage I/O to make sure important VMs get priority.
5. DRS VM Affinity – Restrict VM placement to particular hosts inside of an ESX cluster.
6. New HA dashboard for monitoring your ESX HA cluster.
7. Up to 8 simultaneous VMotions.
8. Microsoft Volume Shadow Service (VSS) in Windows Server 2008 and 2008 R2 guest operating systems for VADP
9. QoS engine that can prioritize between virtual machines, vMotion, Fault Tolerance (FT) and IP storage traffic.
10. Support for USB pass-through

Should be a very nice release!

The new Veeam Sure Backup Technology that checks that your VM will be able to recover from a backup is shown in this video. VERY cool, good example of how you can backup a system that is corrupted and have a false sense of security that it can be recovered without the use of SureBackup:

Storage I/O Control is a new feature coming in vSphere 4.1 that will allow the admin to control and throttle VM throughput to the datastores when a VMs latency crosses a certain threshold.

This is an entertaining video made by someone else that demonstrates this ( pre-release ) technology:

This is a great site for keeping track of VMware build information:
VMware Release Build Information

Tags:

[vSphere 4.0 Update 2 is out which includes updates for ESX/ESXi hosts (build 261974) as well as the vCenter Server (build 258672).

This update primarily has more hardware support and guest OS support, and the ability to see more performance statistics for NFS datastores, along with the usual bug fixes.

To see the new NFS stats use (r)esxtop and hit "u" to see the NFS datastore stats like below:

esxtop

esxtop

NOTE: If you are using VMware View, do NOT upgrade at this time. See this article for the latest info on this issue: 1022830

UPDATE June 14th: If you are using EMC CLARiiON, make sure and read This EMC Post Before upgrading.

Update July 9th: Another nice feature in this update is how “Delete All” snapshots is handled, reducing the amount of space required and lowering the chance you fill up the datastore. See Yellow Bricks for more details

HiperLogic offers hardware solutions from HP ( and other vendors ), and for Virtualization we are a big fan of the HP DL380 G6/G7. We have noted during our VMware health check service that many DL380 installs that were done by our competitors were not ordered and/or configured for maximum performance. This post is to assist customers in getting it right.

Memory
Not having an optimal memory configuration can drastically reduce performance. HP has engineered the DL380 G6 to support large amounts of RAM and still keep the memory speed at 1333MHz (With 95W CPUs ). The memory configuration on new Nehalem based systems is no longer a simple matter of just picking a capacity and running with it like in the old days. If you want to configure your own memory see www.hp.com/go/ddr3memory-configurator

Disk

Next is disk performance. You should purchase a Battery Backed Write Cache Module with any DL380 G6 for vSphere ESXi or you will have performance problems, see this HP advisory. Even if you are running ESX today, ESXi may in the very near future be the only path forward, so better to spend a few dollars to future proof your investment.

BIOS

Having the wrong BIOS settings can really make a huge difference in performance. For example you may consider setting:

Turbo Boost On
HP Power Profile to Maximum Performance
HP Power Regulator to Static High Performance
Thermal Configuration Increased Cooling

Local Disk

HiperLogic also likes to have some fast local disk, the DL380G6 has plenty of internal storage for putting lots of fast disk. Having fast local disk is great even if you use a SAN as you can always storage vmotion to a local VMFS volume when testing or debugging VM disk I/O or VM disk performance issues, and for temporarily working around SAN issues etc. Disk is cheap, and it is nice to have the flexibility.

Finally always check the VMware HCL, for the DL380 G6 that is here

There is also a HP VMware HCL and vSphere 5 for HP Servers

Tags:

« Older entries

© 2006-2010 HiperLogic, LLC.  |  Serving the Ann Arbor, Southeast Michigan, and Ohio region  |  (888)-268-3930  |