Performances are the greatest concerns CIO/CTO usually have approaching virtualization.
You surely would compare a virtual machine performance against a physical server, but you could also be in need of exploring how different virtualization technologies perform.
The first aspect you should test is I/O performances: physical raw partitions, proprietary filesystems, remoted SANs systems, local virtual IDE or SCSI disk subsystem. All of these configurations should be tested and compared with each other and against physical machines I/O performances.
Another second aspect you could test is network performances since virtual network adapters devices can handle traffic in different ways and be more or less efficient.
The best way to stress test a VM is to use standard tool for physical machines stress testing.
And just in case you are new to this practice below I compiled a list of great, ready to go, free tools:
Memory reliability
I/O performances
Network performances
- DeNamik LoadGen
- JMeter
- Microsoft Exchange Server 2003 Load Simulator
- Microsoft Exchange Server 2007 Load Generator (64bit version)
- Microsoft Exchange Server 2007 Stress and Performance Tool (64bit version)
- Microsoft SharePoint Server 2007 (and SharePoint Services 3.0) Test Data Load Tool
- Microsoft Web Application Stress Tool
- Microsoft Web Capacity Analysis Tool
- OpenSTA
- SLAMD Distributed Load Generation Engine
- Soap-Stone
I found Intel IOMeter and Microsoft Web Application Stress Tool both great for stress tests.
You could also take a look at a new interesting liveCD Linux distribution, StressLinux, providing a complete stressing test suite.
If you are going to test VMware ESX performances you should also absolutely check the VMware ESX Server Performance Troubleshooting lab manual released at VMworld 2006.