There’s no doubt that the virtualization industry needs a standard benchmarking platform. The only two alternatives we have today are simply ignored (Intel vConsolidate) or are not recognized by all the vendors (VMware VMmark).
Now even the specialized press is questioning about the value of these platforms, we are talking specifically about Anandtech, suggesting that they may not use real-world workloads to test the hypervisors:
There are only two consolidation benchmarks out there: Intel’s vConsolidate and VMware’s VMmark. Both are cumbersome to set up and both are based on industry benchmarks (SPECJbb2005) that are only somewhat or even hardly representative of real-world applications. The result is that VMmark, despite the fact that it is a valuable benchmark, has turned into yet another OEM benchmark(et)ing tool. The only goal of the OEMs seems to be to produce scores as high as possible; that is understandable from their point of view, but not very useful for the IT professional. Without an analysis of where the extra performance comes from, the scores give a quick first impression but nothing more.
To prove its point Anandtech has developed its own benchmark, vApus Mark I (developed by the academic group Sizing Server Lab), that is useful to compare the of different CPUs in a virtual infrastructure running Windows guest operating systems. This is a fine first step as most of the virtual machines deployed in the world run Windows, but just in case some customers are not satisfied the group is already developing a new version that features Windows and Linux virtual machines.
The beauty of this work, assuming it has no flaws, is that it can be used with any hypervisor, including VMware ESX, Citrix XenServer and Microsoft Hyper-V.
Read more
