Vendors’ marketing departments spend a lot of time promoting performance analysis when a new virtualization platform or a new server hardware hits the market, but are the customers interested?
Right now we have only two benchmarks methodologies to measure virtual infrastructures: the VMware VMmark (launched in July 2007) and the Intel vConsolidate (launched in December 2006).
There’s also an ongoing activity at the Standard Performance Evaluation Corp. (SPEC) to define a standard practice but the development is so slow that nobody really knows if hardware virtualization will still be around for its release.
Additionally, virtualization.info got a tip that Intel discontinued vConsolidate during the second half of 2009 (we are still waiting for an official answer on this).
This means that the VMware’s benchmark platform is the only option customers have to compare servers and hypervisors.
It could be fair enough if every vendor would recognize and support VMmark, turning it into a de facto industry standard. Unfortunately it’s not the case because the VMware EULAs prohibits to openly use the tool and publish independent comparisons. After three years (VMmark 1.0 beta was released in December 2006) VMware still requires to review and approve any VMmark-based analysis.
VMware is not the one to blame for this.
In any open and competitive market customers would complain as long as VMware doesn’t change its policy or, more likely, the VMware competitors wouldn’t work together to offer something better.
In three years the amount of customers that openly complained about the VMmark EULA is minimal.
In part it depends on the fact that VMware ESX has led the market for so long that just a few really needed something to compare hypervisors’ performance. In part it depends on the fact that performance measurement isn’t a key aspect of product evaluation for most companies.
The lack of alternatives from other vendors (namely Microsoft, Citrix, Oracle and Parallels) instead may depend on the fact that no vendor has an interest in highlighting its product’s deficiencies compared to the market leader.
It’s much more comfortable to blame VMware for its absurd EULA and use this as an excuse to not recognize any comparison unless it turns out to be positive.
But now we are in 2010. VMware’s competitors release hypervisors that are getting more and more mature, customers are seriously evaluating alternatives, sites with heterogeneous virtual infrastructures are no more an unlikely scenario.
Is it time for an open comparison or performance measurement became just a marketing exercise?