virtualization.info started this year questioning the industry interest in virtualization benchmarks.
In that article we reported that Intel discontinued its vConsolidate platform, giving the customers no other choice that using the VMware VMMark system and its non-competitive EULA.
Intel was kind enough to answer, confirming that vConsolidate was discontinued in early 2009, and providing some insights behind the choice, that we quote here integrally:
Early 2009, we stopped development and maintenance of the benchmark. It is however still used by several companies for their own internal testing and evaluation of server configurations. We no longer support external publication of vConsolidate benchmark results.
Prior to this, it was available to customers via direct contact with me and my team. I have a list (that I cannot share, sorry) of companies (OEM’s, ISV’s, Finance, Tech, Medical, Auto, Insurance, and others) who requested access over the last few years and used vConsolidate in their own test lab environments. That was after all a primary goal of the benchmark; internal evaluation of server systems for a virtualization environment. Its goal from the start was not to become an industry standard benchmark. But rather, it was always designed and maintained essentially as a test tool. Although, for a lack of any other solution, it crossed that boundry on some occasions (with approvals of course).
In terms of virtualization solutions, VMMark came about at roughly the same time period and had roughly the same test scenario’s involved as did vConsolidate. The most often discussed comparison is that VMMark focuses on VMWare environments, where vConsolidate allowed for multiple hypervisor configurations. One key difference that most people do not often call out is that VMWare made a benchmark for publicly evaluating server configurations, we made vConsolidate as a test tool for internal labs to evaluate server configurations. Customers needed something to test with in non-VMWare test environment, so vConsolidate was updated slightly, given a GUI, and offered as a tool they could use. But it was still primarily only for internal evaluation.
Why did we stop development, maintenance, and essentially offering vConsolidate out as a virtualization environment test solution? Well, the short answer is that SPEC has had their own version of a virtualization benchmark forthcoming that would supersede what vConsolidate offers. We fed a lot of what we learned from our efforts on vConsolidate to the SPEC committee… as members, we leverage our experience. Ultimatly, we decided that supporting an industry accepted/developed benchmark is better for customers.
As far as the question of ‘does anyone care about virtualization benchmarks in 2010’. I guarantee the answer is yes.