Anandtech challenges VMware with its own independent multi-hypervisor benchmark tool

Posted by virtualization.info Staff   |   Monday, May 25th, 2009   |  

vmware logo

There’s no doubt that the virtualization industry needs a standard benchmarking platform. The only two alternatives we have today are simply ignored (Intel vConsolidate) or are not recognized by all the vendors (VMware VMmark).

Now even the specialized press is questioning about the value of these platforms, we are talking specifically about Anandtech, suggesting that they may not use real-world workloads to test the hypervisors:

There are only two consolidation benchmarks out there: Intel’s vConsolidate and VMware’s VMmark. Both are cumbersome to set up and both are based on industry benchmarks (SPECJbb2005) that are only somewhat or even hardly representative of real-world applications. The result is that VMmark, despite the fact that it is a valuable benchmark, has turned into yet another OEM benchmark(et)ing tool. The only goal of the OEMs seems to be to produce scores as high as possible; that is understandable from their point of view, but not very useful for the IT professional. Without an analysis of where the extra performance comes from, the scores give a quick first impression but nothing more.

To prove its point Anandtech has developed its own benchmark, vApus Mark I (developed by the academic group Sizing Server Lab), that is useful to compare the of different CPUs in a virtual infrastructure running Windows guest operating systems. This is a fine first step as most of the virtual machines deployed in the world run Windows, but just in case some customers are not satisfied the group is already developing a new version that features Windows and Linux virtual machines.
The beauty of this work, assuming it has no flaws, is that it can be used with any hypervisor, including VMware ESX, Citrix XenServer and Microsoft Hyper-V.

The benchmark measures the performance of four virtual machines (each equipped with 4 vCPUs and 4GB vRAM) with four enterprise grade applications:

  • 1 x OLAP database, based on SQL Server 2008 x64 running on Windows 2008 64-bit, which runs Nieuws.be (over 100GB data organized in hundreds of tables)
  • 2 x MCS eFMS portals running PHP, IIS on Windows 2003 R2, described here.
  • 1 x OLTP database, based on Oracle 10G Calling Circle benchmark of Dominic Giles.

Because of this selection the Sizing Server Lab firmly claims that its vApus Mark I is not made to replace VMmark as it mimics the average virtual data center usage, while their own platform only replicates resource intensive services.

The results are extremely interesting as they highlight a performance difference between different CPUs that is not matching at all the numbers obtained with VMmark:

vApusMarkI 
The rest of long article includes key information about impact on performance of CPU components like dual and quad core architectures, the cache, the memory bandwidth and the clock speed.

The conclusions are surprising: if you have a VMware ESX 3.5 Update 4 (ESX 4.0 will be used in future tests) then the Xeon Nehalem is without a doubt the fastest platform, but the latest quad-core Opteron is not far behind.

The entire Anandtech benchmark analysis is definitively worth a read while waiting the likely outraged reaction of VMware.



blog comments powered by Disqus


virtualization.info Newest articles
Release: VMware vRealize Log Insight 4.5

June 13th, 2017

Log Insight is a log aggregation, management and analysis tool, that VMware first introduced in 2013 and considered a competitor of Splunk.
Yesterday VMware announced the release of version 4.5, available for…

Release: VMware vRealize Automation 7.3

June 6th, 2017

Today VMware announced the latest release of its cloud management platform vRealize Automation, former vCloud Automation Center.
VMware vRealize Automation 7.3 release notes can be found at this link.

The…

Paper: Introducing the NSX-T Platform

February 9th, 2017

“We see greater potential strategic opportunity in NSX over the next decade than our franchise product vSphere has had for the past decade.”
said VMware’s CEO Pat Gelsinger talking about…

Paper: VMware vSphere Virtual Machine Encryption Performance

November 22nd, 2016

Encryption of virtual machines is something that has been requested for years by the security community. VMware continued to postpone its implementation due to the negative operational impact that many…

Quest Software leaves Dell

November 1st, 2016

In September 2012 Dell announced to have completed the acquisition of Quest Software, a Californian company with an history in systems management, security, business intelligence and, falling back in our…

Citrix announces Q3 2016 results

October 21st, 2016

Citrix announced its financial results for third quarter 2016.
The revenues for the second quarter were $841 million for an increase of 3% compared to Q3 2015.
Net income was $132…

Release: VMware vSphere 6.5 & Virtual SAN 6.5

October 19th, 2016

2016 edition of VMworld US has been quite turbulent, on the other hand during VMworld Europe, happening these days in Barcelona, the company announced a few more products for the…

Release: VMware vRealize Log Insight 4.0

October 18th, 2016

Log Insight is a log aggregation, management and analisys tool, that VMware first introduced in 2013 and now is usually compared with Splunk.
Yesterday VMware announced Log Insight’s new major…

Release: Windows Server 2016 with support for Window Server & Hyper-V containers

October 13th, 2016

Yesterday Microsoft announced the general availability of Windows Server 2016 which the company defines as a cloud-ready OS.
Beside fancy definitions, one of the most relevant perks of this release…

Release: Oracle VM 3.4.2

September 22nd, 2016

During Oracle OpenWorld 2016 the company released version 3.4.2 of its enterprise virtualization solution.
Oracle VM is available for both x86 and SPARC based processor architectures and uses the Xen hypervisor…

VMworld US 2016 Wrap-up

September 1st, 2016

Today was the last day of VMware’s flagship conference VMworld in Las Vegas, an highly controversial edition which left a good chunk of the audience disoriented if not properly disappointed….

Gartner releases its Magic Quadrant for Cloud Infrastructure as a Service for 2016

August 11th, 2016

Last week Gartner updated its Magic Quadrant for Cloud Infrastructure as a Service (IaaS) for the year 2016. The Magic Quadrant for the year 2015 was released in May last year…

Release: Ansible Tower 3 by Red Hat

August 2nd, 2016

Ansible is one of the four main players in the automation market, younger then the well known Chef and Puppet, has been launched in 2013 in Durham, N.C. and acquired…

IBM announces earnings for Q2 2016

July 19th, 2016

Yesterday IBM announced its results for Q2 2016.

If we compare with the same quarter in 2015 earnings per share, from continuing operations, decreased 22%. Net income, from continuing operations,…

 
Monthly Archive