SPEC releases the first standard benchmark for hardware and OS virtualization – UPDATED

Posted by virtualization.info Staff   |   Thursday, July 15th, 2010   |  

Finally, the Standard Performance Evaluation Corporation (SPEC) released the first industry standard benchmark for hardware virtualization and OS virtualization: SPECvirt_sc2010
The standard body is working on this since November 2006.

The benchmark designed to simulate the activity of three typical workloads: a web server, a Java application server and a mail server.
To do so it leverages existing SPEC benchmarks, modified to measure performance in a virtualization environment: SPECweb2005, SPECjAppServer2004 and SPECmail2008.

SPECvirt_sc2010 adopts the same approach used by the VMware benchmark VMmark, the tiles, to measure scalability: the framework deploys additional tiles until overall throughput reaches a peak and all virtual machines continue to meet required quality of service (QoS) criteria.

The web server workload is made by three tiers: a file server, an infrastructure server and of course the web server. They are hosted in two virtual machines (file server and web server are together).
The application server workload is made by two tiers: a backed database server and of course the application server. They are hosted in two VMs.
The mail server workload is made by just a single tier: the IMAP mail server, which is hosted inside a single VM.
A sixth, additional workload called SPECpoll has been created and put into an idle virtual machines. It sends and acknowledges network pings against the idle server in 100% load phase to measure its responsiveness and to all VMs in the 0% load phase (active idle) during power-enabled runs.

SPECvirt_sc2010_Tile

The benchmark is also able to measure power consumption and power/performance relationship.

It’s not available for free: it costs $3,000 with discounts for qualified non-profit and educational institutions.

Customers may believe that the availability of this standard finally closes the endless discussions about the value of VMmark, originally introduced more than three years ago, the VMware’s restriction in using it to compare vSphere against other products, and the quality of independent alternatives like the Project Virtual Reality Check (VRC) or the Anandtech vApus Mark.

Unfortunately, it may not be the case yet. The SPEC virtualization subcommittee that developed it in fact includes the following members and contributors: AMD, Dell, Fujitsu Siemens Computers, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware.
For some reasons anyway, the press announcement issued by the SPEC only includes a subset of the names: AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys and VMware.

Microsoft is not there. And Citrix is not even a member of the subcommittee. It’s unclear if the two support this benchmark or not.
If they don’t, then that customers will get only some benefits: now will be able to use SPECvirt_sc2010 to compare multiple hypervisors, but until all players will validate the benchmark, the results will always be questionable.

Update: IBM is the first and only OEM that already published some SPECvirt_sc2010 scores, and they are quite interesting because the benchmark has been used to measure the performance of KVM.

The System x x3650 M3 achieved an overall performance score of 1169 @ 72 VMs served by Red Hat Enterprise Linux (RHEL) 5.5 and KVM.

The x3650 M3 was configured with the Intel Xeon Processor X5680 (3.33GHz with 256KB L2 cache per
core and 12MB L3 cache per processor—2 chips/12 cores/6 cores per chip), 144GB of memory and ninety-six 73GB disk drives.



blog comments powered by Disqus


virtualization.info Newest articles
VMware results for Q1 2014

April 23rd, 2014

On April 22, VMware released the results about its growth for Q1 2014.
VMware, announced a total  revenue growth of $1.36 billion, for an increase of 14 percent compared to Q1…

Release: Splunk App for VMware 3.1

April 17th, 2014

On April 15 Splunk announced the availability of Splunk App for VMware 3.1, the system that is competing with VMware Log Insight solution to provide administrators with a real-time…

Release: Ubuntu 14.04 LTS

April 15th, 2014

Today Canonical announced the new Ubuntu Linux 14.04 LTS with a press release intriguingly focused on its role as an OpenStack platform.
This release, that will be available on…

VMware introduces vCloud Hybrid Service – Disaster Recovery

April 15th, 2014

VMware vCloud Hybrid Service (vCHS) is the VMware’s owned and operated public Infrastructure as a Service (IaaS) platform, launched in 2013 in the US and extended to Europe in February…

Paper: 3D Graphics for Virtual Desktops Smackdown

April 11th, 2014

PQR, a dutch technology company has released a paper titled:"3D Graphics for Virtual Desktops Smackdown". The paper which contains 139 pages is written by virtualization experts, Benny Tritsch, Ruben Spruijt,…

Microsoft releases preview of Microsoft Azure Automation

April 10th, 2014

Microsoft has announced the release of a preview of Azure Automation. Azure Automation provides an orchestration engine for use within Microsoft Azure. Azure Automation allows you to automate the creation,…

Release: Citrix XenClient 5.1

April 10th, 2014

Citrix has released version 5.1 of its client hypervisor XenClient. XenClient consists of two technologies, the XenClient, which is a type-1 client hypervisor running on selected hardware and the XenClient…

Release: Microsoft Virtual Machine Converter 2.0

April 10th, 2014

In October 2012 Microsoft released version 1.0 of its Virtual Machine Converter tool (MVMC) allowing the conversion of VMware based virtual machines (VM’s) to Hyper-V based VM’s and virtual hard…

Paper: What’s New in VMware vSphere 5.5 Networking

April 9th, 2014

VMware vSpere 5.5 was released in September 2013 and introduced a couple of improvements to the networking capabilities of the vSphere platform.
These enhancements could be resumed as follows:

More…

VMware announces Horizon 6, adding Server Based Computing as a solution

April 9th, 2014

VMware today announced the release of version 6 of its end user computing suite: Horizon.
Starting with this version VMware not only provides a Virtual Desktop Infrastructure (VDI) solution based…

VMware elects Paul Sagan in the board of directors

April 8th, 2014

Yesterday VMware announced that David Goulden, CEO of EMC Information Infrastructure and CFO of EMC, has left the board of directors.
Goulden will be replaced by Paul Sagan, former Akamai’s…

VMware announces Q1 vExperts 2014

April 2nd, 2014

VMware vExpert is the program, started in 2009, that “rewards” the individuals who has been recognized as active contributors of the community that rotates around the VMware ecosystem.
The program…

Red Hat releases beta of Enterprise Virtualization version 3.4

April 1st, 2014

Red Hat has released a beta for an upcoming release of Red Hat Enterprise Virtualization (RHEV) platform version 3.4. Red Hat Enterprise Virtualization (RHEV) is Red Hats virtualization platform based…

Release: VMware vCenter Log Insight 2.0-beta

March 27th, 2014

Log Insight is VMware’s product for log aggregation, management and analysis. Introduced in June 2013, Log Insight is kept updated with a fast pace in order to be competitive on…

 
Monthly Archive