SPEC releases the first standard benchmark for hardware and OS virtualization – UPDATED

Posted by virtualization.info Staff   |   Thursday, July 15th, 2010   |  

Finally, the Standard Performance Evaluation Corporation (SPEC) released the first industry standard benchmark for hardware virtualization and OS virtualization: SPECvirt_sc2010
The standard body is working on this since November 2006.

The benchmark designed to simulate the activity of three typical workloads: a web server, a Java application server and a mail server.
To do so it leverages existing SPEC benchmarks, modified to measure performance in a virtualization environment: SPECweb2005, SPECjAppServer2004 and SPECmail2008.

SPECvirt_sc2010 adopts the same approach used by the VMware benchmark VMmark, the tiles, to measure scalability: the framework deploys additional tiles until overall throughput reaches a peak and all virtual machines continue to meet required quality of service (QoS) criteria.

The web server workload is made by three tiers: a file server, an infrastructure server and of course the web server. They are hosted in two virtual machines (file server and web server are together).
The application server workload is made by two tiers: a backed database server and of course the application server. They are hosted in two VMs.
The mail server workload is made by just a single tier: the IMAP mail server, which is hosted inside a single VM.
A sixth, additional workload called SPECpoll has been created and put into an idle virtual machines. It sends and acknowledges network pings against the idle server in 100% load phase to measure its responsiveness and to all VMs in the 0% load phase (active idle) during power-enabled runs.

SPECvirt_sc2010_Tile

The benchmark is also able to measure power consumption and power/performance relationship.

It’s not available for free: it costs $3,000 with discounts for qualified non-profit and educational institutions.

Customers may believe that the availability of this standard finally closes the endless discussions about the value of VMmark, originally introduced more than three years ago, the VMware’s restriction in using it to compare vSphere against other products, and the quality of independent alternatives like the Project Virtual Reality Check (VRC) or the Anandtech vApus Mark.

Unfortunately, it may not be the case yet. The SPEC virtualization subcommittee that developed it in fact includes the following members and contributors: AMD, Dell, Fujitsu Siemens Computers, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware.
For some reasons anyway, the press announcement issued by the SPEC only includes a subset of the names: AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys and VMware.

Microsoft is not there. And Citrix is not even a member of the subcommittee. It’s unclear if the two support this benchmark or not.
If they don’t, then that customers will get only some benefits: now will be able to use SPECvirt_sc2010 to compare multiple hypervisors, but until all players will validate the benchmark, the results will always be questionable.

Update: IBM is the first and only OEM that already published some SPECvirt_sc2010 scores, and they are quite interesting because the benchmark has been used to measure the performance of KVM.

The System x x3650 M3 achieved an overall performance score of 1169 @ 72 VMs served by Red Hat Enterprise Linux (RHEL) 5.5 and KVM.

The x3650 M3 was configured with the Intel Xeon Processor X5680 (3.33GHz with 256KB L2 cache per
core and 12MB L3 cache per processor—2 chips/12 cores/6 cores per chip), 144GB of memory and ninety-six 73GB disk drives.



blog comments powered by Disqus


virtualization.info Newest articles
VMware acquires CloudVolumes

August 20th, 2014

Application containerization is one of the hot topics of 2014, the idea of abstracting applications from the underlying OSes is intriguing and suggests a more simple and agile way to…

Release: Parallels Desktop 10 for Mac

August 20th, 2014

Today Parallels announced the availability of version 10 of its desktop virtualization platform for Mac: Parallels Desktop.
The new release obviously includes the support for OS X 10.10 Yosemite, better…

Paper: Intel DPDK with VMware vSphere

August 14th, 2014

The Intel Data Plane Development Kit (DPDK) is an open source software library for Linux User Space applications that enables higher levels of packet processing throughput, that gives application developers…

Release: Teradici PCoIP Workstation Access Software

August 13th, 2014

Teradici is the developer of the PC-over-IP (PCoIP) remote desktop protocol, which is leveraged in several VDI solutions and provided with Teradici hardware solutions which are OEM’ed by several vendors….

Release: Veeam Task Manager for Hyper-V

August 13th, 2014

Veeam released Veeam Task Manager for Hyper-V, a free tool for monitoring and management of Microsoft Hyper-V.
You all know Windows Task Manager, the system monitor and startup – task…

Platform9 exits from stealth mode

August 12th, 2014

Platform9, founded in 2013 from former VMware Engineers Sirish Raghuram (former Staff Engineer and now Platform9’s CEO), Madhura Maskasky (also former Staff Engineer and now Head of Product), Roopak…

Tech: Microsoft Cloud Ecosystem Poster

August 11th, 2014

This month Microsoft made available one of its famous posters that visually represents its entire Cloud Ecosystem and called indeed Cloud Ecosystem Poster: Microsoft Azure, Windows Server 2012 R2, System…

Release: Login Virtual Session Indexer 4.1

August 6th, 2014

Login VSI has released version 4.1 of its vendor independent Login Virtual Session Indexer (VSI) benchmarking utility. Version 4.1 is the follow up of version 4.0 which was released in…

Video: VMware Horizon Mirage Bootcamp series

July 31st, 2014

Horizon Mirage is VMware’s centralized image management for Windows desktops with backup and OS migration capabilities, the product uses a layering technology to segment OS, applications and user personalizations into…

Microsoft announces Q2 2014 earnings

July 24th, 2014

On July 22 Microsoft announced its financial results for the second quarter of 2014.
Total revenue for the quarter ended June 30 was $23.38 billion. Looking at the financials details we…

Citrix announces Q2 2014 earnings

July 24th, 2014

On July 23, Citrix announced its financial results for second quarter of fiscal year 2014, ended June 30, 2014.
Citrix announced a total revenue of $782 million, for an increase of 7%…

Paper: Reference Architecture for Horizon with View and Virtual SAN

July 23rd, 2014

VMware has released a paper titled:"Reference Architecture for Horizon with View and Virtual SAN". The paper which contains 50 pages contains a reference architecture for VMware Horizon with View virtual…

Xen Project releases Mirage OS 2.0

July 23rd, 2014

The Xen Project, the community which develops the Xen hypervisor under the GNU Public License (GPLv) and which is now part of the Linux foundation has released version 2.0 of…

VMware results for Q2 2014

July 23rd, 2014

On July 22, VMware released the results about its growth for Q2 2014.
VMware, announced a total  revenue growth of $1.46 billion, for an increase of 17 percent compared to Q2 2013.
Operating…

 
Monthly Archive