SPEC releases the first standard benchmark for hardware and OS virtualization – UPDATED

Posted by virtualization.info Staff   |   Thursday, July 15th, 2010   |  

Finally, the Standard Performance Evaluation Corporation (SPEC) released the first industry standard benchmark for hardware virtualization and OS virtualization: SPECvirt_sc2010
The standard body is working on this since November 2006.

The benchmark designed to simulate the activity of three typical workloads: a web server, a Java application server and a mail server.
To do so it leverages existing SPEC benchmarks, modified to measure performance in a virtualization environment: SPECweb2005, SPECjAppServer2004 and SPECmail2008.

SPECvirt_sc2010 adopts the same approach used by the VMware benchmark VMmark, the tiles, to measure scalability: the framework deploys additional tiles until overall throughput reaches a peak and all virtual machines continue to meet required quality of service (QoS) criteria.

The web server workload is made by three tiers: a file server, an infrastructure server and of course the web server. They are hosted in two virtual machines (file server and web server are together).
The application server workload is made by two tiers: a backed database server and of course the application server. They are hosted in two VMs.
The mail server workload is made by just a single tier: the IMAP mail server, which is hosted inside a single VM.
A sixth, additional workload called SPECpoll has been created and put into an idle virtual machines. It sends and acknowledges network pings against the idle server in 100% load phase to measure its responsiveness and to all VMs in the 0% load phase (active idle) during power-enabled runs.

SPECvirt_sc2010_Tile

The benchmark is also able to measure power consumption and power/performance relationship.

It’s not available for free: it costs $3,000 with discounts for qualified non-profit and educational institutions.

Customers may believe that the availability of this standard finally closes the endless discussions about the value of VMmark, originally introduced more than three years ago, the VMware’s restriction in using it to compare vSphere against other products, and the quality of independent alternatives like the Project Virtual Reality Check (VRC) or the Anandtech vApus Mark.

Unfortunately, it may not be the case yet. The SPEC virtualization subcommittee that developed it in fact includes the following members and contributors: AMD, Dell, Fujitsu Siemens Computers, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware.
For some reasons anyway, the press announcement issued by the SPEC only includes a subset of the names: AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys and VMware.

Microsoft is not there. And Citrix is not even a member of the subcommittee. It’s unclear if the two support this benchmark or not.
If they don’t, then that customers will get only some benefits: now will be able to use SPECvirt_sc2010 to compare multiple hypervisors, but until all players will validate the benchmark, the results will always be questionable.

Update: IBM is the first and only OEM that already published some SPECvirt_sc2010 scores, and they are quite interesting because the benchmark has been used to measure the performance of KVM.

The System x x3650 M3 achieved an overall performance score of 1169 @ 72 VMs served by Red Hat Enterprise Linux (RHEL) 5.5 and KVM.

The x3650 M3 was configured with the Intel Xeon Processor X5680 (3.33GHz with 256KB L2 cache per
core and 12MB L3 cache per processor—2 chips/12 cores/6 cores per chip), 144GB of memory and ninety-six 73GB disk drives.



blog comments powered by Disqus


virtualization.info Newest articles
Release: Veeam Management Pack for System Center Operations Manager 7.0

July 23rd, 2014

Veeam today released version 7.0 of the Management Pack for monitoring and management of VMware ESX and Microsoft Hyper-V using Microsoft System Center Operations Manager 2012.
New in this release…

OpenNebula 4.8 Beta introduces support for Azure and SoftLayer

July 22nd, 2014

OpenNebula has just released the Beta version of its OpenNebula 4.8, codename “Lemon Slice”, that finally introduces the long awaited support for public clouds providers others than Amazon AWS….

Video: VMware Horizon 6 and AirWatch

July 21st, 2014

BYOD (Bring Your Own Device) is one of the hot challenges that nowadays enterprises are facing, that’s why AirWatch acquisition, made by VMware earlier this year, is extremely significative together…

Paper: VMware Virtual SAN Design and Sizing Guide for Horizon View Virtual Desktop Infrastructures

July 17th, 2014

VMware has released a paper titled:"VMware Virtual SAN Design and Sizing Guide for Horizon View Virtual Desktop Infrastructures". The paper which contains 16 pages focuses on the definitions, sizing guidelines,…

Citrix releases Tech Preview of next version of XenServer

July 16th, 2014

Citrix has released a technical preview of XenServer codenamed Creedence the next major version of XenServer, which is now at version 6.2. When released this version will probably be version…

Release: VMUnify v3.0

July 16th, 2014

The Indian company VMUnify, originated from MindTree’s innovation program, announced this week the general availability of version 3.0 of its homonymous solution for managing Virtual Infrastructure Enterprises, Private or…

VMware announces general availability of vCloud Hybrid Service in Asia-Pacific region

July 15th, 2014

vCloud Hybrid Service is VMware’s owned and operated IaaS (Infrastructure as a Service) offering, released in September 2013 in the US only and extended to Europe (UK) in February 2014….

Paper: Virtualization Fabric Design Considerations Guide

July 10th, 2014

Microsoft has released a paper titled:"Virtualization Fabric Design Considerations Guide". The paper which contains 53 pages helps you to understand how to design a virtualization fabric that is able to…

Gartner releases its 2014 Magic Quadrant for x86 Server Virtualization Infrastructure

July 10th, 2014

Gartner has released its annual Magic Quadrant for x86 Server Virtualization Infrastructure, again positioning VMware and Microsoft in the leaders Quadrant. Leaders in this market have a clear strategy and…

Release: Red Hat Enterprise Linux OpenStack Platform 5

July 10th, 2014

Red Hat has released its OpenStack platform version 5. The Red Hat Enterprise Linux OpenStack Platform is able to create, deploy, and scale a public or private OpenStack cloud. The…

Paper: HP Reference Architecture for OpenStack on Ubuntu 14.04 LTS

July 3rd, 2014

HP has released a paper titled: “HP Reference Architecture for OpenStack on Ubuntu 14.04 LTS“. The paper which contains 15 pages provides guidance in the deployment of an OpenStack Icehouse…

Release: Oracle VM 3.3

July 3rd, 2014

Oracle has released version 3.3 of its enterprise virtualizatoin solution. The virtualization solution is available for both x86 and SPARC based processor architectures and uses the Xen hypervisor technology, supporting…

VMware releases public beta of Virtual Volumes

July 1st, 2014

Besides releasing the public beta 2 for vSphere 6.0 today, VMware also released a beta for Virtual Volumes (VVOLs).
Virtual Volumes allows a Virtual Machine and its disk to become…

Release: Microsoft Assessment and Planning Toolkit 9.1

July 1st, 2014

Microsoft has released version 9.1 of its capactiy planning tool the Assessment and Planning Toolkit (MAP). This version is the follow-up of version 9.0 which was released in December last…

 
Monthly Archive