SPEC releases the first standard benchmark for hardware and OS virtualization – UPDATED

Posted by virtualization.info Staff   |   Thursday, July 15th, 2010   |  

Finally, the Standard Performance Evaluation Corporation (SPEC) released the first industry standard benchmark for hardware virtualization and OS virtualization: SPECvirt_sc2010
The standard body is working on this since November 2006.

The benchmark designed to simulate the activity of three typical workloads: a web server, a Java application server and a mail server.
To do so it leverages existing SPEC benchmarks, modified to measure performance in a virtualization environment: SPECweb2005, SPECjAppServer2004 and SPECmail2008.

SPECvirt_sc2010 adopts the same approach used by the VMware benchmark VMmark, the tiles, to measure scalability: the framework deploys additional tiles until overall throughput reaches a peak and all virtual machines continue to meet required quality of service (QoS) criteria.

The web server workload is made by three tiers: a file server, an infrastructure server and of course the web server. They are hosted in two virtual machines (file server and web server are together).
The application server workload is made by two tiers: a backed database server and of course the application server. They are hosted in two VMs.
The mail server workload is made by just a single tier: the IMAP mail server, which is hosted inside a single VM.
A sixth, additional workload called SPECpoll has been created and put into an idle virtual machines. It sends and acknowledges network pings against the idle server in 100% load phase to measure its responsiveness and to all VMs in the 0% load phase (active idle) during power-enabled runs.

SPECvirt_sc2010_Tile

The benchmark is also able to measure power consumption and power/performance relationship.

It’s not available for free: it costs $3,000 with discounts for qualified non-profit and educational institutions.

Customers may believe that the availability of this standard finally closes the endless discussions about the value of VMmark, originally introduced more than three years ago, the VMware’s restriction in using it to compare vSphere against other products, and the quality of independent alternatives like the Project Virtual Reality Check (VRC) or the Anandtech vApus Mark.

Unfortunately, it may not be the case yet. The SPEC virtualization subcommittee that developed it in fact includes the following members and contributors: AMD, Dell, Fujitsu Siemens Computers, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware.
For some reasons anyway, the press announcement issued by the SPEC only includes a subset of the names: AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys and VMware.

Microsoft is not there. And Citrix is not even a member of the subcommittee. It’s unclear if the two support this benchmark or not.
If they don’t, then that customers will get only some benefits: now will be able to use SPECvirt_sc2010 to compare multiple hypervisors, but until all players will validate the benchmark, the results will always be questionable.

Update: IBM is the first and only OEM that already published some SPECvirt_sc2010 scores, and they are quite interesting because the benchmark has been used to measure the performance of KVM.

The System x x3650 M3 achieved an overall performance score of 1169 @ 72 VMs served by Red Hat Enterprise Linux (RHEL) 5.5 and KVM.

The x3650 M3 was configured with the Intel Xeon Processor X5680 (3.33GHz with 256KB L2 cache per
core and 12MB L3 cache per processor—2 chips/12 cores/6 cores per chip), 144GB of memory and ninety-six 73GB disk drives.



blog comments powered by Disqus


virtualization.info Newest articles
Paper: Cisco UCS C240-M3 Rack Server with NVIDIA GRID GPU cards on Citrix XenServer 6.2 and XenDesktop 7.5

October 21st, 2014

Cisco has released a paper titled: “Cisco UCS C240-M3 Rack Server with NVIDIA GRID GPU cards on Citrix XenServer 6.2 and XenDesktop 7.5“.
The paper which contains 38 pages will…

Microsoft announces updates to its public and private cloud portfolio

October 20th, 2014

Microsoft today announced several upcoming features to both its public Microsoft Azure services, as its private cloud solution based on Windows Server and System Center. CEO Satya Nadella stated that…

OpenStack releases the 10th version of its IaaS platform called Juno

October 20th, 2014

OpenStack, the open source cloud computing project has released its 10th version of its IaaS platform for public, private and hybrid clouds. This version has 342 new features and…

VMware decides to disable TPS in future ESXi releases by default

October 17th, 2014

In a knowledge base article titled: “Security considerations and disallowing inter-Virtual Machine Transparent Page Sharing (2080735)” published on October 16th, VMware states that it will disable the Transparant Page Sharing…

Paper: Citrix Virtual Desktop Handbook 7.x

October 16th, 2014

Based on the recent releases of Citrix XenApp and XenDesktop 7.6, Citrix has updated its design guidance called the "Citrix Virtual Desktop Handbook 7.x". The handbook which contains 202 pages…

VMware announces vRealize Log Insight 2.5

October 16th, 2014

VMware has announced the release of version 2.5 of its log aggregation, management and analysis product Log Insight. This version will be the follow up of version 2.0 which was…

Release: Oracle VM VirtualBox 4.3.18

October 16th, 2014

Oracle has released a new version of its virtualization platform VM Virtualbox. Version 4.3.18 is considered a maintenance release which can be installed on top of version 4.3. The update…

Paper: Performance and Scalability of Microsoft SQL Server on VMware vSphere 5.5

October 16th, 2014

VMware has released a paper titled: "Performance and Scalability of Microsoft SQL Server on VMware vSphere 5.5". The Paper which contains 33 pages demonstrates that large Microsoft SQL Server databases…

Paper: Microsoft Exchange Server Performance on VMware Virtual SAN

October 16th, 2014

VMware has released a paper titled: "Microsoft Exchange Server Performance on VMware Virtual SAN". The paper which contains 9 pages shows the results of performance tests of Microsoft Exchange Server…

Microsoft announces support for Docker container virtualization for next version of Windows Server

October 15th, 2014

Microsoft has announced that it will support for Docker in its next version of Windows Server. Docker which provides a so called container virtualization solution currently receives a lot of…

Paper: Achieving Over 1-Million IOPS from Hyper-V VMs in a Scale-Out File Server Cluster Using Windows Server 2012 R2

October 15th, 2014

Microsoft has released a paper titled:"Achieving Over 1-Million IOPS from Hyper-V VMs in a Scale-Out File Server Cluster Using Windows Server 2012 R2". The paper which contains 24 pages demonstrates…

VMworld Europe 2014 Wrap-Up

October 15th, 2014

VMworld Europe 2014 in Barcelona has seen Pat Gelsinger (VMware CEO), Bill Fathers (EVP and GM, Hybrid Cloud Services Business Unit) and Sanjay Poonen (EVP and GM, End-User Computing) in…

Release: Microsoft Virtual Machine Converter 3.0

October 14th, 2014

Microsoft has released version 3.0 of its converter tool from the VMware platform to Hyper-V and Windows Azure, the Microsoft Virtual Machine Converter (MVMC). Version 3.0 is the follow-up of…

Paper: VMware Mirage Large-Scale Reference Architecture

October 13th, 2014

VMware has released a paper titled: "VMware Mirage Large-Scale Reference Architecture". The paper, which contains 30 pages is a provides a reference architecture and real-world testing results for image management,…

 
Monthly Archive