SPEC releases the first standard benchmark for hardware and OS virtualization – UPDATED

Posted by virtualization.info Staff   |   Thursday, July 15th, 2010   |  

Finally, the Standard Performance Evaluation Corporation (SPEC) released the first industry standard benchmark for hardware virtualization and OS virtualization: SPECvirt_sc2010
The standard body is working on this since November 2006.

The benchmark designed to simulate the activity of three typical workloads: a web server, a Java application server and a mail server.
To do so it leverages existing SPEC benchmarks, modified to measure performance in a virtualization environment: SPECweb2005, SPECjAppServer2004 and SPECmail2008.

SPECvirt_sc2010 adopts the same approach used by the VMware benchmark VMmark, the tiles, to measure scalability: the framework deploys additional tiles until overall throughput reaches a peak and all virtual machines continue to meet required quality of service (QoS) criteria.

The web server workload is made by three tiers: a file server, an infrastructure server and of course the web server. They are hosted in two virtual machines (file server and web server are together).
The application server workload is made by two tiers: a backed database server and of course the application server. They are hosted in two VMs.
The mail server workload is made by just a single tier: the IMAP mail server, which is hosted inside a single VM.
A sixth, additional workload called SPECpoll has been created and put into an idle virtual machines. It sends and acknowledges network pings against the idle server in 100% load phase to measure its responsiveness and to all VMs in the 0% load phase (active idle) during power-enabled runs.

SPECvirt_sc2010_Tile

The benchmark is also able to measure power consumption and power/performance relationship.

It’s not available for free: it costs $3,000 with discounts for qualified non-profit and educational institutions.

Customers may believe that the availability of this standard finally closes the endless discussions about the value of VMmark, originally introduced more than three years ago, the VMware’s restriction in using it to compare vSphere against other products, and the quality of independent alternatives like the Project Virtual Reality Check (VRC) or the Anandtech vApus Mark.

Unfortunately, it may not be the case yet. The SPEC virtualization subcommittee that developed it in fact includes the following members and contributors: AMD, Dell, Fujitsu Siemens Computers, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware.
For some reasons anyway, the press announcement issued by the SPEC only includes a subset of the names: AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys and VMware.

Microsoft is not there. And Citrix is not even a member of the subcommittee. It’s unclear if the two support this benchmark or not.
If they don’t, then that customers will get only some benefits: now will be able to use SPECvirt_sc2010 to compare multiple hypervisors, but until all players will validate the benchmark, the results will always be questionable.

Update: IBM is the first and only OEM that already published some SPECvirt_sc2010 scores, and they are quite interesting because the benchmark has been used to measure the performance of KVM.

The System x x3650 M3 achieved an overall performance score of 1169 @ 72 VMs served by Red Hat Enterprise Linux (RHEL) 5.5 and KVM.

The x3650 M3 was configured with the Intel Xeon Processor X5680 (3.33GHz with 256KB L2 cache per
core and 12MB L3 cache per processor—2 chips/12 cores/6 cores per chip), 144GB of memory and ninety-six 73GB disk drives.



blog comments powered by Disqus


virtualization.info Newest articles
Release: Oracle VM VirtualBox 4.3.26

March 25th, 2015

Oracle has released a new version of its virtualization platform VM Virtualbox. Version 4.3.26 is considered a maintenance release which can be installed on top of version 4.3. The update…

Paper: Virtual SAN 6.0 Performance: Scalability and Best Practices

March 25th, 2015

VMware has released a paper titled: "Virtual SAN 6.0 Performance: Scalability and Best Practices". The paper which contains 23 pages investigates how VMware Virtual SAN performs and scales for multiple…

Release: Parallels Virtuozzo 6.0

March 23rd, 2015

Parallels today announced a new version of its container and hardware virtualization platform Virtuozzo. Appearantly the product was renamed earlier to Parallels Cloud Server, but with this version Parallels decided…

Release: VMware Horizon 6.1

March 18th, 2015

VMware last week released version 6.1 of its Virtual Desktop Infrastructure (VDI) and Server Based Computing (SBC) platform Horizon. Version 6.1 is the follow up of version 6 which was…

Book: Microsoft System Center Deploying Hyper-V with Software-Defined Storage & Networking

March 18th, 2015

Microsoft has published a free ebook titled: “Microsoft System Center Deploying Hyper-V with Software-Defined Storage & Networking“. The book which contains 236 pages provides a step-by-step guide for a…

Microsoft hires former Forrester Analyst as Cloud Chief Strategist

March 18th, 2015

Today, March 17, Microsoft disclosed that the former Vice President & Principal Analyst at Forrester Research: James Staten, was elected as chief strategist for its Cloud & Enterprise group. Staten…

VMware announces support for CoreS on vSphere 5.5. and vCloud Air

March 11th, 2015

VMware has announced that it now supports CoreOS on both vSphere 5.5. and vCloud Air. CoreOS is a Linux distribution designed to provide minimal functionality required for deploying applications inside…

Paper: VMware vSphere 6 and Oracle Database Scalability Study

March 11th, 2015

VMware has released a paper titled:"VMware vSphere 6 and Oracle Database Scalability Study". The paper which contains 12 pages details the outcome of a series of tests to demonstrate the…

Book: Microsoft Azure Essentials – Azure Automation

March 11th, 2015

Microsoft has published a free ebook titled:”Microsoft Azure Essentials: Azure Automation“. The book which contains 113 pages introduces a fairly new feature of Microsoft Azure called Azure Automation. Using…

Release: Unidesk Corporation Unidesk 3.0

March 11th, 2015

Unidesk Corporation, provider of layering software for Virtual Desktop Infrastructure (VDI) environments yesterday released version 3.0 of Unidesk. The Unidesk layering solution provides a way to divide a disk image…

SimpliVity raises $175M in Series D Funding

March 11th, 2015

SimpliVity, is one of the competitors in the crowdy hyperconverged infrastructure and a data architecture market. Yesterdat the company announced that it has closed a $175 million Series D…

GigaOM shuts down

March 10th, 2015

Today it’s a sad day for the world of the information technology websites, a very well known name on the scene, GigaOM, just announced that it is shutting down…

Docker acquires SocketPlane

March 10th, 2015

Few days after announcing the availability of its orchestration tools in beta, Docker also disclosed the acquisition of SocketPlane, a software-defined networking (SDN) startup founded during last quarter of…

Release: 5nine Cloud Security 5.1

March 10th, 2015

5nine has released version 5.1 of its Cloud Security for Hyper-V product. Cloud Security for Hyper-V is a solution which helps to ensure security and compliance for Hyper-V environments. This…

 
Monthly Archive