SPEC releases the first standard benchmark for hardware and OS virtualization – UPDATED

Posted by virtualization.info Staff   |   Thursday, July 15th, 2010   |  

Finally, the Standard Performance Evaluation Corporation (SPEC) released the first industry standard benchmark for hardware virtualization and OS virtualization: SPECvirt_sc2010
The standard body is working on this since November 2006.

The benchmark designed to simulate the activity of three typical workloads: a web server, a Java application server and a mail server.
To do so it leverages existing SPEC benchmarks, modified to measure performance in a virtualization environment: SPECweb2005, SPECjAppServer2004 and SPECmail2008.

SPECvirt_sc2010 adopts the same approach used by the VMware benchmark VMmark, the tiles, to measure scalability: the framework deploys additional tiles until overall throughput reaches a peak and all virtual machines continue to meet required quality of service (QoS) criteria.

The web server workload is made by three tiers: a file server, an infrastructure server and of course the web server. They are hosted in two virtual machines (file server and web server are together).
The application server workload is made by two tiers: a backed database server and of course the application server. They are hosted in two VMs.
The mail server workload is made by just a single tier: the IMAP mail server, which is hosted inside a single VM.
A sixth, additional workload called SPECpoll has been created and put into an idle virtual machines. It sends and acknowledges network pings against the idle server in 100% load phase to measure its responsiveness and to all VMs in the 0% load phase (active idle) during power-enabled runs.

SPECvirt_sc2010_Tile

The benchmark is also able to measure power consumption and power/performance relationship.

It’s not available for free: it costs $3,000 with discounts for qualified non-profit and educational institutions.

Customers may believe that the availability of this standard finally closes the endless discussions about the value of VMmark, originally introduced more than three years ago, the VMware’s restriction in using it to compare vSphere against other products, and the quality of independent alternatives like the Project Virtual Reality Check (VRC) or the Anandtech vApus Mark.

Unfortunately, it may not be the case yet. The SPEC virtualization subcommittee that developed it in fact includes the following members and contributors: AMD, Dell, Fujitsu Siemens Computers, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware.
For some reasons anyway, the press announcement issued by the SPEC only includes a subset of the names: AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys and VMware.

Microsoft is not there. And Citrix is not even a member of the subcommittee. It’s unclear if the two support this benchmark or not.
If they don’t, then that customers will get only some benefits: now will be able to use SPECvirt_sc2010 to compare multiple hypervisors, but until all players will validate the benchmark, the results will always be questionable.

Update: IBM is the first and only OEM that already published some SPECvirt_sc2010 scores, and they are quite interesting because the benchmark has been used to measure the performance of KVM.

The System x x3650 M3 achieved an overall performance score of 1169 @ 72 VMs served by Red Hat Enterprise Linux (RHEL) 5.5 and KVM.

The x3650 M3 was configured with the Intel Xeon Processor X5680 (3.33GHz with 256KB L2 cache per
core and 12MB L3 cache per processor—2 chips/12 cores/6 cores per chip), 144GB of memory and ninety-six 73GB disk drives.



blog comments powered by Disqus


virtualization.info Newest articles
Citrix CEO Mark Templeton plans to retire

July 30th, 2015

On July 28 Citrix disclosed that Mark Templeton, the company CEO, has plans to retire after over 20 years of service, an unexpected news that drew attention on the next…

Citrix announces Q2 2015 results

July 30th, 2015

Citrix announced its financial results for second quarter 2015.
The revenues for the second quarter were $797 million for an increase of 2% compared to Q2 2014.
Net income was $103…

VMware announces results for Q2 2015

July 24th, 2015

VMware released the results about its results for Q2 2015.
VMware, announced a GAAP total revenue growth of $1.52 billion, for an increase of 4 percent compared to Q2 2014. License revenues…

Gartner releases its 2015 Magic Quadrant for x86 Server Virtualization Infrastructure

July 21st, 2015

Gartner released its Magic Quadrant for x86 Server Virtualization Infrastructure for year 2015, once more positioning VMware and Microsoft in the leaders Quadrant.
No surprises in the leaders quadrant, as…

Release: Parallels Mac Management 4.0 for Microsoft SCCM

July 21st, 2015

Parallels announced the availability of Parallels Mac Management 4.0 for Microsoft System Center Configuration Manager (SCCM), extending its range of capabilities to Mac computers.
This new release increases the flexibility…

Avast Software Acquires Remotium

July 10th, 2015

On July 8, Avast Software, a Czech private limited company provider of antivirus and security software products announced to have acquired Silicon Valley based Remotium, a company founded in…

Release: Oracle VM VirtualBox 5.0 Candidate 1 (RC1)

July 8th, 2015

Oracle has announced to have released version 5.0 Candidate 1 (RC1) for its virtualization platform VM VirtualBox. The release brings enhancement and bug fixes to the previous versions, improving stability…

IBM announces Docker Based Container Services

July 1st, 2015

Containers continue to gain traction, IBM announced the release of enterprise class containers based on Docker and built on its Platform as a Service, Bluemix for hybrid environments. It will…

Release: Xen Project Hypervisor 4.5.1

July 1st, 2015

The Xen Project, the community which develops the Xen hypervisor under the GNU General Public License (GPLv2) announced the availability of a new maintenance release, version 4.5.1 of the Xen…

Red Hat Summit 2015 Wrap-up

June 29th, 2015

Last week, Red Hat held its premiere event in Boston, Massachusetts from June 23 to 26, the Red Hat Summit,  where several announcements have been made.
The summit offered over…

Midokura expands OpenStack Team with Takashi Yamamoto

June 29th, 2015

Network virtualization is one of the hot topics within the OpenStack community. Last week japanese startup Midokura focused on network virtualization announced the assignment of Takashi Yamamoto as OpenStack…

Red Hat announces Q1 2015 earnings

June 24th, 2015

On June 18 Red Hat announced its financial results for its fiscal year 2016 and for the first quarter of 2015.
Total revenue for the first quarter ended May 31,…

Rancher Labs raises $10M in Series A funding

June 11th, 2015

Rancher Labs is a startup providing management on top of the Docker containers, today the company has announced that it has raised $10 million in a series A funding…

EMC acquires Virtustream

May 26th, 2015

EMC has acquired Virtustream, a Infrastructure as a Service (IaaS) provider originally funded among the others from SAP for 1.2 billion US dollars. Virtustream also provides a Cloud Management Platform…

 
Monthly Archive