SPEC releases the first standard benchmark for hardware and OS virtualization – UPDATED

Posted by virtualization.info Staff   |   Thursday, July 15th, 2010   |  

Finally, the Standard Performance Evaluation Corporation (SPEC) released the first industry standard benchmark for hardware virtualization and OS virtualization: SPECvirt_sc2010
The standard body is working on this since November 2006.

The benchmark designed to simulate the activity of three typical workloads: a web server, a Java application server and a mail server.
To do so it leverages existing SPEC benchmarks, modified to measure performance in a virtualization environment: SPECweb2005, SPECjAppServer2004 and SPECmail2008.

SPECvirt_sc2010 adopts the same approach used by the VMware benchmark VMmark, the tiles, to measure scalability: the framework deploys additional tiles until overall throughput reaches a peak and all virtual machines continue to meet required quality of service (QoS) criteria.

The web server workload is made by three tiers: a file server, an infrastructure server and of course the web server. They are hosted in two virtual machines (file server and web server are together).
The application server workload is made by two tiers: a backed database server and of course the application server. They are hosted in two VMs.
The mail server workload is made by just a single tier: the IMAP mail server, which is hosted inside a single VM.
A sixth, additional workload called SPECpoll has been created and put into an idle virtual machines. It sends and acknowledges network pings against the idle server in 100% load phase to measure its responsiveness and to all VMs in the 0% load phase (active idle) during power-enabled runs.

SPECvirt_sc2010_Tile

The benchmark is also able to measure power consumption and power/performance relationship.

It’s not available for free: it costs $3,000 with discounts for qualified non-profit and educational institutions.

Customers may believe that the availability of this standard finally closes the endless discussions about the value of VMmark, originally introduced more than three years ago, the VMware’s restriction in using it to compare vSphere against other products, and the quality of independent alternatives like the Project Virtual Reality Check (VRC) or the Anandtech vApus Mark.

Unfortunately, it may not be the case yet. The SPEC virtualization subcommittee that developed it in fact includes the following members and contributors: AMD, Dell, Fujitsu Siemens Computers, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware.
For some reasons anyway, the press announcement issued by the SPEC only includes a subset of the names: AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys and VMware.

Microsoft is not there. And Citrix is not even a member of the subcommittee. It’s unclear if the two support this benchmark or not.
If they don’t, then that customers will get only some benefits: now will be able to use SPECvirt_sc2010 to compare multiple hypervisors, but until all players will validate the benchmark, the results will always be questionable.

Update: IBM is the first and only OEM that already published some SPECvirt_sc2010 scores, and they are quite interesting because the benchmark has been used to measure the performance of KVM.

The System x x3650 M3 achieved an overall performance score of 1169 @ 72 VMs served by Red Hat Enterprise Linux (RHEL) 5.5 and KVM.

The x3650 M3 was configured with the Intel Xeon Processor X5680 (3.33GHz with 256KB L2 cache per
core and 12MB L3 cache per processor—2 chips/12 cores/6 cores per chip), 144GB of memory and ninety-six 73GB disk drives.



blog comments powered by Disqus


virtualization.info Newest articles
Docker releases Docker Security Scanning

May 12th, 2016

Containers’ security is one of the emerging topics in those companies moving this technology into production. A few small players emerged to compete exclusively in this portion of the…

RightScale announces Docker Container Management

May 3rd, 2016

RightScale is a Santa Barbara, CA based company, provider of a Software as a Service (SaaS) management solution that so far only supported standard Infrastructure as a Service (IaaS) cloud…

Release: WinDocks 1.0

April 5th, 2016

Yesterday, Bellevue (WA) based company WinDocks, released version 1.0 of its homonymous Docker engine for Windows.
The company, founded by a small group of former Microsoft’s employees, rides Docker’s…

LANDesk Acquires AppSense

March 17th, 2016

LANDesk Software, founded in 1985 and headquartered in Salt Lake City, Utah , provides systems management, security management, service management, asset management and process management solutions with a strong focus…

Release: Red Hat Enterprise Virtualization 3.6

March 15th, 2016

Last week open source giant Red Hat announced the availability of version 3.6 of its KVM-based virtualization platform Red Hat Enterprise Virtualization (RHEV).
While this new release provides the expected…

Docker acquires Conductant

March 4th, 2016

Yesterday Docker announced to have acquired a semi-stealth startup called Conductant, focused on workloads orchestration.

Both Conductant’s founders, Bill Farner and David Chung, have significant enterprise experience coming from…

Cisco acquires CliQr

March 1st, 2016

Today Cisco announced the intent to acquire CliQr Technologies Inc., a privately held company based in San Jose, CA.
CliQr is one of the most promising startups in the Cloud…

Release: VMware vCloud Suite 7 and vRealize Suite 7

February 11th, 2016

Yesterday VMware announced version 7 of both its vCloud and vRealize suites, confirming its efforts to be relevant in the CMPs (Cloud Management Platforms) space.
vRealize Suite 7 is made…

Platform9 adds Multi-Region & Multi-Hypervisor management into Managed OpenStack

February 11th, 2016

Platform9 solutions leverages a mix of SaaS and on premises Virtual Appliance to provide into supported environments capabilities typical of Cloud Management Platforms (CMPs) such self-service provisioning, monitoring, configuration…

VMware announces vRealize Log Insight 3.3

February 10th, 2016

Log Insight is a log aggregation, management and analisys tool, that VMware first introduced in 2013 and now is fiercely competing against Splunk.
Today the company announced Log Insight 3.3…

Microsoft releases Azure Stack TP1

February 1st, 2016

During Ignite 2015, back in September, Microsoft announced Microsoft Azure Stack as the building block for its hybrid cloud strategy.
What Microsoft is saying with this move is that, despite…

Release: Ansible 2.0

January 14th, 2016

Ansible is an IT automation tool especially popular within the developers’ community, thanks to its simplicity and agentless nature.
Three months away from its acquisition by Red Hat the company announced…

Nutanix goes Public

December 23rd, 2015

Nutanix is a provider of converged infrastructure, basically a physical server containing both compute and storage driven by an Installed Hypervisor of choice, this server, called a node can…

Release: VMware vRealize Automation 7 and VMware vRealize Business Standard 7

December 23rd, 2015

Last week VMware finally completed the first major update of VMware vRealize Suite, its well known cloud management solution, releasing in GA both vRealize Automation 7 and vRealize Business Standard…

 
Monthly Archive