Release: VMware VMmark 1.0

Posted by Staff   |   Monday, July 23rd, 2007   |  

After almost one year of beta testing VMware finally release first version of its benchmarking platform: VMmark.

Measuring performances in a virtualization platform is much more complex than doing the same on a traditional physical system because of many factors impacting results. One of them is workloads amount and arrangement, meaning how many virtual machines are running on the same host and which kind of applications they are serving.

For this reason VMware approach to benchmarking uses concept of tiles, groups of six virtual machines each running a typical business application, and overall score:

The unit of work for a benchmark of virtualized consolidation environments can be naturally defined as a collection of virtual machines executing a set of diverse workloads. The VMmark Benchmark refers to this unit of work as a tile. The total number of tiles that a physical system can accommodate gives a coarse-grain measure of that system’s consolidation capacity. This concept is similar to some server benchmarks, such as TPC-C, which scale the workload in a step-wise fashion to increase the system load.

Tiles are relatively heavyweight objects that cannot by themselves capture small variations in system performance. To address this, both the number of tiles and the performance of each individual workload determine the overall benchmark score.

In VMmark also metrics are collected in an uncommon way, considering client-server transactions more than raw numbers achieved in local system interaction between software and hardware:

These metrics are collected at frequent intervals during the course of the run. The standard VMmark workload is designed to run for 3 hours with workload metrics reported every 60 seconds. This means that rather than having a single number upon completion of a test run, the user will have a series of numbers for each of the workloads. However, each workload score is defined as a single number: the average of a consecutive subset of the series of datapoints for that workload.

The steady state for the benchmark is defined as the middle two hours of the three-hour run. The first and last half hours are the ramp-up and ramp-down times, respectively. The steady state is further divided into three 40-minute sections. For each of the 40-minute sections we compute the result for the tile and select the median score of the three as the score for the tile.

The resulting per-tile scores are then summed to create the final metric. Normalization allows the integration of the different component metrics into an overall score.

This approach implies benchmarkers have to setup a complex client-server infrastructure, mimicking a production environment.

As in the past VMware doesn’t allow publishing of VMmark-based analysis without a formal approval:

VMware encourages all VMmark benchmarkers to formally submit a full disclosure report of their VMmark results and supporting documentation to VMware for review and publication on the VMmark website. VMware has established the review process described below to insure that all run and reporting rules have been followed and that the disclosure includes the details needed to reproduce the result.

Any VMmark result that the benchmarker wishes to formally submit for publication on the VMmark website should provide the full disclosure report and the supporting documentation to VMware a minimum of ten (10) business days prior to the planned publication date.

the VMmark test was not run in full compliance with these rules, VMware will not accept that result for publication and the benchmarker must not use that result in any public disclosure.

On top of that VMware put severe restrictions on how VMmark can be used to compare other benchmarking approaches or other benchmarked platforms:

Comparisons of VMmark metrics and submetrics to any other benchmark metrics are not allowed. VMmark utilizes other benchmarking software as load generators and produces results which are not comparable to the original benchmarks’ metrics.

Competitive comparisons using VMmark results in academic or research papers are not allowed.

Download VMmark free of charge here.

It will be interesting to see how other vendors (XenSource, Virtual Iron and Microsoft mainly) will react to this methodoly which is not stardard and severely restricted.

VMmark is a remarkable work trying to address a growing need in the virtualization industry, but without a broad acceptance and against a solid technical opposition, results it will generate may be useless for customers.

This is the reason why VMware (along with SWsoft, IBM, Sun, and others) is also working with SPEC to develop a standard benchmarking framework, which will hopefully grant customers more unbiased results.

blog comments powered by Disqus Newest articles
Release: VMware vRealize Log Insight 4.5

June 13th, 2017

Log Insight is a log aggregation, management and analysis tool, that VMware first introduced in 2013 and considered a competitor of Splunk.
Yesterday VMware announced the release of version 4.5, available for…

Release: VMware vRealize Automation 7.3

June 6th, 2017

Today VMware announced the latest release of its cloud management platform vRealize Automation, former vCloud Automation Center.
VMware vRealize Automation 7.3 release notes can be found at this link.


Paper: Introducing the NSX-T Platform

February 9th, 2017

“We see greater potential strategic opportunity in NSX over the next decade than our franchise product vSphere has had for the past decade.”
said VMware’s CEO Pat Gelsinger talking about…

Paper: VMware vSphere Virtual Machine Encryption Performance

November 22nd, 2016

Encryption of virtual machines is something that has been requested for years by the security community. VMware continued to postpone its implementation due to the negative operational impact that many…

Quest Software leaves Dell

November 1st, 2016

In September 2012 Dell announced to have completed the acquisition of Quest Software, a Californian company with an history in systems management, security, business intelligence and, falling back in our…

Citrix announces Q3 2016 results

October 21st, 2016

Citrix announced its financial results for third quarter 2016.
The revenues for the second quarter were $841 million for an increase of 3% compared to Q3 2015.
Net income was $132…

Release: VMware vSphere 6.5 & Virtual SAN 6.5

October 19th, 2016

2016 edition of VMworld US has been quite turbulent, on the other hand during VMworld Europe, happening these days in Barcelona, the company announced a few more products for the…

Release: VMware vRealize Log Insight 4.0

October 18th, 2016

Log Insight is a log aggregation, management and analisys tool, that VMware first introduced in 2013 and now is usually compared with Splunk.
Yesterday VMware announced Log Insight’s new major…

Release: Windows Server 2016 with support for Window Server & Hyper-V containers

October 13th, 2016

Yesterday Microsoft announced the general availability of Windows Server 2016 which the company defines as a cloud-ready OS.
Beside fancy definitions, one of the most relevant perks of this release…

Release: Oracle VM 3.4.2

September 22nd, 2016

During Oracle OpenWorld 2016 the company released version 3.4.2 of its enterprise virtualization solution.
Oracle VM is available for both x86 and SPARC based processor architectures and uses the Xen hypervisor…

VMworld US 2016 Wrap-up

September 1st, 2016

Today was the last day of VMware’s flagship conference VMworld in Las Vegas, an highly controversial edition which left a good chunk of the audience disoriented if not properly disappointed….

Gartner releases its Magic Quadrant for Cloud Infrastructure as a Service for 2016

August 11th, 2016

Last week Gartner updated its Magic Quadrant for Cloud Infrastructure as a Service (IaaS) for the year 2016. The Magic Quadrant for the year 2015 was released in May last year…

Release: Ansible Tower 3 by Red Hat

August 2nd, 2016

Ansible is one of the four main players in the automation market, younger then the well known Chef and Puppet, has been launched in 2013 in Durham, N.C. and acquired…

IBM announces earnings for Q2 2016

July 19th, 2016

Yesterday IBM announced its results for Q2 2016.

If we compare with the same quarter in 2015 earnings per share, from continuing operations, decreased 22%. Net income, from continuing operations,…

Monthly Archive