In the last few days virtualization.info recommended a couple of new/refreshed, very interesting papers from VMware: The Design and Evaluation of a Practical System for Fault-Tolerant Virtual Machines and Understanding Memory Resource Management in VMware ESX 4.1.
There’s at least a third one that is worth a read: Enhanced VMware ESX 4.1 CPU Scheduler.
The 22-pages paper starts with a brief introduction on the role of the CPU scheduler and then goes on with an in-depth analysis of the design changes applied to ESX 4.1:
ESX 4.1 introduces wide-VM NUMA support, which improves memory locality for memory-intensive workloads. Based on testing with micro benchmarks, the performance benefit can be up to 11–17 percent.
In ESX 4, many improvements have been introduced in the CPU scheduler. This includes further relaxed co-scheduling, lower lock contention, and multicore-aware load balancing. Co-scheduling overhead has been further reduced by the accurate measurement of the co-scheduling skew and by allowing more scheduling choices. Lower lock contention is achieved by replacing the scheduler cell lock with finer-grained locks. By eliminating the scheduler cell, a virtual machine can get higher aggregated cache capacity and memory bandwidth. Lastly, multicore-aware load balancing achieves high CPU utilization while minimizing the cost of migrations.
It also includes an entire benchmark section with the performance evaluation of the updated design, comparing ESX 3.5 with ESX 4.