Linux 2.6 kernel-based (Ubuntu) virtual machines experience slow disk I/O performance

Below is an issue i resolved of slow disk IO with ubuntu lucid.

Symptoms

Linux 2.6 kernel-based virtual machines experience slow storage performance as compared to physical hosts.

Cause

As of the Linux 2.6 kernel, the default I/O Scheduler is Completely Fair Queuing (CFQ). The scheduler is an effective solution for nearly all workloads.

With the release of Linux 2.6, the kernel has these schedulers as configurable options:

• Completely Fair Queuing (cfq): CFQ is the default under many Linux distributions. CFQ places synchronous requests submitted by processes into a number of per-process queues and then allocates time slices for each of the queues to access the disk. The length of the time slice and the number of requests a queue is allowed to submit depends on the I/O priority of the given process. Asynchronous requests for all processes are batched together in fewer queues, one per priority.
  
• NOOP (noop): NOOP is the simplest I/O scheduler for the Linux kernel based upon the FIFO queue concept. The NOOP scheduler inserts all incoming I/O requests into a simple FIFO queue and implements request merging. The scheduler assumes I/O performance optimization will be handled at some other layer of the I/O hierarchy.
  
• Anticipatory (anticipatory): Anticipatory is an algorithm for scheduling hard disk I/O. It seeks to increase the efficiency of disk utilization by "anticipating" synchronous read operations.
  
• Deadline (deadline): The goal of the Deadline scheduler is to guarantee a start service time for a request. It does this by imposing a deadline on all I/O operations to prevent starvation of requests.

The default scheduler will affect all disk I/O for VMDK and RDM-based virtual storage solutions. In virtualized environments, it is often not beneficial to schedule I/O at both the host and guest layers. If multiple guests use storage on a filesystem or block device managed by the host operating system, the host may be able to schedule I/O more efficiently because it is aware of requests from all guests and knows the physical layout of storage, which may not map linearly to the guests' virtual storage.

Resolution

Testing has shown that NOOP or Deadline perform better for virtualized Linux guests. ESX uses an asynchronous intelligent I/O scheduler, and for this reason virtual guests should see improved performance by allowing ESX to handle I/O scheduling.

To implement this change, please refer to the documentation for your Linux distribution.

Note: All scheduler tuning should be tested under normal operating conditions as synthetic benchmarks typically do not accurately compare performance of systems using shared resources in virtual environments.

For example, this change can be implemented by:

• The scheduler can be set for each hard disk unit. To check which scheduler is being used for particular drive, run this command:
  
  cat /sys/block/disk/queue/scheduler
  
  For example, to check the current I/O scheduler for sda:
  
  # cat /sys/block/sda/queue/scheduler
  [noop] anticipatory deadline cfq
  
  In this example, the sda drive scheduler is set to NOOP.
  
• To change the scheduler on a running system, run this command:
  
  # echo scheduler > /sys/block/disk/queue/scheduler
  
  For example, to set the sda I/O scheduler to NOOP:
  
  # echo noop > /sys/block/sda/queue/scheduler
  
  Note: This command will not change the scheduler permanently. The scheduler will be reset to the default on reboot. To make the system use a specific scheduler by default, add an elevator parameter to the default kernel entry in the GRUB boot loader menu.lstfile.
  
  For example, to make NOOP the default scheduler for the system, the /boot/grub/menu.lst kernel entry would look like this:
  
  title CentOS (2.6.18-128.4.1.el5)
  root (hd0,0)
  kernel /vmlinuz-2.6.18-128.4.1.el5 ro root=/dev/VolGroup00/LogVol00 elevator=noop
  initrd /initrd-2.6.18-128.4.1.el5.img
  
  With the elevator parameter in place, the system will set the I/O scheduler to the one specified on every boot.

The noop scheduler is changed  by following commands.

Edit /etc/default/grub, such as gksudo gedit /etc/default/grub, here you need to add elevator=noop.

Change GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" to GRUB_CMDLINE_LINUX_DEFAULT="quiet splash elevator=noop".

Then run sudo update-grub2 and restart.