9.7. Optimizing Disk I/O Usage

When you determine that disk I/O is a problem, it can be helpful to determine which application is causing the I/O.

Figure 9-5 shows the steps we take to determine the cause of disk I/O usage.

To begin the investigation, jump to Section 9.7.1

9.7.1. Is the System Stressing a Particular Disk?

Run iostat in the extended statistic mode and look for partitions that have an average wait (await) greater than zero. await is the average number of milliseconds that requests are waiting to be filled. The higher this number, the more the disk is overloaded. You can confirm this overload by looking at the amount of read and write traffic on a disk and determining whether it is close to the maximum amount that the drive can handle.

If many files are accessed on a single drive, it may be possible to increase performance by spreading out these files to multiple disks. However, it is first necessary to determine what files are being accessed.

Proceed to Section 9.7.2.

9.7.2. Which Application Is Accessing the Disk?

As mentioned in the chapter on disk I/O, this is where it can be difficult to determine which process is causing a large amount of I/O, so we must try to work around the lack of tools to do this directly. By running top, you first look for processes that are nonidle. For each of these processes, proceed to Section 9.7.3.

9.7.3. Which Files Are Accessed by the Application?

First, use strace to trace all the system calls that an application is making that have to do with file I/O, using strace -e trace=file. We can then strace using summary information to see how long each call is taking. If certain read and write calls are taking a long time to complete, this process may be the cause of the I/O slowdown. By running strace in normal mode, it is possible to see which file descriptors it is reading and writing from. To map these file descriptors back to files on a file system, we can look in the proc file system. The files in /proc/<pid>/fd/ are symbolic links from the file descriptor number to the actual files. An ls -la of this directory shows which files this process is using. By knowing which files the process is accessing, it might be possible to reduce the amount of I/O the process is doing, spread it more evenly between multiple disks, or even move it to a faster disk.

After you determine which files the process is accessing, go to Section 9.9.