If the problem is at CPU, mpstat (multiple processor statistics) gives a cleaner report. Especially mpstat reports separately information about each CPU separately so you can see if a single CPU is overloaded while others are free. The syntax for mpstat is:
mpstat <interval> <count>
Interval is the time in seconds between printing out a line of statistics. Count is the number of lines of output you want.
The report generated by the mpstat command has the following format:
- CPU: Processor number, starts with 0. The keyword all indicates that statistics are calculated as averages among all processors.
- %user: Percentage of CPU utilization that occurred while executing at the user level (application).
- %nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority.
- %sys: Percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing interrupts or softirqs.
- %iowait: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
- %irq: Percentage of time spent by the CPU or CPUs to service interrupts.
- %soft: Percentage of time spent by the CPU or CPUs to service softirqs. A softirq (software interrupt) is one of up to 32 enumerated software interrupts which can run on multiple CPUs at once.
- %steal: Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
- %idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
- intr/s: Total number of interrupts received per second by the CPU or CPUs.
Linux 2.6.9-220.127.116.11.2.ELsmp (soaserver) 10/14/2012
04:03:03 PM CPU %user %nice %system %iowait %irq %soft %idle intr/s
04:03:05 PM all 0.25 0.00 0.25 0.00 0.25 0.00 99.25 1129.65
04:03:07 PM all 0.25 0.00 0.00 0.00 0.00 0.00 99.75 1114.50
04:03:09 PM all 3.75 0.00 1.25 0.25 0.00 0.00 94.75 1124.00
Average: all 1.42 0.00 0.50 0.08 0.08 0.00 97.92 1122.70
On SMP machines a processor that does not have any activity at all is a disabled (offline) processor.
Below are some general tips, which you can use while interpreting the output -
If %user is very high then your application is consuming the CPUs and it is being overburdened.
If %sys is high then your server is burnened by the system (kernel) calls.
If %iowait constantly a non-zero number, then you may have some disk I/O contention. It is recommended to check the “Time spent waiting for IO (wa)” of vmstat to see whether there is any waiting on disk storage subsystem. You should also consult the iostat output.