sar is a great general performance monitoring tool. sar can output data for almost everything Linux does. The sar command is delivered in the sysstat rpm.
sar can display performance data for CPU, run queue, disk I/O, paging (swap), memory, CPU interrupts, networking, and more. The most important sar feature is the capability to create a data file. Every Linux system should collect sar data from cron jobs. The sar data file provides a system administrator with historical performance information. This feature is very important, and it separates sar from the other performance tools. If a nightly batch job runs twice as long as normal, you won’t find out until the next morning (unless you get paged). You need the ability to look at performance data from 12 hours ago. The sar data collector provides this ability. Many reporting syntaxes exist, but let’s look at data collection first.
sar Data Collector
sar data collection is done with a binary executable and two scripts in /usr/lib/sa. The sar data collector is a binary executable located at /usr/lib/sa/sadc. The job of sadc is to write to the data collection file /var/log/sa/. Several options can be supplied to sadc. A common syntax is:
/usr/lib/sa/sadc interval iterations file name
interval is the number of seconds between sampling. iterations is the number of samples to take. file name specifies the output file. A simple sadc syntax is /usr/lib/ sa/sadc 360 5 /tmp/sadc.out. This command takes five samples at five-minute intervals and stores them in /tmp/sadc.out. We should collect the samples on a regular basis, so we need a script to be run by cron. We should put the samples in a place that makes sense, like we did with the top script from the previous section. Fortunately, the sysstat rpm provides the /usr/lib/sa/sa1 script to do all this.
The sa1(8) man page is much longer than the sa1 script itself. /usr/lib/sa/sa1 is a very simple script that runs sadc with the syntax sadc -F -L 1 1 /var/log/sa/sa## where ## is the day of the month. Older versions of sa1 use the output from date +.%Y_%m_%d as the file suffix. The -F option causes sadc to force creation of the output file if necessary. The -L locks the output file before writing to it to prevent corrupting the file when two sadc processes are running at the same time. Older versions of sadc didn’t have the -L option, so the sa1 script performed manual locking. The only options for the sa1 script are the interval between samples and the number of iterations to sample. A cron file (/etc/cron.d/sysstat) is supplied with sysstat. It differs between sysstat versions. The following are the entries for the version 5.0.5 of sysstat:
# cat /etc/cron.d/sysstat# run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib/sa/sa1 1 1 # generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib/sa/sa2 -A
CPU Statistics
The sar -u output shows CPU information. The -u option is the default for sar. The output shows CPU utilization as a percentage.
| Field | Description | CPU | CPU number |
|---|---|
| %user | Time spent running processes in user mode |
| %nice | Time spent running niced processes |
| %system | Time spent running processes in kernel mode (system) |
| %iowait | Time during which the processor was waiting for I/O to complete while no process was executing on the CPU |
| %idle | Time during which no process was executing on the CPU |
This should look familiar. It is the same CPU information as in top reports. The following shows the output format:
[root@fisher dave]# sar 5 10Linux 2.4.21-27.EL (fisher) 04/30/2005 02:03:20 PM CPU %user %nice %system %iowait %idle 02:03:25 PM all 36.80 0.00 4.20 0.00 59.00 02:03:30 PM all 37.80 0.00 5.20 2.20 54.80 02:03:35 PM all 55.40 0.00 4.40 3.00 37.20 02:03:40 PM all 53.60 0.00 6.20 0.00 40.20 02:03:45 PM all 37.20 0.00 6.60 1.00 55.20 02:03:50 PM all 36.60 0.00 4.40 2.20 56.80 02:03:55 PM all 51.00 0.00 4.20 1.00 43.80 02:04:00 PM all 55.60 0.00 4.20 0.00 40.20 02:04:05 PM all 40.60 0.00 8.00 1.20 50.20 02:04:10 PM all 36.60 0.00 4.20 2.20 57.00 Average: all 44.12 0.00 5.16 1.28 49.44
The 5 10 causes sar to take 10 samples at 5-second intervals. The first column of any sar report is a timestamp.
We could have looked at the file created with sadc by using the -f option. This sar syntax shows the output from sar -f /var/log/sa/sa21:
[dave@fisher dave]$ sar -f /var/log/sa/sa21head -n 20Linux 2.4.21-20.EL (fisher) 04/21/2005 12:00:00 AM CPU %user %nice %system %iowait %idle 12:10:00 AM all 0.23 0.00 0.23 0.02 99.52 12:20:00 AM all 0.22 0.00 0.20 0.01 99.57 12:30:01 AM all 0.21 0.00 0.19 0.01 99.59 12:40:00 AM all 0.23 0.00 0.22 0.02 99.54 12:50:01 AM all 0.19 0.00 0.28 0.01 99.52 01:00:00 AM all 0.22 0.00 0.18 0.01 99.59 01:10:00 AM all 0.40 0.00 0.25 0.02 99.34 01:20:00 AM all 0.20 0.00 0.25 0.01 99.53 01:30:00 AM all 0.20 0.00 0.23 0.01 99.56 01:40:00 AM all 0.22 0.00 0.21 0.02 99.56 01:50:00 AM all 0.22 0.00 0.20 0.02 99.56 02:00:00 AM all 0.22 0.00 0.19 0.01 99.58 02:10:00 AM all 0.23 0.00 0.24 0.02 99.50 02:20:01 AM all 0.22 0.00 0.25 0.01 99.52 02:30:00 AM all 0.19 0.00 0.22 0.01 99.57 02:40:00 AM all 0.22 0.00 0.22 0.01 99.55 02:50:00 AM all 0.21 0.00 0.21 0.01 99.56
The sar command can break down this information for each CPU in a multi-CPU Linux box too, as the following sar -u -P ALL 5 5 output demonstrates:
Linux 2.4.21-20.ELsmp (doughboy) 04/13/200510:17:56 AM CPU %user %nice %system %iowait %idle 10:18:01 AM all 26.41 0.00 23.46 0.00 50.13 10:18:01 AM 0 23.20 0.00 32.80 0.00 44.00 10:18:01 AM 1 26.60 0.00 20.80 0.00 52.60 10:18:01 AM 2 25.80 0.00 21.00 0.00 53.20 10:18:01 AM 3 30.06 0.00 19.24 0.00 50.70 10:18:01 AM CPU %user %nice %system %iowait %idle 10:18:06 AM all 17.70 0.00 24.15 0.00 58.15 10:18:06 AM 0 22.40 0.00 26.40 0.00 51.20 10:18:06 AM 1 15.20 0.00 24.60 0.00 60.20 10:18:06 AM 2 19.00 0.00 20.00 0.00 61.00 10:18:06 AM 3 14.20 0.00 25.60 0.00 60.20 10:18:06 AM CPU %user %nice %system %iowait %idle 10:18:11 AM all 13.69 0.00 23.74 0.05 62.52 10:18:11 AM 0 9.00 0.00 27.40 0.00 63.60 10:18:11 AM 1 19.40 0.00 20.40 0.20 60.00 10:18:11 AM 2 13.20 0.00 21.00 0.00 65.80 10:18:11 AM 3 13.17 0.00 26.15 0.00 60.68 10:18:11 AM CPU %user %nice %system %iowait %idle 10:18:16 AM all 16.40 0.00 23.00 0.00 60.60 10:18:16 AM 0 16.60 0.00 18.00 0.00 65.40 10:18:16 AM 1 15.00 0.00 23.00 0.00 62.00 10:18:16 AM 2 19.40 0.00 19.80 0.00 60.80 10:18:16 AM 3 14.60 0.00 31.20 0.00 54.20 10:18:16 AM CPU %user %nice %system %iowait %idle 10:18:21 AM all 32.60 0.00 22.10 0.00 45.30 10:18:21 AM 0 30.80 0.00 24.40 0.00 44.80 10:18:21 AM 1 34.80 0.00 24.00 0.00 41.20 10:18:21 AM 2 32.00 0.00 20.20 0.00 47.80 10:18:21 AM 3 32.80 0.00 19.80 0.00 47.40 Average: CPU %user %nice %system %iowait %idle Average: all 21.36 0.00 23.29 0.01 55.34 Average: 0 20.40 0.00 25.80 0.00 53.80 Average: 1 22.20 0.00 22.56 0.04 55.20 Average: 2 21.88 0.00 20.40 0.00 57.72 Average: 3 20.96 0.00 24.40 0.00 54.64
Disk I/O Statistics
sar is a good tool for looking at disk I/O. The following shows a sample of sar disk I/O output.
[dave@fisher dave]$ sar -d 5 2Linux 2.4.21-27.EL (fisher) 04/29/2005 04:22:03 PM DEV tps rd_sec/s wr_sec/s 04:22:08 PM dev3-0 1.40 0.00 784.00 04:22:08 PM dev3-1 0.00 0.00 0.00 04:22:08 PM dev3-2 1.40 0.00 784.00 04:22:08 PM dev3-3 0.00 0.00 0.00 04:22:08 PM dev3-64 0.00 0.00 0.00 04:22:08 PM dev3-65 0.00 0.00 0.00 04:22:08 PM DEV tps rd_sec/s wr_sec/s 04:22:13 PM dev3-0 34.60 0.00 4219.20 04:22:13 PM dev3-1 0.00 0.00 0.00 04:22:13 PM dev3-2 34.60 0.00 4219.20 04:22:13 PM dev3-3 0.00 0.00 0.00 04:22:13 PM dev3-64 0.00 0.00 0.00 04:22:13 PM dev3-65 0.00 0.00 0.00 Average: DEV tps rd_sec/s wr_sec/s Average: dev3-0 18.00 0.00 2501.60 Average: dev3-1 0.00 0.00 0.00 Average: dev3-2 18.00 0.00 2501.60 Average: dev3-3 0.00 0.00 0.00 Average: dev3-64 0.00 0.00 0.00 Average: dev3-65 0.00 0.00 0.00
The -d shows disk I/O information. The 5 2 options are interval and iterations, just like the sar data collector.
| Field | Description | DEV | Disk device | tps | Transfers per second (or IOs per second) |
|---|---|
| rd_sec/s | 512-byte reads per second |
| wr_sec/s | 512-byte writes per second |
512 is just a unit of measure. It doesn’t imply that all disk I/O is in 512-byte chunks. The DEV column is the disk device in the format dev#-#, where the first # is a device major number, and the second # is a minor or sequential number. sar uses the minor number with kernels greater than 2.5. For example, we saw dev3-0 and dev3-1 in the sar -d output. These correspond to /dev/hda and /dev/hda1. Look at the following entries in /dev:
brw-rw---- 1 root disk 3, 0 Jun 24 2004 hdabrw-rw---- 1 root disk 3, 1 Jun 24 2004 hda1
/dev/hda has major number 3 and minor number 0. hda1 has a major number of 3 and a minor number of 1.
Networking Statistics
sar offers four different syntax options to display networking information. The -n option takes four different switches: DEV, EDEV, SOCK, and FULL. DEV displays networking interface information, EDEV shows statistics about network errors, SOCK shows socket information, and FULL shows all three switches. They can be used separately or together.
| Field | Description | IFACE | LAN interface |
|---|---|
| rxpck/s | Packets received per second |
| txpck/s | Packets transmitted per second |
| rxbyt/s | Bytes received per second |
| txbyt/s | Bytes transmitted per second |
| rxcmp/s | Compressed packets received per second |
| txcmp/s | Compressed packets transmitted per second |
| rxmcst/s | Multicast packets received per second |
The following is the sar output using the -n DEV option:
# sar -n DEV 5 3Linux 2.4.21-20.EL (fisher) 04/20/2005 03:39:18 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s 03:39:23 AM lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 03:39:23 AM eth0 36.80 0.00 8412.20 10.80 0.00 0.0 0 0.00 03:39:23 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s 03:39:28 AM lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 03:39:28 AM eth0 50.20 1.80 9798.20 526.60 0.00 0.0 0 0.00 03:39:28 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s 03:39:33 AM lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 03:39:33 AM eth0 38.20 1.80 8882.60 400.80 0.00 0.0 0 0.00 Average: IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/ s rxmcst/s Average: lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 Average: eth0 41.73 1.20 9031.00 312.73 0.00 0.0 0 0.00
Information about networking errors can be displayed with sar -n EDEV.
# sar -n EDEV 5 3Linux 2.4.21-20.EL (fisher) 04/17/2005 10:41:44 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 10:41:49 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:49 AM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:49 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 10:41:54 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:54 AM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:54 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 10:41:59 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:41:59 AM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
| Field | Description | IFACE | LAN interface |
|---|---|
| rxerr/s | Bad packets received per second |
| txerr/s | Bad packets transmitted per second |
| coll/s | Collisions per second |
| rxdrop/s | Received packets dropped per second because buffers were full |
| txdrop/s | Transmitted packets dropped per second because buffers were full |
| txcarr/s | Carrier errors per second while transmitting packets |
| rxfram/s | Frame alignment errors on received packets per second |
| rxfifo/s | FIFO overrun errors per second on received packets |
| txfifo/s | FIFO overrun errors per second on transmitted packets |
The SOCK argument displays IPCS socket information.
# sar -n SOCK 5 3Linux 2.4.21-144-default (sawnee) 04/17/05 16:00:56 totsck tcpsck udpsck rawsck ip-frag 16:01:01 117 11 8 0 0 16:01:06 117 11 8 0 0 16:01:11 117 11 8 0 0 Average: 117 11 8 0 0
| Field | Description | totsck | Total number of sockets used |
|---|---|
| tcpsck | Number of TCP sockets used |
| udpsck | Number of UDP sockets used |
| rawsck | Number of raw sockets used |
| ip-frag | Number of IP fragments used |
sar can generate many other reports. It is worth reading through the sar(1) man page to see whether there are any others you want to start using.
Tags: performance, sar, tools