Performance tools: sar

sar is a great general performance monitoring tool. sar can output data for almost everything Linux does. The sar command is delivered in the sysstat rpm.

sar can display performance data for CPU, run queue, disk I/O, paging (swap), memory, CPU interrupts, networking, and more. The most important sar feature is the capability to create a data file. Every Linux system should collect sar data from cron jobs. The sar data file provides a system administrator with historical performance information. This feature is very important, and it separates sar from the other performance tools. If a nightly batch job runs twice as long as normal, you won’t find out until the next morning (unless you get paged). You need the ability to look at performance data from 12 hours ago. The sar data collector provides this ability. Many reporting syntaxes exist, but let’s look at data collection first.

sar Data Collector

sar data collection is done with a binary executable and two scripts in /usr/lib/sa. The sar data collector is a binary executable located at /usr/lib/sa/sadc. The job of sadc is to write to the data collection file /var/log/sa/. Several options can be supplied to sadc. A common syntax is:

/usr/lib/sa/sadc interval iterations file name

interval is the number of seconds between sampling. iterations is the number of samples to take. file name specifies the output file. A simple sadc syntax is /usr/lib/ sa/sadc 360 5 /tmp/sadc.out. This command takes five samples at five-minute intervals and stores them in /tmp/sadc.out. We should collect the samples on a regular basis, so we need a script to be run by cron. We should put the samples in a place that makes sense, like we did with the top script from the previous section. Fortunately, the sysstat rpm provides the /usr/lib/sa/sa1 script to do all this.

The sa1(8) man page is much longer than the sa1 script itself. /usr/lib/sa/sa1 is a very simple script that runs sadc with the syntax sadc -F -L 1 1 /var/log/sa/sa## where ## is the day of the month. Older versions of sa1 use the output from date +.%Y_%m_%d as the file suffix. The -F option causes sadc to force creation of the output file if necessary. The -L locks the output file before writing to it to prevent corrupting the file when two sadc processes are running at the same time. Older versions of sadc didn’t have the -L option, so the sa1 script performed manual locking. The only options for the sa1 script are the interval between samples and the number of iterations to sample. A cron file (/etc/cron.d/sysstat) is supplied with sysstat. It differs between sysstat versions. The following are the entries for the version 5.0.5 of sysstat:

# cat /etc/cron.d/sysstat# run system activity accounting tool every 10 minutes

*/10 * * * * root /usr/lib/sa/sa1 1 1

# generate a daily summary of process accounting at 23:53

53 23 * * * root /usr/lib/sa/sa2 -A

CPU Statistics

The sar -u output shows CPU information. The -u option is the default for sar. The output shows CPU utilization as a percentage.

Field Description CPU CPU number
%user Time spent running processes in user mode
%nice Time spent running niced processes
%system Time spent running processes in kernel mode (system)
%iowait Time during which the processor was waiting for I/O to complete while no process was executing on the CPU
%idle Time during which no process was executing on the CPU

This should look familiar. It is the same CPU information as in top reports. The following shows the output format:

[root@fisher dave]# sar 5 10Linux 2.4.21-27.EL (fisher)     04/30/2005

02:03:20 PM       CPU     %user     %nice   %system   %iowait      %idle

02:03:25 PM       all     36.80      0.00      4.20      0.00      59.00

02:03:30 PM       all     37.80      0.00      5.20      2.20      54.80

02:03:35 PM       all     55.40      0.00      4.40      3.00      37.20

02:03:40 PM       all     53.60      0.00      6.20      0.00      40.20

02:03:45 PM       all     37.20      0.00      6.60      1.00      55.20

02:03:50 PM       all     36.60      0.00      4.40      2.20      56.80

02:03:55 PM       all     51.00      0.00      4.20      1.00      43.80

02:04:00 PM       all     55.60      0.00      4.20      0.00      40.20

02:04:05 PM       all     40.60      0.00      8.00      1.20      50.20

02:04:10 PM       all     36.60      0.00      4.20      2.20      57.00

Average:          all     44.12      0.00      5.16      1.28      49.44

The 5 10 causes sar to take 10 samples at 5-second intervals. The first column of any sar report is a timestamp.

We could have looked at the file created with sadc by using the -f option. This sar syntax shows the output from sar -f /var/log/sa/sa21:

[dave@fisher dave]$ sar -f /var/log/sa/sa21head -n 20Linux 2.4.21-20.EL (fisher)   04/21/2005

12:00:00 AM       CPU     %user     %nice   %system   %iowait      %idle

12:10:00 AM       all      0.23      0.00      0.23      0.02      99.52

12:20:00 AM       all      0.22      0.00      0.20      0.01      99.57

12:30:01 AM       all      0.21      0.00      0.19      0.01      99.59

12:40:00 AM       all      0.23      0.00      0.22      0.02      99.54

12:50:01 AM       all      0.19      0.00      0.28      0.01      99.52

01:00:00 AM       all      0.22      0.00      0.18      0.01      99.59

01:10:00 AM       all      0.40      0.00      0.25      0.02      99.34

01:20:00 AM       all      0.20      0.00      0.25      0.01      99.53

01:30:00 AM       all      0.20      0.00      0.23      0.01      99.56

01:40:00 AM       all      0.22      0.00      0.21      0.02      99.56

01:50:00 AM       all      0.22      0.00      0.20      0.02      99.56

02:00:00 AM       all      0.22      0.00      0.19      0.01      99.58

02:10:00 AM       all      0.23      0.00      0.24      0.02      99.50

02:20:01 AM       all      0.22      0.00      0.25      0.01      99.52

02:30:00 AM       all      0.19      0.00      0.22      0.01      99.57

02:40:00 AM       all      0.22      0.00      0.22      0.01      99.55

02:50:00 AM       all      0.21      0.00      0.21      0.01      99.56

The sar command can break down this information for each CPU in a multi-CPU Linux box too, as the following sar -u -P ALL 5 5 output demonstrates:

Linux 2.4.21-20.ELsmp (doughboy)      04/13/200510:17:56 AM       CPU     %user     %nice   %system   %iowait      %idle

10:18:01 AM       all     26.41      0.00     23.46      0.00      50.13

10:18:01 AM         0     23.20      0.00     32.80      0.00      44.00

10:18:01 AM         1     26.60      0.00     20.80      0.00      52.60

10:18:01 AM         2     25.80      0.00     21.00      0.00      53.20

10:18:01 AM         3     30.06      0.00     19.24      0.00      50.70

10:18:01 AM       CPU     %user     %nice   %system   %iowait      %idle

10:18:06 AM       all     17.70      0.00     24.15      0.00      58.15

10:18:06 AM         0     22.40      0.00     26.40      0.00      51.20

10:18:06 AM         1     15.20      0.00     24.60      0.00      60.20

10:18:06 AM         2     19.00      0.00     20.00      0.00      61.00

10:18:06 AM         3     14.20      0.00     25.60      0.00      60.20

10:18:06 AM       CPU     %user     %nice   %system   %iowait      %idle

10:18:11 AM       all     13.69      0.00     23.74      0.05      62.52

10:18:11 AM         0      9.00      0.00     27.40      0.00      63.60

10:18:11 AM         1     19.40      0.00     20.40      0.20      60.00

10:18:11 AM         2     13.20      0.00     21.00      0.00      65.80

10:18:11 AM         3     13.17      0.00     26.15      0.00      60.68

10:18:11 AM       CPU     %user     %nice   %system   %iowait      %idle

10:18:16 AM       all     16.40      0.00     23.00      0.00      60.60

10:18:16 AM         0     16.60      0.00     18.00      0.00      65.40

10:18:16 AM         1     15.00      0.00     23.00      0.00      62.00

10:18:16 AM         2     19.40      0.00     19.80      0.00      60.80

10:18:16 AM         3     14.60      0.00     31.20      0.00      54.20

10:18:16 AM       CPU     %user     %nice   %system   %iowait      %idle

10:18:21 AM       all     32.60      0.00     22.10      0.00      45.30

10:18:21 AM         0     30.80      0.00     24.40      0.00      44.80

10:18:21 AM         1     34.80      0.00     24.00      0.00      41.20

10:18:21 AM         2     32.00      0.00     20.20      0.00      47.80

10:18:21 AM         3     32.80      0.00     19.80      0.00      47.40

Average:          CPU     %user     %nice   %system   %iowait      %idle

Average:          all     21.36      0.00     23.29      0.01      55.34

Average:            0     20.40      0.00     25.80      0.00      53.80

Average:            1     22.20      0.00     22.56      0.04      55.20

Average:            2     21.88      0.00     20.40      0.00      57.72

Average:            3     20.96      0.00     24.40      0.00      54.64

Disk I/O Statistics

sar is a good tool for looking at disk I/O. The following shows a sample of sar disk I/O output.

[dave@fisher dave]$ sar -d 5 2Linux 2.4.21-27.EL (fisher)     04/29/2005

04:22:03 PM       DEV       tps  rd_sec/s   wr_sec/s

04:22:08 PM    dev3-0      1.40      0.00     784.00

04:22:08 PM    dev3-1      0.00      0.00       0.00

04:22:08 PM    dev3-2      1.40      0.00     784.00

04:22:08 PM    dev3-3      0.00      0.00       0.00

04:22:08 PM   dev3-64      0.00      0.00       0.00

04:22:08 PM   dev3-65      0.00      0.00       0.00

04:22:08 PM       DEV       tps  rd_sec/s   wr_sec/s

04:22:13 PM    dev3-0     34.60      0.00    4219.20

04:22:13 PM    dev3-1      0.00      0.00       0.00

04:22:13 PM    dev3-2     34.60      0.00    4219.20

04:22:13 PM    dev3-3      0.00      0.00       0.00

04:22:13 PM   dev3-64      0.00      0.00       0.00

04:22:13 PM   dev3-65      0.00      0.00       0.00

Average:          DEV       tps  rd_sec/s   wr_sec/s

Average:       dev3-0     18.00      0.00    2501.60

Average:       dev3-1      0.00      0.00       0.00

Average:       dev3-2     18.00      0.00    2501.60

Average:       dev3-3      0.00      0.00       0.00

Average:      dev3-64      0.00      0.00       0.00

Average:      dev3-65      0.00      0.00       0.00

The -d shows disk I/O information. The 5 2 options are interval and iterations, just like the sar data collector.

Field Description DEV Disk device tps Transfers per second (or IOs per second)
rd_sec/s 512-byte reads per second
wr_sec/s 512-byte writes per second

512 is just a unit of measure. It doesn’t imply that all disk I/O is in 512-byte chunks. The DEV column is the disk device in the format dev#-#, where the first # is a device major number, and the second # is a minor or sequential number. sar uses the minor number with kernels greater than 2.5. For example, we saw dev3-0 and dev3-1 in the sar -d output. These correspond to /dev/hda and /dev/hda1. Look at the following entries in /dev:

brw-rw----    1 root     disk       3,   0 Jun 24  2004 hdabrw-rw----    1 root     disk       3,   1 Jun 24  2004 hda1

/dev/hda has major number 3 and minor number 0. hda1 has a major number of 3 and a minor number of 1.

Networking Statistics

sar offers four different syntax options to display networking information. The -n option takes four different switches: DEV, EDEV, SOCK, and FULL. DEV displays networking interface information, EDEV shows statistics about network errors, SOCK shows socket information, and FULL shows all three switches. They can be used separately or together.

Field Description IFACE LAN interface
rxpck/s Packets received per second
txpck/s Packets transmitted per second
rxbyt/s Bytes received per second
txbyt/s Bytes transmitted per second
rxcmp/s Compressed packets received per second
txcmp/s Compressed packets transmitted per second
rxmcst/s Multicast packets received per second

The following is the sar output using the -n DEV option:

# sar -n DEV 5 3Linux 2.4.21-20.EL (fisher)     04/20/2005

03:39:18 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s

txcmp/

s  rxmcst/s

03:39:23 AM        lo      0.00      0.00      0.00      0.00      0.00

0.0

0      0.00

03:39:23 AM      eth0     36.80      0.00   8412.20     10.80      0.00

0.0

0      0.00

03:39:23 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s    rxcmp/s

txcmp/

s  rxmcst/s

03:39:28 AM        lo      0.00      0.00      0.00      0.00       0.00

0.0

0      0.00

03:39:28 AM      eth0     50.20      1.80   9798.20    526.60       0.00

0.0

0      0.00

03:39:28 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s    rxcmp/s

txcmp/

s  rxmcst/s

03:39:33 AM        lo      0.00      0.00      0.00      0.00       0.00

0.0

0      0.00

03:39:33 AM      eth0     38.20      1.80   8882.60    400.80       0.00

0.0

0      0.00

Average:        IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s    rxcmp/s

txcmp/

s  rxmcst/s

Average:           lo      0.00      0.00      0.00      0.00       0.00

0.0

0      0.00

Average:         eth0     41.73      1.20   9031.00    312.73       0.00

0.0

0      0.00

Information about networking errors can be displayed with sar -n EDEV.

# sar -n EDEV 5 3Linux 2.4.21-20.EL (fisher)     04/17/2005

10:41:44 AM     IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s

txcarr/s  rxfram/s  rxfifo/s  txfifo/s

10:41:49 AM        lo      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00

10:41:49 AM      eth0      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00

10:41:49 AM     IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s

txcarr/s  rxfram/s  rxfifo/s  txfifo/s

10:41:54 AM        lo      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00

10:41:54 AM      eth0      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00

10:41:54 AM     IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s

txcarr/s  rxfram/s  rxfifo/s  txfifo/s

10:41:59 AM        lo      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00

10:41:59 AM      eth0      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00

Average:        IFACE   rxerr/s   txerr/s     coll/s  rxdrop/s  txdrop/s

txcarr/s  rxfram/s  rxfifo/s  txfifo/s

Average:           lo      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00

Average:         eth0      0.00      0.00      0.00      0.00       0.00

0.00      0.00      0.00      0.00
Field Description IFACE LAN interface
rxerr/s Bad packets received per second
txerr/s Bad packets transmitted per second
coll/s Collisions per second
rxdrop/s Received packets dropped per second because buffers were full
txdrop/s Transmitted packets dropped per second because buffers were full
txcarr/s Carrier errors per second while transmitting packets
rxfram/s Frame alignment errors on received packets per second
rxfifo/s FIFO overrun errors per second on received packets
txfifo/s FIFO overrun errors per second on transmitted packets

The SOCK argument displays IPCS socket information.

# sar -n SOCK 5 3Linux 2.4.21-144-default (sawnee)       04/17/05

16:00:56       totsck    tcpsck    udpsck    rawsck    ip-frag

16:01:01          117        11         8         0          0

16:01:06          117        11         8         0          0

16:01:11          117        11         8         0          0

Average:          117        11         8         0          0
Field Description totsck Total number of sockets used
tcpsck Number of TCP sockets used
udpsck Number of UDP sockets used
rawsck Number of raw sockets used
ip-frag Number of IP fragments used

sar can generate many other reports. It is worth reading through the sar(1) man page to see whether there are any others you want to start using.