This sounds more complex than it really is, because as you’ll see in this script, the archive is just a file emailed to a remote mailbox and could even be pointed to a Yahoo! or Hotmail mailbox. The list of files is kept in a separate data file, with shell wildcards allowed therein. Filenames can contain spaces too, something that rather complicates the script, as you’ll see.

#!/bin/sh

# remotebackup – Takes a list of files and directories,
# builds a single archive, compressed, then emails it off to a
# remote archive site for safekeeping. It’s intended to be run
# every night for critical user files, but not intended to
# replace a more rigorous backup scheme. You should strongly
# consider using unpacker, Script #88, on the remote end too.

uuencode=”/usr/bin/uuencode”
outfile=”/tmp/rb.$$.tgz”
outfname=”backup.$(date +%y%m%d).tgz”
infile=”/tmp/rb.$$.in”

trap “/bin/rm -f $outfile $infile” 0

if [ $# -ne 2 -a $# -ne 3 ] ; then
echo “Usage: $(basename $0) backup-file-list remoteaddr {targetdir}” >&2
exit 1
fi

if [ ! -s "$1" ] ; then
echo “Error: backup list $1 is empty or missing” >&2
exit 1
fi

# Scan entries and build fixed infile list. This expands wildcards
# and escapes spaces in filenames with a backslash, producing a
# change: “this file” becomes this\ file so quotes are not needed.

while read entry; do
echo “$entry” | sed -e ’s/ /\\ /g’ >> $infile
done < “$1″

# The actual work of building the archive, encoding it, and sending it

tar czf – $(cat $infile) | \
$uuencode $outfname | \
mail -s “${3:-Backup archive for $(date)}” “$2″

echo “Done. $(basename $0) backed up the following files:”
sed ’s/^/ /’ $infile
echo -n “and mailed them to $2 ”
if [ ! -z "$3" ] ; then
echo “with requested target directory $3″
else
echo “”
fi

exit 0

After the basic validity checks, the script processes the file containing the list of critical files, which is supplied as the first command argument, to ensure that spaces embedded in its filenames will work in the while loop (remember, by default spaces delimit arguments, so without some additional help, the shell will think that “test file” is two arguments, not one). It does this by prefacing every space with a backslash. Then it builds the archive with the primitive but useful tar command, which lacks the ability to read standard input for its file list and thus must be fed the filenames via a cat invocation.

tar czf – $(cat $infile)

The tar invocation automatically compresses the archive, and uuencode is then utilized to ensure that the resultant archive data file can be successfully emailed without corruption. The end result is that the remote address receives an email message with the uuencoded tar archive as an attachment. This should be a straightforward script.

This script expects two arguments: the name of a file that contains a list of files to archive and back up, and the destination email address for the compressed, uuencoded archive file. The file list can be as simple as

$ cat filelist
*.sh
*.html

As a result, most sysadmins have a set of instructions that they place at the top of their log file analysis utilities, similar to the following:

mv $log.2 $log.3
mv $log.1 $log.2
mv $log $log.1
touch $log

If run weekly, this would produce a rolling one-month archive of log file information divided into week-size portions of data. However, it’s just as easy to create a script that accomplishes this for all log files in the /var/log directory at once, thereby relieving any log file analysis scripts of the burden.

The script steps through each file in the /var/log directory that matches a particular set of criteria, checking each matching file’s rotation schedule and last-modified date to see if it’s time for it to be rotated.

#!/bin/sh
# rotatelogs – Rolls logfiles in /var/log for archival purposes.
# Uses a config file to allow customization of how frequently
# each log should be rolled. The config file is in
# logfilename=duration
# format, where duration is in days. If, in the config
# file, an entry is missing for a particular logfilename,
# rotatelogs won’t rotate the file more frequently than every seven days.

logdir=”/var/log”
config=”/var/log/rotatelogs.conf”
mv=”/bin/mv”
default_duration=7 count=0

duration=$default_duration

if [ ! -f $config ] ; then
echo “$0: no config file found. Can’t proceed.” >&2; exit 1
fi

if [ ! -w $logdir -o ! -x $logdir ] ; then
echo “$0: you don’t have the appropriate permissions in $logdir” >&2
exit 1
fi

cd $logdir

# While we’d like to use ‘:digit:’ with the find, many versions of
# find don’t support POSIX character class identifiers, hence [0-9]

for name in $(find . -type f -size +0c ! -name ‘*[0-9]*’ \
! -name ‘\.*’ ! -name ‘*conf’ -maxdepth 1 -print | sed ’s/^\.\///’)
do

count=$(( $count + 1 ))

# Grab this entry from the config file

duration=”$(grep “^${name}=” $config|cut -d= -f2)”
if [ -z $duration ] ; then
duration=$default_duration
elif [ "$duration" = "0" ] ; then
echo “Duration set to zero: skipping $name”
continue
fi

back1=”${name}.1″; back2=”${name}.2″;
back3=”${name}.3″; back4=”${name}.4″;

# If the most recently rolled log file (back1) has been modified within
# the specific quantum, then it’s not time to rotate it.

if [ -f "$back1" ] ; then
if [ -z $(find \"$back1\" -mtime +$duration -print 2>/dev/null) ]
then
echo -n “$name’s most recent backup is more recent than $duration ”
echo “days: skipping” ; continue
fi
fi

echo “Rotating log $name (using a $duration day schedule)”

# Rotate, starting with the oldest log
if [ -f "$back3" ] ; then
echo “… $back3 -> $back4″ ; $mv -f “$back3″ “$back4″
fi
if [ -f "$back2" ] ; then
echo “… $back2 -> $back3″ ; $mv -f “$back2″ “$back3″
fi
if [ -f "$back1" ] ; then
echo “… $back1 -> $back2″ ; $mv -f “$back1″ “$back2″
fi
if [ -f "$name" ] ; then
echo “… $name -> $back1″ ; $mv -f “$name” “$back1″
fi
touch “$name”
chmod 0600 “$name”
done

if [ $count -eq 0 ] ; then
echo “Nothing to do: no log files big enough or old enough to rotate”
fi
exit 0

To truly be useful, the script needs to work with a configuration file that lives in /var/log, which allows different log files to be set to different rotation schedules. The contents of a typical configuration file are as follows:

# Configuration file for the log rotation script.
# Format is name=duration where ‘name’ can be any
# filename that appears in the /var/log directory. Duration
# is measured in days.

ftp.log=30
lastlog=14
lookupd.log=7
lpr.log=30
mail.log=7
netinfo.log=7
secure.log=7
statistics=7
system.log=14
# Anything with a duration of zero is not rotated
wtmp=0

The heart of this script is the find statement:

for name in $(find . -type f -size +0c ! -name ‘*[0-9]*’ \
! -name ‘\.*’ ! -name ‘*conf’ -maxdepth 1 -print | sed ’s/^\.\///’)

This creates a loop, returning all files in the /var/log directory that are greater than 0 characters in size, don’t contain a number in their name, don’t start with a period (Mac OS X in particular dumps a lot of oddly named log files in this directory; they all need to be skipped), and don’t end with the word “conf” (we don’t want to rotate out the rotatelogs.conf file, for obvious reasons!). The maxdepth 1 ensures that find doesn’t step into subdirectories. Finally, the sed invocation removes any leading ./ sequences.

The Results

$ sudo rotatelogs
ftp.log’s most recent backup is more recent than 30 days: skipping
Rotating log lastlog (using a 14 day schedule)
… lastlog -> lastlog.1
lpr.log’s most recent backup is more recent than 30 days: skipping

Notice that of all the log files in /var/log, only three matched the specified find criteria, and of those only one, lastlog, hadn’t been backed up sufficiently recently, according to the duration values in the configuration file shown earlier.

One example of how this script could be even more useful is to have the oldest archive file, the old $back4 file, emailed to a central storage site before it’s over-written by the mv command in the following statement:

echo “… $back3 -> $back4″ ; $mv -f “$back3″ “$back4″

Another useful enhancement to rotatelogs would be to compress all rotated logs to further save on disk space, which would also require that the script recognize and work properly with compressed files as it proceeded.

One of the reasons so many systems neglect backups is that many of the backup tools are crude and difficult to understand. The dump and restore commands (called ufsdump and restore in Solaris) are typical, with five “dump levels” and an intimidating configuration file required.

A shell script can solve this problem. This script backs up a specified set of directories, either incrementally (that is, only those files that have changed since the last backup) or full backup (all files). The backup is compressed on the fly to minimize space usage, and the script output can be directed to a file, a tape device, a remotely mounted NFS partition, or even a CD burner on compatible systems.

#!/bin/sh

# backup – Creates either a full or incremental backup of a set of
# defined directories on the system. By default, the output
# file is saved in /tmp with a timestamped filename, compressed.
# Otherwise, specify an output device (another disk, or a
# removable storage device).

usageQuit()
{
cat << “EOF” >&2
Usage: $0 [-o output] [-i|-f] [-n]
-o lets you specify an alternative backup file/device
-i is an incremental or -f is a full backup, and -n prevents
updating the timestamp if an incremental backup is done.
EOF
exit 1
}

compress=”bzip2″ # change for your favorite compression app
inclist=”/tmp/backup.inclist.$(date +%d%m%y)”
output=”/tmp/backup.$(date +%d%m%y).bz2″
tsfile=”$HOME/.backup.timestamp”
btype=”incremental” # default to an incremental backup
noinc=0 # and an update of the timestamp

trap “/bin/rm -f $inclist” EXIT

while getopts “o:ifn” arg; do
case “$arg” in
o ) output=”$OPTARG”; ;;
i ) btype=”incremental”; ;;
f ) btype=”full”; ;;
n ) noinc=1; ;;
? ) usageQuit ;;
esac
done

shift $(($OPTIND – 1))

echo “Doing $btype backup, saving output to $output”

timestamp=”$(date +’%m%d%I%M’)”

if [ "$btype" = "incremental" ] ; then
if [ ! -f $tsfile ] ; then
echo “Error: can’t do an incremental backup: no timestamp file” >&2
exit 1
fi
find $HOME -depth -type f -newer $tsfile -user ${USER:-LOGNAME} | \
pax -w -x tar | $compress > $output
failure=”$?”
else
find $HOME -depth -type f -user ${USER:-LOGNAME} | \
pax -w -x tar | $compress > $output
failure=”$?”
fi

if [ "$noinc" = "0" -a "$failure" = "0" ] ; then
touch -t $timestamp $tsfile
fi
exit 0

For a full system backup, the pax command does all the work, piping its output to a compression program (bzip2 by default) and then to an output file or device. An incremental backup is a bit more tricky because the standard version of tar doesn’t include any sort of modification time test, unlike the GNU version of tar. The list of files modified since the previous backup is built with find and saved in the inclist temporary file. That file, emulating the tar output format for increased portability, is then fed to pax directly.

Choosing when to mark the timestamp for a backup is an area in which many backup programs get messed up, typically marking the “last backup time” when the program has finished the backup, rather than when it started. Setting the timestamp to the time of backup completion can be a problem if any files are modified during the backup process (which can take quite a while if the backup is being fed to a tape device). Because files modified under this scenario would have a last-modified date older than the timestamp date, they would not be backed up the next night.

However, timestamping before the backup takes place is wrong too, because if the backup fails, there’s no way to reverse the updated timestamp. Both of these problems are avoided by saving the date and time before the backup starts (in the timestamp variable), but applying the value of $timestamp to $tsfile using the -t flag to touch only after the backup has succeeded.

This script has a number of options, all of which can be ignored to perform the default incremental backup based on the timestamp for the last incremental backup. The flags allow you to specify a different output file or device (-o output), to choose a full backup (-f), to actively choose an incremental backup (-i), or to prevent the timestamp file from being updated in the case of an incremental backup (-n).

$ backup
Doing incremental backup, saving output to /tmp/backup.140703.bz2

As you would expect, the output of a backup program isn’t very scintillating. But the resulting compressed file is sufficiently large that it shows plenty of data is within:

$ ls -l /tmp/backup*
-rw-r–r– 1 taylor wheel 61739008 Jul 14 07:31 backup.140703.bz2

#!/bin/sh

# archivedir – Creates a compressed archive of the specified directory.

maxarchivedir=10 # size, in blocks, of ‘big’ directory
compress=gzip # change to your favorite compress app
progname=$(basename $0)

if [ $# -eq 0 ] ; then
echo “Usage: $progname directory” >&2 ;exit 1
fi

if [ ! -d $1 ] ; then
echo “${progname}: can’t find directory $1 to archive.” >&2; exit 1
fi

if [ "$(basename $1)" != "$1" -o "$1" = "." ] ; then
echo “${progname}: You must specify a subdirectory” >&2
exit 1
fi

if [ ! -w . ] ; then
echo “${progname}: cannot write archive file to current directory.” >&2
exit 1
fi

dirsize=”$(du -s $1 | awk ‘{print $1}’)”

if [ $dirsize -gt $maxarchivedir ] ; then
echo -n “Warning: directory $1 is $dirsize blocks. Proceed? [n] ”
read answer
answer=”$(echo $answer | tr ‘[:upper:]‘ ‘[:lower:]‘ | cut -c1)”
if [ "$answer" != "y" ] ; then
echo “${progname}: archive of directory $1 canceled.” >&2
exit 0
fi
fi
archivename=”$(echo $1 | sed ’s/$/.tgz/’)”

if tar cf – $1 | $compress > $archivename ; then
echo “Directory $1 archived as $archivename”
else
echo “Warning: tar encountered errors archiving $1″
fi

exit 0

This script is almost all error-checking code, to ensure that it never causes a loss of data or creates an incorrect snapshot. In addition to the typical tests to validate the presence and appropriateness of the starting argument, this script also forces the user to be in the parent directory of the subdirectory to be compressed and archived, which ensures that the archive file is saved in the proper place upon completion. The conditional if [ ! -w . ] ; then verifies that the user has write permission on the current directory. And this script even warns users before archiving if the resultant backup file would be unusually large.

Finally, the actual command that archives the specified directory is

tar cf – $1 | $compress > $archivename

The return code of this command is tested to ensure that the script never deletes the directory if an error of any sort occurs.

This script should be invoked with the name of the desired directory to archive as its only argument. To ensure that the script doesn’t try to archive itself, it requires that a subdirectory of the current directory be specified as the argument, rather than “.”.

$ archivedir scripts
Warning: directory scripts is 2224 blocks. Proceed? [n] n
archivedir: archive of directory scripts canceled.

This seemed as though it might be a big archive, so I hesitated to create it, but thinking about it, there’s no reason not to proceed after all:

$ archivedir scripts
Warning: directory scripts is 2224 blocks. Proceed? [n] y
Directory scripts archived as scripts.tgz

The results:

$ ls -l scripts.tgz
-rw-r–r– 1 taylor staff 325648 Jul 14 08:01 scripts.tgz

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hdb2 25695892 1871048 22519564 8% /
/dev/hdb1 101089 6218 89652 7% /boot
none 127744 0 127744 0% /dev/shm

What would be much more useful is a version of df that summarizes the available capacity values in column four and then presents the summary in a way that is easily understood. It’s a task easily accomplished in a script.

#!/bin/sh

# diskspace – Summarizes available disk space and presents it in a logical
# and readable fashion.

tempfile=”/tmp/available.$$”
trap “rm -f $tempfile” EXIT

cat << ‘EOF’ > $tempfile
{ sum += $4 }
END { mb = sum / 1024
gb = mb / 1024
printf “%.0f MB (%.2fGB) of available disk space\n”, mb, gb
}
EOF

df -k | awk -f $tempfile

exit 0

This script can be run as any user and produces a succinct one-line summary of available disk space.

On the same system on which the df output shown earlier was generated, the script reports the following:

$ diskspace
96199 MB (93.94GB) of available disk space

Another issue to consider is whether it’s more useful to know about the available disk space on all devices, including those partitions that cannot grow (like /boot), or whether reporting on user volumes is sufficient. If the latter is the case, you can improve this script by making a call to grep immediately after the df call. Use grep with the desired device names to include only particular devices, or use grep -v followed by the unwanted device names to screen out devices you don’t want included.

#!/bin/sh
# diskhogs – Disk quota analysis tool for Unix; assumes all user
# accounts are >= UID 100. Emails message to each violating user
# and reports a summary to the screen.MAXDISKUSAGE=20
violators=”/tmp/diskhogs0.$$”

trap “/bin/rm -f $violators” 0

for name in $(cut -d: -f1,3 /etc/passwd | awk -F: ‘$2 > 99 { print $1 }’)
do
echo -n “$name ”
# You might need to modify the following list of directories to match
# the layout of your disk. Most likely change: /Users to /home
find / /usr /var /Users -user $name -xdev -type f -ls | \
awk ‘{ sum += $7 } END { print sum / (1024*1024) }’

done | awk “\$2 > $MAXDISKUSAGE { print \$0 }” > $violators

if [ ! -s $violators ] ; then
echo “No users exceed the disk quota of ${MAXDISKUSAGE}MB”
cat $violators
exit 0
fi

while read account usage ; do

cat << EOF | fmt | mail -s “Warning: $account Exceeds Quota” $account
Your disk usage is ${usage}MB, but you have been allocated only
${MAXDISKUSAGE}MB. This means that you need to either delete some of
your files, compress your files (see ‘gzip’ or ‘bzip2′ for powerful and
easy-to-use compression programs), or talk with us about increasing
your disk allocation.

Thanks for your cooperation in this matter.

Dave Taylor @ x554
EOF
echo “Account $account has $usage MB of disk space. User notified.”

done < $violators

exit 0

This script has no starting arguments and should be run as root for accurate results. This can most safely be accomplished by using the sudo command.

$ sudo diskhogs
Account linda has 39.7 MB of disk space. User notified.
Account taylor has 21799.5 MB of disk space. User notified.

If we now peek into the linda account mailbox, we’ll see that a message from the script has been delivered:

Subject: Warning: linda Exceeds Quota

Your disk usage is 39.7MB, but you have been allocated only 20MB. This means
that you need to either delete some of your files, compress your files (see
‘gzip’ or ‘bzip2′ for powerful and easy-to-use compression programs), or talk
with us about increasing your disk allocation.

Thanks for your cooperation on this matter.

Dave Taylor @ x554