#83 Synchronizing Files with SFTP

While the ftp program is quite widely available, it's really something you should avoid like the plague. There are two reasons for this. First, ftp servers are notorious for being buggy and having security holes, and second, and much more problematic, ftp transfers all data between the server and client in the clear. This means that when you transmit files to your server, your account name and password are sent along without any encryption, making it relatively trivial for someone with a packet sniffer to glean this vital information. That's bad. Very bad.

Instead, all modern servers should support the considerably more secure ssh (secure shell) package, a login and file transfer pair that supports end-to-end encryption. The file transfer element of the encrypted transfer is sftp, and it's even more primitive than ftp, but we can still rewrite ftpsyncup to work with sftp, as shown in this script.

Downloading an ssh package

If you don't have ssh on your system, complain to your vendor and administrative team. There's no excuse. You can also obtain the package and install it yourself by starting at http://www.openssh.com/

The Code

#!/bin/sh

# sftpsync - Given a target directory on an sftp server, makes sure that
#   all new or modified files are uploaded to the remote system. Uses
#   a timestamp file ingeniously called .timestamp to keep track.
timestamp=".timestamp"
tempfile="/tmp/sftpsync.$$"
count=0

trap "/bin/rm -f $tempfile" 0 1 15      # zap tempfile on exit &sigs

if [ $# -eq 0 ] ; then
  echo "Usage: $0 user@host { remotedir }" >&2
  exit 1
fi

user="$(echo $1 | cut -d@ -f1)"
server="$(echo $1 | cut -d@ -f2)"

if [ $# -gt 1 ] ; then
  echo "cd $2" >> $tempfile
fi

if [ ! -f $timestamp ] ; then
  # no timestamp file, upload all files
  for filename in *
  do
    if [ -f "$filename" ] ; then
      echo "put -P \"$filename\"" >> $tempfile
      count=$(($count + 1))
    fi
  done
else
  for filename in $(find . -newer $timestamp -type f -print)
  do
     echo "put -P \"$filename\"" >> $tempfile
     count=$(($count + 1))
  done
fi

if [ $count -eq 0 ] ; then
  echo "$0: No files require uploading to $server" >&2
  exit 1
fi

echo "quit" >> $tempfile

echo "Synchronizing: Found $count files in local folder to upload."

if ! sftp -b $tempfile "$user@$server" ; then
  echo "Done. All files synchronized up with $server"
  touch $timestamp
fi
exit 0

How It Works

Like ftp, sftp allows a series of commands to be fed to it as a pipe or input redirect, which makes this script rather simple to write: Almost the entire script focuses on building the sequence of commands necessary to upload changed files. At the end, the sequence of commands is fed to the sftp program for execution.

As with Scripts #81 and #82, if you have a version of sftp that doesn't properly return a nonzero failure code to the shell when a transfer fails, simply remove the conditional block at the end of the script, leaving only

sftp -b $tempfile "$user@$server"
touch $timestamp

Because sftp requires the account to be specified as user@host, it's actually a bit simpler than the equivalent ftp script shown in Script #81, ftpsyncup. Also notice the -P flag added to the put commands; it causes sftp to retain the local permission, creation, and modification times for all files transferred.

Running the Script

This script is simple to run: Move into the local source directory, ensure that the target directory exists, and invoke the script with your username, server name, and remote directory. For simple situations, I have an alias called ssync (source sync) that moves into the directory I need to keep in sync and invokes sftpsync automatically:

alias ssync=sftpsync taylor@intuitive.com /wicked/scripts

The "Hacking the Script" section shows a more sophisticated wrapper that makes the synchronization script even more helpful.

The Results

$ sftpsync taylor@intuitive.com /wicked/scripts
Synchronizing: Found 2 files in local folder to upload.
Connecting to intuitive.com...
taylortaylor@intuitive.com's password:
sftp> cd /wicked/scripts
sftp> put -P "./003-normdate.sh"
Uploading ./003-normdate.sh to /usr/home/taylor/usr/local/etc/httpd/htdocs/
intuitive/wicked/scripts/003-normdate.sh
sftp> put -P "./004-nicenumber.sh"
Uploading ./004-nicenumber.sh to /usr/home/taylor/usr/local/etc/httpd/htdocs/
intuitive/wicked/scripts/004-nicenumber.sh
sftp> quit
Done. All files synchronized up with intuitive.com

Hacking the Script

The wrapper script that I use to invoke sftpsync is a tremendously useful script, and I have used it throughout the development of this book to ensure that the copies of the scripts in the web archive (see http://www.intuitive.com/wicked/) are exactly in sync with those on my own servers, all the while adroitly sidestepping the insecurities of the ftp protocol.

This wrapper, ssync, contains all the necessary logic for moving to the right local directory (see the variable localsource) and creating a file archive that has the latest versions of all the files in a so-called tarball (named for the tar, tape archive, command that's used to build it). The last line of the script calls sftpsync:

#!/bin/sh

# ssync - If anything's changed, creates a tarball and syncs a remote
#    directory via sftp using sftpsync.

sftpacct="taylor@intuitive.com"
tarballname="AllFiles.tgz"
localsource="$HOME/Desktop/Wicked Cool Scripts/scripts"
remotedir="/wicked/scripts"
timestamp=".timestamp"
count=0

sftpsync="$HOME/bin/sftpsync"

# First off, let's see if the local dir exists and has files

if [ ! -d "$localsource" ] ; then
  echo "$0: Error: directory $localsource doesn't exist?" >&2
  exit 1
fi

cd "$localsource"

# Now let's count files to ensure something's changed:

if [ ! -f $timestamp ] ; then
  for filename in *
  do
    if [ -f "$filename" ] ; then
      count=$(($count + 1))
    fi
  done
else
  count=$(find . -newer $timestamp -type f -print | wc -l)
fi
if [ $count -eq 0 ] ; then
  echo "$(basename $0): No files found in $localsource to sync with remote."; exit
0
fi

echo "Making tarball archive file for upload"

tar -czf $tarballname ./*

# Done! Now let's switch to the sftpsync script

exec $sftpsync $sftpacct $remotedir

With one command, a new archive file is created, if necessary, and all files (including the new archive, of course) are uploaded to the server as needed:

$ ssync
Making tarball archive file for upload
Synchronizing: Found 2 files in local folder to upload.
Connecting to intuitive.com...
taylor@intuitive.com's password:
sftp> cd shellhacks/scripts
sftp> put -P "./AllFiles.tgz"
Uploading ./AllFiles.tgz to shellhacks/scripts/AllFiles.tgz
sftp> put -P "./ssync"
Uploading ./ssync to shellhacks/scripts/ssync
sftp> quit
Done. All files synchronized up with intuitive.com

This script can doubtless be hacked further. One obvious tweak would be to have ssync invoked from a cron job every few hours during the work day so that the files on a remote backup server are invisibly synchronized to your local files without any human intervention.