#89 Tracking FTP Usage

If you're running an anonymous FTP server, you should already be constantly monitoring what happens in the ~ftp/pub directory (which is usually where uploads are allowed), but any FTP server requires you to keep an eye on things.

The ftp daemon's transfer log (xferlog) file format is definitely one of the most cryptic in Unix, which makes analyzing it in a script rather tricky. Worse, there's a standard, common xferlog file format that just about everyone uses (and which this script expects), and there's an abbreviated ftpd.log format that some BSD versions of ftpd use that's just about impossible to analyze in a script.

So we'll focus on the xferlog format. The columns in an xferlog are as shown in Table 10-2.

Table 10-2: Field values in the xferlog file
Column	Value
1–5	Current time
6	Transfer time (secs)
7	Remote host
8	File size
9	Filename
10	Transfer type
11	Special action flag
12	Direction
13	Access mode
14	Username
15	Service name
16	Authentication method
17	Authenticated user ID
18-?	Additional codes as added by the specific fptd program (usually omitted)

A sample line from an xferlog is as cryptic as you might expect:

Mon Nov  4 12:22:46 2002 2 192.168.124.152 2170570 \
/home/ftp/pub/openssl-0.9.5r.tar.gz b _ i r leoftp 0 * c

This script quickly scans through xferlog, highlighting connections and files uploaded and downloaded, and producing other useful statistics.

The Code

#!/bin/sh

# xferlog - Analyzes and summarizes the FTP transfer log. A good doc
# detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm.

stdxferlog="/var/log/xferlog"
temp="/tmp/$(basename $0).$$"
nicenum="$HOME/bin/nicenumber"    # Script #4

trap "/bin/rm -f $temp" 0

extract()
{
  # Called with $1 = desired accessmode, $2 = section name for output

  if [ ! -z "$(echo $accessmode | grep $1)" ] ; then

    echo "" ; echo "$2"

    if [ "$1" = "a" -o "$1" = "g" ] ; then
      echo "  common account (entered password) values:"
    else
      echo "  user accounts accessing server: "
    fi
    awk "\$13 == \"$1\" { print \$14 }" $log | sort | \
        uniq -c | sort -rn | head -10 | sed 's/^/    /'

    awk "\$13 == \"$1\" && \$12 == \"o\" { print \$9 }" $log | sort | \
      uniq -c | sort -rn | head -10 | sed 's/^/    /' > $temp
    if [ -s $temp ] ; then
      echo " files downloaded from server:" ; cat $temp
    fi

    awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log | sort | \
      uniq -c | sort -rn | head -10 | sed 's/^/    /' > $temp

    if [ -s $temp ] ; then
      echo " files uploaded to server:" ; cat $temp
    fi
  fi
}

###### The main script block

case $# in
  0 ) log=$stdxferlog           ;;
  1 ) log="$1"                  ;;
  * ) echo "Usage: $(basename $0) {xferlog name}" >&2
      exit 1
esac

if [ ! -r $log ] ; then
  echo "$(basename $0): can't read $log." >&2
  exit 1
fi
# Ascertain whether it's an abbreviated or standard ftp log file format. If
# it's the abbreviated format, output some minimal statistical data and quit:
# The abbreviated format is too difficult to analyze in a short script,
# unfortunately.

if [ ! -z $(awk '$6 == "get" { short=1 } END{ print short }' $log) ] ; then
   bytesin="$(awk 'BEGIN{sum=0} $6=="get" {sum+=$9} END{print sum}' $log)"
  bytesout="$(awk 'BEGIN{sum=0} $6=="put" {sum+=$9} END{print sum}' $log)"

  echo -n "Abbreviated ftpd xferlog from "
  echo -n $(head -1 $log | awk '{print $1, $2, $3 }')
  echo " to $(tail -1 $log | awk '{print $1, $2, $3}')"
  echo " bytes in: $($nicenum $bytesin)"
  echo " bytes out: $($nicenum $bytesout)"
  exit 0
fi

 bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )"
bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )"
time="$(awk 'BEGIN{sum=0} {sum += $6} END{ print sum }' $log)"

echo -n "Summary of xferlog from "
echo -n $(head -1 $log | awk '{print $1, $2, $3, $4, $5 }')
echo    " to $(tail -1 $log | awk '{print $1, $2, $3, $4, $5}')"
echo "       bytes in: $($nicenum $bytesin)"
echo "       bytes out: $($nicenum $bytesout)"
echo "  transfer time: $time seconds"

accessmode="$(awk '{print $13}' $log | sort -u)"

extract "a" "Anonymous Access"
extract "g" "Guest Account Access"
extract "r" "Real User Account Access"

exit 0

How It Works

In an xferlog, the total number of incoming bytes can be calculated by extracting just those lines that have direction="i" and then summing up the eighth column of data. Outgoing bytes are in the same column, but for direction="o".

 bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )"
bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )"

Ironically, the slower the network connection, the more accurate the total connection time is. On a fast network, smaller transfers are logged as taking zero seconds, though clearly every transfer that succeeds must be longer than that.

Three types of access mode are possible: a is anonymous, g is for users who utilize the guest account (usually password protected), and r is for real or regular users. In the case of anonymous and guest users, the account value (field 14) is the user's password. People connecting anonymously are requested by their FTP program to specify their email address as their password, which is then logged and can be analyzed.

Of this entire xferlog output stream, the most important entries are those with an anonymous access mode and a direction of i, indicating that the entry is an upload listing. If you have allowed anonymous connections and have either deliberately or accidentally left a directory writable, these anonymous upload entries are where you'll be able to see if skript kiddies, warez hackers, and other characters of ill repute are exploiting your system. If such an entry lists a file uploaded to your server, it needs to be checked out immediately, even if the file-name seems quite innocuous.

This test occurs in the following statement in the extract function:

awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log | sort | \
  uniq -c | sort -rn | head -10 | sed 's/^/    /' > $temp

In this rather complex awk invocation, we're checking to see whether field 13 matches the anonymous account code (because extract is called as extract "a" "Anonymous Access") and whether field 12 indicates that it's an upload with the code i. If both of these conditions are true, we process the value of field 9, which is the name of the file uploaded.

If you're running an FTP server, this is definitely a script for a weekly (or even daily) cron job.

Running the Script

If invoked without any arguments, this script tries to read and analyze the standard ftpd transfer log /var/log/xferlog. If that's not the correct log file, a different filename can be specified on the command line.

The Results

The results depend on the format of the transfer log the script is given. If it's an abbreviated form, some minimal statistics are generated and the script quits:

$ xferlog  succinct.xferlog
Abbreviated ftpd xferlog from Aug 1 04:20:11 to Sep 1 04:07:41
       bytes in: 215,300,253
       bytes out: 30,305,090

When a full xferlog in standard format is encountered, considerably more information can be obtained and displayed by the script:

$ xferlog
Summary of xferlog from Mon Sep 1 5:03:11 2003 to Tue Sep 30 17:38:50 2003
       bytes in: 675,840
      bytes out: 3,989,488
  transfer time: 11 seconds

Anonymous Access
  common account (entered password) values:
       1 taylor@intuitive.com
       1 john@doe
  files downloaded from server:
       1 /MySubscriptions.opml
  files uploaded to server:
       1 /tmp/Find.Warez.txt

Real User Account Access
  user accounts accessing server:
       7 rufus
       2 taylor
  files downloaded from server:
       7 /pub/AllFiles.tgz
       2 /pub/AllFiles.tar

Security Alert! Did you notice that someone using anonymous FTP has uploaded a file called /tmp/Find.Warez.txt? "Warez" are illegal copies of licensed software — not something you want on your server. Upon seeing this, I immediately went into my FTP archive and deleted the file.