Network File Transfer: FTP

You can use File Transfer Protocol (FTP) clients to transfer extremely large files directly from one site to another. FTP can handle both text and binary files. This is one of the TCP/IP protocols, and it operates on systems connected to networks that use the TCP/IP protocols, such as the Internet. FTP performs a remote login to another account on another system connected to you on a network. Once logged into that other system, you can transfer files to and from it. To log in, you need to know the login name and password for the account on the remote system. For example, if you have accounts at two different sites on the Internet, you can use FTP to transfer files from one to the other. Many sites on the Internet allow public access using FTP, however. Such sites serve as depositories for large files anyone can access and download. These sites are often referred to as FTP sites, and in many cases, their Internet address begins with the word ftp, such as ftp.gnome.org or ftp.redhat.com. Others begin with other names, such as metalab.unc.edu. These public sites allow anonymous FTP login from any user. For the login name, you use the word "anonymous," and for the password you use your Internet address. You can then transfer files from that site to your own system.

You can perform FTP operations using any one of a number of FTP client programs. For Linux systems, you can choose from several FTP clients. Many now operate using GUI interfaces such as GNOME. Some, such as Mozilla, have limited capabilities, whereas others, such as NcFTP, include an extensive set of enhancements. The original FTP client is just as effective, though not as easy to use. It operates using a simple command line interface and requires no GUI or cursor support, as do other clients.

The Internet has a great many sites open to public access. They contain files anyone can obtain using file transfer programs, such as NcFTP. Unless you already know where a file is located, however, finding it can be difficult. To search for files on FTP sites, you can use search engines provided by Web sites, such as Yahoo!, Excite, AltaVista, Google, or Lycos. For Linux software, you can check sites such as freshmeat.net, sourceforge.net, rpmfind.net, apps.kde.com, and www.gnome.org. These sites usually search for both Web pages and FTP files.

Web Browser–Based FTP: Mozilla

You access an FTP site and download files from it with any Web browser. A Web browser is effective for checking out an FTP site to see what files are listed there. When you access an FTP site with a Web browser, the entire list of files in a directory is listed as a Web page. You can move to a subdirectory by clicking its entry. With Mozilla, you can easily browse through an FTP site to download files. To download a file with Mozilla, hold down the SHIFT key and then double-click the file. This will start the transfer operation. This opens a box for selecting your local directory and the name for the file. The default name is the same as on the remote system. You can manage your downloads with the download manager, which will let you cancel a download operation in progress or remove other downloads requested. The manager will show the time remaining, speed, and the amount transferred for the current download. Browsers are useful for locating individual files, though not for downloading a large set of files, as is usually required for a system update.

The K Desktop File Manager: Konqueror

On the K Desktop, the desktop file manager (Konqueror) has a built-in FTP capability. The FTP operation has been seamlessly integrated into standard desktop file operations. Downloading files from an FTP site is as simple as copying files by dragging them from one directory window to another, but one of the directories happens to be located on a remote FTP site. On the K Desktop, you can use a file manager window to access a remote FTP site. Files in the remote directory are listed just as your local files are. To download files from an FTP site, you open a window to access that site, entering the URL for the FTP site in the window's location box. Open the directory you want, and then open another window for the local directory to which you want the remote files copied. In the window showing the FTP files, select the ones you want to download. Then simply click and drag those files to the window for the local directory. A pop-up menu appears with choices for Copy, Link, or Move. Select Copy. The selected files are then downloaded. Another window then opens, showing the download progress and displaying the name of each file in turn, and a bar indicating the percentage downloaded so far.

GNOME Desktop FTP: Nautilus

The easiest way to download files is to use the built-in FTP capabilities of the GNOME file manager, Midnight Commander. You can also use several GNOME-based FTP clients that offer more features, including gFTP. Check the GNOME Web site at www.gnome.org for more. gFTP is included with the current GNOME release.

On GNOME, the desktop file manager—Nautilus—has a built-in FTP capability much like the KDE file manager. The FTP operation has been seamlessly integrated into standard desktop file operations. Downloading files from an FTP site is as simple as dragging files from one directory window to another, where one of the directories happens to be located on a remote FTP site. Use the GNOME file manager to access a remote FTP site, listing files in the remote directory, just as local files are. Just enter the FTP URL following the prefix ftp:// and press ENTER. The top directory of the remote FTP site will be displayed. Simply use the file manager to progress through the remote FTP site's directory tree until you find the file you want. Then open another window for the local directory to which you want the remote files copied. In the window showing the FTP files, select those you want to download. Then CTRL-click and drag those files to the window for the local directory. CTRL-click performs a copy operation, not a move. As files are downloaded, a dialog window appears showing the progress.

gFTP

The gFTP program is a simpler GNOME FTP client designed to let you make standard FTP file transfers. It has an interface similar to WS_FTP used on Windows. The gFTP window consists of several panes. The top-left pane lists files in your local directory, and the top-right pane lists your remote directory. Subdirectories have folder icons preceding their names. The parent directory can be referenced by the double period entry (..) with an up arrow at the top of each list. Double-click a directory entry to access it. The pathnames for all directories are displayed in boxes above each pane. You can enter a new pathname for a different directory to change to it, if you want.

Two buttons between the panes are used for transferring files. The left arrow button, <-, downloads selected files in the remote directory, and the right arrow button, ->, uploads files from the local directory. To download a file, click it in the right-side pane and then click the left arrow button, <-. When the file is downloaded, its name appears in the left pane, your local directory. Menus across the top of the window can be used to manage your transfers. A connection manager enables you to enter login information about a specific site. You can specify whether to perform an anonymous login or to provide a username and password. Click the Connect button to connect to that site. A drop-down menu for sites enables you to choose the site you want.

NcFTP

The NcFTP program has a screen-based interface that can be run from any shell command line. It does not use a desktop interface. FTP operations are executed using commands you enter at a prompt. Options and bookmarks can be selected using cursor-based menus. To start up NcFTP, you enter the ncftp command on the command line. If you are working in a window manager, such as KDE, GNOME, or FVWM, open a shell terminal window and enter the command at its prompt. The main NcFTP screen consists of an input line at the bottom of the screen with a status line above it. The remainder of the screen is used to display commands and responses from remote systems. For example, when you download files, a message specifying the files to be downloaded is displayed in the status line. NcFTP lets you set preferences for different features, such as anonymous login, progress meters, or a download directory. Enter the pref command to open the preferences screen. From there, you can select and modify the listed preferences.

To connect to an FTP site, you enter the open command on the input line, followed by the site's address. The address can be either an IP address or a domain name, such as ftp.gnome.org. If you don't supply an address, a list of your bookmarked sites is displayed, and you can choose one from there. By default, NcFTP attempts an anonymous login, using the term "anonymous" as your username and your e-mail address as the password. When you successfully connect, the status bar displays the remote site's name on the left and the remote directory name on the right.

open ftp.gnome.org

If you want to log into a specific account on a remote site, have yourself prompted for the username and password by using the -u option with the open command. The open command remembers the last kind of login you performed for a specific site and repeats it. If you want to change back to an anonymous login from a user login, you use the -a option with the open command. For busy sites, you may be unable to connect on the first try and you must repeat the open process. NcFTP has a redial capability you turn on with the -r option. The -d option sets the delay for the next attempt, and the -g option sets the maximum number of connection attempts. With the lookup command, you can obtain the IP and domain name addresses for an FTP site. The lookup command takes as an argument either the IP or domain name address and then displays both. This is useful for finding a site's IP address. With the -v option, more information, such as aliases, is retrieved. The NcFTP open options are shown in Table 14-5.

Once connected, you enter commands on the input line to perform FTP operations such as displaying file lists, changing directories, or downloading files. With the ls command, you can list the contents of the current remote directory. Use the cd command to change to another remote directory. The dir command displays a detailed listing of files. With the page command, you view the contents of a remote file, a screen at a time. To download files, you use the get command, and to upload files, you use the put command. During a download, a progress meter above the status bar displays how much of the file has been downloaded so far. The get command has several features described in more detail in the following section. When you finish, you can disconnect from the site with the close command. You can then use open to connect to another site, or quit the NcFTP program with the quit command. The help command lists all NcFTP commands. You can use the help command followed by the name of a command to display specific information on it.

The NcFTP program supports several commands that operate on your local system. These are usually standard FTP command names preceded by an l. lcd changes your local working directory, lls lists the contents of your local directory, lpage displays the contents of a local file a screen at a time, and lpwd displays the local directory's full pathname. For any other local commands or scripts you need to execute, use the shell escape command, !. Simply precede the shell command or script with a !. The NcFTP program also provides commands for managing files and directories on your remote site, provided you have the permission to do so. You can use mkdir to create a remote directory, and rmdir to remove one. Use the rm command to erase remote files. With the rename command, you can change their names. The NcFTP commands are listed in Table 14-6.

The NcFTP program also has a colon mode of operation that enables you to issue a single ncftp command to download a file. Enter the ncftp command, followed by a URL, for the file you want. You can enter the command on the shell command line or place it within a script. For example, the following command downloads the readme file on the Red Hat FTP site:

$ ncftp ftp.redhat.com/pub/README

In the colon mode, the -c option sends the file to the standard output and the -m option pipes it to your pager:

$ ncftp -c ftp.redhat.com/pub/README > ~/redhatinfo/readme
$ ncftp -m ftp.redhat.com/pub/README 

NcFTP Download Features

The NcFTP get command differs significantly from the original FTP client's get command. Whereas the original FTP client uses two commands, get and mget, to perform download operations, NcFTP uses only the get command. However, the NcFTP get command combines the capabilities of both mget and get into the get command, as well as adding several new features. Table 14-7 lists the various get command options. By default, the NcFTP get command performs wildcard matching for filenames. If you enter only part of a filename, the get command tries to download all files beginning with that name. You can turn off wildcard matching with the -G option, in which case you must enter the full names of the files you want. The following example downloads all files with names beginning with "Xfree86" and is similar to using mget Xfree86* in the original FTP:

get Xfree86

The get command checks to see if you already have a file you are trying to download. If so, it skips the download. The get command also checks if the file you already have is a newer version, in which case it also skips the download. This is a helpful feature for easily maintaining upgrade files. You can simply access the update directory on the remote site, and then use the get command with the * to download to the directory you are using to keep your upgrade file. Only newer versions or newly added upgrade files are downloaded, instead of the entire set. If you want to download a file, even though you have it already, you can force the download with the -f option. For example, to download upgrades for Red Hat manually, you can connect to the Red Hat upgrade directory in the Red Hat FTP site and then issue the following get command:

get *

Certain features require you to enter an option on the command line after the get command. For example, adding the -R command specifies a recursive capability, enabling you to download and create subdirectories and their files. This command is particularly helpful in downloading upgrade directories, such as Red Hat's, which contain several subdirectories. The following example downloads the i386 directory and all its subdirectories:

get -R i386

If you want to give a file a different name on your local system, use the -z option. Enter the local filename you want after the remote filename. The following example downloads the readme file and renames it calinfo. If you did not use the -z option, both names would be taken as files to be downloaded, instead of only the first.

get -z readme calinfo

To obtain recent files only, you can use the -n option, which takes as its argument a number of days. Files older than the specified number of days are not retrieved. The following example downloads files posted within the last 30 days:

get -n 30 *

Bookmarks and Macros

When you disconnect (close) from a site, NcFTP automatically saves information about it. This includes the site address, the directory you were in, and the login information. This information is placed in a file called bookmarks in your .ncftp directory. The site information is given a bookmark name you can use to access the site easily again. The bookmark name is usually the key name in the site's address. You can use this name to connect to the site. For example, ftp.redhat.com could be named redhat. You could then connect to it with the following command:

open redhat

You can edit your bookmark entries using the bookmark editor. Enter the command bookmarks to bring up the editor. Remote systems you have accessed are listed on the right side of the screen. Bookmark commands are listed on the left. You can change the bookmark name or edit login information, such as the username or password, the remote directory, or the transfer mode.

The NcFTP program supports macros for simple operations. You create macros by entering macro definitions in the macros file located in your .ncftp directory. Initially, no such file will exist, so you have to create one using any text editor. The macros file is a simple text file you can edit with any text editor. The syntax for a macro definition follows:

macro macro-name
 ftp-commands
end

A macro executes NcFTP commands. Remember, however, the ! is an NcFTP command that enables you to execute any Linux command or script. With a preceding !, you can define an NcFTP macro that executes any shell command or any script you have written. A simple example of a macro is as follows:

macro ascii
 type ascii
end

Macros support parameters similar to those used by shell programs. Arguments entered after a macro name can be referenced in the macro using a $ sign and the number of the argument in the argument list. $1 references the first argument, $2 the second, and so on. $* is a special parameter that references all arguments, and $@ references all arguments, encasing each in double quotes.

macro cdls
 cd $1
 ls
end

The NcFTP program also supports a limited numbers of event macros. These are macros executed when a certain event is detected, such as when the program starts or shuts down. For example, a macro defined with the name .start.ncftp has its commands executed every time you start NcFTP; .quit.ncftp executes its commands when you quit. Site-specific macros also execute whenever it is necessary to access or disconnect from certain sites. These macros begin with either the open or close event, followed by the site's bookmark. For example, a macro defined with the name .open.redhat would execute its commands whenever you connected to the Red Hat site. A macro named .open.any has its commands executed whenever you connect to any site, and one named .close.any executes whenever you disconnect from a site.

ftp

The name ftp designates the original FTP client used on Unix and Linux systems. ftp uses a command line interface, and it has an extensive set of commands and options you can use to manage your FTP transfers. You start the ftp client by entering the command ftp at a shell prompt. If you have a specific site you want to connect to, you can include the name of that site on the command line after the ftp keyword. Otherwise, you need to connect to the remote system with the ftp command open. You are then prompted for the name of the remote system with the prompt "(to)". Upon entering the remote system name, ftp connects you to the system and then prompts you for a login name. The prompt for the login name consists of the word "Name" and, in parentheses, the system name and your local login name. Sometimes the login name on the remote system is the same as the login name on your own system. If the names are the same, press ENTER at the prompt. If they are different, enter the remote system's login name. After entering the login name, you are prompted for the password. In the next example, the user connects to the remote system garnet and logs into the robert account:

$ ftp
ftp> open
(to) garnet
Connected to garnet.berkeley.edu.
220 garnet.berkeley.edu FTP server ready.
Name (garnet.berkeley.edu:root): robert
password required
Password:
user robert logged in
ftp>

Once logged in, you can execute Linux commands on either the remote system or your local system. You execute a command on your local system in ftp by preceding the command with an exclamation point. Any Linux commands without an exclamation point are executed on the remote system. One exception exists to this rule. Whereas you can change directories on the remote system with the cd command, to change directories on your local system, you need to use a special ftp command called lcd (local cd). In the next example, the first command lists files in the remote system, while the second command lists files in the local system:

ftp> ls
ftp> !ls

The ftp program provides a basic set of commands for managing files and directories on your remote site, provided you have the permission to do so (see Table 14-8). You can use mkdir to create a remote directory, and rmdir to remove one. Use the delete command to erase a remote file. With the rename command, you can change the names of files. You close your connection to a system with the close command. You can then open another connection if you want. To end the ftp session, use the quit or bye command.

ftp> close
ftp> bye
Good-bye
$

To transfer files to and from the remote system, use the get and put commands. The get command receives files from the remote system to your local system, and the put command sends files from your local system to the remote system. In a sense, your local system gets files from the remote and puts files to the remote. In the next example, the file weather is sent from the local system to the remote system using the put command:

ftp> put weather
PORT command successful.
ASCII data connection
ASCII Transfer complete.
ftp>

If a download is ever interrupted, you can resume the download with reget. This is helpful for an extremely large file. The download resumes from where it left off, so the whole file needn't be downloaded again. Also, be sure to download binary files in binary mode. For most FTP sites, the binary mode is the default, but some sites might have ASCII (text) as the default. The command ascii sets the character mode, and the command binary sets the binary mode. Most software packages available at Internet sites are archived and compressed files, which are binary files. In the next example, the transfer mode is set to binary, and the archived software package mydata.tar.gz is sent from the remote system to your local system using the get command:

ftp> binary
ftp> get mydata.tar.gz
PORT command successful.
Binary data connection
Binary Transfer complete.
ftp>

You may often want to send several files, specifying their names with wildcard characters. The put and get commands, however, operate only on a single file and do not work with special characters. To transfer several files at a time, you have to use two other commands, mput and mget. When you use mput or mget, you are prompted for a file list. You can then either enter the list of files or a file-list specification using special characters. For example, *.c specifies all the files with a .c extension, and * specifies all files in the current directory. In the case of mget, files are sent one by one from the remote system to your local system. Each time, you are prompted with the name of the file being sent. You can type y to send the file or n to cancel the transmission. You are then prompted for the next file. The mput command works in the same way, but it sends files from your local system to the remote system. In the next example, all files with a .c extension are sent to your local system using mget:

ftp> mget
(remote-files) *.c
mget calc.c? y
PORT command successful
ASCII data connection
ASCII transfer complete
mget main.c? y
PORT command successful
ASCII data connection
ASCII transfer complete
ftp>

Answering the prompt for each file can be a tedious prospect if you plan to download a large number of files, such as those for a system update. In this case, you can turn off the prompt with the prompt command, which toggles the interactive mode on and off. The mget operation then downloads all files it matches, one after the other.

ftp> prompt
Interactive mode off.
ftp> mget
(remote-files) *.c
 PORT command successful
ASCII data connection
ASCII transfer complete
PORT command successful
ASCII data connection
ASCII transfer complete
ftp>

Automatic Login and Macros: .netrc

The ftp client has an automatic login capability and support for macros. Both are entered in a user's ftp configuration file called .netrc. Each time you connect to a site, the .netrc file is checked for connection information, such as a login name and password. In this way, you needn't enter a login name and password each time you connect to a site. This feature is particularly useful for anonymous logins. Instead of your having to enter the username "anonymous" and your e-mail address as your password, they can be automatically read from the .netrc file. You can even make anonymous login information your default so that, unless otherwise specified, an anonymous login is attempted for any FTP site to which you try to connect. If you have sites you must log into, you can specify them in the .netrc file and, when you connect, either automatically log in with your username and password for that site or be prompted for them.

Entries in the .netrc file have the following syntax. An entry for a site begins with the term "machine," followed by the network or Internet address, and then the login and password information.

machine system-address login remote-login-name password password

The following example shows an entry for logging into the dylan account on the turtle.trek.com system:

machine golf.mygames.com login dylan password legogolf

For a site you would anonymously log into, you enter the word "anonymous" for the login name and your e-mail address for the password.

machine ftp.redhat.com login anonymous password dylan@turtle.trek.com

In most cases, you are using ftp to access anonymous FTP sites. Instead of trying to make an entry for each one, you can make a default entry for anonymous FTP login. When you connect to a site, ftp looks for a machine entry for it in the .netrc file. If none exists, ftp looks for a default entry and uses that. A default entry begins with the word "default" with no network address. To make anonymous logins your default, enter anonymous and your e-mail address as your login and password.

default login anonymous password dylan@turtle.trek.com

A sample .netrc file with a machine definition and a default entry is shown here.

.netrc

machine golf.mygames.com login dylan password legogolf
default login anonymous password dylan@turtle.trek.com

You can also define macros in your .netrc file. With a macro, you can execute several ftp operations at once using only the macro name. Macros remain in effect during a connection. When you close a connection, the macros are undefined. Although a macro can be defined on your ftp command line, defining them in .netrc entries makes more sense. This way, you needn't redefine them again. They are read automatically from the .netrc file and defined for you. You can place macro definitions within a particular machine entry in the .netrc file or in the default entry. Macros defined in machine entries are defined only when you connect to that site. Macros in the default entry are defined whenever you make a connection to any site.

The syntax for a macro definition follows. It begins with the keyword macdef, followed by the macro name you want to give it, and ends with an empty line. ftp macros can take arguments, referenced within the macro with $n, where $1 references the first argument, and $2 the second, and so on. If you need to use a $ character in a macro, you have to quote it using the backslash, \$.

macdef macro-name
ftp commands
empty-line

The redupd macro, defined next, changes to a directory where it then downloads Red Hat updates for the current release. It also changes to a local directory where the update files are to be placed. The prompt command turns off the download prompts for each file. The mget command then downloads the files. The macro assumes you are connected to the Red Hat FTP site.

defmac redupd
cd pub/redhat/current
lcd /root/redupdate
prompt
mget *

A sample .netrc file follows with macros defined for both specific and default entries. An empty line is placed after each macro definition. You can define several macros for a machine or the default entry. The macro definitions following a machine entry up to the next machine entry are automatically defined for that machine connection.

machine updates.redhat.com login anonymous password dylan@turtle.trek.com
# define a macro for downloading updated from the Red Hat site
defmac redupd
 cd pub/redhat/current
 lcd /root/redupdate
 prompt
 mget *
   
default login anonymous password dylan@turtle.trek.com
defmac lls
!ls

lftp

The lftp program is an enhanced FTP client with advanced features such as the capabilities to download mirror sites and to run several FTP operation in the background at the same time. It uses a command set similar to that for the ftp client. You use get and mget commands to download files, with the -o option to specify local locations for them. Use lcd and cd to change local and remote directories.

To manage background commands, you use many of the same commands as for the shell (see Chapter 8). The & placed at the end of a command puts it into the background. Use CTRL-Z to put a job already running into the background. Commands can be grouped with parentheses and placed together into the background. Use the jobs command to list your background jobs and the wait or fg command to move jobs from the background to the foreground. When you exit lftp, the program will continue to run any background jobs. In effect, lftp becomes a background job itself.

When you connect to a site, you can queue commands with the queue command, setting up a list of FTP operations to perform. With this feature, you could queue several download operations to a site. The queue can be reordered and entries deleted if you wish. You can also connect to several sites and set up a queue for each one. The mirror command lets you maintain a local version of a mirror site. You can download an entire site or just update newer files, as well as removing files no longer present on the mirror.

You can tailor lftp with options set in the .lftprc file. System-wide settings are placed in the /etc/lftp.conf file. Here, you can set features like the prompt to use and your anonymous password. The .lftp directory holds support files for command history, logs, bookmarks, and startup commands. lftp also supports the .netrc file, checking it for login information.