Section 4.3. Objective 3: Make and Install Programs from Source

4.3. Objective 3: Make and Install Programs from Source

Open source software is credited with offering value that rivals or even exceeds that of proprietary vendors' products. While binary distributions make installation simple, you sometimes won't have access to a binary package. In these cases, you'll have to compile the program from scratch.

4.3.1. Getting Open Source and Free Software

Source code for the software that makes up a Linux distribution is available from a variety of sources. Your distribution media contain both source code and compiled binary forms of many software projects. Since much of the code that comes with Linux originates from the Free Software Foundation (FSF), the GNU web site contains a huge array of software. Not just for Linux, either. Although Linux distributions are largely made up of GNU software, that software runs on many other Unix and Unix-like operating systems, including the various flavors of BSD (e.g., FreeBSD, NetBSD, and OpenBSD). Major projects, such as Apache (http://www.apache.org), distribute their own code. Whatever outlet you choose, the source code must be packaged for your use, and among the most popular packaging methods for source code is the tarball.

4.3.1.1. What's a tarball?

Code for a significant project that a software developer wishes to distribute is originally stored in a hierarchical tree of directories. Included are the source code (in the C language), a Makefile, and some documentation. To share the code, the entire tree must be encapsulated in a way that is efficient and easy to send and store electronically. A common method of doing this is to use tar to create a single tarfile containing the directory's contents, and then use gzip or bzip2 to compress it for efficiency. The resulting compressed file is referred to as a tarball. This method of distribution is popular because both tar and gzip are widely available and understood, ensuring a wide audience. A tarball is usually indicated by the use of the multiple extensions .tar and .gz, put together into .tar.gz. A combined single extension of .tgz is also popular. (bzip2 offers significantly smaller compressed files at a cost of additional CPU overhead. Files compressed with bzip2 usually use the extension .bz2.)

4.3.1.2. Opening a tarball

The contents of a tarball are obtained through a two-step process. The file is first uncompressed with gzip and then extracted with tar. The following is an example, starting with tarball.tar.gz:

$ gzip -d tarball.tar.gz
$ tar xvf tarball.tar

The -d option to gzip indicates "decompress mode." If you prefer, you can use gunzip in place of gzip -d to do the same thing:

$ gunzip tarball.tar.gz
$ tar xvf tarball.tar

You can also skip the intermediate unzipped file by piping the output of gzip straight into tar:

$ gzip -dc tarball.tar.gz | tar xv

In this case, the -c option to gzip tells it to keep the compressed file in place. This saves disk space. For even more convenience, avoid using gzip entirely and use the decompression capability in tar (GNU tar offers compression; older tar programs don't):

$ tar zxvf tarball.tar.gz

Files compressed with bzip2 can be opened with exactly the same options that gzip uses.

$ bzip2 -dc tarball.tar.bz2 | tar xv

Recent versions of GNU tar also support the j option to run bzip2 directly.

$ tar jxvf tarball.tar.bz2

On the Exam
All of these methods achieve the same result. Be sure you understand that tar can archive directly to files (not just to a tape drive) and that a compressed version of a tarfile is made with gzip. Be familiar with the various ways you could extract files from a tarball, including gzip -d ; tar, gunzip ; tar, gzip -d | tar, and tar z. You should be comfortable using tar and gzip and their more common options.

4.3.2. Compiling Open Source Software

Once you've extracted the source code, you're ready to compile it. You'll need to have the appropriate tools available on your system, such as gcc and make.

4.3.2.1. configure

Most larger source code packages include a configure script located at the top of the source code tree. configure is produced for you by the programmer using the autoconf utility. autoconf is beyond the scope LPIC Level 1 exams. This script needs no modification or configuration from the user. When it executes, it examines your system to verify the existence of a compiler, libraries, utilities, and other items necessary for a successful compile. It uses the information it finds to produce a custom Makefile for the software package on your particular system. If configure finds that something is missing, it fails and gives you a terse but descriptive message. configure succeeds in most cases, leaving you ready to begin the actual compile process.

4.3.2.2. make

make is a utility for controlling and scripting software compilation. When multiple source code files are used in a project, it is rarely necessary to compile all of them for every build of the executable. Instead, only the source files that have changed since the last compilation really need to be compiled again.

make works by defining targets and their dependencies as specified in the Makefile. The ultimate target in a software build is the executable file or files. They depend on object files, which in turn depend on source code files. When a source file is edited, its date is more recent than that of the last compiled object. make is designed to automatically handle these dependencies and do the right thing.

Typical usage for make is:

make [-f  makefile  ]
[ option  [...] ] [ target  ]

Commonly used options include:

-f filename: Use filename as a Makefile.
-j number_of_jobs: Tells make how many processes to run simultaneously.

4.3.2.3. Installing the compiled software

Most mature source code projects come with a predetermined location in the filesystem for the executable files created by compilation. In many cases, they're expected to go to /usr/local/bin. To facilitate installation to these default locations, many Makefiles contain a special target called install. By executing the make install command, files are copied and set with the appropriate attributes.

Warning: The default installation directory included in a project's Makefile may differ from that defined by your Linux distribution. If you upgrade software you are already using, this could lead to confusion over versions.

On the Exam
A basic understanding of make is sufficient for Exam 101. In addition, be prepared to add to or modify the contents of variables in a Makefile, such as include directories or paths. This could be necessary, for example, if additional libraries must be included in the compilation or if a command must be customized.

4.3.2.4. Example:Compiling bash

GNU's bash shell is presented here as an example of the process of compiling. You can find a compressed tarball of the bash source at the GNU FTP site, ftp://ftp.gnu.org/gnu/bash/. Multiple versions might be available. Version 2.03 is used in this example (you will find more recent versions). The compressed tarball is bash-2.03.tar.gz. As you can see by its name, it is a tar file that has been compressed with gzip. To uncompress the contents, use the compression option in tar:

# tar zxvf bash-2.03.tar.gz
bash-2.03/
bash-2.03/CWRU/
bash-2.03/CWRU/misc/
bash-2.03/CWRU/misc/open-files.c
bash-2.03/CWRU/misc/sigs.c
bash-2.03/CWRU/misc/pid.c
... (extraction continues) ...

Next move into the new directory, take a look around, and read some basic documentation:

# cd bash-2.03
# ls
AUTHORS        NEWS
CHANGES        NOTES
COMPAT         README
COPYING        Y2K
CWRU           aclocal.m4
INSTALL        alias.c
MANIFEST       alias.h
Makefile.in    ansi_stdlib.h
... (listing continues) ...
# less README

The build process for bash is started by using the dot-slash prefix to launch configure:

# ./configure
creating cache ./config.cache
checking host system type... i686-pc-linux-gnu
Beginning configuration for bash-2.03 for i686-pc-linux-gnu
checking for gcc... gcc
checking whether the C compiler (gcc  ) works... yes
checking whether the C compiler (gcc  ) is a
  cross-compiler... no
checking whether we are using GNU C... yes
checking whether gcc accepts -g... yes
checking whether large file support needs explicit
  enabling... yes
checking for POSIXized ISC... no
checking how to run the C preprocessor... gcc -E # make
... (configure continues) ...

Next, compile:

# make
/bin/sh ./support/mkversion.sh -b -s release -d 2.03 \
  -p 0 -o newversion.h && mv newversion.h version.h
***********************************************************
*                                                         *
* Making Bash-2.03.0-release for a i686 running linux-gnu
*                                                         *
***********************************************************
rm -f shell.o
gcc  -DPROGRAM='"bash"' -DCONF_HOSTTYPE='"i686"' \
  -DCONF_OSTYPE='"linux-gnu"' -DCONF_MACHTYPE='"i686
-pc-linux-gnu"' -DCONF_VENDOR='"pc"' -DSHELL \
  -DHAVE_CONFIG_H  -D_FILE_OFFSET_BITS=64  -I.  -I. -I./
lib -I/usr/local/include -g -O2 -c shell.c
rm -f eval.o
... (compile continues) ...

If the compile yields fatal errors, make terminates and the errors must be addressed before installation. Errors might include problems with the source code (unlikely), missing header files or libraries, and other problems. Error messages will usually be descriptive enough to lead you to the source of the problem.

The final step of installation requires that you are logged in as root to copy the files to the system directories:

# make install
/usr/bin/install -c -m 0755 bash /usr/local/bin/bash
/usr/bin/install -c -m 0755 bashbug /usr/local/bin/bashbug
( cd ./doc ; make  \
        man1dir=/usr/local/man/man1 man1ext=1 \
        man3dir=/usr/local/man/man3 man3ext=3 \
        infodir=/usr/local/info install )
make[1]: Entering directory '/home/ftp/bash-2.03/doc'
test -d /usr/local/man/man1 || /bin/sh ../support/mkdirs /usr/local/man/man1
test -d /usr/local/info || /bin/sh ../support/mkdirs
  /usr/local/info
/usr/bin/install -c -m 644 ./bash.1
  /usr/local/man/man1/bash.1
/usr/bin/install -c -m 644 ./bashbug.1
  /usr/local/man/man1/bashbug.1
/usr/bin/install -c -m 644 ./bashref.info
  /usr/local/info/bash.info
if /bin/sh -c 'install-info --version'
  >/dev/null 2>&1; then \
  install-info --dir-file=/usr/local/info/dir
  /usr/local/info/bash.info; \
else true; fi
make[1]: Leaving directory '/home/ftp/bash-2.03/doc'

The installation places the new version of bash in /usr/local/bin. Now, two working versions of bash are available on the system:

# which bash
/bin/bash
# /bin/bash -version
GNU bash, version 1.14.7(1)
# /usr/local/bin/bash -version
GNU bash, version 2.03.0(1)-release (i686-pc-linux-gnu)
Copyright 1998 Free Software Foundation, Inc.

On the Exam
Familiarize yourself with the acquisition, configuration, compilation, and installation of software from source. Be prepared to answer questions on make and Makefile, the function of the configure script, gzip, and tar.