30.7 TCP Preforked Server, File Locking Around accept

The implementation that we just described for 4.4BSD, which allows multiple processes to call accept on the same listening descriptor, works only with Berkeley-derived kernels that implement accept within the kernel. System V kernels, which implement accept as a library function, may not allow this. Indeed, if we run the server from the previous section on such a system, soon after the clients start connecting to the server, a call to accept in one of the children returns EPROTO, which means a protocol error.

The reasons for this problem with the SVR4 library version of accept arise from the STREAMS implementation (Chapter 31) and the fact that the library accept is not an atomic operation. Solaris fixes this, but the problem still exists in most other SVR4 implementations.

The solution is for the application to place a lock of some form around the call to accept, so that only one process at a time is blocked in the call to accept. The remaining children will be blocked trying to obtain the lock.

There are various ways to provide this locking around the call to accept, as we described in the second volume of this series. In this section, we will use POSIX file locking with the fcntl function.

The only change to the main function (Figure 30.9) is adding a call to our my_lock_init function before the loop that creates the children.

+    my_lock_init("/tmp/lock.XXXXXX"); /* one lock file for all children */
     for (i = 0; i < nchildren; i++)
         pids[i] = child_make(i, listenfd, addrlen); /* parent returns */

The child_make function remains the same as Figure 30.11. The only change to our child_main function (Figure 30.12) is to obtain a lock before calling accept and release the lock after accept returns.

     for ( ; ; ) {
         clilen = addrlen;
+        my_lock_wait();
         connfd = Accept(listenfd, cliaddr, &clilen);
+        my_lock_release();

         web_child(connfd);        /* process request */
         Close(connfd);

Figure 30.16 shows our my_lock_init function, which uses POSIX file locking.

Figure 30.16 my_lock_init function using POSIX file locking.

server/lock_fcntl.c

 1 #include    "unp.h"

 2 static struct flock lock_it, unlock_it;
 3 static int lock_fd = -1;
 4                     /* fcntl() will fail if my_lock_init() not called */

 5 void
 6 my_lock_init(char *pathname)
 7 {
 8     char     lock_file[1024];

 9         /* must copy caller's string, in case it's a constant */
10     strncpy(lock_file, pathname, sizeof(lock_file));
11     lock_fd = Mkstemp(lock_file);

12     Unlink(lock_file);          /* but lock_fd remains open */

13     lock_it.l_type = F_WRLCK;
14     lock_it.l_whence = SEEK_SET;
15     lock_it.l_start = 0;
16     lock_it.l_len = 0;

17     unlock_it.l_type = F_UNLCK;
18     unlock_it.l_whence = SEEK_SET;
19     unlock_it.l_start = 0;
20     unlock_it.l_len = 0;
21 }

9–12 The caller specifies a pathname template as the argument to my_lock_init, and the mktemp function creates a unique pathname based on this template. A file is then created with this pathname and immediately unlinked. By removing the pathname from the directory, if the program crashes, the file completely disappears. But as long as one or more processes have the file open (i.e., the file's reference count is greater than 0), the file itself is not removed. (This is the fundamental difference between removing a pathname from a directory and closing an open file.)

13–20 Two flock structures are initialized: one to lock the file and one to unlock the file. The range of the file that is locked starts at byte offset 0 (a l_whence of SEEK_SET with l_start set to 0). Since l_len is set to 0, this specifies that the entire file is locked. We never write anything to the file (its length is always 0), but that is fine. The advisory lock is still handled correctly by the kernel.

It may be tempting to initialize these structures using

static struct flock lock_it = { F_WRLCK, 0, 0, 0, 0 };
static struct flock unlock_it = { F_UNLCK, 0, 0, 0, 0 };

but there are two problems. First, there is no guarantee that the constant SEEK_SET is 0. But more importantly, there is no guarantee by POSIX as to the order of the members in the structure. The l_type member may be the first one in the structure, but not on all systems. All POSIX guarantees is that the members that POSIX requires are present in the structure. POSIX does not guarantee the order of the members, and POSIX also allows additional, non-POSIX members to be in the structure. Therefore, initializing a structure to anything other than all zeros should always be done by actual C code, and not by an initializer when the structure is allocated.

An exception to this rule is when the structure initializer is provided by the implementation. For example, when initializing a Pthread mutex lock in Chapter 26, we wrote

pthread_mutex_t mlock = PTHREAD_MUTEX_INITIALIZER;

The pthread_mutex_t datatype is often a structure, but the initializer is provided by the implementation and can differ from one implementation to the next.

Figure 30.17 shows the two functions that lock and unlock the file. These are just calls to fcntl, using the structures that were initialized in Figure 30.16.

This new version of our preforked server now works on SVR4 systems by assuring that only one child process at a time is blocked in the call to accept. Comparing rows 2 and 3 in Figure 30.1 shows that this type of locking adds to the server's process control CPU time.

The Apache Web server, http://www.apache.org, preforks its children and then uses either the technique in the previous section (all children blocked in the call to accept), if the implementation allows this, or file locking around the accept.

Effect of Too Many Children

We can check this version to see if the same thundering herd problem exists, which we described in the previous section. We check by increasing the number of (unneeded) children and noticing that the timing results get worse proportionally.

Figure 30.17 my_lock_wait and my_lock_release functions using fcntl.

server/lock_fcntl.c

22 void
23 my_lock_wait()
24 {
25     int     rc;

26     while ( (rc = fcntl(lock_fd, F_SETLKW, &lock_it)) < 0) {
27         if (errno == EINTR)
28             continue;
29         else
30             err_sys("fcntl error for my_lock_wait");
31     }
32 }

33 void
34 my_lock_release()
35 {
36     if (fcntl(lock_fd, F_SETLKW, &unlock_it) < 0)
37         err_sys("fcntl error for my_lock_release");
38 }

Distribution of Connections to the Children

We can examine the distribution of the clients to the pool of available children by using the function we described with Figure 30.14. Figure 30.2 shows the result. The OS distributes the file locks uniformly to the waiting processes (and this behavior was uniform across several operating systems we tested).