Hack 51. Display HTTP Headers

See the raw information exchanged between Firefox and web servers.

This hack explains how to inspect the HTTP information that goes back and forth between Firefox and web servers. The simplest way to see web requests is, of course, to examine the server's logs. That strategy doesn't yield every byte of information, though, so here are some alternatives. There are also many tools that can help with cookies [Hack #52] . You might also want Firefox diagnostics [Hack #96] .

5.9.1. Get the Live HTTP Headers Extension

The Live HTTP Headers extension is available at http://livehttpheaders.mozdev.org/. After installation, HTTP requests and response headers are logged to a separate window. Figure 5-7 shows sample output after a single request. Requests are captured to this window only when the window is open.

Figure 5-7. Live HTTP Headers at work on the Google home page

Click any line in the display to select that request. Click the Replay... button to bring up the Live HTTP Replay window. From there, you can send the same request over and over again, with original dates intact. Click the content in the Replay window to place the cursor in the headers. You can modify the headers to include whatever text you require. If the request is a POST (form submission) request, you can modify the form values sent in the POST Content pane at the bottom. All in all, it's a very useful tool.

This Replay functionality reminds us of the difference between GET and POST HTTP requests. GET requests are supposed to be idempotent, meaning that repeating the request has no effect on the server. POST requests, on the other hand, are expected to change the server's state, so repetition is a trickier matter. For example, you can't easily repeat a POST request that adds a unique key to a database table; such a key can be added only once. Comment out the SQL commit statement in the server code to make this easier.

5.9.2. Analyze Headers in the Raw

If you don't like client-side diagnostics [Hack #96], then to see what Firefox sends to the server on Linux/Unix, you can use nc(1) to act as a fake HTTP listener. To do so, make requests as needed to a real HTTP server, and then on the server side (as root), shut down the web server and run nc as a listener for the next request. Then, make another web page request with Firefox. Here's a suitable command line:

nc -l -v -v -p 80

This small script will keep nc running across repeated request attempts, but all requests will be aborted without response because nc is not a web server and closes its opened socket without sending a single byte:

while true; do
  nc -l -v -v -p 80 |
  while read line; do
    echo "$line"
    if [ "$line" = "^M" ]; then  # ^M is a Control-M character
      pkill -x nc
    fi
  done
done

An interesting use of nc is to install it in place of your web proxy server and watch the requests that Firefox makes on startup. Of course, those can also be seen from the proxy server logs. An alternative to nc is to run a full proxy such as squid (http://www.squid-cache.org) in diagnostic mode. Here's a suitable command line:

squid -d 9 -X -N -a 80

Because squid implements HTTP caching policies, its use is not as transparently simple as nc. It also requires a bit of configuration.

For a more transparent solution that shows traffic in both directions, use a packet sniffer or protocol analyzer such as Ethereal (whose output is quite cryptic, however) or else write a tiny HTTP proxy logger using Perl.