#61 Extracting URLs from a Web Page

A straightforward shell script application of lynx is to extract a list of URLs on a given web page, which can be quite helpful in a variety of situations.

The Code

#!/bin/sh

# getlinks - Given a URL, returns all of its internal and
#   external links.
if [ $# -eq 0 ] ; then
  echo "Usage: $0 [-d|-i|-x] url"  >&2
  echo "-d=domains only, -i=internal refs only, -x=external only" >&2
  exit 1
fi

if [ $# -gt 1 ] ; then
  case "$1" in
    -d) lastcmd="cut -d/ -f3 | sort | uniq"
        shift
        ;;
    -i) basedomain="http://$(echo $2 | cut -d/ -f3)/"
        lastcmd="grep \"^$basedomain\" | sed \"s|$basedomain||g\" | sort | uniq"
        shift
        ;;
    -x) basedomain="http://$(echo $2 | cut -d/ -f3)/"
        lastcmd="grep -v \"^$basedomain\" | sort | uniq"
        shift
        ;;
     *) echo "$0: unknown option specified: $1" >&2; exit 1
  esac
else
  lastcmd="sort | uniq"
fi

lynx -dump "$1" | \
  sed -n '/^References$/,$p' | \
  grep -E '[[:digit:]]+\.' | \
  awk '{print $2}' | \
  cut -d\? -f1 | \
  eval $lastcmd

exit 0

How It Works

When displaying a page, lynx shows the text of the page, formatted as best it can, followed by a list of all hypertext references, or links, found on that page. This script simply extracts just the links by using a sed invocation to print everything after the "References" string in the web page text, and then processes the list of links as needed based on the user-specified flags.

The one interesting technique demonstrated by this script is the way the variable lastcmd is set to filter the list of links that it extracts according to the flags specified by the user. Once lastcmd is set, the amazingly handy eval command is used to force the shell to interpret the content of the variable as if it were a command, not a variable.

Running the Script

By default, this script outputs a list of all links found on the specified web page, not just those that are prefaced with http:. There are three optional command flags that can be specified to change the results, however: -d produces just the domain names of all matching URLs, -i produces a list of just the internal references (that is, those references that are found on the same server as the current page), and -x produces just the external references, those URLs that point to a different server.

The Results

A simple request is a list of all links on a specified website home page:

$ getlinks http://www.trivial.net/
http://www.intuitive.com/
http://www.trivial.net/kudos/index.html
http://www.trivial.net/trivial.cgi
mailto:nerds@trivial.net

Another possibility is to request a list of all domain names referenced at a specific site. This time let's first use the standard Unix tool wc to check how many links are found overall:

$ getlinks http://www.amazon.com/ | wc -l
     136

Amazon has 136 links on its home page. Impressive! Now, how many different domains does that represent? Let's generate a full list with the -d flag:

$ getlinks -d http://www.amazon.com/
s1.amazon.com
www.amazon.com

As you can see, Amazon doesn't tend to point anywhere else. Other sites are different, of course. As an example, here's a list of all external links in my weblog:

$ getlinks -x http://www.intuitive.com/blog/
LYNXIMGMAP:http://www.intuitive.com/blog/#headermap
http://blogarama.com/in.php
http://blogdex.media.mit.edu/
http://booktalk.intuitive.com/
http://chris.pirillo.com/
http://cortana.typepad.com/rta/
http://dylan.tweney.com/
http://fx.crewtags.com/blog/
http://geourl.org/near/
http://hosting.verio.com/index.php/vps.html
http://imajes.info/
http://jake.iowageek.com/
http://myst-technology.com/mysmartchannels/public/blog/214/
http://smattering.org/dryheat/
http://www.101publicrelations.com/blog/
http://www.APparenting.com/
http://www.backupbrain.com/
http://www.bloghop.com/
http://www.bloghop.com/ratemyblog.htm
http://www.blogphiles.com/webring.shtml
http://www.blogshares.com/blogs.php
http://www.blogstreet.com/blogsqlbin/home.cgi
http://www.blogwise.com/
http://www.gnome-girl.com/
http://www.google.com/search/
http://www.icq.com/
http://www.infoworld.com/
http://www.mail2web.com/
http://www.movabletype.org/
http://www.nikonusa.com/usa_product/product.jsp
http://www.onlinetonight.net/ethos/
http://www.procmail.org/
http://www.ringsurf.com/netring/
http://www.spamassassin.org/
http://www.tryingreallyhard.com/
http://www.yahoo.com/r/p2

Hacking the Script

You can see where getlinks could be quite useful as a site analysis tool. Stay tuned: Script #77, checklinks, is a logical follow-on to this script, allowing a quick link check to ensure that all hypertext references on a site are valid.