Hack 95. Loop Around the 10-Result Limit

If you want more than 10 results, you'll have to loop.

The Google API returns only 10 results per query, plenty for some queries, but for most applications, 10 results barely scratches the surface. If you want more than 10 results, you're going to have to loop, querying for the next set of 10 each time. The first query returns the top 10. The next, 11 through 20. And so forth.

This hack builds on the basic Perl script googly.pl [Hack #92] that we showed you in the previous hack. To get at more than the top 10 results, no matter the programming language you're using, you'll have to create a loop.

Bear in mind that each and every query counts against your daily allotment. Loop three times and you've used up three queries. Ten, and you're down 10. While this doesn't seem like much given your quota of 1,000 queries a day, you'd be surprised how quickly you can reach the bottom of the cookie jar without knowing where they all went.

9.12.1. The Code

Save the following code to a text file named looply.pl. Again, remember to replace insert key here with your Google API key, as explained in "Using Your Google API Key" earlier in this chapter.

In addition to the Google API Developer's Kit, you'll need the SOAP::Lite Perl module [Hack #93] installed before running this hack.

The alterations to the previous hack needed to support looping through more than the first 10 results are called out in bold.

#!/usr/local/bin/perl

# looply.pl

# A typical Google Web API Perl script.

# Usage: perl looply.pl <query>

     

# Your Google API developer's key.

my $google_key='insert key here ';

     

# Location of the GoogleSearch WSDL file.

my $google_wdsl = "./GoogleSearch.wsdl";

     

# Number of times to loop, retrieving 10 results at a time.

my $loops = 3; # 3 loops x 10 results per loop = top 30 results

     

use strict;

     

# Use the SOAP::Lite Perl module.

use SOAP::Lite;

     

# Take the query from the command line.

my $query = shift @ARGV or die "Usage: perl looply.pl  <query>\n";

     

# Create a new SOAP::Lite instance, feeding it GoogleSearch.wsdl.

     

my $google_search = SOAP::Lite->service("file:$google_wdsl");

     

# Keep track of result number.

my $number = 0;

     

for (my $offset = 0; $offset <= ($loops-1)*10; $offset += 10) {

 # Query Google.

 my $results = $google_search -> 

  doGoogleSearch(

    $google_key, $query, $offset , 10, "false", "", "false",

    "", "latin1", "latin1"

  );

 

 # No sense continuing unless there are more results.

 last unless @{$results->{resultElements}};

 

 # Loop through the results.

 foreach my $result (@{$results->{'resultElements'}}) {

 

  # Print out the main bits of each result.

  print

   join "\n",  

   ++$number,

   $result->{title} || "no title",

   $result->{URL},

   $result->{snippet} || 'no snippet',

   "\n";

 

 }

}

Notice that the script tells Google which set of 10 results it's after by passing an offset ($offset). The offset is increased by 10 each time ($offset += 10).

9.12.2. Running the Script

Run this script from the command line ["How to Run the Hacks" in the Preface], passing it your Google search:

$ perl looply.pl  "query keywords"

9.12.3. The Results

Here's a sample run. The first attempt doesn't specify a query and so triggers a usage message and doesn't go any further. The second searches for learning perl and prints out the results. Output is just the same as for the googly.pl script in the prior hack, but now the number of results you net is limited only by your specified loop count (in this case 3, netting 3 10 or 30 results).

$ perl googly.pl 

Usage: perl looply.pl <query>

% perl looply.pl "learning perl"

1

oreilly.com -- Online Catalog: Learning Perl , 3rd Edition

http://www.oreilly.com/catalog/lperl3/

... Learning Perl , 3rd Edition Making Easy Things 

Easy and Hard Things Possible By Randal<br> L. Schwartz, Tom Phoenix 

3rd Edition July 2001 0-596-00132-0, Order Number ... 

 

...

29

Intro to Perl  for CGI 

http://hotwired.lycos.com/webmonkey/98/47/index2a.html

...  Some people feel that the benefits of learning  

Perl  scripting are few.<br> But ...  part. That's right. 

Learning Perl  is just like being a cop. ...    

30

WebDeveloper.com ®: Where Web Developers and Designers Learn How ... 

http://www.webdeveloper.com/reviews/book6.html

...  Registration CreditCard Processing Compare Prices. 

Learning Perl .  Learning <br>  Perl , 2nd Edition.  

Publisher: O'Reilly Author: Randal Schwartz 

...

9.12.4. Hacking the Hack

Alter the value assigned to the $loops variable to change the number of results. For instance, to loop 9 times and grab the top 90 results, change things like so:

# Number of times to loop, retrieving 10 results at a time.

my $loops = 9; # 9 loops x 10 results per loop = top 90 results

< Day Day Up >