14.7 Administrative Tools

Now that we have finished implementing a Python-powered, web-enabled, concurrently accessible report database, and published web pages and scripts that make that database accessible to the cyberworld at large, we can sit back and wait for reports to come in. Or almost; there still is no way for the site owner to view or delete records offline. Moreover, all records are tagged as "not yet verified" on submission, and must somehow be verified or rejected.

This section lists a handful of tersely documented PyErrata scripts that accomplish such tasks. All are Python programs shipped in the top-level AdminTools directory and are assumed to be run from a shell command line on the server (or other machine, after database downloads). They implement simple database content dumps, database backups, and database state-changes and deletions for use by the errata site administrator.

These tasks are infrequent, so not much work has gone into these tools. Frankly, some fall into the domain of "quick and dirty" hackerage and aren't as robust as they could be. For instance, because these scripts bypass the database interface classes and speak directly to the underlying file structures, changes in the underlying file mechanisms will likely break these tools. Also in a more polished future release, these tools might instead sprout GUI- or web-based user interfaces to support over-the-net administration. For now, such extensions are left as exercises for the ambitious reader.

14.7.1 Backup Tools

System backup tools simply spawn the standard Unix tar and gzip command-line programs to copy databases into single compressed files. You could write a shell script for this task too, but Python works just as well, as shown in Examples Example 14-24 and Example 14-25.

Example 14-24. PP2E\Internet\Cgi-Web\PyErrata\AdminTools\backupFiles.py

#!/usr/bin/python
import os
os.system('tar -cvf DbaseFiles.tar ../DbaseFiles')
os.system('gzip DbaseFiles.tar')

Example 14-25. PP2E\Internet\Cgi-Web\PyErrata\AdminTools\backupShelve.py

#!/usr/bin/python
import os
os.system('tar -cvf DbaseShelve.tar ../DbaseShelve')
os.system('gzip DbaseShelve.tar')

14.7.2 Display Tools

The scripts in Examples Example 14-26 and Example 14-27 produce raw dumps of each database structure's contents. Because the databases use pure Python storage mechanisms (pickles, shelves), these scripts can work one level below the published database interface classes; whether they should depends on how much code you're prepared to change when your database model evolves. Apart from printing generated record filenames and shelve keys, there is no reason that these scripts couldn't be made less brittle by instead calling the database classes' loadSortedTable methods. Suggested exercise: do better.

Example 14-26. PP2E\Internet\Cgi-Web\PyErrata\AdminTools\dumpFiles.py

#!/usr/bin/python
import glob, pickle
 
def dump(kind):
    print '\n', kind, '='*60, '\n'
    for file in glob.glob("../DbaseFiles/%s/*.data" % kind):
        print '\n', '-'*60
        print file
        print pickle.load(open(file, 'r'))

dump('errataDB')
dump('commentDB')

Example 14-27. PP2E\Internet\Cgi-Web\PyErrata\AdminTools\dumpShelve.py

#!/usr/bin/python
import shelve
e = shelve.open('../DbaseShelve/errataDB')
c = shelve.open('../DbaseShelve/commentDB')

print '\n', 'Errata', '='*60, '\n'
print e.keys(  )
for k in e.keys(  ): print '\n', k, '-'*60, '\n', e[k]

print '\n', 'Comments', '='*60, '\n'
print c.keys(  )
for k in c.keys(  ): print '\n', k, '-'*60, '\n', c[k]

Running these scripts produces the following sorts of results (truncated at 80 characters to fit in this book). It's not nearly as pretty as the web pages generated for the user in PyErrata, but could be piped to other command-line scripts for further offline analysis and processing. For instance, the dump scripts' output could be sent to a report-generation script that knows nothing of the Web:

[mark@toy .../Internet/Cgi-Web/PyErrata/AdminTools]$ python dumpFiles.py

errataDB ============================================================


------------------------------------------------------------
../DbaseFiles/errataDB/937907956.159-5157.data
{'Page number': '42', 'Type': 'Typo', 'Severity': 'Low', 'Chapter number': '3'...

------------------------------------------------------------
...more...

commentDB ============================================================


------------------------------------------------------------
../DbaseFiles/commentDB/937908410.203-5352.data
{'Submit date': '1999/09/21, 06:06:50', 'Submitter email': 'bob@bob.com',...

------------------------------------------------------------
...more...

[mark@toy .../Internet/Cgi-Web/PyErrata/AdminTools]$ python dumpShelve.py

Errata ============================================================

['938245136.363-20046', '938244808.434-19964']

938245136.363-20046 ------------------------------------------------------------
{'Page number': '256', 'Type': 'Program bug', 'Severity': 'High', 'Chapter nu...

938244808.434-19964 ------------------------------------------------------------
{'Page number': 'various', 'Type': 'Suggestion', 'Printing Date': '', 'Chapte...

Comments ============================================================

['938245187.696-20054']

938245187.696-20054 ------------------------------------------------------------
{'Submit date': '1999/09/25, 03:39:47', 'Submitter email': 'bob@bob.com', 'Re...

14.7.3 Report State-Change Tools

Our last batch of command-line tools allows the site owner to mark reports as verified or rejected and to delete reports altogether. The idea is that someone will occasionally run these scripts offline, as time allows, to change states after investigating reports. And this is the end to our quest for errata automation: the investigation process itself is assumed to require both time and brains.

There are no interfaces in the database's classes for changing existing reports, so these scripts can at least make a case for going below the classes to the physical storage mediums. On the other hand, the classes could be extended to support such update operations too, with interfaces that could also be used by future state-change tools (e.g., web interfaces).

To minimize some redundancy, let's first define state-change functions in a common module listed in Example 14-28, so they may be shared by both the file and shelve scripts.

Example 14-28. PP2E\Internet\Cgi-Web\PyErrata\AdminTools\verifycommon.py

#################################################################
# put common verify code in a shared module for consistency and
# reuse; could also generalize dbase update scan, but this helps
#################################################################

def markAsVerify(report):
    report['Report state'] = 'Verified by author'

def markAsReject(report):
    reason = ''                               # input reject reason text
    while 1:                                  # prepend to original desc
        try:
            line = raw_input('reason>')
        except EOFError:
            break
        reason = reason + line + '\n'
    report['Report state'] = 'Rejected - not a real bug'
    report['Description']  = ('Reject reason: ' + reason + 
                 '\n[Original description=>]\n' + report['Description'])

To process state changes on the file -based database, we simply iterate over all the pickle files in the database directories, as shown in Example 14-29.

Example 14-29. PP2E\Internet\Cgi-Web\PyErrata\AdminTools\verifyFiles.py

#!/usr/bin/python
########################################################
# report state change and deletion operations;
# also need a tool for anonymously publishing reports
# sent by email that are of general interest--for now,
# they can be entered with the submit forms manually;
# this is text-based: the idea is that records can be
# browsed in the errata page first (sort by state to 
# see unverified ones), but an edit gui or web-based
# verification interface might be very useful to add;
########################################################

import glob, pickle, os
from verifycommon import markAsVerify, markAsReject

def analyse(kind):
    for file in glob.glob("../DbaseFiles/%s/*.data" % kind): 
        data = pickle.load(open(file, 'r'))
        if data['Report state'] == 'Not yet verified':
             print data
             if raw_input('Verify?') == 'y':
                 markAsVerify(data)
                 pickle.dump(data, open(file, 'w'))
             elif raw_input('Reject?') == 'y':
                 markAsReject(data)
                 pickle.dump(data, open(file, 'w'))
             elif raw_input('Delete?') == 'y':
                 os.remove(file)  # same as os.unlink
                
print 'Errata...';   analyse('errataDB')
print 'Comments...'; analyse('commentDB')

When run from the command line, the script displays one report's contents at a time and pauses after each to ask if it should be verified, rejected, or deleted. Here is the beginning of one file database verify session, shown with line wrapping so you can see what I see (it's choppy but compact):

[mark@toy .../Internet/Cgi-Web/PyErrata/AdminTools]$ python verifyFiles.py 
Errata...
{'Page number': '12', 'Type': 'Program bug', 'Printing Date': '', 'Chapter numbe
r': '', 'Submit date': '1999/09/21, 06:17:13', 'Report state': 'Not yet verified
', 'Submitter name': 'Lisa Lutz', 'Submitter email': '', 'Description': '1 + 1 =
 2, not 3...\015\012', 'Submit mode': '', 'Part number': '', 'Severity
': 'High'}
Verify?n 
Reject?n 
Delete?n 
{'Page number': '', 'Type': 'Program bug', 'Printing Date': '', 'Chapter number'
: '16', 'Submit date': '1999/09/21, 06:20:22', 'Report state': 'Not yet verified
', 'Submitter name': 'jerry', 'Submitter email': 'http://www.jerry.com', 'Descri
ption': 'Help! I just spilled coffee all over my\015\012computer...\015\012     
     ', 'Submit mode': '', 'Part number': '', 'Severity': 'Unknown'}
Verify?n 
Reject?y 
reason>It's not Python's fault 
reason>(ctrl-d) 
 ...more...

Verifications and rejections change records, but deletions actually remove them from the system. In verifycommon, a report rejection prompts for an explanation and concatenates it to the original description. Deletions delete the associated file with os.remove; this feature may come in handy if the system is ever abused by a frivolous user (including me, while writing examples for this book). The shelve -based version of the verify script looks and feels similar, but deals in shelves instead of flat files, as shown in Example 14-30.

Example 14-30. PP2E\Internet\Cgi-Web\PyErrata\AdminTools\verifyShelve.py

#!/usr/bin/python
########################################################
# like verifyFiles.py, but do it to shelves; 
# caveats: we should really obtain a lock before shelve 
# updates here, and there is some scan logic redundancy
########################################################

import shelve
from verifycommon import markAsVerify, markAsReject

def analyse(dbase):
    for k in dbase.keys(  ): 
        data = dbase[k]
        if data['Report state'] == 'Not yet verified':
             print data
             if raw_input('Verify?') == 'y':
                 markAsVerify(data)
                 dbase[k] = data
             elif raw_input('Reject?') == 'y':
                 markAsReject(data)
                 dbase[k] = data
             elif raw_input('Delete?') == 'y':
                 del dbase[k]

print 'Errata...';    analyse(shelve.open('../DbaseShelve/errataDB'))
print 'Comments...';  analyse(shelve.open('../DbaseShelve/commentDB'))

Note that the verifycommon module helps ensure that records are marked consistently and avoids some redundancy. However, the file and shelve verify scripts still look very similar; it might be better to further generalize the notion of database update scans by moving this logic into the storage-specific database interface classes shown earlier.

Short of doing so, there is not much we can do about the scan-logic redundancy or storage-structure dependencies of the file and shelve verify scripts. The existing load-list database class methods won't help, because they don't provide the generated filename and shelve key details we need to rewrite records here. To make the administrative tools more robust, some database class redesign would probably be in order -- which seems as good a segue to the next section as any.