11.1. SerializationPython supplies a number of modules dealing with I/O operations that serialize (save) entire Python objects to various kinds of byte streams and deserialize (load and recreate) Python objects back from such streams. Serialization is also known as marshaling. 11.1.1. The marshal ModuleThe marshal module supports the specific serialization tasks needed to save and reload compiled Python files (.pyc and .pyo). marshal handles only fundamental built-in data types: None, numbers (int, long, float, complex), strings (plain and Unicode), code objects, and built-in containers (tuple, list, dict) whose items are instances of elementary types. marshal does not handle sets nor user-defined types and classes. marshal is faster than other serialization modules, and is the one such module that supports code objects. Module marshal supplies the following functions.
11.1.1.1. A marshaling exampleSay you need to read several text files, whose names are given as your program's arguments, recording where each word appears in the files. What you need to record for each word is a list of (filename, line-number) pairs. The following example uses marshal to encode lists of (filename, line-number) pairs as strings and store them in a DBM-like file (as covered in "DBM Modules" on page 285). Since these lists contain tuples, each containing a string and a number, they are within marshal's abilities to serialize. import fileinput, marshal, anydbm wordPos = {} for line in fileinput.input( ): pos = fileinput.filename( ), fileinput.filelineno( ) for word in line.split( ): wordPos.setdefault(word,[]).append(pos) dbmOut = anydbm.open('indexfilem', 'n') for word in wordPos: dbmOut[word] = marshal.dumps(wordPos[word]) dbmOut.close( ) We also need marshal to read back the data stored in the DBM-like file indexfilem, as shown in the following example: import sys, marshal, anydbm, linecache dbmIn = anydbm.open('indexfilem') for word in sys.argv[1:]: if not dbmIn.has_key(word): sys.stderr.write('Word %r not found in index file\n' % word) continue places = marshal.loads(dbmIn[word]) for fname, lineno in places: print "Word %r occurs in line %s of file %s:" % (word,lineno,fname) print linecache.getline(fname, lineno), 11.1.2. The pickle and cPickle ModulesThe pickle and cPickle modules supply factory functions, named Pickler and Unpickler, to generate objects that wrap file-like objects and supply serialization mechanisms. Serializing and deserializing via these modules is also known as pickling and unpickling. The difference between the modules is that, in pickle, Pickler and Unpickler are classes, so you can inherit from these classes to create customized serializer objects, overriding methods as needed. In cPickle, on the other hand, Pickler and Unpickler are factory functions that generate instances of non-subclassable types, not classes. Performance is much better with cPickle, but inheritance is not feasible. In the rest of this section, I'll be talking about module pickle, but everything applies to cPickle too. Note that unpickling from an untrusted data source is a security risk; an attacker could exploit this to execute arbitrary code. Don't unpickle untrusted data! Serialization shares some of the issues of deep copying, covered in deepcopy on page 172. Module pickle deals with these issues in much the same way as module copy does. Serialization, like deep copying, implies a recursive walk over a directed graph of references. pickle preserves the graph's shape when the same object is encountered more than once: the object is serialized only the first time, and other occurrences of the same object serialize references to a single copy. pickle also correctly serializes graphs with reference cycles. However, this means that if a mutable object o is serialized more than once to the same Pickler instance p, any changes to o after the first serialization of o to p are not saved. For clarity and simplicity, avoid altering objects that are being serialized while serialization to a Pickler instance is in progress. pickle can serialize in an ASCII format or in either of two compact binary ones. The ASCII format is the default, for backward compatibility, but you should normally request binary format 2, which saves time and storage space. When you reload objects, pickle transparently recognizes and uses any format. I recommend you always specify binary format 2: the size and speed savings can be substantial, and binary format has basically no downside except loss of compatibility with very old versions of Python. pickle serializes classes and functions by name, not by value. pickle can therefore deserialize a class or function only by importing it from the same module where the class or function was found when pickle serialized it. In particular, pickle can serialize and deserialize classes and functions only if they are top-level names for their module (i.e., attributes of their module). For example, consider the following: def adder(augend): def inner(addend, augend=augend): return addend+augend return inner plus5 = adder(5) This code binds a closure to name plus5 (as covered in "Nested functions and nested scopes" on page 77)a nested function inner plus an appropriate nested scope. Therefore, trying to pickle plus5 raises a pickle.PicklingError exception: a function can be pickled only when it is top-level, and function inner, whose closure is bound to name plus5 in this code, is not top-level but rather nested inside function adder. Similar issues apply to all pickling of nested functions and nested classes (i.e., classes that are not top-level). 11.1.2.1. Functions of pickle and cPickleModules pickle and cPickle expose the following functions.
11.1.2.2. A pickling exampleThe following example handles the same task as the marshal example shown earlier but uses cPickle instead of marshal to encode lists of (filename, line-number) pairs as strings: import fileinput, cPickle, anydbm wordPos = { } for line in fileinput.input( ): pos = fileinput.filename( ), fileinput.filelineno( ) for word in line.split( ): wordPos.setdefault(word,[ ]).append(pos) dbmOut = anydbm.open('indexfilep','n') for word in wordPos: dbmOut[word] = cPickle.dumps(wordPos[word], 1) dbmOut.close( ) We can use either cPickle or pickle to read back the data stored to the DBM-like file indexfilep, as shown in the following example: import sys, cPickle, anydbm, linecache dbmIn = anydbm.open('indexfilep') for word in sys.argv[1:]: if not dbmIn.has_key(word): sys.stderr.write('Word %r not found in index file\n' % word) continue places = cPickle.loads(dbmIn[word]) for fname, lineno in places: print "Word %r occurs in line %s of file %s:" % (word,lineno,fname) print linecache.getline(fname, lineno), 11.1.2.3. Pickling instancesIn order for pickle to reload an instance x, pickle must be able to import x's class from the same module in which the class was defined when pickle saved the instance. Here is how pickle saves the state of instance object x of class T and later reloads the saved state into a new instance y of T (the first step of the reloading is always to make a new empty instance y of T, except where I explicitly say otherwise in the following):
All the items in the d or t object that pickle saves and reloads (normally a dictionary or tuple) must in turn be instances of types suitable for pickling and unpickling (i.e., pickleable objects), and the procedure just outlined may be repeated recursively, if necessary, until pickle reaches primitive pickleable built-in types (dictionaries, tuples, lists, sets, numbers, strings, etc.). As mentioned in "The copy Module" on page 172, special methods _ _getinitargs_ _, _ _getnewargs_ _, _ _getstate_ _, and _ _setstate_ _ also control the way instance objects are copied and deep-copied. If a new-style class defines _ _slots_ _, and therefore its instances do not have a _ _dict_ _, pickle does it best to save and restore a dictionary equivalent to the names and values of the slots. However, such a new-style class should define _ _getstate_ _ and _ _setstate_ _; otherwise, its instances may not be correctly pickleable and copy-able through such best-effort endeavors. 11.1.2.4. Pickling customization with the copy_reg moduleYou can control how pickle serializes and deserializes objects of an arbitrary type (or new-style class) by registering factory and reduction functions with module copy_reg. This is particularly, though not exclusively, useful when you define a type in a C-coded Python extension. Module copy_reg supplies the following functions.
11.1.3. The shelve ModuleThe shelve module orchestrates modules cPickle (or pickle, when cPickle is not available in the current Python installation), cStringIO (or StringIO, when cStringIO is not available in the current Python installation), and anydbm (and its underlying modules for access to DBM-like archive files, as discussed in "DBM Modules" on page 285) in order to provide a simple, lightweight persistence mechanism. shelve supplies a function open that is polymorphic to anydbm.open. The mapping object s returned by shelve.open is less limited than the mapping object a returned by anydbm.open. a's keys and values must be strings. s's keys must also be strings, but s's values may be of any pickleable types or classes. pickle customizations (e.g., copy_reg, _ _getinitargs_ _, _ _getstate_ _, and _ _setstate_ _) also apply to shelve, since shelve delegates serialization to pickle. Beware of a subtle trap when you use shelve and mutable objects. When you operate on a mutable object held in a shelf, the changes don't "take" unless you assign the changed object back to the same index. For example: import shelve s = shelve.open('data') s['akey'] = range(4) print s['akey'] # prints: [0, 1, 2, 3] s['akey'].append('moreover') # trying direct mutation print s['akey'] # doesn't take; prints: [0, 1, 2, 3] x = s['akey'] # fetch the object x.append('moreover') # perform mutation s['akey'] = x # store the object back print s['akey'] # now it takes, prints: [0, 1, 2, 3, 'moreover'] You can finesse this issue by passing named argument writeback=True when you call shelve.open, but beware: if you do pass that argument, you may seriously impair the performance of your program. 11.1.3.1. A shelving exampleThe following example handles the same task as the earlier pickling and marshaling examples, but uses shelve to persist lists of (filename, line-number) pairs: import fileinput, shelve wordPos = { } for line in fileinput.input( ): pos = fileinput.filename( ), fileinput.filelineno( ) for word in line.split( ): wordPos.setdefault(word,[ ]).append(pos) shOut = shelve.open('indexfiles','n') for word in wordPos: shOut[word] = wordPos[word] shOut.close( ) We must use shelve to read back the data stored to the DBM-like file indexfiles, as shown in the following example: import sys, shelve, linecache shIn = shelve.open('indexfiles') for word in sys.argv[1:]: if not shIn.has_key(word): sys.stderr.write('Word %r not found in index file\n' % word) continue places = shIn[word] for fname, lineno in places: print "Word %r occurs in line %s of file %s:" % (word,lineno,fname) print linecache.getline(fname, lineno), These two examples are the simplest and most direct of the various equivalent pairs of examples shown throughout this section. This reflects the fact that module shelve is higher-level than the modules used in previous examples. ![]() |