7.7. Lightweight Protocols

XML and HTTP are all very well but are not the solution to every problem. Sometimes a lightweight protocol and format are the best solution to your particular problems. Any protocol or format for which you need to call out to a complex library for handling is not lightweight. Let's look at a number of the problems with using XML and HTTP.

7.7.1. Memory Usage

The memory usage of our applications is often an issue. Invoking an XML parser or HTTP library isn't a zero-cost action, especially when you need to do it thousands of times per minute. If we can use simple protocols and formatting that we can handle completely in a few hundred lines of code, then we're using less memory.

In the same way, large documents over HTTP can be a waste of memory, depending on your HTTP library. If you're sending a 1 MB binary file over HTTP, then your library will probably receive the entire file, create a structure in memory representing the response, and then pass it along to you. The entire file has been received before you can start to do anything. A protocol where you can receive data as it's sent would allow you to write the data straight to disk as it arrives.

7.7.2. Network Speed

The speed and amount of data sent over the network can also be an issue. If you're issuing a request many times a second that are comprised of a single number, the few bytes used to represent that data won't be the only thing sent. Wrapping the number in XML might add another 100 bytes (for the XML prolog and root element tags). Adding the HTTP request headers and the response saying the data was received might add another few hundred bytes. Our sending of the number 20 now transfers the following data across the wire:

POST /counter/ HTTP/1.0
Content-Length: 40
Content-Type: text/xml; charset=UTF-8
<?xml version="1.0"?>
<value>20</value>
HTTP/1.0 200 OK
Date: Thu, 06 Oct 2005 23:56:01 GMT
Content-Length: 65
Content-Type: text/xml; charset=UTF-8
<?xml version="1.0"?>
<status>ok</status>

At over 200 bytes, this is in stark contrast to the same exchange using a dedicated protocol where we simply send the data and close the connection (or send an EOF signal of some sort):

20

This difference in size can start to really make a difference for requests that are small in size and repeated often. Multiplying the data size by a factor of one hundred is no small addition.

7.7.3. Parsing Speed

When we're parsing XML, we need to read and examine each character one by one, checking for special escape sequences. In a data format that pre-declares the length of a piece of data, we can just slurp in the specified number of bytes, ignoring the contents. This becomes important when we're exchanging huge pieces of data, such as files. Embedding a large file in an XML document means that we'll need to check every single byte of it as it comes in, looking for an ending tag.

Before we even get the XML delivered to us, the same can be true at the HTTP level. If the content length isn't specified for a body segment, then we need to constantly check if we've reached the end of the body segment or hit a multipart boundary. This can be averted by always specifying a content length, but the same process always applies to parsing headersyou need to carefully read each character until you come across the CR LF CR LF sequence delimiting the body.

7.7.4. Writing Speed

It's not just the reading speed we need to concern ourselves with eitherfor large binary documents, writing XML can be a slow process, too. We can't use CDATA segments for including binary files lest the file contain the ]]> escape sequence or contain some of the byte sequences that are either invalid XML (code points below U+0020, excluding tab, carriage return, and linefeed) or invalid in the current encoding (if you're using UTF-8). We have to add binary files as PCDATA segments, but this means going through the files byte by byte, escaping the four special characters, escaping the characters not allowed in XML, and pandering to the encoding scheme. For a large file, this can be a long process, and we won't know the end data size until we're done (since each escaped special character expands the data). Because we won't know the size, we'll have to either skip the HTTP Content-Length header (which makes reading at the other end slower), or we'll have to escape the whole file before we can start sending it (which means we'll need more memory to buffer the escaped version, or even disk space depending on the size). The safest method in this case is base64 encoding the data, which produces a chunk of data of known size (4 bytes for every 3 in the source data, rounding up). The downside to base64 encoding is that we need to process the entire file, performing bitwise operations on every byte.

7.7.5. Downsides

So we've seen a few compelling reason to avoid using HTTP and XML in situations where we either have many tiny messages, or some very large messages. But creating our own lightweight protocol isn't all roses either; there are some serious issues to consider before starting down that path.

Designing your own protocol tends to mean that you control both ends of the exchange. Within your own closed system this is fine, but if you ever need to interoperate with other services, then using a proprietary protocol isn't a great thing. Anybody else using your services is going to have to write their own handling library from scratch or use yours if it's written in a suitable language. You'll have to document your protocol and perhaps start adding features for usage patterns you hadn't previously intended.

This road leads to creating a much larger protocol with all the problems of existing ones, but with the added problem that it wasn't designed as a coherent whole from the beginning. This can all be avoided by either not using the protocol external to your own services or by not extending it to facilitate further functionality.

Developing your own protocol takes both time and resources. You'll first need to design and then implement it, which may take a significant portion of time. But the real time suck comes later whenever you experience a problem. For every problem with the remote services, you'll need to consider that there is a problem with the protocol. Even if the protocol is well defined, the implementation may not be. Your protocol library may contain small issues that you don't come across for a long time, but every time you do see a problem, you need to check things are working fine at the protocol level.

This problem can also occur with established protocols thoughyou will always have to consider that there's a bug in your HTTP library, or a problem creating or parsing your XML. The difference is that the HTTP and XML libraries have been used by thousands of people for many yearsthousands of people testing the code to make sure it works before you touch it.

It's not all bad news though. Much of this can be avoided by creating very specialized, very simple lightweight protocols.

7.7.6. Rolling Your Own

Within the Flickr system, we had a middleware layer that dealt with file storage. When a file needed to be stored, we would pass it to the middleware that would then choose where to write it, perform the write operation, and notify us of where the file had been stored. The protocol for communicating needed to be fast to read and write, since we were sending a lot of large files.

There were already existing file transfer protocols available, but all of them were ruled out for various reasons. FTP uses two sockets, NFS and SMB require a persistent connection, SCP requires encrypting the stream, and so on.

We designed our own protocol that consisted of two basic building blocksa string and a file. A string was specified by a leading byte giving the length in bytes (from 0 to 255), followed by the UTF-8-encoded representation of the string. The string "hello world" is encoded as byte 11, followed by the 11 bytes of the string. A file was specified by a string object, in which the contents of the string were the file length in bytes, followed by the file contents. A 1 MB binary file would be encoded the byte 7, the string "1048576," and then 1048576 bytes of file contents.

With these basic building blocks, the protocol was then defined as a sequence of strings and files. The process of storing a file was to first send the string "STORE," then a string containing the number of files. For each file, a string containing the filename, then the file itself were sent. Once all files were sent, the remote server responded with either the string "OK" or the string "FAILED." When the string "OK" was sent, it was followed by a string containing the volume onto which the file set was written. When the string "FAILED" was returned, a following string contained the failure message.

The basis of the design was that no data was sent without first sending its length. The length of strings was limited to 255 bytes so that we would know in advance that we only had to read 1 byte to find the length (a concept borrowed from Pascal's strings). When we know how long the contents of a file will be, we can stream it across the socket without having to encode or decode it, increasing both reading and writing time.

Because of the simplicity of the protocol, the full library code weighs in at around 600 lines of PHP. This includes opening and closing sockets, hot failover to redundant servers, and safe read and writes, which take up most of the 600 lines. On top of this, the library provides support for several possible file operationsstore, replace, and deleteall built on top of the protocol primitives.

Developing your own protocol is, for most tasks, a bad idea. But for certain domain-specific tasks with small functionality sets, where there are compelling reasons to avoid established protocols (such as speed concerns), creating your own protocol can be a big win.

It's worth noting that prior to creating our own protocol, Flickr first used NFS and later SCP to transfer files around the system. It was only when we got to the point of coming up against problems with these protocols that we moved toward creating our own. As with most components of an application, a good first rule of thumb is to avoid doing anything complicated or time-consuming, and instead build on the existing work of others.