Interoperability: The "Holy Grail" of Web Services

The promise of Web services lies in the ability to exchange data and functionality among partners using standards-based messages—and arbitrary technological infrastructures. This not only buys you the ability to talk to many new partners, but it also allows both you and your partners to avoid "lock in"to a particular development platform. If you've defined your outward-facing interface to the world in terms of XML, SOAP, WSDL, and UDDI, you can feel free to switch from Java to .NET and back again (not that you would be likely to really do so!) without ever disturbing your ongoing Web service–enabled partner relationships. Ideally, you should also be cleanly insulated from worrying about achieving interoperability with different implementations of these protocols. Sounds great, doesn't it? Well, in reality, things aren't always quite so easy—but they are getting there.

The SOAP 1.1 specification is somewhat vague about several issues and provides a lot of flexibility (too much, many say) in the way you choose to do things. For instance:

• Section 5 of the spec (which, if you'll recall from Chapter 3, "Simple Object Access Protocol (SOAP)," is the data encoding section) states that null values can be represented either by the omission of the XML element representing that value, or by an XML element with the xsi:nil attribute present and set to true (this is the ninth rule in the "rules for encoding" list in section 5.1). This can be a problem in several ways—an example would be a toolkit that doesn't care about the names of method arguments, but uses ordinal positioning of the XML-encoded arguments to match them up with the real arguments. If this toolkit has published a service, and another toolkit decides to simply omit arguments whose values are nil when making a request, we have a potential problem.

• The spec is not clear (in section 7) about how to represent a Remote Procedure Call (RPC) response with a void return type and no output parameters. This could, for instance, be an empty SOAP envelope, an empty SOAP method response element, or even an HTTP 204 (No Response) with no SOAP envelope. If faults are generated when toolkits don't see the version they expect, this could cause some trouble.

• It's not apparent how to decide exactly which body entry in an RPC-encoded message is the main one that would be processed as an RPC method call. (There can be multiple body entries in the case of multi-ref serialization, for instance.)

• It is possible with SOAP to make messages self-describing, in which case metadata about the contents of the message (for example, the xsi:type attribute on RPC parameters) is embedded within the message itself. It is equally possible to assume that such metadata is conveyed out-of-band in a Web Services Description Language (WSDL) document or via some other means. The problem is that when you're talking to new parties, it might be hard to know what is appropriate, especially because some toolkits rely heavily on the xsi:type element's presence in messages on the wire.

• The SOAPAction HTTP header is somewhat vaguely defined in section 6.1.1 of the spec as representing the "intent" of the message. Some packages don't use this value at all, and others absolutely require it for dispatching to the correct service. SOAPAction has been a classic source of interoperability problems.

These issues, and others like them, can provide some serious challenges when you are trying to create a world where any SOAP implementation can talk to any other with a reasonable chance of success. If we decide to use xsi:nil="true" for nulls, and you decide to omit elements, there is a significant chance that we might encounter some problems communicating. How can we begin to resolve these issues and move towards a maximally interoperable world? Luckily, there are at least two different answers to that question—and both are making great progress in parallel.

The Soapbuilders Community

Clearly, two things need to occur if different implementations hope to truly interoperate. The first is that the developers of these implementations need to be able to talk to each other in order to iron out problems. The second is that there need to be some sort of compatibility tests that can be used to judge whether a particular implementation can successfully talk to another.

A solution to both of these issues appeared in the form of the soapbuilders list, a very active community of SOAP developers on Yahoo. The list was started in January 2001 by Tony Hong of Xmethods as a place for SOAP implementers to congregate and a forum in which to discuss interoperability issues.

Some of the particular areas that have been discussed fruitfully on soapbuilders include:

• The various ways to express arrays in WSDL (see Chapter 6, "Describing Web Services").

• Differences in dealing with xsi:type requirements in various engines.

• Issues regarding encoding styles, and whether it's reasonable to assume a default encodingStyle in the absence of an explicit encodingStyle attribute.

• The purpose and structure of the SOAPAction HTTP header.

• Lots of others—you can read the soapbuilders archives (see the resources section at the end of the chapter) yourself to check it out, and join the list to follow it into the future.

The Interoperability Lab

As we mentioned, soapbuilders also provided the community with a testing ground for SOAP implementations, in the form of the Interop Lab. You can find the Web pages for the tests at http://www.xmethods.net/ilab.

The Interop Lab represented the coming together of a number of similar efforts—Dave Winer and the Userland software team had developed a SOAP Validator test suite, there was discussion of how to do conformance tests on the various mailing lists, and the Apache SOAP development team had also been sketching out a test suite with Microsoft's .NET group.

Essentially, the Interop Lab tests involve a set of services that everyone agrees to implement at their nodes. The first round of these services test basic SOAP compatibility and the ability to serialize and deserialize simple data types. Many of the tests are of the form "echo this [string, integer, float, etc]"—this tests basic SOAP connectivity (envelopes are parsed and understood), deserialization (I understood the data you passed), and serialization (I sent you back something which you recognize is the same as what you passed me).

Several notable benefits have arisen from the xmethods/soapbuilders Interop Lab:

• Many bugs in various implementations have been found and fixed—in most cases quite rapidly due to quick feedback from the community.

• There is a well-known place that developers and consumers can visit to get a snapshot of the current state of features and bugs in a variety of implementations.

• When a new package comes out, the developers can rapidly get feedback from many sources telling them how their implementation plays well (or doesn't) with others.

• In May 2001, an Interopathon event was held at NetWorld/Interop in Las Vegas. Many vendors, both small and large, showed up for a couple of days of interoperability demonstrations and conversation. More than 20 different implementations supplied implementations of a simple digital marketplace system, simulating bidding and buying items from each other.

We're still ironing out a lot of issues, but the progress towards interoperability that has been achieved in a very short time is remarkable. There are more than 50 SOAP 1.1 implementations as of August 2001, and they're all learning to play well together.

The soapbuilders community is now working on higher-level interop testing, which includes testing of SOAP header functionality and WSDL as well as core SOAP. The community is also integrating an automatic (SOAP-based, of course) registry service for new endpoints—so when an implementation changes, they can make a SOAP call to the Interop Lab hub that will cause all the other implementations to re-run their tests against that endpoint.

The W3C: The Emergence of a Standardized SOAP

In parallel with the grass-roots, developer-driven soapbuilders effort, the World Wide Web consortium (W3C) is in the midst of a standardization effort to give SOAP a good cleaning. The W3C are the standards-keepers for XML technologies—XML, XSL, X-Link, XSLT, and so on.

The W3C XML Protocol group has two interesting qualities; first, it's the largest group (more than 70 members) that the W3C has yet run, and second, unlike most other W3C activities, the group is doing almost all its work in the open—there is a public mailing list, xml-dist-app (for "XML distributed applications"), on which most of the group discussions take place. Both of these points serve to illustrate the high level of interest in SOAP and Web services across the Internet community.

As we mentioned in Chapter 3, the XML Protocol group has produced a working draft of the SOAP 1.2 spec, which attempts to address some of the issues that were somewhat unclear in SOAP 1.1. The new spec can be found at the URL listed at the end of the chapter. SOAP 1.2 is divided into two specifications: part 1, the "Messaging Framework," is really core SOAP. Part 2, "Adjuncts," describes all of the normative, but non-core, portions of the SOAP framework, such as the HTTP binding and the RPC convention.

The new spec includes:

• A more precise definition of the SOAP processing model, which clears up some ambiguities regarding MustUnderstand in particular. This is in section 2, specifically sections 2.4 and 2.5.

• A standardized way of providing information about exactly which headers were the cause of a MustUnderstand fault (section 4.4.2).

• A versioning model that should make it possible for processors to negotiate version conflicts, or at least be more clear about what specific versioning problems occur (section 4.1.2 and the referenced versioning fault).

• An Abstract Model (actually a separate document) of the SOAP processing system, which begins to clarify how SOAP systems work in general, and can be used as a design tool.

The XML Protocol group keeps an up-to-date list of issues against the SOAP spec, which you can find linked from the group Web page. Many of these issues, due to the public nature of the group's work, have also benefited greatly from conversations on the soapbuilders list. The combination of a dedicated group within the W3C and a vibrant community of developers who are actually building SOAP software today will hopefully result in a standard that not only addresses the concerns raised by SOAP 1.1, but also makes migration and implementation easy.

Future work within W3C might include standardizing WSDL and clearing up the ambiguities and issues with that spec, as well. If the work in these areas is successful, the resulting recommendations would go a long way toward resolving many of the interoperability problems that have been experienced thus far.