This message constitutes notice of a Last Call for comments on XEP-0244 (IO Data).

Abstract: This specification defines an XMPP protocol extension for handling the input to and output from a remote entity.

URL: http://www.xmpp.org/extensions/xep-0244.html

This Last Call begins today and shall end at the close of business on 2009-09-01.

Please consider the following questions during this Last Call and send your feedback to the standards @ xmpp.org discussion list:

1. Is this specification needed to fill gaps in the XMPP protocol stack or to clarify an existing protocol?
2. Does the specification solve the problem stated in the introduction and requirements?
3. Do you plan to implement this specification in your code? If not, why not?
4. Do you have any security concerns related to this specification?
5. Is the specification accurate and clearly written?

Your feedback is appreciated!

There remains quite a lot to do, and you are more than welcome to join in the project. There is a Java library and we’ve integrated the specs into Bioclipse and Taverna (see Details behind the “Calling XMPP cloud services from Taverna2” ), but there is no support for BioCatalogue yet, and no libraries for other programming language yet.

Here’s the solution (yes, suboptimal, but since we were hacking on XMPP support in Bioclipse) which shows the structure in JChemPaint and Jmol as bonus (gist:107507):

// Today, Johannes challenged me to use Bioclipse and XMPP to calculate the Lipinski Rule of Five for
// http://en.wikipedia.org/wiki/Farnesol
query = "Farnesol"

// Zero: clear the console
js.clear();
js.print("Query: " + query + "\n");

// One: connect to the XMPP hive, and make contact with the CDK descriptor service here in Uppsala
xmpp.connect();
var service = xmpp.getService("descriptor.ws1.bmc.uu.se");
service.discoverSync(5000);
service.getFunctions();
var func = service.getFunction("LipinskiRuleOfFive");

// Two: take advantage of RDF, DBPedia
store = rdf.createStore()
rdf.importURL(store, "http://dbpedia.org/data/" + query + ".rdf")
rdf.importURL(store, "http://dbpedia.org/data/" + query + "/section1/Chembox_Identifiers.rdf")

// Three: run the SPARQL query and extract the SMILES from the List<List<String>>, and remove
// the '@en' suffix
var sparql = "PREFIX dbprop: <http://dbpedia.org/property/> SELECT ?o WHERE { ?s dbprop:smiles ?o }"
smiles = rdf.sparql(store, sparql).get(0).get(0)
smiles = smiles.substring(0, smiles.length()-3)

// Four: create a CML document
propane = cdk.fromSMILES(smiles);
js.print("Molecule SMILES: " + smiles + "\n");

// Five: call the function
result = func.invokeSync(propane.getCML(), 900000);
cmlReturned = xmpp.toString(result);

// Six: tune the CML so that the Bioclipse CML reader is happy
cmlReturned = cmlReturned.replace("xsd:int", "xsd:integer")

// Seven: extract the Lipinski Rule of Five score
propertyList = cml.fromString(cmlReturned);
value = propertyList.getPropertyElements().get(0).
  getScalarElements().get(0).getValue()
js.print("Lipinski Rule of Five: " + value + "\n")

// Eight: while at it, let's create a 2D and open in JChemPaint
service = xmpp.getService("cdk.ws1.bmc.uu.se");
service.discoverSync(5000);
service.getFunctions();
func = service.getFunction("generate2Dcoordinates");
mol = cdk.fromSMILES(smiles)
result = func.invokeSync(mol.getCML(), 900000);
cmlReturned = xmpp.toString(result);
mol2d = cdk.fromCml(cmlReturned);
ui.open(mol2d)

// Nine: oh, and a 3D model in Jmol
func = service.getFunction("addExplicitHydrogens");
result = func.invokeSync(mol.getCML(), 900000);
mol = cdk.fromCml(xmpp.toString(result));
func = service.getFunction("generate3Dcoordinates");
result = func.invokeSync(mol.getCML(), 900000);
mol3d = cdk.fromCml(xmpp.toString(result));
file = "/Virtual/foo.cml";
ui.remove(file)
cdk.saveCML(mol3d, file);
ui.open(file)

Earlier in this series:

• Bioclipse2 Scripting #1: from SMILES to a UFF optimized structure in Jmol
• Bioclipse2 Scripting #2: searching PubChem

• Details behind the “Calling XMPP cloud services from Taverna2”
• Calling XMPP cloud services from Taverna2
• Next generation asynchronous webservices

So, I set up XMPP services for QSAR descriptor calculation, 2D diagram and 3D geometry calculations and a few more, using the CDK. Chemical Markup Language has been my primary choice for some 10 years now (see Peter’s blog) as it allows me to do things I cannot do in other formats.

Now, our XMPP services publish themselves what data types the allow as input and what they output in return. They do this by publishing XML Schema to describe the input and output types. My CDK services use CML, so they return the CML schema. Johannes’ xws4j implementation of the IO-DATA specification has an add on that can build bindings to the schema on the fly. Now, CML comes with a good XOM-based binding (called CMLXOM) so this is not strictly necessary, but for less common schemata it is worthwhile: you can always create bindings for brand new schemata, for older versions, for whatever. Services can even create their own local schemata, and people will still be able to easily use them. This is to me a big plus for this architecture.

Anyway, while CMLXOM exists, we wanted to show that the on-the-fly creation of bindings works, even for large schemata, such as CML. However, one of the older flavours had an small error in a regular expression in a data type CML defines. Johannes therefore asked me to test building bindings for the CML schema version used in my services. He adviced me to use scomp for this, which is a command line utility around the XMLBeans library used for the binding generation.

As I am running Ubuntu, I preferred installing the packaged version instead of installing the binary provided by XMLBeans. Now, after I did this, I noticed that this .deb did not install the scomp utility, so I filled a wishlist bug report. Earlier this week I already encountered another bug, but this package being Java, I had a good idea on how to fix the bug.

And so I implemented my own wishlist. I’m sure there is room for improvement, as my .deb packaging skills are a bit rusty (a very long time ago I have been in the Debian New Maintainers queue, but by the time they solved the long queue delays, I was too occupied with other things. Yes, this was a long time ago already :). Anyway, Ubuntu’s LaunchPad has a nice feature, called the Personal Package Archives. This service will, after I have finished hacking on the packaging specs in the famous debian/ folder and tested the .debs build from it, will rebuild it and put the resulting package up for download.

Conclusion: a perfect opportunity to finally gives this a try. The learning curve was surprisingly shallow, and the result can be seen in my personal package archive:

Now, you can easily imagine that I will soon work on packaging stuff I did in the past too, such as update libcdk-java and now that OpenJDK in main can run Jmol reasonably, finally package Jmol for main. I just hope I remember my Alioth account, so that I can properly contribute to the debichem project.

Getting back to running scomp on the CML scheme, it works with one minor problem:

$ scomp -src . -d .  cml.xsd
/home/egonw/tmp/cml/cml.xsd:10098:9: warning: p-props-correct.2.2: maxOccurs must be greater than or equal to 1.
Time to build schema type system: 1.792 seconds
Time to generate code: 3.297 seconds
Time to compile code: 9.658 seconds

The problem is reflected by line 10098 which goes like:

<xsd:sequence minOccurs="0" maxOccurs="0">

which can be traced down to line 23 in schema2/trunk/elements/tableHeaderCell.xsd. I filled a bug report about this.

I promised details, but realize I have actually already posted a lot of them in October :

Johannes ideas led to the IO-DATA proposal (XEP-0244), which is currently marked experimental and being discussed on the ws-xmpp mailing list. He gathered a few people around him to get it going, resulting in working stuff! Yeah!

Joerg asked Could you post more results, what is it, why do we need it, e.g. why are you mentioning SOAP and cloud? Do not know enough to see the bonus right now.

What is it? IO-DATA is a protocol on top of the XMPP protocol to allow machine-to-machine communication. Actually, much like SOAP, RPC, and other platforms. How IO-DATA differs lies to some extend to the transport layer: instead of using HTTP, it used the XMPP transport protocol, also used for Jabber chat clients. It basically allows clients like Taverna to chat with services running elsewhere.

Why do we need it? Most services run over HTTP, making them web services. This is convenient, because there is much infrastructure around, like web browsers. REST services also take advantage of this. However, for heavy computing this sometimes leads to problems. For example, routers are known to have time outs on HTTP connections. To solve this, SOAP services often introduce a polling mechanism. IO-DATA takes a different approach. Instead of having to ask all the time how a calculation is doing, you can just wait for the service to send you a message when it is done. Instead of working around the lack of asynchronous aspects, IO-DATA introduces these in the protocol.

Other interesting features include that the IO-DATA integrates the interface formats for services into the service itself, SOAP needs WSDL for this, and that it features service discovery via DISCO. The latter is done with SOAP too, for example with UDDI and BioMoby. The latter also adds strong data typing for input and output of services.

IO-DATA addresses the data typing by allowing asking the service what XML Schema it uses for input and output. While XML Schema has alternative, and which may be prefered in some situations, it does allow strong data typing and supports a lot of formats in life sciences (which I’ll summarise soon).

Moreover, if there just happens not to be a suitable schemata around, you can just define one yourself, which can be as simple as a single element wrapper around some custom text-based format. You worry about supporting many formats? Well, no need. Johannes’ xws4j library, which I used for the Taverna plugin too, allows compiling a Java binding code. Bioclipse’s script environment allows you do to this on the fly: you find a service, ask for the schema, compile bindings for input and output, set up the input with the input binding, send it of to the service, and use the output binding for convenient access to the computation results. Without having to reboot Bioclipse. Isn’t that cool? Can your software do that? (See this example Gist: the io factory creates the binding).

Why do I mention SOAP and the cloud? It should be clear from the above why I mention SOAP: it offer the same functionality, but more conveniently, we think. I mention cloud here, to refer to cloud computing which is doing computation on the cloud, which is a synonym for the internet (see Cloud Computing @ Wikipedia). Because it does not use HTTP, we do not feel we can call it web service. Instead, cloud computing is a more general term, not tied to any particular architecture. IO-DATA is just one possible architecture, one we think is promising for life science applications.

Details will follow, but here’s the source code.

xws.connect();
service = xws.getService("cdk.ws1.bmc.uu.se");
service.discoverSync(9000);
service.getFunctions();
f = service.getFunction("calculateMass");
ios = f.getIoSchemataSync(9000);
iof = xws.getIoFactory(ios);
smiDoc = iof.createSmilesDocument();
smiDoc.setSmiles("CCC");
result = f.invokeSync(smiDoc.toString(), 9000);
obj = iof.getOutputObject(result);
print("Mass: " +  obj.getStringValue())

At first, it might look a bit verbose to just calculate the mass of a molecule, and it is, and it is not even written in XML. Hahahaha

Anyway, the code rocks, thanx to Johannes’ great work on his xws4j library! I’ll explain the script. The first line gets Bioclipse online using a Jabber account, which you set via Bioclipse’ preferences pages. The next few lines allows you to connect to a cloud service, this one running on ws1.bmc.uu.se and called cdk. With the getFunctions() method we query which functions are available, called ports in WDSL if not mistaken, from which we pick the calculateMass one.

And then the action joins in. One nice feature of the IO-DATA proposal is that the function itself defines the XML Schema it uses for input and output, and does not rely on WDSL to do that (or maybe recent SOAP specs allows that too). So, we query the function for its schemata, and the xws4j library then something funky happens: we order the library to create a data model on the fly for this service! From this we get a Java data model for the service. This allows us to use createSmilesDocument() and setSmiles(). That’s function-specific stuff!

Of course, we do not have to do that. For example, the second function I wrote (generate3Dcoordinates) eats and spits CML, and I’d rather rely on CMLDOM or CDK as data model then. But more on that later…

The Bioclipse xws4j plugin actually puts the data model in my workspace, so that I can easily introspect the API:

The last three lines invoke the function (synchronously, as it’s cheap), and get the mass from the function output. BTW, I should stress that a function does not require any specific implementation regarding synchronous or asynchronous calls. You write one function, and can call it in either way you like. The library hides all IO-DATA details around that.

Johannes ideas solve this limitation by using the general XMPP protocol for chatting:

client

he, can you do something for me?

service

sure, I can do generate3Dcoordinates and generateSMILES.

client

ah, nice! what input does generateSMILES take? and the output?

service

input: CML, output a simple string.

client

ok, here's the CML

service

I'm done now. sorry that it took 10 minutes, but I'm running Vista...

client

excellent, please send me the results

service

ok, here is the SMILES for lacosamide: CC(=O)N[C@H](COC)C(=O)NCC1=CC=CC=C1

Well, the important bit is in the last line. A job may take lone, even on clusters. The client might have to reboot meanwhile (possibly because of critical security updates)… the service will just continue, and send you a message when done. If you just happen to be offline, it will send a message when you are back online.

Johannes ideas led to the IO-DATA proposal (XEP-0244), which is currently marked experimental and being discussed on the ws-xmpp mailing list. He gathered a few people around him to get it going, resulting in working stuff! Yeah!

Chemistry Development Kit XWS

Besides contributing to the proposal, I am also involved in this project by writing XMPP-webservices, for the CDK. This brings me to cdk-xws, which is the project to bring CDK functionality online as webservices using IO-DATA.

This shows three nodes, the first being the CDK service, with two functions, of which I only implemented one yet.

For the curious, this is what the XMPP messages look like:

<iq from="egonw@ws1.bmc.uu.se/home"
    id="JSO-0.12.5-6"
    to="cdk.ws1.bmc.uu.se"
    type="set">
  <command xmlns="http://jabber.org/protocol/commands"
           action="execute"
           node="calculateMass">
    <iodata xmlns="urn:xmpp:tmp:io-data"
            type="input">
      <in>
        <smiles xmlns="urn:xws:cdk:input">CCC</smiles>
      </in>
    </iodata>
  </command>
</iq>
<iq from="cdk.ws1.bmc.uu.se"
    id="JSO-0.12.5-6"
    to="egonw@ws1.bmc.uu.se/home"
    type="result"
    xml:lang="en">
  <command xmlns="http://jabber.org/protocol/commands"
           node="calculateMass"
           sessionid="XWS-1"
           status="completed">
    <iodata xmlns="urn:xmpp:tmp:io-data"
            type="output">
      <out>
        <mass>36.032207690364004</mass>
      </out>
    </iodata>
    <note type="info">Done</note>
  </command>
</iq>