Bar chart showing the number of compounds with a particular chemical identifier.

Wikidata has many sub projects, such as WikiCite, which captures the collective of primary literature. Another one is the WikiProject Chemistry. The two nicely match up, I think, making a public database linking chemicals to literature (tho, very much needs to be done here), see my recent ICCS 2018 poster (doi:10.6084/m9.figshare.6356027.v1, paper pending).

But Wikidata is also a great resource for identifier mappings between chemical databases, something we need for our metabolism pathway research. The mapping, as you may know, are used in the latter via BridgeDb and we have been using Wikidata as one of three sources for some time now (the others being HMDB and ChEBI). WikiProject Chemistry has a related ChemID effort, and while the wiki page does not show much recent activity, there is actually a lot of ongoing effort (see plot). And I’ve been adding my bits.

Limitations of the links

But not each identifier in Wikidata has the same meaning. While they are all classified as ‘external-id’, the actual link may have different meaning. This, of course, is the essence of scientific lenses, see this post and the papers cited therein. One reason here is the difference in what entries in the various databases mean.

Wikidata has an extensive model, defined by the aforementioned WikiProject Chemistry. For example, it has different concepts for chemical compounds (in fact, the hierarchy is pretty rich) and compound classes. And these are differently modeled. Furthermore, it has a model that formalizes that things with a different InChI are different, but even allows things with the same InChI to be different, if need arises. It tries to accurately and precisely capture the certainty and uncertainty of the chemistry. As such, it is a powerful system to handle identifier mappings, because databases are not clear, and chemistry and biological in data is even less: we measure experimentally a characterization of chemicals, but what we put in databases and give names, are specific models (often chemical graphs).

That model differs from what other (chemical) databases use, or seem to use, because not always do databases indicate what they actually have in a record. But I think this is a fair guess.

ChEBI

ChEBI (and the matching ChEBI ID) has entries for chemical classes (e.g. fatty acid) and specific compounds (e.g. acetate).

PubChem, ChemSpider, UniChem

These three resources use the InChI as central asset. While they do not really have the concept of compound classes so much (though increasingly they have classifications), they do have entries where stereochemistry is undefined or unknown. Each one has their own way to link to other databases themselves, which normally includes tons of structure normalization (see e.g. doi:10.1186/s13321-018-0293-8 and doi:10.1186/s13321-015-0072-8).

HMDB

HMDB (and the matching P2057) has a biological perspective; the entries reflect the biology of a chemical. Therefore, for most compounds, they focus on the neutral forms of compounds. This makes linking to/from other databases where the compound is not neutral chemically less precise.

CAS registry numbers

CAS (and the matching P231) is pretty unique itself, and has identifiers for substances (see Q79529), much more than chemical compounds, and comes with a own set of unique features. For example, solutions of some compound, by design, have the same identifier. Previously, formaldehyde and formalin had different Wikipedia/Wikidata pages, both with the same CAS registry number.

Limitations of the links #2

Now, returning to our starting point: limitations in linking databases. If we want FAIR mappings, we need to be as precise as possible. Of course, that may mean we need more steps, but we can always simplify at will, but we never can have a computer make the links more complex (well, not without making assumptions, etc).

And that is why Wikidata is so suitable to link all these chemical databases: it can distinguish differences when needed, and make that explicit. It make mappings between the databases more FAIR.

Update: Mark wrote up a blog post on the RDF that the ChEMBL team itself.

Yesterday, the paper “The ChEMBL database as linked open data” (doi:10.1186/1758-2946-5-23) by Andra Waagmeester (@andrawaag), Ola Spjuth (@ola_spjuth), Peter Ansell (@p_ansell), Antony Williams (@chemconnector), Valery Tkachenko, Janna Hastings, Bin Chen (@binchenindiana), David J Wild (@davidjohnwild), and me appeared in the OA JChemInf journal.

I am also indebted to the ChEMBL team (@chembl) for both providing such valuable data under a liberal Open Access license and their critical reading of the manuscript! Additionally, I would like to stress that the ChEMBL team will create their own RDF version of ChEMBL and that this paper is not describing the version they will release.

BTW, the source of the paper is available from GitHub. And the (original) scripts to create RDF from the MySQL dump of ChEMBL are also on GitHub.

This paper outlines the RDF as it has evolved from various earlier projects. The above diagram visualizes the basic structure (red), various Linked Data resources linked too (blue) and illustrates how various ontologies are used, such as the CHEMINF, BIBO, and CiTO ontologies.

Additionally, various applications and links are described developed by various co-authors. For example, Peter worked on the use in Bio2RDF and Bin and David on Chem2Bio2RDF. Andra developed an extension for his (#altmetric) CitedIn resource, giving credit to a paper when data in it is extracted into ChEMBL. Ola, Valery, and Anthony developed a Bioclipse Decision Support extension, which supports a nearest neighbor search in ChEMBL using ChemSpider. Of course, Ola also hosts the SPARQL end point of which you can monitor the uptime at the also cool mondeca.com service:

(Yes, I think I have all the cool buzzwords covered in this paper. Sadly, marketing is needed nowadays as a scientist. Where is the time that you could rant on page after page in all your domain specific jargon, not having to worry if your reader would understand it immediately, or without a university degree…)

What this paper does not describe, is all the things I did with ChEMBL-RDF in the Open PHACTS project (@Open_PHACTS), which includes the use of QUDT and the jQUDT library for unit normalization outlined in this document and the use of VoID for link sets as described in this document.

• McPrinciple #1: access to data
• McPrinciple #2: be clear in what you mean

Peter’s post #solo10: Green Chain Reaction; where to store the data? DSR? IR? BioTorrent, OKF or ??? gives me enough basis to write up a third principle:

Molecular Chemometrics Principles #3: We make scientific progress if we build on past achievements.

Sounds logical, right? Practically, the way we share our cheminformatics knowledge makes this standing on shoulders pretty difficult. But there is one particular aspect I would like to ask your attention for: you can contribute by making clear what shoulders you would like to stand on. That is, where do you prefer to put your effort, and what message would you like to give to your user community.

In the aforelinked post, Peter asks where he should upload his data, and he suggest BioTorrent (see my review BitTorrents for Science ), DSpace, and CKAN. Now, his Green Chain Reaction is picked up (see these few blog posts), and the resulting data should be distributed as much as possible. The exact location does not really matter…

But…

By picking where you upload, you make a statement to your community: “Look guys, we are distributing our data via Foo, because we believe those guys are doing good work! Perhaps you can support them too.”.

This principle does not only apply to data, it applies to things too. For example, when iChemLabs and RSC ChemSpider Announce Partnership they do not just improve the user experience of ChemSpider (which I certainly won’t object against), but they also imply “Look dudes, your product is just not good enough and we do not want to help you improve it either”. Of course, ChemSpider has every right, and for them to succeed it is crucial to make decisions like this. Fortunately, ChemDoodle is GPL.

Every project with a user base has the opportunity to support shoulders, if they only visibly stand on them. By merely discussion the Green Chain Reaction, I show to support this social web experiment. You can too. Use these powers wisely. May the McPrinciples be with you.

There has been some talk about a ChemSpider Bugzilla, but I don’t think this has materialized yet, and I’ll have the default to info-at-chemspider-dot-com …

Reading blogs used to be to get the news, but this has changed. Still, blogging gives more freedom, more space. Blogs did soon follow. Chris was the first to blog about it:

This is great news and I’m confident that it will be a move to even more openess in chemistry and cheminformatics. It will also allow the RSC to use Tony fantastic tools for even more semantic markup of articles. I’m looking forward to talking to everyone about the implications. For now, congratulations, Tony, and congratulations, RSC, for this great deal.

I think Tony himself was next:

This is good for us for a number of reasons. Specifically we will no longer have to deal with our very significant resource limitations but more than that it lends credence and validation to the work that we have been doing over the past 2 years. It seems so long ago now but ChemSpider was first unveiled to the world at the ACS Spring meeting 2007. What began then only as a hobby project is now being recognized by the community as one of the primary resources for internet chemistry.

His network and insight in required data curation is what I think made ChemSpider a success.

Later views followed from Peter, Rich and Neil. I have only congratulations, which I hereby join, and expect that only future will tell us if our cheers are correct.

Where next?

As Tony indicated, the deal will practically mean better support for ChemSpider in terms of computing power, making if easier for them to make upgrades, hence better uptime, etc. It may, indeed, also mean more data, provided from RSC archives, as suggested by Neil. More practically, I can imagine seeing Project Prospect contributing InChI-DOI links to ChemSpider very soon.

And this would be one of the two recommendations I have to ChemSpider at this moment:

1. now linked to a publisher, and with both text mining efforts and expertise, focus on these InChI-DOI links, and, in particular, focus on those InChI-DOI links which involve papers that describe measured properties of the molecules;
2. with the increased support, finish the Open Data work done, by making it easy for people to download the ChemSpider-OpenData subset. This, I believe, is crucial for a wider adoption in the OpenData community, as OpenData which is practically made impossible to easily download is not Open enough. Previous priorities may have been focused on setting up a viable commercial alternative, but with the RSC backing, this can no longer be a reason to not do this.

Once more, congratulations to the ChemSpider-team and the involved RSC people, and very much looking forward to seeing how this will change chemistry for the better!

Now, my practical in next weeks CDK Workshop will use Groovy (please install it on your laptop!), and am hacking up example scripts for the course material, and came up with this script to download the structure of domoic acid from PubChem (CID:5282253):

We are very excited by what we have put together so far. There are currently 457 H NMR, 389 C NMR, 11 IR and 29 NIR spectra. This is only possible because of people who submitted their spectra to ChemSpider as Open Data - please keep uploading!

Now, the NMRShiftDB also hosts quite a number of NMR spectra, and I have a hobby to submit spectra, particularly for rare nuclei. In particular, I think it is fun to to have as many as possible structures which have spectra for all the nuclei in that structure. Benzene is a simple example for which NMR spectra are available for all nuclei (see this entry).

Now, the main difference between the NMRShiftDB and ChemSpider spectral data is the the first are annotated peak lists (each shift is assigned to an atom), and the latter are full, but unannotated, spectral curves. So, there are quite a few things you could do here. For example, see which structures which NMR curves are not yet annotated in NMRShiftDB. Antony pointed me to this page which is an overview of all spectral data in ChemSpider, but that page is difficult to machine process. Partly, because it is a mix of Open and Proprietary data, and partly because it uses JavaScript to navigate the table. (BTW, RDF interfaces to both resources would be much more helpful, and simply allow me to query all molecules which have a spectrum which is Open, and which is not found in the NMRShiftDB. I am working on a RDF interface to NMRShiftDB.)

Antony also asked me to advertise the option to upload Open spectral curves to ChemSpider. So, hereby. However, I really do hope ChemSpider will make it easier for others to reuse all the Open Data, as having to machine browsing the linked HTML interface is a waste of ChemSpider computing resources.

Update: the game is now available from spectralgame.com.

<script type="text/javascript" src="http://www.chemspider.com/csjsapi.ashx?op=img&amp;tk=3d178e75-a272-4d60-8ca9-5b1183a0e746&amp;id=171&amp;w=120&amp;p=1&amp;eid=%22azijnzuur%22"></script>

There is an option to have ChemSpider link back to blog, and I will have to figure out how to enable Chemical blogspace to extract the InChI from the underlying JavaScripts.

Update: I noticed that the ChemSpider server was a bit sluggish this morning, and that loading my blog page halts at loading the JavaScript… Tony, I suggest to use some Ajax magic here, with a really fast JavaScript download (using an almost static bit of JavaScript), and then a Ajax to access to slower bits, which might involve image generation and database lookup.

Update2: the feature was already under development before Cameron asked about it.

Update3: the script is no longer working, and I made the code visible instead, for historic reasons.

Anyway, the key thing which ONS and CDK and Jmol share, is that they use an Open Notebook. Not every Open Source or Open Data project does. Actually, many scientific Open Source are not open Projects! They are more like the Cathedral than the wished-for Bazaar (see The Cathedral and the Bazaar). So, Open Source (science) projects are certainly not ONS projects by default!

Now, the CDK actually is ONS, it is a Bazaar. The notebooks we use include:

• open project via mailing lists
• open methods/results via subversion
• informal reporting via blogs (e.g. Rajarshi, Christoph, Thomas, mine)
• informal reporting via CDK News

What more would you wish for? That’s not a rhetorical question. Remember that every reader of this blog is in my advisory board !

Unfortunately, I do not create work at a workbench myself, so I do not produce new knowledge myself, other than extracted from existing data. That’s really a shame, and I really do hope that Jean-Claude or Cameron will send me a box to measure solubilities (see here, here, and here, here for first data exploration), even though I cannot participate in the challenge. (hint, hint :)

From Cathedral to Bazaar in Life Sciences

One Cathedral we ran into with Bioclipse was BioCatalogue, which will serve as website where people can annotate and categorize (web) services. While the project has been around for a while, the website was rather uninformative. Fortunately, the projects is going to open up, and be more Bazaar-like. For example, they now started a wiki and a mailing list. I hope these efforts will continue, so that I can contribute from my point of view!

The EMBRACE Registry is a project with similar goals and a rather nice outcome (which I learned about on Monday ). It is actually anticipate to be replaced by or merge with BioCatalogue. So, all data I entered, cheminformatics workflows (look, no ‘o’ ), will later be available from BioCatalogue too. That is already my first contribution to BioCatalogue. One enormously interesting feature of the Registry, is that is allows uploading of code to test the service. This will mean the Registry will not only poll if the service is still online (by checking the WSDL file), it will also test if the service behaves properly. Now, immediate thoughts are mashups with MyExperiment. Each WSDL entry in the Registry points to MyExperiment workflows that use them, and the workflow page would indicate the status of all used WDSL services. This integration was already anticipated long before I thought about it, as the involved Cathedrals were nicely located in the same floor in Manchester.

Below is a screenshot from the EMBRACE Registry for the ChemSpider WDSL entry for a workspace I uploaded about a year ago to MyExperiment:

BTW, ChemSpider has an Advisory Board of which I am member, but it is also a classical (and intentional) Cathedral project. We do share common interests though, which makes us collaborate.

Why Important?

One recurrent theme in Open Source is given enough eyeballs, all bugs are shallow. This surely applies to science as well. The difference between the two is that in current science the eyes only inspect with a delay of at least 6 months. Current practice is that research is finished (delay), and when decided publishable written up a paper (delay, and loosing valuable information in the process, as you can read in my blog all the time), and published (even more delay). ONS changes that, and so do Bazaar-like open source projects, such as the CDK, Jmol and Bioclipse. They bugs are present, whether we like it or not, not just in source code, but in science too. Theories get overthrown, but why should we like the long delays current scientific good practice? Hate it! Work around it. Use the Bazaar. Use ONS!

Now, ONS actually needs Open Source, allowing them to deal effectively with the data they produce; to allow extraction of new scientific knowledge from the measurements. If Rajarshi and Pierre would not have made their efforts, other could not easily join in, leading to those much hated delays. Bugs should be shallow, and openness allows us to make those bugs visible. We can prove that there is a bug, without having to reproduce data ourselves, leading to those nasty delays again. Just copy the data, compare it to your own, do your analysis.

One recent project in open source chemistry dealing with making bugs visible, is the web page set up by Andreas Tille for the DebiChem project. His page summarizes the bugs listed for the chemistry in Debian (which includes the Blue Obelisk projects Avogadro, BODR, CDK, Chemical MIME Data, Kalzium and OpenBabel):

This data analysis helps the projects being analyzed.

Packaging

This brings me to a last topic, for this blog: packaging using Open Standards. In order to allow those eyeballs to spot bugs, it is of the utmost importance to package your results in Open Standards, and not just one, but likely many. For Open Source projects this ultimately means Distribution Packages (deb or rpm). If that goal has been achieved, you know your results can be read by anyone. Software should be installable (make, ant, cmake, etc), and Data should be readable (no PDF, but RDF, XML, JSON, or whatever standard). Preferably not Excel, as this is too free format (as Rajarshi also indicated), but with some added conventions it may do well. Blue Obelisk project are generally doing well in terms of packaging.

For the CDK, which already is reasonably well packaged, I am currently working on Eclipse and Maven2 packages. The former is already being used by Bioclipse, while the second aims at Jumbo (which has just seen a new release. Jim, I’m happy to see the CMLDOM/Jumbo split!), CDK-Taverna, and possibly a third (Paula, what for do you plan to use it?). The POM export is not fully working yet, but with four research sites involved in this Open Project, I’m sure we’ll work it out.

The bottom line is, scientific progress would benefit so much from a Bazaar approach. And the key thing is not collaboration; that’s something you can do in a Cathedral-like fashion too. No, the key thing is to be Open and allow anyone, even your worst nightmare, to comment on what you do. Let him prove you wrong, openly, that is.

OK, there it is. My open notebook entry for this week. Now you know what I have been up to this week.

I would add to it that I’d like to see a meaningful discussion of the risks of Share Alike and Attribution on data integration. Chemspider’s move to CC-BY-SA fits into this discussion nicely - it’s a total violation of the open data protocol we laid out at SC, which says “Don’t Use CC Licenses on Data” - but it does conform inside the broader OKD.

Now, let’s take this into pieces.

1. John notes that ChemSpider is in compliance with the OKD. This means, that ChemSpider thinks about Open Data just like the Open Knowledge Foundation does. I’ve scanned through the OKD, and it indeed seems to support the BY and SA clauses of the CC. So, Chemspider did not do a bad thing.
2. Data integration is tricky: you have to keep track of license information on an entry-by-entry level. For each fact, you keep to track the source, and associate the source with it’s original license. For example, the NMRShiftDB information in ChemSpider should be GNU FDL.
3. OpenX licenses may be viral. This holds for the GNU GPL as well as for the CC-BY-SA. Nothing new there. It just requires that when you would like to incorporate the ChemSpider data into a larger database, that database has to be CC-BY-SA too, or likely at least CC-SA.

Summarizing, I think ChemSpider did a good thing, and that ChemSpider does not violate the OpenData idea, but instead, that the CC-BY-SA and the OKD violates John’s requirements for integrating data resources (apparently based on a two year legal study). That has nothing to do with ChemSpider.

Now, people will always have different opinions on Openness. The original BSD clause had a restrictive ‘advertisement’ clause, not Open enough for at least the Debian Free Software Guidelines (DFSG), while still open source. The clause was later removed from the BSD license.

Another Debian example is Firebox, which is named IceWeasel in Debian, because the ‘license’ on the Firefox name is not open enough.

Another problem with the definition of Openness, is the viral aspect of some licenses (see earlier). For some, the GPL is not open enough, because it does not give people the freedom to license their software they like themselves, something the BSD and MIT licenses do allow. There is ongoing debate (and that should be ongoing) on how much freedom a license must provide to be called Open. The whole OpenAccess discussion is similar (see e.g. Peter’s story on this), where the discussion on the minimal amount of freedom is even worse.

Should we worry about ChemSpider being ‘only’ CC-BY-SA? Maybe. Data is not software, but I disagree that viral license would be OK for software, but NOT for data. That’s just BSD-versus-GPL all over again. I am happy about OpenBabel being GPL, and I am happy about ChemSpider being CC-BY-SA too.

All that said, these discussion are important. And creating good definitions of what freedoms are required, are crucial in deciding whether something is Open. The Blue Obelisk does not have/use such definitions yet, and we should start discussing this, and define a Blue Obelisk ODOSOS Guidelines. Please no funny jokes about how we can boogy then :)

Now, looking forward to hearing what you think about these issues… Looking forward to the other blog items!