The current count actually is just above 230 thousand IUPAC names, but further growth may require new approaches, such as the four ideas I posted earlier. I have gone through all full-text Open Access articles provided by the Europe PMC API. Now, this list is not static, but I wanted to start using their bulk downloads anyway.

The current results

I have been looking at the names coming in. Some are short, others long. The complexity is fascinating and I will have to brush up my cheminformatics skills to make chemical space splots and visualize the structural diversity. I also note the current workflow does a good job at unicode characters, and we have plenty of names like ε,ε-carotene-3,3’-dione. There are also names that I do not expect to be really valid, like hydroxymethyl methacrylate- that end with a hyphen (41 in total), but their overall count is low. And OPSIN is happy with it, so the name fits the rules.

The ten longest names (so far) are these (with the lengths 322, 324, 332, 357, 371, 373, 376, 421, 429, and 626):

(5Z)-3-ethyl-5-[[4-[15-[7-[(Z)-(3-ethyl-4-oxo-2-sulfanylidene-1,3-thiazolidin-5-ylidene)methyl]-2,1,3-benzothiadiazol-4-yl]-9,9,18,18-tetra(nonyl)-5,14-dithiapentacyclo[10.6.0.03,10.04,8.013,17]octadeca-1(12),2,4(8),6,10,13(17),15-heptaen-6-yl]-2,1,3-benzothiadiazol-7-yl]methylidene]-2-sulfanylidene-1,3-thiazolidin-4-one
(Z)-[[4-[[(Z)-N’-carbamoyl-N-[2-[2-[2-[[3-[(4S)-6,8-dichloro-2-methyl-3,4-dihydro-1H-isoquinolin-4-yl]phenyl]sulfonylamino]ethoxy]ethoxy]ethyl]carbamimidoyl]amino]butylamino]-[2-[2-[2-[[3-[(4S)-6,8-dichloro-2-methyl-3,4-dihydro-1H-isoquinolin-4-yl]phenyl]sulfonylamino]ethoxy]ethoxy]ethylamino]methylene]urea dihydrochloride
2-((Z)-2-((6-(4-(6-((Z)-(1-(dicyanomethylene)-5,6-difluoro-3-oxo-1H-inden-2(3H)-ylidene)methyl)-4,4-bis(2-ethylhexyl)-4H-cyclopenta[1,2-b:5,4-b′]dithiophen-2-yl)-2,3-bis(hexyloxy)phenyl)-4-(5,7-diethylundecan-6-yl)-4H-cyclopenta[1,2-b:5,4-b′]dithiophen-2-yl)methylene)-5,6-difluoro-3-oxo-2,3-dihydro-1H-inden-1-ylidene)malononitrile
(2S,4S,5R,6R)‐5‐acetamido‐2‐[(2S,3R,4R,5S,6R)‐5‐[(2S,3R,4R,5R,6R)‐3‐acetamido‐4,5‐dihydroxy‐6‐(hydroxymethyl)oxan‐2‐yl]oxy‐2‐[(2R,3S,4R,5R,6R)‐4,5‐dihydroxy‐2‐(hydroxymethyl)‐6‐[(E,2S,3R)‐3‐hydroxy‐2‐(octadecanoylamino)octadec‐4‐enoxy]oxan‐3‐yl]oxy‐3‐hydroxy‐6‐(hydroxymethyl)oxan‐4‐yl]oxy‐4‐hydroxy‐6‐[(1R,2R)‐1,2,3‐trihydroxypropyl]oxane‐2‐carboxylic acid
(2R,3S,4R,5R,7S,9S,10S,11R,12S,13R)-7-[(benzylcarbamoyl)oxy]-2-(1-{[(2R,3R,4R,5R,6R)-5-hydroxy-3,4-dimethoxy-6-methyltetrahydro-2H-pyran-2-yl]oxy}propan-2-yl)-10-{[(2S,3R,6R)-3-hydroxy-4-(methoxyimino)-6-methyltetrahydro-2H-pyran-2-yl]oxy}-3,5,7,9,11,13-hexamethyl-6,14-dioxo-12-{[(2S,5R,7R)-2,4,5-trimethyl-1,4-oxazepan-7-yl]oxy}oxacyclotetradecan-4-yl 3-methylbutanoate
2-[4-[2-[[(2R)-1-[[(4R,7S,10S,13R,16S,19R)-10-(4-aminobutyl)-4-[[(2R,3R)-1,3 dihydroxybutan-2-yl]carbamoyl]-7-[(1R)-1-hydroxyethyl]-16-[(4-hydroxyphenyl)methyl]-13-(1H-indol3-ylmethyl)-6,9,12,15,18-pentaoxo-1,2-dithia-5,8,11,14,17-pentazacycloicos-19-yl]amino]-1-oxo-3 phenylpropan-2-yl]amino]-2-oxoethyl]-7,10-bis(carboxymethyl)-1,4,7,10-tetrazacyclododec-1-yl]acetic acid
(2R,3S,4R,5R,7S,9S,10S,11R,12S,13R)-12-{[(2R,4R,5S,6S)-4,5-dihydroxy-4,6-dimethyltetrahydro-2H-pyran-2-yl]oxy}-7-hydroxy-2-(1-{[(2R,3R,4R,5R,6R)-5-hydroxy-3,4-dimethoxy-6-methyltetrahydro-2H-pyran-2-yl]oxy}propan-2-yl)-10-{[(2S,3R,6R)-3-hydroxy-4-(methoxyimino)-6-methyltetrahydro-2H-pyran-2-yl]oxy}-3,5,7,9,11,13-hexamethyl-6,14-dioxooxacyclotetradecan-4-yl 3-methylbutanoate
(2S,4S,5R,6R)‐5‐acetamido‐2‐[(2S,3R,4R,5S,6R)‐5‐[(2S,3R,4R,5R,6R)‐3‐acetamido‐5‐hydroxy‐6‐(hydroxymethyl)‐4‐[(2R,3R,4S,5R,6R)‐3,4,5‐trihydroxy‐6‐(hydroxymethyl)oxan‐2‐yl]oxyoxan‐2‐yl]oxy‐2‐[(2R,3S,4R,5R,6R)‐4,5‐dihydroxy‐2‐(hydroxymethyl)‐6‐[(E,2S,3R)‐3‐hydroxy‐2‐(octadecanoylamino)octadec‐4‐enoxy]oxan‐3‐yl]oxy‐3‐hydroxy‐6‐(hydroxymethyl)oxan‐4‐yl]oxy‐4‐hydroxy‐6‐[(1R,2R)‐1,2,3‐trihydroxypropyl]oxane‐2‐carboxylic acid
(2R,3S,4R,5R,7S,9S,10S,11R,12S,13R)-12-{[(2R,4R,5S,6S)-4,5-dihydroxy-4,6-dimethyltetrahydro-2H-pyran-2-yl]oxy}-2-(1-{[(2R,3R,4R,5R,6R)-5-hydroxy-3,4-dimethoxy-6-methyltetrahydro-2H-pyran-2-yl]oxy}propan-2-yl)-10-{[(2S,3R,6R)-3-hydroxy-4-(methoxyimino)-6-methyltetrahydro-2H-pyran-2-yl]oxy}-3,5,7,9,11,13-hexamethyl-7-({[2-(2-methyl-5-nitro-1H-imidazol-1-yl)ethyl]carbamoyl}oxy)-6,14-dioxooxacyclotetradecan-4-yl 3-methylbutanoate
N-[(2S,3R,4R,5S,6R)-5-[(2S,3R,4R,5S,6R)-3-amino-5-[(2S,3R,4R,5S,6R)-3-amino-5-[(2S,3R,4R,5S,6R)-3-amino-5-[(2S,3R,4R,5S,6R)-3-amino-5-[(2S,3R,4R,5S,6R)-3-amino-5-[(2S,3R,4R,5S,6R)-3-amino-4,5-dihydroxy-6-(hydroxymethyl)oxan-2-yl]oxy-4-hydroxy-6-(hydroxymethyl)oxan-2-yl]oxy-4-hydroxy-6-(hydroxymethyl)oxan-2-yl]oxy-4-hydroxy-6-(hydroxymethyl)oxan-2-yl]oxy-4-hydroxy-6-(hydroxymethyl)oxan-2-yl]oxy-4-hydroxy-6-(hydroxymethyl)oxan-2-yl]oxy-2-[(2R,3S,4R,5R,6S)-5-amino-6-[(2R,3S,4R,5R,6R)-5-amino-4,6-dihydroxy-2-(hydroxymethyl)oxan-3-yl]oxy-4-hydroxy-2-(hydroxymethyl)oxan-3-yl]oxy-4-hydroxy-6-(hydroxymethyl)oxan-3-yl]carbamate

That last compound has the InChIKey DKPKDPKJVDQUPD-XGBIXEJNSA-M and cannot be found in Google nor in PubChem. It looks like this:

There are some closely related compounds, though.

Chemicals only published about once

Some related data was blogged by Henry Rzepa last week, with this quote by Lee from CAS:

38.5% of the current substances have only 1 reference

Apparently, based on CAS Registry data, about 1 in 3 chemical structures are only published about once. And two in three are published about at least twice. I agree with Henry here, with organic chemistry literature in mind, I would have expected that 38.5% to be higher.

Anyway, since this project is not tracking in which articles IUPAC names are found, I have nothing to study this.

1 million IUPAC names

So, the primary goal of this project is to reach one million IUPAC names. We are currently at around 23%. Not bad, considering we started in Februari. And we have plenty of untouched literature left.

But I also applied idea 1, the varying names. The idea is that this was I can explode the number of compounds. In that compounds above, just the number of variations by enumerating all OH replacements with OMe and OEt would help a lot.

Because I wanted to make sure I could answer positively at the ICCS if we made it to one million CCZero IUPAC names, I implemented a very simple enumeration script. Really dumb approach. But the results are interesting. I started with the 200026 names from the milestone. If I explode these names, I get 1,377,127 IUPAC names, well above the target. Even if I remove name variations due to unicode variations for hyphens, I still have 1,162,107 IUPAC names.

Something interesting I cannot fully understand at this moment yet, however, is the following. When I calculate the number of unique InChIKeys for the milestone, I get 117,726 keys, and when I do this for the list of name variations, I get 203,979 keys. So, while the IUPAC name list is about five times as long, the list of InChIKeys is not even twice as long. Well, I guess that is why this is called research.

This milestone release took a bit longer. Going from 50 to 100 thousand is a bigger step than from 10 to 50 thousand, but the open access chemistry literature was already done by then. Basically, I ran out of open access chemistry publications. The scripts are now finding names in all (open access) literature, and the number of new names per articles is a lot lower. Still about 1 in every twenty to 30 articles. But the diversity in names is not really going down, which is important.

The first few weeks, I used the Google Colab to run a Jupyter notebook, initial created by Magnus, but having to process more articles to get a reasonable number of new IUPAC names required longer and longer jobs, and then Google Colab is not really fit (well, the free version anyway). So, I started using a local script. That turned out to be able to handle up to 20 thousand articles in one go and runs at least twice as fast. Moreover, I can run three of them in parallel.

And that had impact. With each commit around 1000 new IUPAC names, the number of commits went up remarkably last week:

At the current speed, I think we’ll make it to 150k soon and I added a new milestone for 200k, which sounds doable in the next three week. That also means that 1M extracted IUPAC names from literature has become a reasonable goal. And we can start thinking about the 2, 5, 10, 50 and 100 million IUPAC names. Those are, at the current speed, rather unlikely to reach from the open access literature anytime soon. That brings us to the question, what will. Well, I have some ideas.

Idea 1: name variations

First, I am figuring out some ways to make variants of names (no, not based on hyphens and spaces; that’s too easy), but actual variations of the chemical structures. For example, I could exhaustively replace “methoxy” with “ethoxy”, and iterate the halogens and acyl chain lengts. I have little doubt that I can grow the list with this approach easily a 5-fold, maybe even a 10-fold.

Idea 2: hallucination

Another idea is that I could use tools that can generate IUPAC names for a limited set of compounds. I once wrote code for alkanes myself and if I can find that, I may be able to generate additional names. But perhaps more realistic is that I train a deep learning model and have it generate names for all compounds in Wikidata (~1.5 million) or PubChem (>100 million). STOUT needed 81 million compounds (doi:10.1186/s13321-021-00512-4), but I don’t need a good model; I just need a model that comes up with new, valid names. Hallucinated names, but valid.

While the list of valid names grows, I can retrain the deep-learned model and repeat. As long as the diversity remains high enough, one could hypothesize that the deep learning will learn new tricks. And then, that should be a near infinite source of additional names.

Idea 3: (semi-)closed access literature

Also, I haven’t touched closed access articles yet. This is all based on the collection of full texts in Europe PMC. For example, I could start with the green open access article in (Dutch) university repositories, particularly those with large chemistry departments. PDF to text tools are mature enough that this will provide a new source. Oh, and perhaps PhD thesis, which are now also increasingly archived in university repository under open access. And that reminds me of a Dutch project two decades ago doing exactly that. I wish I remembered the name.

Idea 4: alternatives to Oscar4 and Europe PMC

So, the first round of named entity recognition was with Europe PMC itself, as explained in the first post. The move to Oscar4 helped a lot. But there exist many other chemical NER tools, like (doi:10.1093/bioinformatics/btn181. And those may find an additional number of names, even with just the literature I already covered.

Well, you get the idea.

ICCS poster rejected

Unfortunately, the ICCS poster abstract did not make the cut. The score was high enough, but they received many abstracts and had to make a selection (of course, I am part of the ICCS organization, and have more details of how it came about). I really like the project, and eager to write up a paper around it.

So, when propagating chemical structures from the Beilstein Bioschemas feed, I was looking for names, IUPAC or not, ideally the name used in the article. When I asked about this, the question came up if they could autogenerate IUPAC names, for which various new tools exist (I think I am missing one from an American team, but cannot find the reference), along with multiple established commerical tools. Because the IUPAC nomenclature is a long list of naming rules, priorities, etc, a rule-based algorithm is logical, but newer methods take a deep-learning approach.

Back to the chemical annotation of chemistry literature. This is of obvious interest: you want to know where we can read more about a certain chemical. We need the chemical structures in a database for that, linked to the articles. This is, of course, one of the original studies of cheminformatics. And when authors of the chemical literature do not provide this routinely (this post shows a few exceptions, but it is still all too rare). And then manual and automated curation is needed, e.g. done by Chemical Abstracts.

Third, Wikidata has about 1.4 million chemical compounds and many names. A property propoal for IUPAC names has been long pending, but once accepted in one form or another, will require IUPAC names too.

One million IUPAC names

Thus, the idea came up, can we create a set of 1 million unique IUPAC names found in literature? I asked on the ELIXIR Europe slack channel if Europe PMC had such a dataset (doi:10.1093/nar/gkad1085). I knew they had been adding chemical named-entity recognition (NER) results in their annotation API. I learned they used ChEBI. Melanie Vollmar and Summer Rosonovski or Europe PMC gave useful information and support. Magnus Palmblad also replied and provided Python code to use the Europe PMC API to fetch names it returns and see if those are IUPAC names. Well, that’s easy. We have OPSIN for that (see doi:10.1021/ci100384d).

Unfortunately, the Europe PMC NER results are not ideal for IUPAC names. Just scanning some 5, 6 organic chemistry journals returned some 8 thousand IUPAC names in open access articles. But it quickly started to be too limited: each set of articles returned increasingly few new names. The reason is simple: the NER is too greedy and as a result, does not easily recognize longer IUPAC names. It is too happy with a substring of the IUPAC name. For example, when it encounters the IUPAC name 5-Bromo-1H-indole-3-carboxylic acid, it settles for indole-3-carboxylic acid:

Open-Source Chemistry Analysis Routines

During my PhD, in 2003, when I worked a few months with Prof. Peter Murray-Rust (University of Cambridge) and Prof. Janet Thornthon (EMBL-EBI), I learned about the research by Sam Adams (doi:10.1039/B411699M), Joe Townsend (doi:10.1039/B411033A), and Peter Corbett (doi:10.1007/11875741_11). One of the tools that used this research was (is) OSCAR, short for Open-Source Chemistry Analysis Routines (see this detailed write up by Peter MR). Later, in 2010 I visted Peter again, as postdoc, in Cambridge, and then worked on the OSCAR project too. And while OSCAR did a lot more, the integration of Corbett’s NER research made OSCAR the obvious follow-up step in finding IUPAC names in literature.

And because OSCAR4 had been integrated into Bioclipse (doi:10.1186/1758-2946-3-41) and I had this ported to Bacting already (doi:10.21105/joss.02558), using this was trivial. The use of Europe PMC is different now, however, and we are no longer using the Annotations API, but just using it to find open access articles, and to get the full text in XML format. That allows a simple XPath search on <p> elements, pass the resulting string to OSCAR4, and the recognized names are checked with OPSIN. And with this approach, processing two of the five or six journals we earlier explored, we find another 40+ thousand IUPAC names. Quite a success, I am tempted to say.

A Blue Obelisk project

So, I started a new Blue Obelisk project, iupac-names, to collect 1M IUPAC names. For researchers to use, learn from, etc. Just IUPAC names. Not even the chemical structure, nor the link to the articles. The first is trivial to do with OPSIN, so the matching SMILES do not need to be stored. Links to literature is tricky because of the aforementioned issues, and we only want to know which (partial) IUPAC names occur in literature. If you really want to know in which articles that IUPAC name is found, you can simply do a search in Europe PMC.

And because we only store IUPAC names, this are very basic facts (this is an IUPAC name, as defined by OPSIN being able to generate a SMILES for this structure) and that that string occurs in some article) and we can share them as CCZero. We defined various milestones, and I am happy that the first two have been reached within two weeks:

• Milestone 10k (doi:10.5281/zenodo.14965762)
• Milestone 50k (doi:10.5281/zenodo.14978557)

This second milestone has 53848 unique names, but as literature goes, there are interesting variations, some likely because of typesetting leading to spaces added and missing. If we ignore spaces and hyphens, we have 50534 names left (hence the milestone). But IUPAC names are also not fully unique, partly because of Unicode character variations and greek letter alternatives, and you may wonder how many different chemical structures this set reflects. While not perfect, the Standard InChI gives some lower limit, and we find 36528 InChIKeys in this second milestone.

Now, we need twenty times as much to reach the 1M IUPAC names, but given we have many, many more open access articles to process. The bottleneck seems to be mostly our workflow.

Can you contribute?

Yes, of course! This is an open science project. But please keep in mind the narrow focus of this project: only IUPAC names which can be found in (open access) literature. This project doed not accept autogenerated names (PubChem would have given use many millions already), nor IUPAC names from existing databases. Ideally, you are able to show the code you use to extract/find those names in literature.

Can I use these names?

First of all, this is what the CCZero license and open science nature of this project is about: reuse. We love to hear how you are using these names, tho, and we encourage you to write up how you are using them. You can use DataCite to cite the release you used, and citing this blog post by DOI is also possible.

Does it support my language too?

No, at this moment it only support IUPAC names in English. Dutch, French, Spanish, or Chinese IUPAC names are valid, but currently not supported. See also this post.

Will there be a publication?

Magnus and I intend so. We already submitted an abstract to the International Conference on Chemical Structures, which has a Collection in the Journal of Cheminformatics. If the abstract gets accepted, of course, we can submit there. Otherwise, we will look for another venue, likely diamond open access.

Where is your script?

Ah, fair point. We did not decide on the final license yet. I have used two scripts based on the template by Magnus. As soon as we have finalized the license, we will make those available.

Because if done right, the publishers can even help forward science, despite crippling progress. That sound harsh, and surely they have done a lot of good for science. In fact, we would not be where we are now without the publishers. But things have changed. With the internet anyone can be publisher. We see this with blogs, we see this with Lulu.com. And, unlike some misinformed people think, this is independent from peer review. Publishers were important because they provide a channel to disseminate knowledge. But paper publishing is no longer the most efficient way. In fact, in terms of value, paper has been overtaken for some years now.

And we need more added value. Not the shipping of the knowledge, but keeping up is the issue. And there too, publishing is inefficient: human language is nice for sharing ideas and concepts, but it fails at disseminating raw facts: measured data. Anyone who has tried creating a data set to find patterns knows this: extracting the information is a lot of effort, mostly caused by the broken paper publishing model. This is most apparent in some research domain where data repositories exist, but sadly this applies to a small minority of data types.

Now, text mining seems in that sense the wrong question: why trying to recover knowledge that should have gone into repositories in the first places. I agree. However, we cannot just throw away all the knowledge kept in these papers, and certainly not as long as people keep insisting on seeing only papers as scientific success. We are slowly seeing this improve, but only very slowly. Things that were apparent to me as a student 20 years ago, are the things that scholars are still struggling with today. Depressing indeed, but it does help you grow a good sense of patience.

And now, Elsevier wants to make a step forward, wants to be leading in science dissemination again. And they come up with an intermediate solution between actual knowledge dissemination and profit: they come up with a license-model, increasing their monopoly on knowledge and trying to lure the scientist into a non-commercial license. From a money-making perspective this is what society expects from them. From someone who likes to see societal problems solves, this is disappointing. They had a great opportunity to lead the field.

Now, is all bad? Not at all. It’s a step, but not the step I would have liked to see. It will be a success: because the CC-BY-NC data that will come out of it, will be part of the web of knowledge. No one will care about the NC part, except all those SMEs in Europe that work on products to help society which will find it much harder to collaborate with other companies, because they cannot share the knowledge the created from analyzing the literature (does Elsevier want a monopoly in this analysis?).

Nor will many in the academic community complain. Surely, those that have worried about this, they will. But the scholar at universities do not care about NC licenses. After all, universities are not commercial. Asking a student to pay 30 thousand euro for a year is surely not commercial. That is the consensus. But I note that this consensus has not be tried in court, and I am looking forward to the day it will happen. Elsevier will likely not challenge this, and silently accept this situation. Just like Microsoft never made a big deal out of people copying office versions of their operating system for at home: you do not bit the hand that feeds you (too hard). You rather go after others, like Academia.edu. It will not be scholar Elsevier will enforce the NC on, and it will not be large companies either: if any, it will be the SMEs. Support them, and do not agree with the license.

Well, it was a nice opportunity for Elsevier. I only see my choice to sign The Cost of Knowledge reaffirmed.

The choice of the NC clause is totally useless in any context of dissemination. I call for Elsevier to at least add this option, if they are serious about improving: text mining is provided to subscribers, via a decent API, adhering to:

1. Facts extracted from literature are licensed CCZero and attribution is paid (facts are copyright free in most parts of the world)
2. Output can contain “snippets” of the original text under international “fair use” concepts, and licensed as CC-BY

Any scientist is expected to attribute the source of information in the first place, and it is kind of sad Elsevier is on such bad foot with their audience that they feel this must be enforced via a contract, but that is not a problem. I also see no reason to deviate from international law about “fair use”; I do understand this is probably an ill defined concept, but 200 characters seems pretty limited to me, as facts can be spread of sentences longer than this.

I know that many will disagree on the CCZero license, and many will feel awkward about giving away data. It has value, right? It’s your property, right? I am not going to argue against that. But personally I do not understand how it aligns with the idea of scientific dissemination. Holding back knowledge as part of making knowledge available? How exactly does that make sense? Importantly, just like with software, Open is not the same as Without-Cost! Hosting and sharing Open Data also costs money (particularly, if it is 1 TB of data). Those are different concepts.

However, I also stress that the scholars have a great responsibility hear: I call for all Elsevier journal editorial boards to not accept this deal either. In fact, all editorial boards have great say in this: it’s them who make a journal valuable. I also call all scholars to be aware the consequences of selling away your copyright. That is a choice in the current era. There are plenty of means to disseminate your science without (much) cost, and APC is a flawed argument.

The current step by Elsevier, after all the effort from many, is not a step forward, it’s a step sideways. Elsevier, I know you can do better. Are you willing?

I am willing, and have been supporting science by making data available as CCZero. However, I also am happy if others are not ready for this, or have other reasons not to. It is not always under their control. For example, I have heard stories where data has been used by politicians as small change to get industry to test their products for safety. I also accept that getting funding as a scholar is hard work, often not paid for, and that it is hard to give away your only security of a future career. Then again, we all know what data is valuable, has already given its value, or is of no use to you anymore. And this latter case I ask you to consider to make data available: data of no use to you anymore, but that could be valuable to others. Make it available, and get cited, and get value out of it, you would not have received when it sat on some hard disk, and probably is lost in five years.

I also fully understand this is my opinion. Thus, not all data I make available is CCZero: I fully respect copyright and license from others; in fact, I often feel I do much more than scientists which object to Open licenses, which just take data as their own as they please. That is why I insist often on clear copyright and license information. Because if missing, default (local) law applies.

If you want to read more analysis, please refer to the following posts:

1. Elsevier opens its papers to text-mining
2. #elsevier’s TDM Terms (TaC): Can they force us to copyright data? (2)
3. Nature’s recent “news” article on Text and Data Mining was unacceptable [redacted]; I ask them to renounce licensing.
4. “Dear Peter,”, Richard van Noorden
5. Reply to Richard van Noorden

• Working on Oscar for three months
• Oscar text mining in Taverna
• Multiple unit test inheritance with JExample
• Oscar4 Java API: chemical name dictionaries
• Oscar4 command line utilities
• Installing Oscar
• Adding a new dictionary to Oscar
• Status update on BJOC analysis with Oscar and ChemicalTagger
• Status update on BJOC analysis with Oscar and ChemicalTagger #2
• Supramolecular chemistry
• Status update on BJOC analysis with Oscar and ChemicalTagger #3
• Oscar: training data, models, etc

These posts will server is a some initial critical mass for a draft report I plan to finish today. I might have to blog some further posts with diagrams, here and there. This post is actually one of them, and discusses something where Oscar can be expected to go next, now that the design is cleaned up (though this effort is not halted now) and it has become possible again to extend it. The over 250 unit tests make this a lot easier too.

One aspect where I expect Oscar to go in 2011 is the support for other languages. To a very large extend this is based on multi-language support in the dictionaries, as well as having training data in a particular language. This also provides some context to my earlier post about the need for a Oscar training data repository .

This extension opens a number of options: analysis of patent literature in other languages, monitoring of press releases in other languages, and news items in local news papers, etc. For example, it could analyse this C2W news item on yeast cells:

There are many use cases for such localized text mining. And it surely matters for determining the impact of research.

Oscar has various places where language specifics are found. For example, in tokenization of a text. One step here is the detection of sentence ends. This is done in most western languages with a period, exclamation mark, question mark, etc. But periods (dots) are also used in abbreviations. Similarly, colons can be used in chemical names. But the every language comes in with different abbreviations that need to be recognized.

Currently, some abbreviations are found in NonSentenceEndings. In the past three months, we have been cleaning up the code, and restructured the source code, making it easier to detect such places. This class will likely undergo further refactoring, to making the list of such non-sentence-endings configurable via files or so. What I expect to see, is that we you initiate Oscar like this:

Oscar oscar = new Oscar(Locale.US);

This might actually even make a nice student summer project. The biggest challenge will be in making a good corpus of training data, like the SciBorg training data that was used for training Oscar3.

But the whole normalization is tainted with English language specifics too. For example, the normalizer will have to ‘normalize’ the question marks, for which there exist several unicode variations. But the normalized variant is language dependent. For example, greek and armenian have different characters (see this page), and then we have not even started talking about the right to left.

Besides localized dictionaries, this Oscar will also benefit from a localized OPSIN. It seem to recognize the Dutch propaan, but not benzeen. I am not going to look at that soon, but if you are interested, I recommend checking out Rich’ posts about forking OPSIN and writing patches.

Getting Oscar going for other languages is a challenge, but also offers new opportunities. Just email the oscar mailing list if you are interested and need help.

N-grams are word parts of n characters. For example, the trigrams of acetic acid include ace, cid, tic, eti, and aci. N-grams of length four include acid, etic, and acet. The MEMM assigns weights to these n-grams, and based on that decided if something is in deed a named entity (in Oscar terminology). For example, consider the acet n-gram: acetone should be matched, but the n-gram facet not.

Put this in perspective in the ongoing refactoring of the Oscar software. We are changing normalization (e.g. converting all unicode hyphen alternatives into one specific hyphen), updating the tokenizer (e.g. changing the list of non-sentence-endings like Prof.). It is clear this changes the n-grams typical for chemical-like things. Worse, the weights are tuned towards to know n-grams, and statistical models are generally a bit overtrained for the data, or, at least, specific for it.

Now, if the distribution of n-grams changes, the weights in the model need to be updated too, to not degrade the model performance. So, Oscar is useless if we cannot retrain its MEMM component after a refactoring. If that would be impossible, we would have effectively created an intellectual monopoly.

Thus, what the Oscar project needs, is one or more free sets of annotated literature, which can be used to train new MEMM models. The SciBorg corpus was used to train the current Oscar3 and Oscar4 models. This data (copyright RSC) will very likely be available under a Creative Commons license (RSC++), but may have the NC clause, which would not be good for developing a business model around the opensource Oscar (such as providing a high-performance web service via a subscription service). I have recently written up the problems the NC clause introduces, and some examples of commercial Open Source cheminformatics projects.

We need not focus only on this SciBorg data, however. In fact, we will need multiple models anyway. For example, the SciBorg papers (42 if not mistaken) are around a particular kind of literature. So, it introduces the risk of using it to analyse papers out of the application domain. Furthermore, I am very interested (and others indicated so too) to use Oscar for other languages. Surely, English is the major language, but there are many use cases for Oscar when useful for other languages.

Therefore, for what we need in the Oscar project, is a registry of training (/test) data, annotated itself with metadata around how that data was created (what quality assurance, what kind of named entity types, how many domain experts were involved, etc), test results for those data sets, etc. My time on the Oscar project is almost over, and I have no clue when I will be able to invest the same amount of time into the project as I did in the past three months. But the creation of this registry is clear step that must be taken in the Oscar4 development.

Oscar4

Most people will be mostly interested into the full Oscar4 program, to extract chemical entities. Oscar3 was also capable of extracting data (like NMR spectra ), but that is not yet being ported. The OscarCLI program takes input, extracts chemicals, and where possible resolves them into connection tables (viz. InChI).

To extract chemicals from a line of text (e.g. “This is propane.”, you do:

$ java -cp oscar4-cli-4.0-SNAPSHOT.jar \
  uk.ac.cam.ch.wwmm.oscar.oscarcli.OscarCLI \
  This is propane.
propane: InChI=1/C3H8/c1-3-2/h3H2,1-2H3

For larger chunks of texts it is easier to route it via stdin, for which we can use the -stdin option:

$ echo "This is propane." | \
  java -cp oscar4-cli-4.0-SNAPSHOT.jar \
  uk.ac.cam.ch.wwmm.oscar.oscarcli.OscarCLI \
  -stdin
propane: InChI=1/C3H8/c1-3-2/h3H2,1-2H3

That way, we can easily process large plain text files (output omitted):

$ cat largeFile.txt | \
  java -cp oscar4-cli-4.0-SNAPSHOT.jar \
  uk.ac.cam.ch.wwmm.oscar.oscarcli.OscarCLI \
  -stdin

If you prefer RDF output, for further integration, use the -output text/turtle:

$ cat largeFile.txt | \
  java -cp oscar4-cli-4.0-SNAPSHOT.jar \
  uk.ac.cam.ch.wwmm.oscar.oscarcli.OscarCLI \
  -stdin -output text/turtle

This returns RDF using the CHEMINF ontology like:

@prefix dc:  .
@prefix rdfs:  .
@prefix ex:  .
@prefix cheminf:  .
@prefix sio: .

ex:entity0
  rdfs:subClassOf cheminf:CHEMINF_000000 ;
  dc:label "propane" ;
  cheminf:CHEMINF_000200 [
    a cheminf:CHEMINF_000113 ;
    sio:SIO_000300 "InChI=1/C3H8/c1-3-2/h3H2,1-2H3" .
  ] .

We can, however, also use Jericho to extract text from HTML pages, made available with the -html option, and pulling in a Beilstein Journal of Organic Chemistry paper with wget:

$ wget -qO- https://doi.org/10.3762/bjoc.6.122 | \
  java -cp oscar4-cli-4.0-SNAPSHOT.jar \
  uk.ac.cam.ch.wwmm.oscar.oscarcli.OscarCLI \
  -stdin -html

This will return 271 chemical entities recognized in the text, matching 48 unique chemical structures.

The source code of Oscar4 is available from this BitBucket project, and you can monitor the code state on this Hudson page. The project I will be working on, is in collaboration with the ChEBI project, and today we met up with various people in the group, and set out some really interesting use cases.

Semantics and Text Mining

Evan Prodromou wrote about RDFa vs microformats. The latter are commonly used in enhancing blog semantics , and for example used by PostGenomic.com. While RDFa is more explicit, e.g. by using namespaced markup, we have to wait until XHTML2 to see it working. I do not think chemists are using tags a log yet, but let me propose the following microformats: 1/CH4/h1H4 and methane. Standard JavaScripts and CSS scripts will then do the rest. (Think: addressing newlines, auto googling-for-inchi, etc).

The reason why using microformats is interesting, is text mining, of various kinds. Whether it is setting up a molecule-article link database, or find hot molecules in blogspace , adding semantics will help tools like OSCAR3 to mine chemistry . Some time ago OTMI was proposed by Nature , and they now set up a dedicated web site to explain there view on text mining. Zack Rosen has a good idea why RDF Semantic web research isn’t working.

Blogspace

There are a few new chemistry blogs I want to mention (and already added to Chemical blogspace ): ChemBark, lirico which has an interesting chemoinformatics section, and The Curious Wavefunction. Worth reading indeed.

Pierre’s YOKOFAKUN deserves a paragraph of his own. He recently blogged about bio2rdf which provides an RDF interface to biochemical knowledge via Life Science Identifiers (LSID), OBOEdit which is a Java-based ontology editor, and Amadea which is a Taverna- and KNIME-like tool for setting up UNIX pipes.

Online EMBL Symposium

A few EMBL PhD students are having the First Online EMBL PhD Symposium (catchy name, or … ;) Anyway, discussions are held on IRC, and it has a rather interesting Web2.0 session. All media is available on the website but requires registration right now. After the conference it will become open access to all. Jean-Claude contributed The UsefulChem Project: Open Source Chemistry Research using Blogs and Wikis to the Participants’ Contributions section, and I had a poster on Distributing molecular information over the Internet, discussing CMLRSS, blog aggregators, CML and other things. The IRC session was logged and is available here.

Literature

Finally, I want to mention three recent articles. First one is a recent write up by Bourne and Friedberg about Ten Simple Rules for Selecting a Postdoctoral Position (DOI: 10.1371/journal.pcbi.0020121). With the end of my current postdoc position nearing, rather useful reading. Some time ago I blogged about a New open access journal Source Code for Biology and Medicine , and the journal is now up and running. Details can be read in the first editorial (DOI: 10.1186/1751-0473-1-1). The third article I would like to mention is Scientific Software Development Is Not an Oxymoron by Baxter (DOI: 10.1371/journal.pcbi.0020087), though I do not think it has new insights.

OK, this was a rather lengthy write up, but really needed to clean up my toblog section :)

Oscar3, however, is capable of more than the name2structure of OPSIN (see also 10.1039/b411033a; it can take a plain text file with an experimental section with details on the synthesis of small organic compounds, and analyze the chemistry in that. This functionality has been available as an RSC authoring tool for some time now (see also 10.1039/b411699m). Unfortunately, what publisher put online (PDF and HTML) is much more difficult to process with Oscar3: those formats are often optimized for display, not for machine processing. The HTML can be cleaned up, but there is no general approach.

Christoph Steinbeck is going to present at the upcoming ACS meeting the use of Oscar3 for extraction of NMR spectra from old journal article, in preperation for submission to the NMRShiftDB.org (see the abstract of CINF 101).

Since the full Oscar3 was not hooked into Bioclipse yet, I had some work to do. It took me some time to figure out how to properly configure Oscar3, and what additional things I had to do to clean up the HTML used by publishers to get Oscar3 to extract NMR spectra (thanx to PeterC for hints!). I also had to tweak the Oscar3 code itself here and there, but that’s what opensource is about :) (Peter, if you are reading this: I have a number of patches for the Oscar3 code in bc_oscar; let me know if you’re interested in them.)

This is the end result:

Note especially the hierarchy in the resource navigator on the left. The misc folder contains all the chemistry found in the article. But more importantly is that for six molecules it fully detected he experimental section! For 3-(2-Oxocyclooctanyl)-3-phenylpropan-1-al (InChI=1/C17H22O2/c18-13-12-15(14-8-4-3-5-9-14)16-10-6-1-2-7-11-17(16)19/h3-5,8-9,13,15-16H,1-2,6-7,10-12H2) it derived the molecular structure (with OPSIN), and a few spectra: H-NMR, high-resolution MS and IR.

So, if you attend the ACS meeting: make sure to visit Christoph’s CINF 101 presentation!