@Grab(group='org.bridgedb', module='org.bridgedb.bio', version='3.0.23')
@Grab(group='org.bridgedb', module='org.bridgedb.rdb.construct', version='3.0.23')

import java.text.SimpleDateFormat;
import java.util.Date;

import org.bridgedb.IDMapperException;
import org.bridgedb.DataSource;
import org.bridgedb.Xref;
import org.bridgedb.bio.DataSourceTxt;
import org.bridgedb.rdb.construct.DBConnector;
import org.bridgedb.rdb.construct.DataDerby;
import org.bridgedb.rdb.construct.GdbConstruct;
import org.bridgedb.rdb.construct.GdbConstructImpl4;

DataSourceTxt.init()

GdbConstruct database = GdbConstructImpl4.createInstance(
  "test", new DataDerby(), DBConnector.PROP_RECREATE
);
database.createGdbTables();
database.preInsert();

inchikeyDS = DataSource.getExistingBySystemCode("Ik")
lmDS = DataSource.getExistingBySystemCode("Lm")
swisslipidsDS = DataSource.getExistingBySystemCode("Sl")

String dateStr = new SimpleDateFormat("yyyyMMdd").format(new Date());
database.setInfo("BUILDDATE", dateStr);
database.setInfo("DATASOURCENAME", "LIPIDMAPS_SWISSLIPIDS");
database.setInfo("DATASOURCEVERSION", "LIPID_TEST");
database.setInfo("DATATYPE", "Metabolite");
database.setInfo("SERIES", "standard_metabolite");

ref1 = new Xref("YECLLIMZHNYFCK-RRNJGNTNSA-J", inchikeyDS, true);
ref2 = new Xref("LMFA07050035", lmDS, false);
database.addGene(ref1)
database.addGene(ref2)
database.addLink(ref1, ref1)
database.addLink(ref1, ref2)

ref3 = new Xref("SLM:000000493", swisslipidsDS, true);
database.addGene(ref3)
database.addLink(ref1, ref3)

database.commit();
database.finalize();

For the people who have worked with BridgeDb Java in the past, note the new SQL schema 4, as used by the GdbConstructImpl4. This schema allows indicating of an identifiers is outdated/retired/etc. This is actually the case for the LMFA07050035 identifiers, and hence the false parameter in the new Xref() call.

So, when I was asked to work with Chemical Abstracts Service (CAS) on a new, bigger than ever version of Common Chemistry (which started as a project between CAS and Wikipedia), I welcomed the project. I don’t quite remember the first meetings, but roughly my task became to work with the new content and match this against Wikidata and Wikipedia. It aligned well with BridgeDb, Scholia, and our metabolomics research, so I even could find sufficient research time for it. This work is now published in the JCIM: CAS Common Chemistry in 2021: Expanding Access to Trusted Chemical Information for the Scientific Community (doi:10.1021/acs.jcim.2c00268).

Figure 2 from the article. Detailed record for caffeine in CAS Common Chemistry (image: CC-BY).

About Wikidata, the paper writes (CC-BY):

The latest release of CAS Common Chemistry has also supported updates and corrections to CAS RNs in Wikidata and Wikipedia. (22) InChIKeys were calculated from CAS SMILES using Bacting 0.0.31 (23) with the Chemistry Development Kit 2.7.1 (24) and were matched with content in Wikidata. The CAS RNs were then compared. References to CAS Common Chemistry were added for CAS RNs that matched. Mismatches have been shared with the Wikidata and Wikipedia communities so that they can manually review and correct the misleading entries using CAS Common Chemistry as a reference. Because Wikidata also curates identifiers from other data sources, validated CAS RNs in Wikidata may also be used to cross-reference with other resources. Scripts are provided in the Supporting Information.

The alignment is a continuous process, as new chemical compounds get added to Wikidata on a weekly basis. The comparison of Common Chemistry with Wikidata and Wikipedia resulted in a wealth of curation data, e.g. inconsistent CAS numbers linked to InChIKeys, where Common Chemistry had a different match than Wikidata or Wikipedia.

CAS registry numbers were not added to Wikidata in this process, only confirmed or reported as different. The latter allowed manual curation by the community, which it did. Reports look like this. When a InChIKey-CAS RN combination in Wikidata was confirmed, it was recorded as a reference, like this:

Screenshot of Wikidata with two references, one reflecting a confirmation by the English Wikipedia (potentially the result of the original Common Chemistry project) and the second as outcome of the now published project.

Thanks to everyone on this project and Andrea Jacobs particularly for leading this open science project.

Some updates by Denise, Martina, Tooba, Helena, and me:

• the project proposal was accepted and published in RIO Journal (doi:10.3897/rio.8.e83031)
• we started drawing various BridgeDb stories as UML diagrams using Mermaid
• updated the documentation in the BridgeDb Webservice repository
• an Ensembl 104-based gene/protein ID mapping database (doi:10.5281/zenodo.6367091)
• better unit test coverage of the BridgeDb Java library
• various CITATION.cff updates

There are some further things cooking, including an updated datasources.tsv and a few pull requests. I expect a new release of the BridgeDb Java library before the end of the month.

With these new results, we also updated the ISAAC database for the two new products (the published proposal and the gene/protein ID mapping database):

Right now, the ISAAC database does not make it easy to add content. Instead, there is a series of forms that have to be manually filled, including separate forms for authors. You cannot simply add a DOI. Well, until recent. Lars Willighagen and I developed a Chrome browser add-on to help out (also works with Brave), using his awesome citation-js (doi:10.7717/peerj-cs.214). The above two entries in the database have been added using this add-on.

We hope it will help other NWO grant holders too and that the add-on becomes obsolete in the near future Because the ISAAC database needs some updates elsewhere too. For example, it does not seem to value open source and open data so much yet:

That is a shame.

• the project started after writing our data management and software sustainability plans (mostly, GitHub+Zenodo)
• the project proposal has been submitted to RIO Journal
• created a private project in the Maastricht University GitLab instance (with all tasks as issues, so that we can monitor progress)
• first patches by Helena to the BridgeDb Java library
• factored out the BridgeDb Webservice into a separate (unpretty, see topright screenshot) repository, so that the BridgeDb Java library compiles again
• Marvin update the BridgeDb Docker with the latest BridgeDb 3.0.13 and the latest mapping files

It should be noted that FAIRplus has funded Chris’s team to work on identifier mapping too. Luc, Lucas, and now Tooba in our team have been working on Ensembl-based gene/protein identifier mappings and FAIRplus Cookbook recipes.

Not bad this progress in the first two weeks. We are ready now to start writing unit tests for much of the BridgeDb code. There were some, but a lot of code is used in production, but not formally tested. So far, the number of regressions due to updated libraries (dependencies) has been quite manageable. But with the work planned in this grant, we need more sustainable software, and therefore more unit testing. With the BridgeDb Webservice factored out, the code is compiling again and so is the code coverage testing.

The BridgeDb Webservice itself needs a rewrite from scratch. At least the mapping between underlying code (which we can reuse) and the REST calls. The library we used here has never been updated and I spent last weekend figuring out how to change the code, but gave up after two days. Rewriting is faster.

• provide a Java API for identifier mapping
• provide ID mappings (two flavors: with and without semantic meaning)
• provide services (R package, OpenAPI webservice)
• track the history of identifiers

The last one is more recent and two aspects are under development here: secondary identifiers and dead identifiers. More about that in some future post. About the first and the third I am also not going to tell much in this post. Just follow the above links.

I do want to say something in this post about the actually identifier mapping databases, in particular those we distribute as Apache Derby files, the storage format used by the Java libraries. These are the files you download if you want mapping databases for PathVisio (doi:10.1371/journal.pcbi.1004085). BridgeDb has mapping files for various things and some example databases the data it maps between:

1. genes and proteins: Ensembl, UniProt, NCBI Gene
2. metabolites; HMDB, ChEBI, LIPID MAPS, Wikidata, CAS
3. publications: DOI, PubMed
4. macromolecular complexes: Complex Portal, Wikidata

The BridgeDb API is agnostic to the things it can map identifiers for.

Downloading mapping files: BridgeDb has an BioSchemas-powered web page with an overview of the latest released mapping files. It looks like this:

This webpage is the result from the cyber attack in late 2019, disrupting a good bit of the infrastructure. This is why we renewed the website, including the download page. The new page actually is hosted on GitHub as a Markdown file, but this is where things get interesting. The Markdown file is actually autogenerated from a JSON file with all the info. Everything, including the BioSchemas annotation is created from that. Basically, JSON gets converted into Markdown (with a custom script), which gets converted into HTML by a GitHub Action/Pages. So, when someone releases a new mapping file on Zenodo or Figshare, they only have to send me a pull request with updated JSON file.

Now, previously, downloading all released mapping files, for example for the BridgeDb webservice, was a bit complicated. The information was a HTML file generated by the webserver for a folder. No metadata. Nuno wrote code to extract the relevant info and download all the files. However, since the information is now available in a public JSON file, it is a lot easier. The following code uses wget and jq, two tools readily available on the popular operating systems. Have fun!

!/bin/bash

wget -nc https://bridgedb.github.io/data/gene.json
wget -nc https://bridgedb.github.io/data/corona.json
wget -nc https://bridgedb.github.io/data/other.json

jq -r '.mappingFiles | .[] | "\(.file)=\(.downloadURL)"' gene.json > files.txt
jq -r '.mappingFiles | .[] | "\(.file)=\(.downloadURL)"' corona.json >> files.txt
jq -r '.mappingFiles | .[] | "\(.file)=\(.downloadURL)"' other.json >> files.txt

for FILE in $(cat files.txt)
do
  readarray -d = -t splitFILE<<< "$FILE"
  echo ${splitFILE[0]}
  wget -nc -O ${splitFILE[0]} ${splitFILE[1]}
done

Actually, while writing this blog post, I notice the code can be further simplified.

Bar chart showing the number of compounds with a particular chemical identifier.

Wikidata has many sub projects, such as WikiCite, which captures the collective of primary literature. Another one is the WikiProject Chemistry. The two nicely match up, I think, making a public database linking chemicals to literature (tho, very much needs to be done here), see my recent ICCS 2018 poster (doi:10.6084/m9.figshare.6356027.v1, paper pending).

But Wikidata is also a great resource for identifier mappings between chemical databases, something we need for our metabolism pathway research. The mapping, as you may know, are used in the latter via BridgeDb and we have been using Wikidata as one of three sources for some time now (the others being HMDB and ChEBI). WikiProject Chemistry has a related ChemID effort, and while the wiki page does not show much recent activity, there is actually a lot of ongoing effort (see plot). And I’ve been adding my bits.

Limitations of the links

But not each identifier in Wikidata has the same meaning. While they are all classified as ‘external-id’, the actual link may have different meaning. This, of course, is the essence of scientific lenses, see this post and the papers cited therein. One reason here is the difference in what entries in the various databases mean.

Wikidata has an extensive model, defined by the aforementioned WikiProject Chemistry. For example, it has different concepts for chemical compounds (in fact, the hierarchy is pretty rich) and compound classes. And these are differently modeled. Furthermore, it has a model that formalizes that things with a different InChI are different, but even allows things with the same InChI to be different, if need arises. It tries to accurately and precisely capture the certainty and uncertainty of the chemistry. As such, it is a powerful system to handle identifier mappings, because databases are not clear, and chemistry and biological in data is even less: we measure experimentally a characterization of chemicals, but what we put in databases and give names, are specific models (often chemical graphs).

That model differs from what other (chemical) databases use, or seem to use, because not always do databases indicate what they actually have in a record. But I think this is a fair guess.

ChEBI

ChEBI (and the matching ChEBI ID) has entries for chemical classes (e.g. fatty acid) and specific compounds (e.g. acetate).

PubChem, ChemSpider, UniChem

These three resources use the InChI as central asset. While they do not really have the concept of compound classes so much (though increasingly they have classifications), they do have entries where stereochemistry is undefined or unknown. Each one has their own way to link to other databases themselves, which normally includes tons of structure normalization (see e.g. doi:10.1186/s13321-018-0293-8 and doi:10.1186/s13321-015-0072-8).

HMDB

HMDB (and the matching P2057) has a biological perspective; the entries reflect the biology of a chemical. Therefore, for most compounds, they focus on the neutral forms of compounds. This makes linking to/from other databases where the compound is not neutral chemically less precise.

CAS registry numbers

CAS (and the matching P231) is pretty unique itself, and has identifiers for substances (see Q79529), much more than chemical compounds, and comes with a own set of unique features. For example, solutions of some compound, by design, have the same identifier. Previously, formaldehyde and formalin had different Wikipedia/Wikidata pages, both with the same CAS registry number.

Limitations of the links #2

Now, returning to our starting point: limitations in linking databases. If we want FAIR mappings, we need to be as precise as possible. Of course, that may mean we need more steps, but we can always simplify at will, but we never can have a computer make the links more complex (well, not without making assumptions, etc).

And that is why Wikidata is so suitable to link all these chemical databases: it can distinguish differences when needed, and make that explicit. It make mappings between the databases more FAIR.

The BridgeDb DataSource Ontology

This example uses the BridgeDb DataSource Ontology, created by BridgeDb developers from Manchester University (Christian, Stian, and Alasdair). The ontology covers describing data sources of identifiers, a technology outlined in the BridgeDb paper by Martijn (see below) as well as terms from the Open PHACTS Dataset Descriptions for the Open Pharmacological Space by Alasdair et al.

Because I needed to put this online for Open PHACTS (BTW, the project won a big award!) and our previous solution did not work well enough anymore. You may also see the HTML of the result first. You may also want to verify it really is HTML: here is the HTML5 validation report. Also, you may be interested in what the ontology in RDF looks like: here is the extracted RDF for the ontology. Now follow the HTML+RDFa snippets. First, the ontology details (actually, I have it split up):

<div about="http://vocabularies.bridgedb.org/ops#"
     typeof="owl:Ontology">
  <h1>The <span property="rdfs:label">BridgeDb DataSource Ontology</span>
    (version <span property="owl:versionInfo">2.1.0</span>)</h1>
  <p>
    This page describes the BridgeDb ontology. Make sure to visit our
    <a property="rdfs:seeAlso" href="http://www.bridgedb.org/">homepage</a> too!
  </p>
</div>
<p about="http://vocabularies.bridgedb.org/ops#">
  The OWL ontology can be extracted
  <a property="owl:versionIRI"
     href="http://www.w3.org/2012/pyRdfa/extract?uri=http://vocabularies.bridgedb.org/ops#">here</a>.
  The Open PHACTS specification on
  <a property="rdf:seeAlso"
    href="http://www.openphacts.org/specs/2013/WD-datadesc-20130912/#bridgedb"
  >Dataset Descriptions</a> is also useful.
</p>

This is the last time I show the custom color coding, but for a first time it is useful. In red are basically the predicates, where @about indicates a new resource is started, @typeof defines the rdf:type, and @property indicates all other predicates. The blue and green blobs are literals and object resources, respectively. If you work this out, you get this OWL code (more or less):

bridgedb: a owl:Ontology;
  rdfs:label "BridgeDb DataSource Ontology"@en;
  rdf:seeAlso
    <http://www.openphacts.org/specs/2013/WD-datadesc-20130912/#bridgedb>;
  rdfs:seeAlso <http://www.bridgedb.org/>;
  owl:versionIRI
    <http://www.w3.org/2012/pyRdfa/extract?uri=http://vocabularies.bridgedb.org/ops#>;
  owl:versionInfo "2.1.0"@en .

An OWL class

Defining OWL classes are using the same approach: define the resource it is @about, define the @typeOf and giving is properties. BTW, note that I added a @id so that ontology terms can be looked up using the HTML # functionality. For example:

<div id="DataSource"
  about="http://vocabularies.bridgedb.org/ops#DataSource"
  typeof="owl:Class">
  <h3 property="rdfs:label">Data Source</h3>
  <p property="dc:description">A resource that defines
    identifiers for some biological entity, like a gene,
    protein, or metabolite.</p>
</div>

An OWL object property

Defining an OWL data property is pretty much the same, but note that we can arbitrary add additional things, making use of <span>, <div>, and <p> elements. The following example also defines the rdfs:domain and rdfs:range:

<div id="aboutOrganism"
  about="http://vocabularies.bridgedb.org/ops#aboutOrganism"
  typeof="owl:ObjectProperty">
  <h3 property="rdfs:label">About Organism</h3>
  <p><span property="dc:description">Organism for all entities
    with identifiers from this datasource.</span>
    This property has
    <a property="rdfs:domain"
      href="http://vocabularies.bridgedb.org/ops#DataSource">DataSource</a>
    as domain and
    <a property="rdfs:range"
      href="http://vocabularies.bridgedb.org/ops#Organism">Organism</a>
    as range.</p>
</div>

So, now anyone can host an OWL ontology with dereferencable terms: to remove confusion, I have used the full URLs of the terms in @about attributes.