Chemical blogspace has seen a lengthy discussion on the quality of a few NMR shift prediction programs ,\nand Ryan wanted to make a final statement. Down his blog item\nhe had this quote from Jeff, discussing the use of the NMRShiftDB as external test set:
\n\n\n\n\n“Of course customers are really interested in how accurately a prediction program can predict THEIR molecules - not a collection of external data such as NMRShiftDB.”
\n
I’m sure none of us knows what weird chemistry people are doing; we will never know what the overlap of the NMRShiftDB test\nset with the customer data set is. The quote suggests it is low, but we simply do not know.
\n\nThe accuracy of prediction models is very difficult to grasp, and one can only estimate it; using a test set.\nIf few data is available, one may opt for using the training set as test set too, and gives an estimate if the\nmodeling method is able to predict at all. However, the outcome of this exercise is the worst possible estimate\nyou can make. So, when possible you use an independent test set, which does not contain any molecules that were\npresent in the training set. (Actually, one could even suggest that this must happen on a shift level, but that\ngives problems with HOSE-code based prediction.)
\n\nNow, what Ryan stresses in his latest blog item\nis that prediction test results for the various available methods does not explicitly state the amount of overlap\nbetween the training and test set, one cannot draw any conclusions. Agreed. I would, however, like to tune this\neven a bit further, after reading the stupid quote (of course, taking out of context). What Jeff probably aimed\nat, is that the prediction accuracy is only meaningful to a customer if there is considerable between the customers\ndata set and the test set, which is what the model makers do not know.
\n\nAnd the overlap actually goes beyond the overlap in terms of molecular identity. It is really the overlap in terms\nof molecular substructures that matters: a database with alkanes but no phenyl rings will more accurately predict\nother alkanes not present in the training set (interpolation), but will not accurately predict compounds with\nphenyl rings (extrapolation). What the customer needs is that his personal data set does not require extrapolation.\nThat is what matters.
\n\nIt is interesting to realize, however, that the NMRShiftDB allows you to upload your molecules, or alternatively,\nyou download the software (it’s open source) and the data (it’s open data) if you don’t want to send your molecules\nover the internet, and the NMRShiftDB software will automatically take into account your own data set.
\n\nThus, if you are working on a series of related molecules, you can extend the NMRShiftDB data set with already\nelucidated structures, reducing the prediction error for your yet related unknowns derivatives. It is that easy\nto include prior/expert knowledge in the NMRShiftDB. I believe the ACD/Labs software allows this too, so the\nquote is really meaningless. Not correct, not wrong, simply says nothing.
\n\nNow, the various releases of the ACD/Labs software show a simple, understandable trend that increasing the number\nof data you use for the training set, reduces the prediction error. That’s because of various reasons I will not\ngo into in this item. The ACD/Labs NMR databases are expensive, because they have to manually extract and validate\nthe data from literature (see The Purgatory Database);\nso, during my PhD I only bought the CNMR and HNMR prediction packages. (Off topic: two weeks after I received my\ncopies of the software, ACD/Labs released a new version, which they kindly sent me a copy of too. Common in\nopensource, but much appreciated at that time. Cheers, ACD/Labs!)
\n\nThe ACD/Labs databases are likely expensive because of various reasons. And this is where the ODOSOS concept of the\nBlue Obelisk comes in. Open Data: if publishers would not copyright their data,\nNMR databases would be much cheaper to set up (see this thread in Peter’s blog );\nassuming ACD/Labs has to pay publishers for actually setting up their database. Open Source: the various Blue\nObelisk projects provide the tools to automatically create a purgatory NMR database ;\nno humans needed for that any more. Open Standards: the data from the NMRShiftDB can be downloaded in various\nformats, among which CMLSpect. Being able to easily read the data, made it possible that we actually have this\ndiscussion. Sure, the open data part of the NMRShiftDB is crucial too! But the database could have used an obscure,\nbinary, undocumented, with many software tweaks and special cases, .doc-like format, which no one could support.
Clearly, ODOSOS gives all, even proprietary, NMR prediction tools a boost, and I am very happy to see that happen.\nIt is the point that we, the Blue Obelisk Movement, are trying to make for some time now.
", "summary": "Chemical blogspace has seen a lengthy discussion on the quality of a few NMR shift prediction programs , and Ryan wanted to make a final statement. Down his blog item he had this quote from Jeff, discussing the use of the NMRShiftDB as external test set:", "date_published": "2007-07-13T00:00:00+00:00", "date_modified": "2025-07-30T00:00:00+00:00", "tags": ["nmr","nmrshiftdb"], "authors": [ { "name": "Egon Willighagen", "url": "https://orcid.org/0000-0001-7542-0286" } ] } ] }