![\"\"](\"/assets/images/c2w.png\")
Oscar is a text miner. It mines in text for chemistry.\nOscar4 is the next iteration of Oscar\ncode that I worked on in the past three months, with Lezan, Sam, and David. I blogged about\naspects of Oscar4 at several occasions:
\n\nThese posts will server is a some initial critical mass for a draft report I plan to finish\ntoday. I might have to blog some further posts with diagrams, here and there. This post is\nactually one of them, and discusses something where Oscar can be expected to go next, now\nthat the design is cleaned up (though this effort is not halted now) and it has become\npossible again to extend it. The over 250 unit tests\nmake this a lot easier too.
\n\nOne aspect where I expect Oscar to go in 2011 is the support for other languages. To a very\nlarge extend this is based on multi-language support in the dictionaries, as well as having\ntraining data in a particular language. This also provides some context to my earlier post\nabout the need for a Oscar training data repository .
\n\nThis extension opens a number of options: analysis of patent literature in other languages,\nmonitoring of press releases in other languages, and news items in local news papers, etc.\nFor example, it could analyse this C2W news item\non yeast cells:
\n\n
There are many use cases for such localized text mining. And it surely matters for determining\nthe impact of research.
\n\nOscar has various places where language specifics are found. For example, in tokenization of a\ntext. One step here is the detection of sentence ends. This is done in most western languages\nwith a period, exclamation mark, question mark, etc. But periods (dots) are also used in\nabbreviations. Similarly, colons can be used in chemical names. But the every language comes in\nwith different abbreviations that need to be recognized.
\n\nCurrently, some abbreviations are found in NonSentenceEndings.\nIn the past three months, we have been cleaning up the code, and restructured the source code,\nmaking it easier to detect such places. This class will likely undergo further refactoring, to\nmaking the list of such non-sentence-endings configurable via files or so. What I expect to see,\nis that we you initiate Oscar like this:
\n\nOscar oscar = new Oscar(Locale.US);\nThis might actually even make a nice student summer project. The biggest challenge will be in making a good\ncorpus of training data, like the SciBorg training data that was used for training Oscar3.
\n\nBut the whole normalization is tainted with English language specifics too. For example, the normalizer\nwill have to ‘normalize’ the question marks, for which there exist several\nunicode variations.\nBut the normalized variant is language dependent. For example, greek and armenian have different characters\n(see this page),\nand then we have not even started talking about the right to left.
\n\nBesides localized dictionaries, this Oscar will also benefit from a localized OPSIN.\nIt seem to recognize the Dutch propaan, but not\nbenzeen. I am not going to look at that soon, but if you are\ninterested, I recommend checking out Rich’\nposts \nabout \nforking \nOPSIN and writing patches.
\n\nGetting Oscar going for other languages is a challenge, but also offers new opportunities. Just email the\noscar mailing list\nif you are interested and need help.
\n\n![propaan](\"https://opsin.ch.cam.ac.uk/opsin/propaan.png\"){kind=link}
![benzeen](\"https://opsin.ch.cam.ac.uk/opsin/benzeen.png\"){kind=link}