![\"\"](\"/assets/images/scholia_comparison.png\")
Scholia is a visual layer on top of Wikidata providing\na rich user experience for browing scholarly research related knowledge. I am using the combinatie\nfor various things, including exploring new research topics (a method, compound, or protein I do not know so much\nabout yet), indexing notable research output (including citations), progress of Citation Typing Ontology\nuptake, etc. This weekend I hope to send around the\nfinal draft for the Scholia Chemistry paper.
\n\nScholia has received a fair share of scholarly and social attention. The Scholia paper has been cited\nover 100 times and\nthe websites received about 200 thousand page views each day (though we do not know how to get Toolforge\nto give us sufficient insight into the how and what of that count). There is a Wikipedia template to link\nto Scholia and some of projects I am involved in link Scholia for articles, such as\nWikiPathways.
\n\nWith that, there is also interest in using it for other Wikibases and perhaps even random SPARQL endpoints.\nThese things are not trivial, as Scholia uses complementary APIs, various URL patterns for some of the\nfunctionality, and generally, all SPARQL queries are tweaked to the Wikidata Blazegraph SPARQL endpoint\nto ensure results are returned in reasonable time. But that last requires use of Blazegraph extensions\nto the SPARQL standard.
\n\nAll this requires Scholia to become more independent, in a better model-view-controller model. And that\nactually turns out very important at this moment. That is, Wikidata is not a RDF-first database, but\na Wikibase-based store. Whenever an edit is made, RDF is generated and the SPARQL endpoint is updated.\nNow, the number of edits in Wikidata is enormous and the notion that the SPARQL endpoint is often minutes\nat most behind is a huge accomplishment. But the Blazegraph platform cannot keep up with Wikidata.\nBlazegraph is open source, but has been bought up and development stopped from one day to another.
\n\nTherefore, a split of the Wikidata SPARQL platform is planned.\nThis split will put one part of\nthe knowledge in on endpoint and the other half in the other. Any query that needs information\nfrom both graphs, will have to do a federated SPARQL query. Basically, there are very few Scholia\nqueries that do not rewriting. My first rewrite actually failed, because the rewriting is not\nobvious and quickly times out. To some extend, this is because now lots of results of subqueries\nneed to be send over the network from one endpoint to the other. When the combined query basically\ncovers half of each endpoint, that’s a lot of network traffic.
\n\nAn immediate use case of the configuration is therefore running Scholia against the current three\nendpoints: the current official endpoint, and the two split endpoints under development. With\na recent patch Finn\nand I worked on, this configuration looks like this (and saved as scholia.ini:
[query-server]\n# Wikidata:\n#sparql_endpoint = https://query.wikidata.org/sparql\n#sparql_editurl = https://query.wikidata.org/#\n#sparql_embedurl = https://query.wikidata.org/embed.html#\n\n# Wikidata Split Main\nsparql_endpoint = https://query-main.wikidata.org/sparql\nsparql_editurl = https://query-main.wikidata.org/#\nsparql_embedurl = https://query-main.wikidata.org/embed.html#\n\n# Wikidata Split Scholar\n#sparql_endpoint = https://query-scholarly.wikidata.org/sparql\n#sparql_editurl = https://query-scholarly.wikidata.org/#\n#sparql_embedurl = https://query-scholarly.wikidata.org/embed.html#\nSo, right now, we can test the impact of the split with Scholia and this patch.\nWe would fire up a local instances of Scholia, running against one of the\nsplit endpoints, and use the Toolforge instance as baseline.
\n\nNow, on my system I need to use Python virtualenv\nso, I first start a Scholia venv:
source ~/.venvs/scholia/bin/activate\nAfter that, I can select an other endpoint, e.g. the main Wikidata split endpoint (query-main-experimental.wikidata.org)\nwere it not they are currently offline as part of the transition\nand run Scholia on a unique port:
scholia run\nThen I can have two browser windows along side and compare Scholia pages againt the current\nScholia instance and when running against another SPARQL endpoint. For now, I can test how well\nScholia runs on the QLever instance of Wikidata (superfast and\nupdated data once a week). Here the configuration I have is not entirely complete, and many\nSPARQL queries do not work against QLever, including anything with graphical depiction. But\nthat said, I can use this configuration:
\n\n[query-server]\n# QLever\n#sparql_endpoint = https://qlever.cs.uni-freiburg.de/api/wikidata\n#sparql_editurl = https://qlever.cs.uni-freiburg.de/wikidata/?query=\n#sparql_embedurl = \nThen, I can compare, for example, the chemicals statistics the main Scholia with one running\nagainst QLever:
\n\n
This query ran without modification. For other queries rewriting is needed, but with this\nsetup we can at least quickly see the differences in the results.
\n\n