submissionamazonec2

JAutomReasoning(2017)59:455–482DOI10.
1007/s10817-017-9406-8TheOWLReasonerEvaluation(ORE)2015CompetitionReportBijanParsia1·NicolasMatentzoglu1·RafaelS.
Gonalves2·BirteGlimm3·AndreasSteigmiller3Received:31March2016/Accepted:1February2017/Publishedonline:21February2017TheAuthor(s)2017.
ThisarticleispublishedwithopenaccessatSpringerlink.
comAbstractTheOWLReasonerEvaluationcompetitionisanannualcompetition(withanassociatedworkshop)thatpitsOWL2compliantreasonersagainsteachotheronvariousstandardreasoningtasksovernaturallyoccurringproblems.
The2015competitionwasthethirdofitssortandhad14reasonerscompetinginsixtrackscomprisingthreetasks(consis-tency,classication,andrealisation)overtwoproles(OWL2DLandEL).
Inthispaper,wediscussthedesign,executionandresultsofthe2015competitionwithparticularattentiontolessonslearnedforbenchmarking,comparativeexperiments,andfuturecompetitions.
KeywordsOWL·Ontologies·Reasoning1IntroductionTheWebOntologyLanguage(OWL)isinitsseconditeration(OWL2)[13,29]andhasseensignicantadoptionespeciallyinHealthCare(see,e.
g.
,[20,38])andLifeSciences(see,e.
g.
,[5,12,48]).
OWL2DLcanbeseenasavariantofthedescriptionlogic(DL)SROIQBNicolasMatentzoglunicolas.
matentzoglu@manchester.
ac.
ukBijanParsiabijan.
parsia@manchester.
ac.
ukRafaelS.
Gonalvesrafael.
goncalves@stanford.
eduBirteGlimmbirte.
glimm@uni-ulm.
deAndreasSteigmillerandreas.
steigmiller@uni-ulm.
de1InformationManagementGroup,UniversityofManchester,Manchester,UK2StanfordCenterforBiomedicalInformaticsResearch,StanfordUniversity,Stanford,CA,USA3InstituteofArticialIntelligence,UniversityofUlm,Ulm,Germany123456B.
Parsiaetal.
[2,18].
ThethreeprolesintroducedinOWL2(calledOWLEL,OWLQL,andOWLRL)[26]correspondtologicalfragmentsofSROIQandweredesignedtoallowforamoresimpleorefcientimplementation.
Finally,OWL2FullisasyntacticextensionofOWL2DLthatdoesnotcorrespondtoadescriptionlogic.
Descriptionlogicsgenerallyaredesignedtobecomputationallypracticalsothat,eveniftheydonothavetractableworst-casecomplexityforkeyservices,theyneverthelessadmitimplementationsthatseemtoworkwellinpractice[10].
UnliketheearlydaysofdescriptionlogicsorevenofthedirectprecursorsofOWL(DAML+OIL[6]),thereasonerlandscape[22,37]forOWLisrich,diverse,andhighlycompliantwithacommon,detailedspecication.
Thus,wehavealargenumberofhighperformance,production-qualityreasonerswithsimilarcorecapacities(withrespecttolanguagefeaturesandstandardinferencetasks).
ResearchonoptimisingOWLreasoningcontinuesapace,thoughempiricalworkstilllagsbehindboththeoreticalandengineeringworkinbreadth,depth,andsophistication.
Thereis,ingeneral,alackofsharedunderstandingoftestcases,testscenarios,infrastructure,andexperimentdesign.
Acommonstrategyinresearchcommunitiestohelpaddresstheseissuesistoholdcompetitions,thatis,experimentsdesignedandhostedbythirdpartiesonaninde-pendent(oftenconstrained,butsometimesexpanded)infrastructure.
Suchcompetitions,incontrasttopublishedbenchmarks,donotalwaysprovideindepthempiricalcharacterisa-tionsofthecompetingtools.
Instead,theyservetwokeyfunctions:(1)theyprovideaclear,motivatingeventthathelpsdrivetooldevelopment(e.
g.
,forcorrectnessorperformance)and(2)componentsofthecompetitionareusefulforsubsequentresearch.
Finally,competitionscanbegreatfunandhelpfosterastrongcommunity.
Theycanbeespeciallyusefulfornew-comersbyprovidingasimplewaytogainsomeprimafacievalidationoftheirtoolswithouttheburdenofdesigningandexecutingcomplexexperimentsthemselves.
Towardtheseends,wehavebeenrunningacompetitionforOWLreasoners(withanassociatedworkshop[3,4,7,15]):theOWLReasonerEvaluation(ORE)competition[33].
OREhasbeenrunning,insubstantivelyitscurrentform,for3years.
Inthispaperwedescribethe2015competition(heldinconjunctionwiththe28thInternationalDescriptionLogicWorkshop(DL2015)1inJune2015.
Thecompetitioncomprisestwodifferentcomponents:thelivecompetition,theheartofORE,pitsanumberofcompetingreasonersagainsteachotheronacarefullycraftedcorpusofOWLontologies,featuringatimeoutof3minandasinglerun;andtheofinecompetition,whichfeaturesparticularlyreasoningintensiveontologiessubmittedbytheontologyengineeringcommunityandrunswitha6htimeoutperontologyandreasoner.
Anoverviewofallresources(reasoners,ontologycorpus,competitionresultdataandanalysisscripts,competitionframework)isalsoavailableonline.
2Thecontributionofthispaperconsistsofadiscussionofthegeneralcompetitiondesignandexecutionaswellasasummaryoftheresultsofthe2015competitionwithparticularattentiontolessonslearnedforbenchmarking,comparativeexperiments,andfuturecom-petitions.
Whilemanyloglesandstatisticsofthecompetitionarepubliclyavailable,theaggregatedresultsandtheiranalysis,aspresentedinthispaper,providein-depthinsightsthatareotherwisequitetime-consumingtoobtain.
Thedescriptionofthecompetitionframeworkallowsdeveloperstoeasilyrerunthecompetitionwithneworupdatedreasonerstogetasenseoftheirrelativeprogress.
Thediscussionofthecompetitiondesignfostersasharedunderstandingoftestcases,testscenarios,infrastructure,andexperimentdesignwithintheDLcommunity.
TheORE2015corpus,whichwedescribeinthispaper,isasignicant1ThewebsitesforDL2015andORE2015arearchivedathttp://dl.
kr.
org/dl2015/andhttps://www.
w3.
org/community/owled/ore-2015-workshop/respectively.
2http://owl.
cs.
manchester.
ac.
uk/publications/supporting-material/ore-2015-report/.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport457anddistinctcorpusforreasonerexperimentationwhetherusedwiththeOREframeworkorinacustomtestharness.
TheOREtoolkitandcorporamayfurtherserveasanucleusforaninfrastructureforcommonexperimentation.
Someofthelessonslearnedmightinspirecompetitionorganisersinothereldsorcommunitieswhowanttoestablishacompetitionfortheirresearcharea.
Theremainderofthispaperisorganisedasfollows:wenextintroducesomepreliminariesregardingOWL.
Section3introducestheoverallcompetitiondesign,thecompilationoftheusedontologycorpus,adescriptionoftheontologiescontributedbyusers,theframeworktorunthecompetitionandtheusedtechnicalenvironment.
Section4describesthepartici-patingsystems.
Sections5and6introducethesetupandoutcomeoftheliveandtheofinecompetition,respectively.
Finally,weconcludeinSect.
7withasummaryofthecompetitionresultsandsomechallengesthatshouldbeaddressedinfuturecompetitions.
2PreliminariesBeforewedescribethecompetitionset-up,werstgivebriefintroductiontoOWLasrelevantfortheremainderofthepaper.
ForafulldenitionofOWL2,pleaserefertotheOWL2StructuralSpecicationandDirectSemantics[28,29].
AdomainofinterestcanbemodelledinOWL2bymeansofindividuals(whichdenoteobjectsfromthedomainofdiscourse),literals(whichdenotedatavalues,suchasstringsorintegers),classes(whichdenotesetsofindividuals),datatypes(whichdenotesetsofdatavalues),objectproperties(whichrelatepairsofindividuals),anddataproperties(whichrelateindividualswithconcretevalues).
Individuals,classes,datatypes,andobjectpropertiescanbeusedtoformclassexpressions,dataranges,andobjectpropertyexpressions,respec-tively;thesearecomplexdescriptionsofsetsofindividuals,setsofliterals,andrelationshipsbetweenindividuals.
Finally,classexpressions,dataranges,objectpropertyexpressions,dataproperties,individuals,andliteralscanbeusedtoformaxioms—statementsthatdescribethedomainbeingmodelled.
Axiomsdescribingindividualsarecommonlycalledassertions.
AnOWL2ontologyOisanitesetofaxioms.
ThesemanticsofaxiomsinanOWLontologyOisgivenbymeansoftwo-sortedinter-pretationsovertheobjectdomainandthedatadomain,wherethelattercontainswell-knowndatavaluessuchasintegersandstrings.
AninterpretationImapsindividualstoelementsoftheobjectdomain,literalstoelementsofthedatadomain,classestosubsetsoftheobjectdomain,datatypestosubsetsofthedatadomain,objectpropertiestosetsofpairsofobjectdomainelements,anddatapropertiestosetsofpairswhoserstcomponentisfromtheobjectdomainandwhosesecondcomponentisfromthedatadomain.
AnindividualiisaninstanceofaclassCinaninterpretationIiftheimageofCcontainstheimageofi.
AninterpretationIisamodelofanontologyOifIsatisesallconditionslistedin[28].
Forexample,ifOcontainsanaxiomstatingthatCisasubclassofD,thentheconditionsfrom[28]requireeachinstanceofCinItoalsobeaninstanceofDinI.
IftheaxiomsofOcannotbesatisedinanyinterpretation(i.
e.
,ifOhasnomodel),thenOisinconsistent;otherwise,Oisconsistent.
IftheinterpretationofaclassCiscontainedintheinterpretationofaclassDinallmodelsofO,thenCisasubclassofD(or,equivalently,DsubsumesC)inO.
IftheinterpretationofanindividualiiscontainedintheinterpretationofaclassCinallmodelsofO,theniisaninstanceofCinO.
Conventionally,thesetofaxiomsisdividedintotwoparts,theTBoxandtheABox.
TheTBoxcomprisesconceptdenitionsandinclusionsandcorrespondstothe"schema"partof123458B.
Parsiaetal.
theontology.
TheABoxisacollectionofgroundassertionswhichcorrespondstothe"data"partoftheontology.
Eachparthascharacteristicreasoningtasks,e.
g.
,classicationfortheTBoxandinstantiationfortheABox.
3CompetitionDesignTheOREcompetitionisinspiredbyandmodelledontheCADEATPSystemCompetition(CASC)[34,44]whichhasbeenrunningfor25yearsandhasbeenheavilyinuentialintheautomatedtheoremprovingcommunity3(especiallyforrst-orderlogic).
Weobservethatcentraltosuchcompetitionsisparticipation,thusvariousincentivestoparticipatearecriticalespeciallyintheearlyyearsofthecompetitionasitistryingtogetestablished.
Hencetheimportanceof"fun"elements,incentives(e.
g.
,prizes,braggingrights),aswellasareasonablechanceofwinningatleastsomething.
ThekeycommonelementsbetweenOREandCASCare:1.
Anumberofdistincttracks/divisions/disciplinescharacterisedbyproblemtype(e.
g.
,"effectivelypropositional"or"OWL2ELontology").
2.
Thetestproblemsarederivedfromalarge,neutral,updatedyearlysetofproblems(e.
g.
,forCASC,theTPTPlibrary[43]).
3.
Reasonerscompete(primarily)onhowmanyproblemstheyareabletosolvewithinagiventimeout.
Asdescriptionlogicshaveavariedsetofcoreinferenceservicessupportedbyessentiallyallreasoners,OREalsohastrackdistinctionsbasedontask(e.
g.
,classicationorrealisation).
OtherCASCinspiredelements:1.
ThereasonerrankingisderivedsolelyfromalivecompetitionrunduringtheDescriptionLogicworkshop,i.
e.
,theofineperformanceevaluationacrossusersubmittedontologiesdoesnotfeedintotheranking.
2.
TherewasasecondarycompetitionamongDLattendeestopredicttheresultsforvariousreasoners.
3.
CompetitorsandorganisersweregivenT-shirtsdesignedspecicallyfortheevent,wherethedesigngoesbeyondthetypicalprintingofeventnamesandlogos.
3.
1TracksORE2015hadsixtracksbasedonthreecentralreasoningservices(consistency,classication,andrealisation)andtwoOWLproles(OWLDLandEL).
Theseservicesarenotubiquitouslysupported,withrealisationnothandledbysomereasoners.
Weusethefollowingdenitionsfortheseservices(thoughanyconsequenceequivalentdenitionwoulddo):–ConsistencycheckingisthetaskofdeterminingwhetheranontologyOisconsistentornot.
–Classicationisthetaskofcomputingallentailedclasssubsumptionsbetweennamedclassesintheontology.
–Ontologyrealisationreferstocomputingallentailedclassassertionsfornamedclassesandindividualnamesoccurringintheontology,i.
e.
,thecomputationofallinstancesforallnamedclassesintheontology.
Thistasksisalsoknownasmaterialisation.
3SeetheCASCwebsitefordetailsonpastcompetitions:http://www.
tptp.
org.
Alsoofinterest,thoughnotdirectlyinspirationalforORE,istheSATcompetitionhttp://www.
satcompetition.
org.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport459Consistencyis,insomesense,themostfundamentalservice.
Classicationis,almostcer-tainly,themostcommonandimportantreasoningserviceforontologiestodate.
Realisationgetsusatleastaminimalformofinstancereasoning.
WeaimtoextendthecompetitionbyotherOWLproleswhenwehaveenoughparticipantsthatarespecicallytunedforthatprole.
InprioryearswealsohadanRLtrack,butthenumberofRL-specicreasonersisverylow.
WehopetointroduceaconjunctivequerytrackinfutureyearsanddiscusssomeofthechallengesinSect.
7.
AllreasonerspurportingtohandletheentiretyofOWL2DLareenteredinalltracks.
Thus,wehavespecialisedELreasonerscompetingagainstfully-edgedOWLDLreasoners.
Foreachtrack,weawardprizestothetopthreeparticipantsforatotalof18possiblewinners.
Awardsareonlygivenforthewinnersofthelivecompetition.
Theofinecom-petitionisaimedatinformingtheontologyandreasonerdevelopersofpotentialissuesaswellasengagingtheontologydevelopmentcommunity.
Forreasonerdevelopers,theofinecompetitiontypicallyfeatureharderandlogicallyexpressiveontologieswhichhaveproventroublesomeforusers.
Forontologydevelopers,inadditiontobringingtheirontologiesinviewofreasonerdevelopers,theyhavetheirontologiestestedonawiderrangeofreasonersinarobustsetting.
3.
2LiveCompetitionCorpusThefulllivecompetitioncorpuscontains1920ontologies.
Eachcompetitioncomeswithitsownrandomstratiedsampleofontologiesfromthisbasecorpusforthelivecompetition—thatisnotall1920ontologiesareactuallyusedinalivecompetition.
Thecompetitioncorpusissampledfromthreesourcecorpora:aJanuary2015snapshotofBioportal[30]containing330biomedicalontologies,theOxfordOntologyLibrary4with793ontologiesthatwerecollectedforthepurposeofontology-relatedtoolevaluation,andMOWLCorp[21],acorpusbasedona2014snapshotofaWebcrawlcontainingaround21,000uniqueontologies.
Theontologiesinthecorpuswerepre-processedusingtheOWLAPI(v3.
5.
1)[14].
Asarststep,theontologiesofallthreesourcecorporawerecollectedandserialisedintoOWL/XMLwiththeirimportsclosuremergedintoasingleontology.
Themergingis,fromacompetitionperspective,necessarytomitigatethebottleneckofloadingpotentiallylargeimportsrepeatedlyoverthenetwork,andbecausethehostsoffrequentlyimportedontologiessometimesimposerestrictionsonthenumberofsimultaneousaccesses.
5Afterthecollection,theentirepoolof21,465ontologieswasdividedintothreegroups:(1)Ontologieswithlessthan50axioms(12,927ontologies),(2)OWL2DLontologies(4199),and(3)OWL2Fullontologies(4339).
Therstgroupwasremovedfromthepool.
Asreasonerdeveloperscouldtunetheirreasonerstowardstheontologiesinthethreepubliclyavailablesourcecorpora,weincludedanumberofapproximationsintoourpool.
TheentiresetofOWL2FullontologieswereapproximatedintoOWL2DL,i.
e.
,weuseda(slightlymodied)versionoftheOWLAPIprolecheckertodropDLprole-violatingaxiomssothattheremainderisinOWL2DL[23].
Becauseofsomeimperfectionsinthe"DLication"process,thisprocesshadtobeperformedtwice.
Forexample,intherstround,theDLexpressivitycheckermayhavenotedamissingdeclarationandanillegalpunning.
Fixingthiswouldresultindroppingtheaxiom(s)causingtheillegalpunningaswellasinjectingthedeclaration—whichcouldresultagaininanillegalpunning.
4http://www.
cs.
ox.
ac.
uk/isg/ontologies/.
5Whichmaybeexceededconsideringthatallreasonersinthecompetitionruninparallel.
123460B.
Parsiaetal.
DLEL020406002040600204060ConsistencyClassificationRealisationVerySmallSmallMediumLargeVeryLargeVerySmallSmallMediumLargeVeryLargeSamplebinsCountFig.
1ThenumberofontologiessampledforeachsizebinTheOWL2DLgroupwasthenapproximatedusingtheOWL2EL/QLapproximationmethodemployedbyTrOWL[35].
Thisresultedina8644successfulapproximations.
Astheonlysyntaxthatisuniformlysupportedbyallreasonersparticipatinginthecompetition,weserialisedtheentirepool(includingtheoriginalOWL2DLontologies,theapproximatedontologies,andthe"DLied"OWL2Fullontologies)intoFunctionalSyntax,andgatheredallrelevantontologymetricsagain.
Assomeontologiesareincludedinmorethanoneofthesourcecorpora,weexcludedatthispoint(asalastpre-processingstep)allduplicates6fromtheentirepoolofontologiesandremovedontologieswithTBoxescontaininglessthan50axioms.
Therandomstratiedsamplingforthecompetitionthenwasdoneasfollows:Allontologieswerebinnedbysizeintothefollowinggroups:Verysmall(50–99axioms),small(100–999axioms),medium(1000–9999axioms),large(10,000–100,000axioms)andverylarge(morethan100,000axioms).
Fromeachgroup,weattemptedtosample60originalontologies,and15approximated(i.
e.
,the"ELied"and"DLied")ontologiesforeachcompetition.
FortheOWL2ELrelatedtracks,theontologieshadtofallundertheOWL2ELprole,fortheOWL2DLcompetitions,theontologieshadtofallunderOWL2DLbutnotunderanyofthethreeOWL2proles,andforthetworealisationchallengesweonlyconsideredthoseontologiesthathadatleast100ABoxaxioms.
Thisprocessresultedinthefollowingsixlivecompetitioncorpora:306forOWLDLConsistencyandClassication,264forOWLDLRealisation,298forOWLELConsistencyandClassication,and109forOWLELRealisation.
Figure1showstheresultsofthesampling,i.
e.
thenumberofontologiesforeachbin.
Thefullcompetitioncorpus(1920uniqueOWL2DLontologies),andtheexecutionorderofthecompetition,canbeobtainedfromZenodo[24].
3.
3UserSubmittedOntologiesTheofinecompetitioncorpusconsistsofontologiessubmittedbyusers.
ForORE2015,wehadfourusersubmissionsconsistingofatotalofsevenontologies.
Theusersubmissionsunderwentthesamepre-processingproceduresasthecorpus(Sect.
3.
2).
Thisoccasionallyhadlargeconsequencesontheontologies,mostimportantlywithrespecttorules(theywerestrippedout)andanyaxiombeyondOWL2DL(forexample,axiomsredeningbuilt-in6Duplicatesarethosethatarebyteidenticalafterbeing"DLied"andserialisedintoFunctionalSyntax.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport461vocabularyorviolatingtheglobalconstraintsonrolehierarchies,see[23]).
Therefore,theresultsoftheofinecompetitioncannotserveasthenalanswertothequestionofwhichreasoneristhebestfortherespectiveontologydevelopers,butwillhopefullygivedirection.
Theuser-submittedcorpushastwoparts:onecontainingthesubmissionsforORE2015,andonefortheusersubmissionsof2014.
7Wewillonlyprovideadetailedbreak-downoftheresultsforthe2015corpus,andpresenttheaggregatedresultsforthe2014corpus.
ThefollowingontologiesweresubmittedtoORE2015:–CellOntology(CO):8COisdesignedasacontrolledvocabularyforcelltypes.
Itisnotorganismspecic,coveringvariouscelltypesfrommammalstoprokaryotes.
9–Drug-DrugInteractionsOntology(DINTO):10DINTOisapharmacologicalontologythatsystematicallyorganisesdrug-druginteraction(DDI)relatedknowledgethatcontainspharmacologicalsubstances,proteinsandrelationshipsamongthem.
DDIsarerepre-sentedatclasslevel.
ThesubmissioncontainedveversionsofDINTO.
Unfortunately,OREreasonersarenotrequiredtodealwithSWRLrules;therefore,onlythebasic(andpre-processed)versionofDINTOwasadmittedtothecompetition.
11–DrosophilaPhenotypeOntology(DPO):8DPOwasdesignedasaqueryableclassicationofphenotypesfromtheFlyBaseknowledgebase[31].
12–GeneOntologyPlus(GO-PLUS):8GO-PLUSisthefullyaxiomatisedpublicreleaseoftheGeneOntology.
Itincludesaxiomsreferencingclassesfrommultipleexternalontologies.
13–VirtualFlyBrainOntologies(VFB):8ThreeoftheVFBontologiesweresubmitted:VFB-KB,combingtheDrosophilaanatomyontology(DAO)withaknowledgebaseofanatomicalindividuals,VFB-EPNT,anexperimentalextensionofDAOincludingspatialdisjointnessaxiomsfortheadultbrain,combinedwithaknowledgebaseofexpressionpatternsincludingexplicitnegation,andVFB-NCT,anexperimentalextensionofDAOincludingspatialdisjointnessaxiomsfortheadultbrainaswellasclosureaxiomsonthesynapsingpatternsofneurons(i.
e.
,thisisintheDLproleofOWL).
14AllontologiessubmittedtoORE2015areproperOWLDLontologies,i.
e.
,theydonotfallintoanyoftheOWL2proles.
Metricsregardingthenumberofaxiomsandtheuseddescriptionlogic(expressivity)fortheontologiescanbefoundinTable1.
Seeingasonly3ofthesubmittedontologiescontainABoxaxioms(theVFBvariants),theseweretheonlyontologiestestedintherealisationtrack.
ThesubmissionsfromORE2014includethefollowing:–CityBenchmark(CB)15[8].
–DataMiningOntologies(DMOP)16[17].
7Wehaveincludedtheusersubmittedontologiesof2014becauseweneglectedtoevaluatethemseparatelyatthetimeofthe2014competition.
8SubmittedbyDr.
DavidOsumi-Sutherland,GOEditorialOfce,EuropeanBioinformaticsInstitute,Euro-peanMolecularBiologyLaboratory,WellcomeTrustGenomeCampus,Hinxton,Cambridge,UK.
9Availableathttps://github.
com/obophenotype/cell-ontology.
10SubmittedbyMaríaHerrero,ComputerScienceDepartment,UnivesidadCarlosIIIdeMadrid.
Leganés,Spain.
11Allversionsareavailableathttps://code.
google.
com/archive/p/dinto/.
12Availableathttps://github.
com/FlyBase/ybase-controlled-vocabulary.
13Availableathttp://purl.
bioontology.
org/ontology/GO-PLUS.
14Allontologiesareavailableathttps://github.
com/VirtualFlyBrain.
15Availableathttps://github.
com/ghxiao/city-bench.
16Availableathttp://www.
e-lico.
eu/DMOP.
html.
123462B.
Parsiaetal.
Table1Breakdownofuser-submittedontologiesintheORE2015corpusOntologyTBoxABoxOntologyTBoxABoxCO75270VFB-EPN33,61263,295DINTO123,9300VFB-KB20,187147,996DPO9170VFB-NCT33,61263,295GO+150,9550Table2Breakdownofuser-submittedontologiesintheORE2014corpusOntologyTBoxABoxOntologyTBoxABoxCB-BERN428209,932MSC-D95320CB-CORK42820,393MSC9532318DCHARS19251728FHKB-V34253307DGO23347,603RMO-A192515,759DMKB19251606FHKB-V13553296DMOP1986765DPC-OLY12235,866DPC-112254,898PD1930973DPC-212279,955FHKB-V24193304FMA-CPFNS123,02486USDA101743602FTC140,7990USDA151765948GALEN-FU37,4110USDA201768600GALEN-H10,6280USDA251779785G-CDS43220USDA51741226G-CDS-D4322140CB-VIENNA428584,266HP12317,027–USDAandDPC17[49].
–SomevariantsofGALENandFMA.
–TheGenomicClinicalDecisionSupportOntology(G-CDS)18[36].
–TheFamilyHistoryKnowledgeBase(FHKB)19[42].
Acompletelistoftheontologiessubmittedin2014withmetricsregardingthenumberofaxiomsintheontologiesispresentedinTable2.
CB-BERN,CB-CORK,andCB-VIENNAfallintotheOWL2QLprole,FTCbelongstotheOWL2ELprole,andallotherontologiesareproperOWLDLontologies,i.
e.
,theydonotfallintoanyoftheOWL2proles.
3.
4TheCompetitionFrameworkThecompetitionframeworkusedinORE2015isaslightlymodiedversionoftheoneusedforORE2014,whichisopensourcedundertheLGPLlicenseandavailableonGithub.
2017Availableathttps://code.
google.
com/archive/p/care-engine/downloads.
18Availableathttps://bioportal.
bioontology.
org/ontologies/GENE-CDS.
19Availableathttps://github.
com/TheOntologist/FHKB.
20https://github.
com/andreas-steigmiller/ore-competition-framework/.
Adetaileddescriptionoftheframe-workandhowtorunitisavailablethere.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport463Theframeworksupportsbothserialandparallelexecutionofacompetition.
Withserialexecutionorserialmode,werefertorunningthecompetitiononasinglecomputer,wherethereasonersarerunoneaftertheotheronallproblems.
Parallelexecutionorparallelmodemeansthatthecompetitionisconguredtorunonaclusterofcomputers,whereonemastermachinedispatchesevaluationtasks(i.
e.
,evaluatingareasoningtaskforaspecicreasoneronagivenontology)toclientmachines,collectstheresultsandservesthemuptoalivedisplay.
Parallel(distributed)modeisusedforthelivecompetition,butserialmodeissufcientfortestingorofineexperiments.
Theframeworkalsologssufcientinformationtoallow"replaying"thecompetition,andincludesscriptsforacompletereplayaswellasdirectlyshowingthenalresults.
TheframeworkisrealisedwithJavaand,therefore,itshouldberunnableonallJavasupportedplatforms.
ReasonersarerequiredtoparseandserialiseOWL'sfunctional-stylesyntax[29].
Thissyntaxisdesignedtoallowforeasyprocessingandwassupportedbyallparticipatingreasoners.
Inordertorunareasonerwithintheframework,reasonerdevelopershavetoprovideascript(ashellscriptand,optionally,aWindowsbatchscript)thatcanbeusedtostartthereasonerwithparameterstoindicatetheinputontologyandthetaskthatistobeperformed.
Reasonersalsoreportprocessingtimes,results,andprocessingerrorsviatheinvocationscript.
Apartfromreportedprocessingerrors(e.
g.
,areasonerstopsprocessinganontologyduetoencounteredunsupporteddatatypes),theframeworkalsorecordscrashes,e.
g.
,duetothememorylimit,aserrorsinlogles.
Finally,theframeworkproducesloglestorecordtimeoutsandwrongresults.
Reasonershavetoreportresultsofareasoningtaskinaspecicoutputformatthatallowsforaneasycomparison(usinghashcodes)ofthereportedresultwithanexpectedone.
Furthermore,thescriptisusedtoenforcethegiventimeandmemorylimits.
SincemanyreasonerssupporttheJava-basedOWLAPI,thereisastandardscriptforOWLAPIbasedreasonersandaJavawrapperclassthatimplementsthefunctionalityforproducingthedesiredresultoutputsandforerrorhandling.
ThismakesiteasytopreparereasonerswithOWLAPIsupportforthecompetitionandweexplicitlyencourageOWLAPIsupportasitsupportsaccesstothereasonersbyaplethoraoftools.
OWLAPIsupportis,however,notrequiredtoparticipateinthecompetition.
TheOWLAPIisaveryrichandratherheavyweightframeworkthatisnottightlyintegratedwithmostreasoners.
Forexample,systemsusingtheOWLAPIgenerallyconsumemorememorybecausetheymaintaintheOWLAPIlevelrepresentationoftheontologyinadditiontotheinternalrepresentationofthereasoner.
Thus,avoidingtheOWLAPIcanhelpcompetitionperformance.
Furthermore,forreasonersnotwritteninJavaOWLAPIsupportcanbedifcultortime-consumingtoimplement.
UsingascriptinsteadofJavacodetostartthereasonersallowsforeasyintegrationofreasonersnotimplementedinJavaorwithoutOWLAPIsupport.
Theframeworkusescongurabletimeoutsforeachreasoningtaskassessedinthecompe-tition.
Forreasonersthatexceedthetimelimitsetforacompetition,theulimitcommandisusedtoenforcetermination.
Thereasonersreportthetimeneededtosolveaproblemthemselvesinwallclocktime.
MethodologicalAspectsItwasdecidedtomeasurethetimesinwallclocktimeinsteadofCPUtime,becauseCPUtimewouldpenaliseparallelreasonerssuchasELK.
RecordingCPUtimeinadditiontothewallclocktimeis,however,anoteworthyextensionofthecurrentframework.
Thetimemeasurementisperformedbythereasonersandthecurrentspecication"recommends"excludingthetimefor"standard"parsingandloadingaswellasthetimeneededforresultserialisation(i.
e.
,writingtheresultstooutputles).
Theideabehindthisistonotpunishreasonersthatofferveryexibleparsing(andserialisation)supportofall123464B.
Parsiaetal.
kindsofsyntaxes,e.
g.
,byusingtherichbutheavyweightOWLAPIforthistask,overthosethathaveaslim,specialisedparserthatjustprocessestheeasy-to-handlefunctional-stylesyntax.
Inaddition,reasonersthatemployspecialisedparserssuchasELK,ELepHant,andKoncludeoftenperformsomekindofreasoning(e.
g.
,whetheraconsistencycheckcanbeomittedbecausetheontologydoesnotusenegation),indexing,andpre-processingalreadyduringparsing.
Thismakesitdifculttoclearlyseparateloadingandreasoningtime.
Hence,ELKchoosestoalwaysincludeloadingtimesinthereportedtime,whileKoncludedoesthisforconsistencychecking,wheretheamountofreasoningtimeismuchlessdominatingthanfortheothertasks.
Asfarasweknow,allotherreasonersdonotincludeparsing/serialisationtimeforanyreasoningtask.
WiththeexceptionofELepHantandRacer,thesesystemsareimplementedinJavaandsimplyusetheOWLAPIforwhichtheparsing/serialisationtimesareeasilyseparable.
Furthermore,thecurrentframeworkutilisesanetworkdrivetoenablethereasonersaccesstotherelevantles(e.
g.
,theontologydocuments).
Hence,readandwriteoperationscanbeinuencedbytheworkloadofthenetworkandshouldbeexcludedorseparatedintheevaluationresults.
ForJava-basedreasonerstheJVMoverheadmightbeadisadvantageduetothe"reandforget"executionstrategyemployedbythecompetitionframework.
Thiswouldparticularlyaffect"easy"problemsthatdonotrequiresignicantcomputationsandrunningtime.
ByusingalongrunningserverbasedapproachtheJVMoverheadforeasycasescouldbeeffectivelyamortised.
Inthecurrentcompetitionset-up,thereasoner-reportedtimeshave,however,alimitedinuence.
Theyareonlyusedforrankingthereasonersthatsolvedanequalnumberofproblems.
Resultsarevalidatedbycomparisonbetweencompetitorsusingamajorityvote/randomtie-breakingfallbackstrategy.
Thisdisputeresolutionmechanismisclearlyunsatisfactory.
Recentwork[19]hasrevealedexamplesinthe2015corpuswherethecorrectreasonerwouldbeunfairlypenalizedforbeingintheminority.
Especiallyproblematicaretwofacts:(1)Thevotesofdeliberatelyincomplete(withrespecttotheirpurportedprole)reasonerssuchasTrOWLcanoutweighvotesofacompletereasonerinthevotingprocedure.
(2)Reasonersmightbeabletovoteseveraltimes.
Forexample,HermiTparticipatedintwoversions(oneusingOWLAPIversion3andoneusingversion4)and,furthermore,itisusedinthecoalitionreasonerMORe.
Hence,abuginHermiTmightresultinthreereasonersdeliveringthesamewrongresult,whichcouldoutweightwoothercorrectreasoners.
AsimilarproblempotentiallyarisesforJfactandFaCT++,asJfactisan(almost)faithfulJavaportofFaCT++.
Note,unlikeCASC,reasonersarenotrequiredtoproduceproofsoftheirresultsasthisisnotastandardfeatureofdescriptionlogicreasoners.
Notethatformanyservices(suchasclassication)proofsforallsubsumptionswouldbeneeded.
Furthermore,(tableau-based)reasonerconstructniterepresentationsofinnitemodelsanditisyetunclearhowsuchpartialmodelscanberepresentedinaformthatallowsforverifyingthemautomatically.
Weare,however,experimentingwithamoresatisfactoryjustication-basedtechniquefordisagreementresolution[19]infuturecompetitions.
3.
5CompetitionEnvironmentsLiveCompetitionThecompetitionwasruninparallelmodeonaclusterof19machines:onemastermachinethatdispatchedreasonerswithproblemstothe18clientmachines,aswellascollectingandservingupresultstoalivedisplay.
EachmachinewasequippedanIntelXeonquad-coreL5410processorrunningat2.
33GHzwith12GBofRAM,forwhich2GBwerereservedfortheoperatingsystem(i.
e.
,10GBcouldbeusedbythereasoners).
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport465TheoperatingsystemwasUbuntu14.
04.
02LTSandtheJavaversionwasOpenJDKv1.
7.
064-bit.
Thereasonerexecutionwaslimitedto180sforeachontologyineachtrack,whereonly150swereallowedforreasoningand30scouldadditionallybeusedforparsingandwritingresultsinordertoreducethepenalisationofreasonerswithslowparsers.
Hence,ifthetimereportedbythereasonerexceeded150s,thenitwasinterpretedasatimeout.
Thesetimelimitswerechosensuchthatthelivecompetitioncouldberunwithin1day(paralleltotheDL/OREworkshopprogram)onthegivenhardwarewithareasonablenumberofontologies(200–300).
21OfineCompetitionTheofinecompetitionforuser-submittedontologieswasrunonanAmazonEC2clusterwheretwentyinstanceswereused,oneofwhichwasthemastermachine,runningthecompetitionserver,andtheremainingnineteenwereclientmachines.
TheAma-zonEC2instancesusedwereoftype"r3.
large",withthefollowingspecications:dual-coreIntelXeonE5-2670(v2)processorrunningat2.
5GHzclockspeed,andwith15GBofRAMmemory,outofwhich2GBwerereservedfortheoperatingsystem,andtheremaining13GBwereavailableforreasoners.
TheoperatingsystemwasUbuntuServer14.
04LTS,andtheJavaversionwasOpenJDKv1.
7.
064-bit.
Thereasonerexecutiontimewaslimitedto6hand10minforeachontology,where6hwereallowedforreasoningandtheadditional10mincouldbeusedforinput–outputoperations,followingthesamerationaleasthelivecompe-titiondescribedabove.
WeknowfrompreviousexperimentsthatclassicationonAmazonEC2instancesisreasonablystable(i.
e.
,lowaveragevariance),soeachtaskwasrunonlyonce.
Runningthecompetitionmultipletimeswouldhaveconsumedconsiderablecomputa-tionalresourcesforonlyamarginalgain—asingleruntookaround75machine-days,i.
e.
ittooktenmachines(runinparallel)morethanaweektoexecutethecompetition.
4CompetitionParticipantsTherewere14reasonersparticipating,with11purportingtocoverOWL2DL,and3beingOWL2ELspecic(seeTable3).
Thereisnospecicpenaltyortestforbeingincompletewithrespecttoaproleand,indeed,onereasoner(TrOWL)isintentionallyincompleteforperformancereasons.
Thenumberofparticipantshasbeenfairlystableoverthepast3years,rangingfrom11to14.
Thereisastablecoreofparticipantswithsomeuctuationonthemargin.
Somereasonersarenotenteredbytheiroriginaldevelopers(e.
g.
,Pellet)andOREcurrentlyhasnopolicyagainstthat.
Weanticipateinthefuturethatmorecoalitionreasonerswillbemadeavailable,thoughcurrentlyonlyMORe,Chainsaw,andPAGOdAusecomponentreasoners(ELKandHermiTareusedbyMORe,FaCT++byChainsaw,andRDFox[27]andHermiTbyPAGOdA)thataremostlyalsocompeting.
Forexample,MORe'scoalitioninvolvespartitioningtheontologyintoanELandDLpart,dispatchingeachparttotherespectivetunedreasoner,andcombiningtheresults[1].
Coalitionreasonersthatdonottransformtheontologyinanyrelevantwaywillneedspecialconsiderationiftheyweretoparticipate.
21Duetothemajorityvoting,atighttimelimitcouldpotentiallybenetreasonersthatguessorapproximateresultsifsoundandcompletereasonerscannotdeterminethecorrectoneswithinthetimelimit.
However,testrunsofthecompetitionhaverevealedthatamoderatelyincreasedtimelimit(e.
g.
,300sinsteadof150s)doesnotseemtosignicantlyinuencetheoverallresults,i.
e.
,mostreasonerscannotsolvemanyadditionalproblemsifthetimelimitisonlymoderatelyincreased.
Thesetestrunsfurthershowedthat30sare(usually)enoughtoparsetheontologies.
123466B.
Parsiaetal.
Table3ParticipantlistwithOWL2DLreasonersinthetopandOWL2ELreasonersinthebottompartReasonerNew2015ConsistencyClassicationRealisationLanguageLicenseOWLDLChainsaw[47]–JavaLGPL2.
0FaCT++[46]–C++LGPL2.
0HermiTa[9]–JavaLGPL3.
0Jfact[32]–JavaLGPL2.
0Konclude[41]–C++LGPL2.
1MORe[1]––JavaLGPL3.
0PAGOdA[50]––JavaAcademiclicensePellet-OA4[40]JavaAGPLv3Racer[11]LISPBSD3-clauselicenseTrOWL[45]–JavaAGPLv3OWLELELepHant[39]–C++ApacheLicence2.
0ELK[16]–JavaApacheLicence2.
0jcel[25]––JavaApacheLicence2.
0aHermiTwassubmittedwithOWLAPI3andOWLAPI4bindingsInthefollowing,wewillintroducetheparticipatingreasoningsystems.
Muchoftheinformationpresentedherecanbefoundonline22aswellasinourrecentlyconductedOWLreasonersurvey[22].
TheversioninformationreectthestateofthesystemasitwassubmittedtoORE2015.
Chainsaw(v.
1.
0)Chainsawisafree(LGPL)OWL2DLreasonerforverylargeontologies.
Itusesamodulardecompositiontotacklethehighcomplexityofreasoning.
Chainsawusesdelegatereasonerstoperformreasoningtasksoverindividualmodules[47].
Developmentstartedin:2012;Supportedlanguage:OWL2DLMaintainedby:UniversityofManchester,UKDownload:https://bitbucket.
org/ignazio1977/chainsawELepHant(v.
0.
5.
7)ELepHantisaconsequence-basedreasonerthataimsatprovidinglightweightandperformantreasoningforOWL2ELontologies[39].
Developmentstartedin:2013;Supportedlanguage:OWL2ELMaintainedby:BarsSertkaya,FrankfurtUniversityofAppliedSciencesDownload:https://github.
com/sertkaya/elephant-reasonerELK(v.
0.
5.
0)ELKisapopularandhighlyperformantconsequence-basedreasonerforOWL2ELontologies.
ItsupportsmostoftheOWL2ELprole[16].
Developmentstartedin:2011;Supportedlanguage:OWL2ELMaintainedby:UniversityofUlm,GermanyDownload:https://github.
com/liveontologies/elk-reasoner22http://owl.
cs.
manchester.
ac.
uk/tools/list-of-reasoners/.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport467FaCT++(v.
1.
6.
4)FaCT++isafree(LGPL),highlyoptimisedopen-sourcetableaurea-sonerforOWL2DLwritteninC++.
FaCT++isatableau-basedreasonerimplementedinC++.
[46].
Developmentstartedin:2003;Supportedlanguage:OWL2DLMaintainedby:UniversityofManchester,UKDownload:https://bitbucket.
org/dtsarkov/factplusplusHermiTandHermiT-OA4(v.
1.
3.
8.
5)Forthecompetition,HermiTwassubmittedtwice:thelatestreleaseversion,whichusesOWLAPI3,andHermiT-OA4,whichisaforkoftheoriginalreasonerwithOWLAPI4bindingsandwasnotsubmittedbyitsoriginaldevelopers.
HermiTisanOWL2DLreasonerbasedonahyper-tableauengineanditaimstofullyandcorrectlysupporttheOWL2DLspecication[9].
Developmentstartedin:2007;Supportedlanguage:OWL2DLMaintainedby:UniversityofOxford,UKDownload:http://www.
hermit-reasoner.
com/jcel(v.
0.
21.
0)jcelisafreeopen-sourceJava-basedreasonersupportingpartsoftheOWL2ELprole.
Itimplementsapolynomial-timemodularconsequence-basedalgorithmforgen-eralTBoxesandABoxes[25].
Developmentstartedin:2010;Supportedlanguage:OWL2ELMaintainedby:TechnischeUniversittDresden,GermanyDownload:http://jcel.
sourceforge.
net/Jfact(v.
4.
0.
1)JfactisapureJavaportofFaCT++withversionsforOWLAPI3.
xand4.
x.
Itiskeptup-to-datewithFaCT++[32].
Developmentstartedin:2011;Supportedlanguage:OWL2DLMaintainedby:UniversityofManchester,UKDownload:https://github.
com/owlcs/jfactKonclude(v.
0.
6.
1)Koncludeisaparallel,high-performancereasonerforOWL2DL.
ItisimplementedinC++andusesareasoningtechniquethatisbasedonahighlyoptimizedtableaualgorithmassistedbyacompletion-basedsaturationprocedure[41].
Developmentstartedin:2013;Supportedlanguage:OWL2DLMaintainedby:UniversityofUlm,derivoGmbH,GermanyDownload:http://konclude.
com/downloadMORe(v.
0.
1.
6)MOReusesmoduleextractiontechniquestoclassifyontologiescombiningreasonersespeciallyoptimisedfordifferentOWL2proles.
ThesubmittedversionofMOReusesHermiTastheOWLDLdelegate,andELKastheOWLELdelegate[1].
Developmentstartedin:2012;Supportedlanguage:OWL2DLMaintainedby:UniversityofOxford,UKDownload:https://github.
com/anaarmas/MORe123468B.
Parsiaetal.
PAGOdA(v.
unversioned)PAGOdAisasoundandcompletequeryansweringengineforOWL2ontologiescombiningthescalabledatalogreasonerRDFoxandtheOWL2reasonerHermiT[50].
Developmentstartedin:2012;Supportedlanguage:OWL2DLMaintainedby:UniversityofOxford,UKDownload:https://github.
com/yujiaoz/PAGOdAPellet-OA4(v.
2.
4.
0)Pelletisafreeopen-sourceJava-basedreasonerforOWL2thatusesatableau-baseddecisionprocedure.
TheversionsubmittedtoOREisaforkoftheoriginalreasonerwithOWLAPI4bindingsandwasnotsubmittedbyitsdevelopingcompanyComplexible[40].
Developmentstartedin:2012;Supportedlanguage:OWL2DLMaintainedby:Complexible(Originalversion)Download:https://github.
com/ignazio1977/pelletRacer(v.
2.
0)Racer(RenamedABoxAndConceptExpressionReasoner)isaknowledgerepresentationsystemthatimplementsahighlyoptimizedtableaucalculusandsupportsmanyfeaturesofOWL2DL[11].
Developmentstartedin:1998;Supportedlanguage:OWL2DLMaintainedby:ConcordiaUniversity,Montreal,CanadaDownload:https://github.
com/ha-mo-we/RacerTrOWL(v.
1.
5)TrOWLisatractablereasoninginfrastructureforOWL2ontologies.
ForTBoxandABoxreasoning,itutilisesasyntacticapproximationfromOWL2DLtoOWL2EL[45].
Developmentstartedin:2012;Supportedlanguage:OWL2DLMaintainedby:UniversityofAberdeen,UKDownload:http://trowl.
org5Results:LiveCompetitionResults,errorreports,andmoredetailsonthecompetitionframeworkareavailableathttp://dl.
kr.
org/ore2015.
Abreak-downofalltracksandthenumbersofcompetingreasonersisshowninTable4.
Figure2showstheresultsofallparticipantsinalltracks,asdisplayedduringthelivecompetition.
Duringthecompetition,thesechartsweredynamicallyupdatedasproblemswerebeingsolvedandreported.
Notethatduetospaceconstraintstheerrorcolumn(labelledwithanexclamationmark)showsthesumofthenumberoferrors,timeouts,andunexpected(wrong)resultsproducedbythereasoner,i.
e.
,thenumberof(processed)ontologiesthatarenotconsideredascorrectlysolved.
ItisworthnotingthatfortheOWLELtasksthereareseveraltiesinthenumberofsolvedproblems.
Inthiscasethereasoningtimeasreportedbythereasonersistakentorankthereasoners.
ForOWLELConsistencythereisatiebetweenELK(rstplace)andKonclude(secondplace),whereELKwasdeterminedthewinnerduetoitsloweraccumulatedreasoning123TheOWLReasonerEvaluation(ORE)2015CompetitionReport469Table4BreakdownofthecompetitionbytrackTaskCompetitorsProblemsOWLDLConsistency10306Classication10306Realisation10264OWLELConsistency13298Classication13298Realisation12109time(425.
1sforELKvs1050.
4sforKonclude).
Inthiscasebothreasonersincludeparsingtimeintothemeasuredtimeand,hence,therankingseemsfair.
AnothertieoccursbetweenHermiT(sixthplace)andHermiT-OA4(seventhplace)with846.
6and874.
7s,respectively.
ItisnotsurprisingthatbothversionsofHermiTperformsimilarlyandsinceloadingtimesarenottakenintoaccountforbothversionstherankingseemsfairalsointhiscase.
ForOWLELClassicationthereisatiebetweenKonclude(secondplace)andMORe(thirdplace)with622.
3and1685.
1s,wherebothreasonersexcludeloadingtimefromthereportedtimes.
Asforclassication,thereisagainatieforHermiT(sixthplace)andHermiT-OA4(seventhplace).
Finally,thereisatieforTrOWL(thirdplace)andPAGOdA(fourthplace)with241.
3and1771.
7s,respectively.
Outofthesixtracks,fourwerewonbythenewhybridreasonerKonclude[41],andtwo(OWLELConsistencyandOWLELClassication)werewonbyELK[16].
Figure3showshowwellthewinningreasonersdidintermsofreasoningtime.
Thereareacoupleofobservationstobemadehere.
First,Konclude,thewinnerofallthreeDLdisciplines,isdoingconsistentlybetteronthemajorityoftheeasierontologies,buttowardstheharderendontheright,otherreasonerscatchup.
ThisisparticularlyobviousfortheELclassicationcompetition.
Upuntilacertainpoint,Koncludeisdoingmuch(sometimesuptoanorderofmagnitude)betterthanELK(thewinnerofthediscipline),buttowardstheharderend,ELKovertakesKonclude.
SomeofthismaybeduetotheJVMoverheadforELKandour"reandforget"executionstrategy.
IfwehadalongrunningserverbasedapproachitmightbethattheJVMoverheadforeasycaseswouldbeeffectivelyamortised.
AnotherinterestingobservationistheperformanceofELepHant's[39]consistencycheck,whichregularlyoutperformsbothELKandKonclude.
Wespeculatethatthisisduetodifferencesinwhetherparsingtimeisincorporatedinthereportedtime(e.
g.
,ELKdoesthisforalltasksandKoncludedoesthisforconsistencychecking).
Afullbreak-downforallreasonersbycompetitionisshowninTable5.
Thecompetitionisreasonablychallenging:inonlytwotracks(ELconsistencyandELclassication)didanyreasonersolvealltheproblemsincompetitionconditions.
Figure4showsadetailedbreakdownofhowmanyproblemsweresolvedbyhowmanyreasoners.
ItisinterestingtoobservethattheunionofallreasonerssuccessfullyprocessallELreasoningproblems.
Asonemightexpect,realisationisstillchallengingforreasoners.
Butinalltracks,forthemajorityofreasoners,theOREproblemsprovideagoodtargetforoptimisation.
Theresultsofthecompetitionsuggestthattheseproblemsare(almost)allinprinciplesolvableonamodestmachinesuchastheonesusedinourcompetition(seeSect.
3.
5)inaround3min.
123470B.
Parsiaetal.
Fig.
2Resultsofthecompetitionbytrackasdisplayedinthelivecompetitiondisplay.
Scoreindicatesthenumberofproblemssolvedoutofthetotalproblemsforthattrack.
Thenumberofunsolvedproblems(whetherbytimeout,crash,or"wrong"results)aredisplayedinthenextcolumn.
Timeindicatesthetimeactuallytakentocompletesolvedproblems.
TimeisusedtoresolvetiesforsolvedproblemsThesmallnumberof(possibly)wrongresultsintheELtracksfurthershowsthatreasoningwithELontologiesalreadyachievedagooddegreeofstabilityandmaturity.
ThisalsoresultsinthefactthatthemajorityvotingisworkingquitewellfortheELdisciplines(tothebestofourknowledge,forallELConsistencyandELClassicationproblems,thecorrectresultsweredetermined).
Incontrast,thereismuchmoredisagreementontheDLtracks,which123TheOWLReasonerEvaluation(ORE)2015CompetitionReport471Fig.
3ReasoningtimeofthethreewinningreasonersineachcategoryfortheDLandELprole(orderedseparatelybyspeedofthereasoner).
Thetophorizontallineillustratesthetimeoutof180sisduetoseveralreasons.
Ontheonehand,reasoningproceduresforOWL2DLaremuchmoreinvolvedandrequiremanyoptimisationstoworksufcientlywellinpractice.
Hence,itcanbedifculttoensurethatimplementationsdonotcontainbugs.
Ontheotherhand,DLontologiesoftencontaindatatypesinawaythataffectsreasoning,butseveralDLreasonershaveonlypartialdatatypesupportandthusmaynotderiveallconsequences.
Furthermore,therearereasoners(e.
g.
,TrOWL)thatapproximatemoreexpressivelanguagefeaturesandare,therefore,morelikelytocomputeanincompleteresult(forthemoreexpressiveDLontologies).
Asaconsequence,themajorityvotingcanidentifywrongresultsascorrectanditisindeedlikelythatthishappenedinafewcases.
Thisalsoseemstobeindicatedbythenumberofties,whichare2fortheDLConsistencytrack,13forDLClassication,and5forDLRealisation(fortheELtracks,therewereonly3tiesintherealisationdiscipline).
Interestingly,mostofthetieswerebetweenKoncludeandTrOWLfor"hard"DLontologiesthatcouldnotbesolvedbyothersystems.
6Results:OfineCompetitionThelivecompetitionhasquitestricttimeconstraintsasitmustnishwithinaworkshopday.
Giventhetimelimitof150sperontology,itisnotpossibletoincludereallyhardontologiesinthecompetition.
Thereare,however,ontologiesusedinreal-worldscenariosthatarestillchallengingstate-of-the-artreasoners.
Toaddressthis,OREallowsuserstosubmittheirchallengingontologiestotheofinecompetition,whichhasamuchmoregeneroustimelimitof6h,andahighermaximummemory(13GBinsteadof10GB).
Theresultsoftheofinecompetitionareparticularlyvaluableforreasonerdevelopersasmanyoptimisations123472B.
Parsiaetal.
Table5Fullbreak-downofsolvedproblemsbyreasonerandtaskoverthe306ontologiesforDLconsistencyandclassication,264forDLrealisation,298forELconsistencyandclassication,and109forELrealisationReasonerSolvedTimeoutErrorWrongSolvedTimeoutErrorWrongDLconsistencyELconsistencyChainsaw29131112761930ELepHant––––296200ELK––––298000FaCT++276161312702260HermiT2948312821600HermiT-OA42938412821600jcel––––2613520Jfact166835252277100Konclude303102298000MORe2534328295300Pellet-OA427826022851300Racer239481182564002TrOWL26603642730250DLclassicationELclassicationChainsaw11917116019194130ELepHant––––291601ELK––––298000FaCT++200871722445130HermiT23667212722600HermiT-OA423766212722600jcel––––13315861Jfact128106591318989218Konclude2887110294004MORe24741216294202Pellet-OA41871051402612890Racer1648625423738023TrOWL201035702750230DLrealisationELrealisationChainsaw79166163436420ELepHant––––841024ELK––––102007FaCT++17258259792730HermiT1639353575200HermiT-OA41629363575200Jfact109894719634303Konclude2472114104005PAGOdA104519514861508Pellet-OA41365424506032215Racer467531403231046TrOWL15004371860221123TheOWLReasonerEvaluation(ORE)2015CompetitionReport473DL101228142628731515112613817271838519211323611472627245333050100150050100150050100150ConsistencyClassificationRealisationcountEL000142962105631222001351095828142567123000613103521626541050100150200250050100150200250050100150200250ConsistencyClassificationRealisation301245678910014235678910111213countFig.
4Numberofreasoningproblems(y-axis)bytotalnumberofreasonerssolvingthem.
Forexample,5DLclassicationtaskswerenotsolvedbyanyreasoner,and123ELclassicationtasksweresolvedbyallreasoners123474B.
Parsiaetal.
areinspiredbyreal-worldproblems.
Furthermore,knowingwhichreasonersareabletohandlesomeinput,whileothersfailto,encouragesdiscussionsbetweendevelopersregardingstrategiesandoptimisationtechniques.
Theresultsofthe2015ofinecompetitionareshowninTables6and7.
Therstthingtonoteisthatatleastonereasonersuccessfullyprocessedeachsubmittedontology,forallthreetasks.
Koncludewastheonlyreasonerthatsuccessfullyterminatedoneveryinput.
TheCOontologywasparticularlychallengingforreasoners(seeTable6)—only3didnottime-outduringclassication:FaCT++,KoncludeandTrOWL.
TrOWLclassiesCOsurprisinglyfast,thoughitreturnswrongresults;FaCT++andKoncludeagreedonadifferentresulttoTrOWL.
VFB-KBwasanotherchallengingontology(seeTable7),whereonlyKoncludecorrectlydeterminedconsistencywithouttimingoutorerring.
RacerseeminglycompletedconsistencycheckingonVFB-KB,buttheresultledidnotcontaintheexpectedresult;thisisarecurringpatternforRaceranditswrongresultsinTables6and7.
23HermiT-OA4andJfact(seemingly)completedclassicationofVFB-KB,buterredortimed-out(respectively)duringconsistencycheckingoverthesameinput.
However,theresultsoutputbythe2reasonerswereincorrect;therewerenosubsumptionsintheresultsleswhenthereare,frommanualinspection,inferencestobecomputed.
Explainingsuchbehaviourisbeyondthescopeofthecompetition.
Notethattheinterfacesforclassicationandstand-aloneconsistencyaredifferent,soabugaffectingonemaynotnecessarilyaffecttheother.
ForRacer,checkingconsistencyofVFB-KB(741.
89s)wasonlyslightlyfaster(by27s)thancomputingclassication—unfortunately,however,leadingtothewrongresult.
ItisinterestingtonotethatHermiT-OA4successfullycompletedclassicationofGO+inlessthanhalfthetimethanHermiT.
Thismayhavehappenedduetoinsufcientorothermemoryproblems,seeingasthemorerecentversionoftheOWLAPIboastsvariousmemoryimprovements.
Thisalsooccursduringconsistencycheckingoverthesameontology.
AstrikingresultinTable6isthat,whilesomereasonerswerefastontheconsistencytask,suchasKoncludethatsuccessfullyterminatedoneveryinputorChainsawandTrOWLinallbutoneinput,othersdidnotevencompletetheconsistencycheckwithinthe6htimeout.
Similartotheclassicationtask,determiningwhetherCOisconsistentwaschallenging;halfthereasonerstimed-out,whiletheremainderterminatedwithin8satmost.
EvenmorechallengingthanCO,onlyonereasonersuccessfullycompletedconsistencycheckingofVFB-KB;Konclude(3.
81s).
Outoftheremainingreasoners,fourtimed-outandtheotherveerred(whereRacerseeminglycompletedthetask,butreportedanerrorandoutputemptyresults).
Ontheotherendofthespectrum,theonlyontologyforwhicheveryreasonersuc-cessfullyterminatedconsistencycheckingwasDPO,typicallywithin4s(withtheexceptionofMORe,whichtook44.
74s).
Outofthethree2015submissionsthatcontainedanABox(seeTable7),onewasnotablydemanding:VFB-KB,onwhichonlytworeasonerswereabletosuccessfullycompleterealisationwithinthetimeout:Pellet-OA4andKonclude,inunder9s.
Unfortunately,theerrorinformationavailablewaslimitedtothatpipedouttothespeciedhook(inthecompetitionframework)forreasonererroroutput.
Fromwhatwecoulddeterminetheissueswiththe2015user-submittedcorpusweremostlyduetoontologyparsingandinternalreasonererrorswhileprocessinginput.
ApartfromthefactthatnotallreasonersmakeuseoftheOWLAPI,parsingerrorscanoccuratvariouspointsintheparsingprocess.
TheOWLAPIdoesnotcheckseveralaspects,e.
g.
,regularityofrolehierarchies,whether23Uponcloserinspection,Raceractuallythrowsastackoverowerrorinthesecases,buttheerrorisnotpipedouttotheframework'serrorhandler.
Asaconsequence,thewrongresultisonlypinneddownuponcorrectnesscheckingratherthanaggedasanerrorstraightaway.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport475Table6Resultsfortheusersubmittedontologies(timesarereportedinseconds)ReasonerCODINTODPOGO+ConsistencyChainsaw1.
804.
860.
766.
05FaCT++2.
32Error0.
811254.
24HermiTTimeout9.
682.
98202.
42HermiT-OA4Timeout12.
663.
6471.
52JfactTimeoutTimeout3.
08TimeoutKonclude0.
423.
600.
186.
38MOReTimeoutError44.
741103.
87Pellet-OA4Timeout35.
532.
55ErrorRacer7.
72Timeout0.
9379.
88(wrong)TrOWL4.
1010.
942.
3838.
02ClassicationChainsawTimeoutError5.
82TimeoutFaCT++6652.
28Error8.
32TimeoutHermiTTimeoutError84.
296227.
97HermiT-OA4TimeoutError93.
512351.
83JfactTimeoutTimeout49.
30(wrong)TimeoutKonclude201.
686.
840.
4669.
75MOReTimeoutError43.
731023.
11Pellet-OA4TimeoutError9.
45ErrorRacerTimeoutTimeoutTimeout78.
92(wrong)TrOWL3.
87(wrong)13.
062.
73(wrong)42.
05(wrong)Italicsindicateafailedattempt,eithertimeout,thrownerror,orwrong(results)rolesusedincardinalityconstraintsaresimple,whethertheuseddatatypesareOWL2datatypes,orwhetheragivenlexicalformcorrespondstoadatavalueinthevaluespaceofthespecieddatatype.
Suchissuesareusuallydetectedbythereasonersduringparsingand,hence,areusuallyreportedasparsingerrors.
Takingintoaccountalluser-submittedontologies,i.
e.
,submissionsfrom2014and2015(seeTable8),everysubmissionwasprocessedbyatleastonereasoner.
ThereweretwoontologiesthateveryDLreasonerclassiedsuccessfullywithinthetimeout:HPandDPC-OLY.
Andtwoontologiesthatonlyonereasoner(TrOWL)terminatedclassicationon:GALEN-FUandG-CDS;theremainingreasonerstimed-out(therewerevetimeoutsonGALEN-FU,andeightonG-CDS)orthrewanerror(fourreasonerserredonGALEN-FU,andoneonG-CDS).
ThereweresevenontologiesthatwereprocessedbyallreasonersexceptChainsaw.
Forconsistencychecking,therewerenineontologiessuccessfullyprocessedbyeveryreasoner.
TheMSC-Dontologyincurredthemosttimeouts;vereasonersdidnotcom-pletethetask.
Therewereadditionalontologiesthatwerechallengingtocheckconsistency,suchasallthreeversionsofFHKBwhereatleastonereasonertimed-out,bothvariantsofGALEN,andallthreeversionsofCBhadtwotothreetimeoutseach.
Therealisationtaskhadtwoontologies(HPandDPC-OLY)astheonlyonessuccessfullyprocessedbyeveryreasoner,theexactsameonesasduringclassication,andsevenotherontologiesthatwereprocessedbyallbutonereasoner(typicallyChainsaw,whichwouldthrowsomereasonerinternalerror).
Severalontologiesprovedchallengingforreasonerstocompleterealisation:123476B.
Parsiaetal.
Table7ResultsfortheusersubmittedontologiesReasonerVFB-EPNTVFB-KBVFB-NCTConsistencyChainsaw12.
82Timeout12.
71FaCT++10.
20Timeout10.
27HermiT11.
11Timeout12.
00HermiT-OA410.
43Error9.
83Jfact121.
92Timeout120.
33Konclude3.
733.
813.
93MOReErrorErrorErrorPellet-OA49.
06Error8.
86Racer134.
28(wrong)741.
89(wrong)129.
21(wrong)TrOWL10.
49Error9.
10ClassicationChainsawErrorErrorErrorFaCT++10.
24Timeout9.
93HermiT9.
59Timeout9.
19HermiT-OA49.
939.
10(wrong)9.
95Jfact121.
719.
22(wrong)121.
38Konclude3.
8724.
453.
40MOReErrorErrorErrorPellet-OA4ErrorErrorErrorRacer132.
82(wrong)768.
53(wrong)132.
41(wrong)TrOWL8.
93Error9.
46RealisationChainsawErrorErrorErrorFaCT++10.
19Timeout10.
04HermiT9.
37Timeout10.
07HermiT-OA49.
40Error9.
77Jfact121.
42Timeout124.
17Konclude3.
924.
243.
93PAGOdAErrorTimeoutErrorPellet-OA4Error8.
76ErrorRacer160.
64(wrong)812.
88(wrong)134.
55(wrong)TrOWL8.
89Error8.
78Italicsindicateafailedattempt,eithertimeout,thrownerror,orwrong(results)bothGALENvariants,MSC-DandMSC,oneachofwhichvereasonerstimed-out,andG-CDSwhereeightreasonerstimedout.
Koncludecompletedrealisationonthemostontolo-gies:23.
Similartotheothertasks,HermiT-OA4processedmoreontologiesthanHermiT,inparticularexhibitingthreetimeoutslessduringrealisationaswellasconsistencychecking.
Intermsoferrorsforthe2014user-submittedcorpus,FaCT++andChainsawwereunabletoprocessGALEN-FUandGALEN-Hduetounsupporteddatatypes,andinadditiontothisweidentiedthesametypesoferrorsasinthe2015corpus,namelyontologyparsingissuesandreasonerinternalerrors.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport477Table8Overallresultsfortheusersubmittedontologieswithnumbersinbracketsshowing%;sortingisbytask,supportedprole(ELreasonersareshownlast)and%ofsolvedproblems;onlyontologieswithanABoxareusedintherealisationtrackReasonerCompletedErrorTimeoutAllConsistencyKonclude33(91.
67)1(2.
78)2(5.
56)36HermiT-OA429(80.
56)5(13.
89)2(5.
56)36HermiT27(75.
00)4(11.
11)5(13.
89)36TrOWL27(75.
00)9(25.
00)0(0.
00)36Chainsaw25(69.
44)8(22.
22)3(8.
33)36Racer24(66.
67)0(0.
00)12(33.
33)36Pellet-OA423(63.
89)8(22.
22)5(13.
89)36Jfact22(61.
11)9(25.
00)5(13.
89)36FaCT++21(58.
33)12(33.
33)3(8.
33)36MORe19(52.
78)9(25.
00)8(22.
22)36ELepHant1(100.
00)0(0.
00)0(0.
00)1ELK1(100.
00)0(0.
00)0(0.
00)1jcel1(100.
00)0(0.
00)0(0.
00)1ClassicationKonclude29(80.
56)2(5.
56)5(13.
89)36TrOWL27(75.
00)9(25.
00)0(0.
00)36HermiT-OA424(66.
67)6(16.
67)6(16.
67)36HermiT23(63.
89)6(16.
67)7(19.
44)36Jfact22(61.
11)9(25.
00)5(13.
89)36FaCT++19(52.
78)13(36.
11)4(11.
11)36Racer19(52.
78)0(0.
00)17(47.
22)36MORe17(47.
22)10(27.
78)9(25.
00)36Pellet-OA414(38.
89)9(25.
00)13(36.
11)36Chainsaw7(19.
44)21(58.
33)8(22.
22)36ELepHant1(100.
00)0(0.
00)0(0.
00)1ELK1(100.
00)0(0.
00)0(0.
00)1jcel0(0.
00)1(100.
00)0(0.
00)1RealisationKonclude23(79.
31)3(10.
34)3(10.
34)29TrOWL21(72.
41)8(27.
59)0(0.
00)29Jfact19(65.
52)9(31.
03)1(3.
45)29FaCT++17(58.
62)10(34.
48)2(6.
90)29HermiT-OA417(58.
62)7(24.
14)5(17.
24)29HermiT16(55.
17)5(17.
24)8(27.
59)29Racer14(48.
28)0(0.
00)15(51.
72)29Pellet-OA413(44.
83)9(31.
03)7(24.
14)29Chainsaw9(31.
03)17(58.
62)3(10.
34)29123478B.
Parsiaetal.
7ConclusionTheORE2015ReasonerCompetitioncontinuedthesuccessofitspredecessors.
Participants,workshopattendees,andinterestedbystandersallhadfun,andtheORE2015corpus,whetherusedwiththeOREframeworkorinacustomtestharness,isasignicantanddistinctcorpusforreasonerexperimentation.
Developerscaneasilyrerunthisyear'scompetitionwithneworupdatedreasonerstogetasenseoftheirrelativeprogress,andwebelievethatsolvingalltheproblemsinthatcorpusinsimilarorsomewhatrelaxedtimeconstraintsisareliableindicatorofaveryhighqualityimplementation.
ThetopslotsinalltrackshavebeendominatedbyKonclude(andtoalesserextentbyELK)for2yearsnow.
Koncludeisahighlyoptimised,veryefcientreasonerwhosedeveloperscontinuouslytestitagainstavastsetofavailableontologies.
Evenso,thereisinterestingjockeyingaroundsecondandthirdplaceforalltracks,andwewereimpressedwithhowwellolderreasoners,whichhavenotbeenupdatedrecently(notablyPellet-OA4andRacer),fared.
Bothacrossuser-submittedontologies(6htimeout)andthelivecompetition(3mintimeout),(almost)everyontologywasprocessedbyatleastonereasoner.
Thisisaconsiderableresultforthecommunityoverall.
Giventhisstasisinresults,wehavedecidedtomovetoa2-yearcycleforcompetitions.
Thisallowsmoretimeforreasonersandthecorpustodevelop,aswellasgivingusmoreresourcestodevelopadditionaltracks.
ItispossiblethatKoncludewillremainthechampion,whichweregardaschallengeforthecompetition.
Weareexperimentingwithdifferentbiasesinourproblemselection(e.
g.
,favouringdifcultproblems)toincreasethecompetitivenessofthecorpus.
Addingadditionaltrackswillalsopotentiallyamelioratethisproblem.
Therobustnessexperimentsin[10]usedamuchlongertimeout(upto2hpertest),thoughtheanalysisclusteredresultsbysubdivisionsofthetimeoutperiod.
Thatsuggeststhataslightlylongertimeoutmightsignicantlyincreasethetotalnumberofsolvedproblemsacrossreasoners.
Increasingthetimeoutneedstobeweighedagainsttheincreasedoverallruntimeofthecompetition(whichisboundedbytheslowestreasoner).
WepreferthebulkofthecompetitiontobeexecutedduringasingledayoftheDLworkshoptofacilitateengagement.
Thisimposesfairlytightlimitsonthetimeoutandnumberofproblems.
24Moreover,sincealmostallontologieswereprocessedbyatleastonereasonerintheORE2015competition,webelievethatourcurrentsettingisreasonablywellbalanced.
Ourofinecompetitionremainsasuitablereasonerbenchmark(withalongertimeout)usingdifcultontologiesfromusersinneedofourservices.
Ideally,theOREtoolkitandcorporawillserveasanucleusforaninfrastructureforcom-monexperimentation.
Tothatend,resultsandanalysisscriptsaremadeavailableonline.
25Thetestharnessseemsperfectlywellsuitedforblackboxhead-to-headcomparisons,andwerecommendexperimentersconsideritbeforewritingahomegrownone.
Thiswillimprovethereliabilityofthetestharnessaswellasreproducibilityofexperiments.
Evenforcaseswheremoreelaborateinternalmeasurementsarerequired,theOREharnesscanserveasthecommandandcontrolmechanism.
Forexample,separatingactualcalculusactivityfromotherbehavior(parsing,serializing,etc.
)requiresadeepdelveintothereasonerinternals.
However,givenasetofreasonersthatcouldseparateoutthosetimings,itwouldbeasimpleextensiontotheharnesstoaccommodatethem.
Whilethetestharnessworkswellfor"head-to-head,re-and-forget"experiments,theanalysisscriptsaremoretunedforcompetitionandnotexperimentation.
Forexample,an24Unfortunatelythisyear,duetotechnicalissues,wehadtorunthecompetitionovernight.
25http://owl.
cs.
manchester.
ac.
uk/publications/supporting-material/ore-2015-report/.
123TheOWLReasonerEvaluation(ORE)2015CompetitionReport479experimentcanhavetworeasonersthatsolveallproblemswithinthetimeout,butoneistwiceasfastastheother.
Mostalgorithmandimplementationcomparisonswillwanttodelveintothatfact.
Thereisnoconsensusofhowtodosuchanalysisatthemoment,butitwouldbestraightforwardtoaddadditionalanalysisscripts(forexample,Fig.
3wasgeneratedfromstandardOREdatabycustomscripts).
TheOREworkshopsolicits"challenge"ontologiesfromontologydeveloperspartlyinthehopesofdirectingreasonerdeveloperattentiontorealuserperformanceneeds.
In2005,wehave,forthersttime,incorporatedan(ofine)challengeinvolvingusersubmittedontologies.
Whiletheresultsdonotcounttowardstheoverallrankingsofthereasoners,wehopethattheyprovideguidanceforontologyuserstoselectappropriatereasonersfortheirproblemsand,perhaps,serveasanincentiveforreasonerdeveloperstodevelopbetteroptimisations.
Themostimportantnextexpansionoftracksistoconjunctivequeryanswering(CQA).
SettingupameaningfulCQAcompetitionissignicantlymoredifcult,becausewedonotonlyhavetoconsiderontologies,butalsoqueriesanddata.
Gatheringsuitable(meaningful)queriesisprobablythemostdifculthurdletoovercome.
However,wemadesignicantprogresstowardareasonabledesignthisyearandhopetoincorporateitinthenextcompe-tition.
Anotherareaofinterestisapplication-stylebenchmarks,whichwouldsituatethereasoningtaskinthecontextofapatternofusethatischaracteristicofarealorrealisticapplication.
Thismightincludemodicationoftheontologyordataduringthecompetitionrun.
Ourcurrentdisputeresolutionmechanismisunsatisfactory.
Recentwork[19]hasrevealedexamplesinthe2015corpuswherethecorrectreasonerwouldbeunfairlypenalizedforbeingintheminority.
Furthermore,incompletereasoners(orunsoundones,shouldanycomeforward)remainaproblem.
Oursolution,indevelopment[19],combinesamoresophisticatedvotingprocedurewithselectmanualverication.
Wehopetoincorporatethemanualvericationstepasaformof"bystander"participation(inadditiontotheresults-predictioncompetition).
Therearestillchallengesinconstructingameaningfulcorpusthatallowsforgeneralisationorproperreasonercomparison,inparticular,sincereasonersdonottypicallyimplementexactlythesamefragmentsofOWL(notably,datatypesupportvarieswidely,andmostELreasonersimplementslightlydifferentsubsetsofOWLEL).
Insomerespects,designingacorpusforacompetitioniseasierinthat,intheend,theresultsofacompetitionarejustthat…theoutcomesofacontest.
AcknowledgementsTheOREcompetitionhasbeentheworkofmanypeopleovertheyearsandwewouldliketoespeciallyacknowledgethecontributionsofErnestoJiménez-RuizforrunningtheveryrstprimitivecompetitionandbeingaPCchairforallworkshops;SamanthaBailforbeingaPCchairaswellasimplementingtherst"liveresults"screen;IanHorrocksforhelpinggettingtheprojectstarted;andYevgenyKazakovforhelpfuldiscussionsonthecompetitiondesignaswellasndingacriticalbugjustbeforethecompetitionstarted.
WealsowouldliketoacknowledgethegeneroussupportofB2iHealthcare(http://b2i.
sg)fortheirrepeateddonationsofprizemoneyandtheDBOntoproject(http://www.
cs.
ox.
ac.
uk/projects/DBOnto/)forfundingcompetitionT-shirts.
Finally,thecompetitionwouldnothavebeenpossiblewithoutthedonationofclustertimebyKonstantinKorovin(FundedbyRoyalSocietyResearchGrantRG080491).
OpenAccessThisarticleisdistributedunderthetermsoftheCreativeCommonsAttribution4.
0Interna-tionalLicense(http://creativecommons.
org/licenses/by/4.
0/),whichpermitsunrestricteduse,distribution,andreproductioninanymedium,providedyougiveappropriatecredittotheoriginalauthor(s)andthesource,providealinktotheCreativeCommonslicense,andindicateifchangesweremade.
123480B.
Parsiaetal.
References1.
ArmasRomero,A.
,CuencaGrau,B.
,Horrocks,I.
:MORe:modularcombinationofOWLreasonersforontologyclassication.
In:TheSemanticWeb—ISWC2012—11thInternationalSemanticWebConfer-ence,Boston,MA,USA,November11–15,2012,Proceedings,PartI,pp.
1–16(2012).
doi:10.
1007/978-3-642-35176-1_12.
Baader,F.
,Calvanese,D.
,McGuinness,D.
,Nardi,D.
,Patel-Schneider,P.
(eds.
):TheDescriptionLogicHandbook:Theory,Implementation,andApplications,2ndedn.
CambridgeUniversityPress,Cambridge(2007)3.
Bail,S.
,Glimm,B.
,Gonalves,R.
S.
,Jiménez-Ruiz,E.
,Kazakov,Y.
,Matentzoglu,N.
,Parsia,B.
(eds.
):InformalProceedingsofthe2ndInternationalWorkshoponOWLReasonerEvaluation(ORE-2013),Ulm,Germany,July22,2013,CEURWorkshopProceedings,vol.
1015.
CEUR-WS.
org.
http://ceur-ws.
org/Vol-1015(2013)4.
Bail,S.
,Glimm,B.
,Jiménez-Ruiz,E.
,Matentzoglu,N.
,Parsia,B.
,Steigmiller,A.
(eds.
):InformalPro-ceedingsofthe3rdInternationalWorkshoponOWLReasonerEvaluation(ORE2014)Co-locatedwiththeViennaSummerofLogic(VSL2014),Vienna,Austria,July13,2014,CEURWorkshopProceedings,vol.
1207.
CEUR-WS.
org.
http://ceur-ws.
org/Vol-1207(2014)5.
Beisswanger,E.
,Schulz,S.
,Stenzhorn,H.
,Hahn,U.
:Biotop:anupperdomainontologyforthelifesciencesadescriptionofitscurrentstructure,contentsandinterfacestoOBOontologies.
Appl.
Ontol.
3(4),205–212(2008).
doi:10.
3233/AO-2008-00576.
Connolly,D.
,vanHarmelen,F.
,Horrocks,I.
,McGuinness,D.
L.
,Patel-Schneider,P.
F.
,Stein,L.
:DAML+OIL(March2001)referencedescription.
W3CNote.
https://www.
w3.
org/TR/daml+oil-reference(2001)7.
Dumontier,M.
,Glimm,B.
,Gonalves,R.
S.
,Horridge,M.
,Jiménez-Ruiz,E.
,Matentzoglu,N.
,Parsia,B.
,Stamou,G.
B.
,Stoilos,G.
(eds.
):InformalProceedingsofthe4thInternationalWorkshoponOWLReasonerEvaluation(ORE-2015)Co-locatedwiththe28thInternationalWorkshoponDescriptionLogics(DL2015),Athens,Greece,June6,2015,CEURWorkshopProceedings,vol.
1387.
CEUR-WS.
org.
http://ceur-ws.
org/Vol-1387(2015)8.
Eiter,T.
,Schneider,P.
,Simkus,M.
,Xiao,G.
:UsingOpenStreetMapdatatocreatebenchmarksfordescriptionlogicreasoners.
In:Bailetal.
[4],pp.
51–57.
http://ceur-ws.
org/Vol-1207/paper_10.
pdf9.
Glimm,B.
,Horrocks,I.
,Motik,B.
,Stoilos,G.
,Wang,Z.
:HermiT:anOWL2reasoner.
J.
Autom.
Reason.
53(3),245–269(2014).
doi:10.
1007/s10817-014-9305-110.
Gonalves,R.
S.
,Matentzoglu,N.
,Parsia,B.
,Sattler,U.
:Theempiricalrobustnessofdescriptionlogicclassication.
In:ProceedingsoftheISWC2013PostersandDemonstrationsTrack,Sydney,Australia,October23,2013,pp.
277–280.
http://ceur-ws.
org/Vol-1035/iswc2013_poster_25.
pdf(2013)11.
Haarslev,V.
,Hidde,K.
,Mller,R.
,Wessel,M.
:TheRacerProknowledgerepresentationandreasoningsystem.
Semant.
Web3(3),267–277(2012).
doi:10.
3233/SW-2011-003212.
Herre,H.
,Hoehndorf,R.
,Kelso,J.
,Loebe,F.
,Schulz,S.
:OBML—ontologiesinbiomedicineandlifesciences.
J.
Biomed.
Semant.
2(S-4),I1.
http://www.
jbiomedsem.
com/content/2/S4/I1(2011)13.
Hitzler,P.
,Krtzsch,M.
,Parsia,B.
,Patel-Schneider,P.
F.
,Rudolph,S.
:OWL2WebOntologyLanguage:Primer,2ndedn.
W3CRecommendation.
https://www.
w3.
org/TR/owl-primer/(2012)14.
Horridge,M.
,Bechhofer,S.
:TheOWLAPI:aJavaAPIforOWLontologies.
Semant.
Web2(1),11–21(2011).
doi:10.
3233/SW-2011-002515.
Horrocks,I.
,Yatskevich,M.
,Jiménez-Ruiz,E.
(eds.
):Proceedingsofthe1stInternationalWorkshoponOWLReasonerEvaluation(ORE-2012),Manchester,UK,July1st,2012,CEURWorkshopProceedings,vol.
858.
CEUR-WS.
org.
http://ceur-ws.
org/Vol-858(2012)16.
Kazakov,Y.
,Krtzsch,M.
,Simancik,F.
:TheincredibleELK—frompolynomialprocedurestoefcientreasoningwithELontologies.
J.
Autom.
Reason.
53(1),1–61(2014).
doi:10.
1007/s10817-013-9296-317.
Keet,C.
M.
,Lawrynowicz,A.
,d'Amato,C.
,Kalousis,A.
,Nguyen,P.
,Palma,R.
,Stevens,R.
,Hilario,M.
:Thedataminingoptimizationontology.
J.
WebSemant.
32,43–53(2015).
doi:10.
1016/j.
websem.
2015.
01.
00118.
Krtzsch,M.
,Simanˇcík,F.
,Horrocks,I.
:Descriptionlogics.
IEEEIntell.
Syst.
29(1),12–19(2014).
doi:10.
1109/MIS.
2013.
12319.
Lee,M.
,Matentzoglu,N.
,Parsia,B.
,Sattler,U.
:Amulti-reasoner,justication-basedapproachtoreasonercorrectness.
In:InternationalSemanticWebConference(2015)20.
Liaw,S.
,Taggart,J.
,Yu,H.
,deLusignan,S.
,Kuziemsky,C.
E.
,Hayen,A.
:Integratingelectronichealthrecordinformationtosupportintegratedcare:practicalapplicationofontologiestoimprovetheaccuracyofdiabetesdiseaseregisters.
J.
Biomed.
Inform.
52,364–372.
http://www.
sciencedirect.
com/science/article/pii/S1532046414001798(2014)123TheOWLReasonerEvaluation(ORE)2015CompetitionReport48121.
Matentzoglu,N.
,Bail,S.
,Parsia,B.
:AsnapshotoftheOWLweb.
In:TheSemanticWeb—ISWC2013—12thInternationalSemanticWebConference,Sydney,NSW,Australia,October21–25,2013,Proceedings,PartI,pp.
331–346(2013).
doi:10.
1007/978-3-642-41335-3_2122.
Matentzoglu,N.
,Leo,J.
,Hudhra,V.
,Sattler,U.
,Parsia,B.
:Asurveyofcurrent,stand-aloneOWLreasoners.
In:Dumontier,M.
,Glimm,B.
,Gonalves,R.
,Horridge,M.
,Jiménez-Ruiz,E.
,Matentzoglu,N.
,Parsia,B.
,Stamou,G.
,Stoilos,G.
(eds.
)InformalProceedingsofthe4thInternationalWorkshoponOWLReasonerEvaluation,vol.
1387.
CEUR-WS(2015)23.
Matentzoglu,N.
,Parsia,B.
:TheOWLfull/DLgapintheeld.
In:Proceedingsofthe11thInternationalWorkshoponOWL:ExperiencesandDirections(OWLED2014)Co-locatedwith13thInternationalSemanticWebConferenceon(ISWC2014),RivadelGarda,Italy,October17–18,2014,pp.
49–60(2014)24.
Matentzoglu,N.
,Parsia,B.
:ORE2015reasonercompetitiondataset(2015).
doi:10.
5281/zenodo.
1857825.
Mendez,J.
:jcel:Amodularrule-basedreasoner.
In:Proceedingsofthe1stInternationalWorkshoponOWLReasonerEvaluation(ORE-2012),Manchester,UK,July1st,2012.
http://ceur-ws.
org/Vol-858/ore2012_paper12.
pdf(2012)26.
Motik,B.
,CuencaGrau,B.
,Horrocks,I.
,Wu,Z.
,Fokoue,A.
,Lutz,C.
:OWL2WebOntologyLanguage:Proles,2ndedn.
W3CRecommendation.
https://www.
w3.
org/TR/owl2-proles/(2012)27.
Motik,B.
,Nenov,Y.
,Piro,R.
,Horrocks,I.
,Olteanu,D.
:ParallelmaterialisationofDatalogprogramsincentralised,main-memoryRDFsystems.
In:ProceedingsoftheTwenty-EighthAAAIConferenceonArticialIntelligence,July27–31,2014,QuébecCity,Québec,Canada,pp.
129–137.
http://www.
aaai.
org/ocs/index.
php/AAAI/AAAI14/paper/view/8505(2014)28.
Motik,B.
,Patel-Schneider,P.
F.
,CuencaGrau,B.
:OWL2WebOntologyLanguage:Directsemantics,2ndedn.
W3CRecommendation.
https://www.
w3.
org/TR/owl2-direct-semantics/(2012)29.
Motik,B.
,Patel-Schneider,P.
F.
,Parsia,B.
,Bock,C.
,Fokoue,A.
,Haase,P.
,Hoekstra,R.
,Horrocks,I.
,Ruttenberg,A.
,Sattler,U.
,Smith,M.
:OWL2WebOntologyLanguage:Structuralspecicationandfunctional-stylesyntax,2ndedn.
W3CRecommendation.
https://www.
w3.
org/TR/owl2-syntax/(2012)30.
Noy,N.
F.
,Shah,N.
H.
,Whetzel,P.
L.
,Dai,B.
,Dorf,M.
,Grifth,N.
,Jonquet,C.
,Rubin,D.
L.
,Storey,M.
A.
D.
,Chute,C.
G.
,Musen,M.
A.
:BioPortal:ontologiesandintegrateddataresourcesattheclickofamouse.
NucleicAcidsRes.
37(Web–Server–Issue),170–173(2009).
doi:10.
1093/nar/gkp44031.
Osumi-Sutherland,D.
,Marygold,S.
J.
,Millburn,G.
H.
,McQuilton,P.
,Ponting,L.
,Stefancsik,R.
,Falls,K.
,Brown,N.
H.
,Gkoutos,G.
V.
:Thedrosophilaphenotypeontology.
J.
Biomed.
Semant.
4,30(2013).
doi:10.
1186/2041-1480-4-3032.
Palmisano,I.
:JFactrepository.
https://github.
com/owlcs/jfact(2015)33.
Parsia,B.
,Matentzoglu,N.
,Gonalves,R.
S.
,Glimm,B.
,Steigmiller,A.
:TheOWLreasonerevaluation(ORE)2015competitionreport.
In:Liebig,T.
,Fokoue,A.
(eds.
)Proceedingsofthe11thInternationalWorkshoponScalableSemanticWebKnowledgeBaseSystemsco-locatedwith14thInternationalSeman-ticWebConference(ISWC2015),CEURWorkshopProceedings,vol.
1457,pp.
2–15.
CEUR-WS.
org(2015)34.
Pelletier,F.
,Sutcliffe,G.
,Suttner,C.
:ThedevelopmentofCASC.
AICommun.
15(2–3),79–90(2002)35.
Ren,Y.
,Pan,J.
Z.
,Zhao,Y.
:SoundnesspreservingapproximationforTBoxreasoning.
In:ProceedingsoftheTwenty-FourthAAAIConferenceonArticialIntelligence,AAAI2010,Atlanta,Georgia,USA,July11–15,2010.
http://www.
aaai.
org/ocs/index.
php/AAAI/AAAI10/paper/view/1955(2010)36.
Samwald,M.
:GenomicCDS:anexampleofacomplexontologyforpharmacogeneticsandclinicaldecisionsupport.
In:Bailetal.
[3],pp.
128–133.
http://ceur-ws.
org/Vol-1015/paper_12.
pdf37.
Sattler,U.
,Matentzoglu,N.
:ListofReasoners(owl.
cs)http://owl.
cs.
manchester.
ac.
uk/tools/list-of-reasoners/.
http://owl.
cs.
manchester.
ac.
uk/tools/list-of-reasoners/(2014)38.
Schulz,S.
,Martínez-Costa,C.
:Howontologiescanimprovesemanticinteroperabilityinhealthcare.
In:Riao,D.
,Lenz,R.
,Miksch,S.
,Peleg,M.
,Reichert,M.
,tenTeije,A.
(eds.
)ProcessSupportandKnowledgeRepresentationinHealthCare-AIME2013JointWorkshop,KR4HC2013/ProHealth2013,Murcia,Spain,June1,2013,RevisedSelectedPapers,LectureNotesinComputerScience,vol.
8268,pp.
1–10.
Springer(2013).
doi:10.
1007/978-3-319-03916-9_139.
Sertkaya,B.
:TheELepHantreasonersystemdescription.
In:InformalProceedingsofthe2ndInternationalWorkshoponOWLReasonerEvaluation(ORE-2013),Ulm,Germany,July22,2013,pp.
87–93.
http://ceur-ws.
org/Vol-1015/paper_15.
pdf(2013)40.
Sirin,E.
,Parsia,B.
,CuencaGrau,B.
,Kalyanpur,A.
,Katz,Y.
:Pellet:apracticalOWL-DLreasoner.
J.
WebSemant.
5(2),51–53(2007).
doi:10.
1016/j.
websem.
2007.
03.
00441.
Steigmiller,A.
,Liebig,T.
,Glimm,B.
:Konclude:systemdescription.
J.
WebSemant.
27,78–85(2014).
doi:10.
1016/j.
websem.
2014.
06.
00342.
Stevens,R.
,Matentzoglu,N.
,Sattler,U.
,Stevens,M.
:AfamilyhistoryknowledgebaseinOWL2.
In:Bailetal.
[4],pp.
71–76.
http://ceur-ws.
org/Vol-1207/paper_11.
pdf123482B.
Parsiaetal.
43.
Sutcliffe,G.
:TheTPTPproblemlibraryandassociatedinfrastructure:theFOFandCNFParts,v3.
5.
0.
J.
Autom.
Reason.
43(4),337–362(2009)44.
Sutcliffe,G.
:TheCADEATPsystemcompetition—CASC.
AIMag.
37(2),99–101(2016)45.
Thomas,E.
,Pan,J.
Z.
,Ren,Y.
:TrOWL:TractableOWL2reasoninginfrastructure.
In:TheSemanticWeb:ResearchandApplications,7thExtendedSemanticWebConference,ESWC2010,Heraklion,Crete,Greece,May30–June3,2010,Proceedings,PartII,pp.
431–435(2010).
doi:10.
1007/978-3-642-13489-0_3846.
Tsarkov,D.
,Horrocks,I.
:FaCT++descriptionlogicreasoner:systemdescription.
In:AutomatedRea-soning,ThirdInternationalJointConference,IJCAR2006,Seattle,WA,USA,August17–20,2006,Proceedings,pp.
292–297(2006).
doi:10.
1007/11814771_2647.
Tsarkov,D.
,Palmisano,I.
:Chainsaw:ametareasonerforlargeontologies.
In:Proceedingsofthe1stInternationalWorkshoponOWLReasonerEvaluation(ORE-2012),Manchester,UK,July1st,2012.
http://ceur-ws.
org/Vol-858/ore2012_paper2.
pdf(2012)48.
Wolstencroft,K.
,Brass,A.
,Horrocks,I.
,Lord,P.
W.
,Sattler,U.
,Turi,D.
,Stevens,R.
:Alittlesemanticwebgoesalongwayinbiology.
In:Gil,Y.
,Motta,E.
,Benjamins,V.
R.
,Musen,M.
A.
(eds.
)TheSemanticWeb—ISWC2005,4thInternationalSemanticWebConference,ISWC2005,Galway,Ireland,November6–10,2005,Proceedings,LectureNotesinComputerScience,vol.
3729,pp.
786–800.
Springer(2005).
doi:10.
1007/11574620_5649.
Wu,J.
,Hudek,A.
,Toman,D.
,Weddell,G.
:AbsorptionforABoxes.
J.
Autom.
Reason.
1–29(2014).
doi:10.
1007/s10817-014-9304-250.
Zhou,Y.
,Nenov,Y.
,Grau,B.
C.
,Horrocks,I.
:Pay-as-you-goOWLqueryansweringusingatriplestore.
In:ProceedingsoftheTwenty-EighthAAAIConferenceonArticialIntelligence,July27–31,2014,QuébecCity,Québec,Canada,pp.
1142–1148(2014)123