ErrorAnalysisofNamedEntityRecognitioninBCCWJMasaakiIchihara1KanakoKomiya1TomoyaIwakura2MaikoYamazaki31IbarakiUniversity3FujitsuLaboratoriesLtd.
,2TokyoInstituteofTechnology{11t4004s@hcs,kkomiya@mx}.
ibaraki.
ac.
jp,iwakura.
tomoya@jp.
fujitsu.
com,yamazaki@lr.
pi.
titech.
ac.
jp1IntroductionNamedEntityRecognitionisaprocessbywhichnamedentities(NEs)suchasthenamesofpersons,locations,andartifactsareextracted.
Mostnamedentityrecognitiontechniqueshavebeenstudiedonnewsarticles,however,theirperformancesondier-entdomaintextssuchasblogs,booksandmaga-zinesarestillnotevaluatedwell.
ThispaperreportsanerroranalysisofKNPonsixdomainsforreveal-ingcausesoferrorsforfurtherimprovementofNErecognition1.
2ErrorAnalysisofKNPonBCCWJJapanesedependencyandcasestructureanalyzerKNP2([2]and[3])wasusedasthenamedentityrecognizer.
TheversionsweusedwereKNPVer.
4.
11andJUMANVer.
7.
0.
Thesixgenres,"Q&Asites","whitepapers","blogs","books","magazines",and"newspaperar-ticles",inBalancedCorpusofContemporaryWrit-tenJapanese(BCCWJ)wereusedasthetargetcor-pora.
OnehundredthirtysixtextsextractedfromBC-CWJ,theyareavailableasClassA3,wereusedfortheexperiments.
TheyweremanuallyannotatedwithninekindsofNEthatweredenedbyInformationRetrievalandExtractionExercise(IREX)4.
TheseNEtypesarethenamesofpersons,locations,artifacts,dates,times,moneys,percents,andoptional5.
Theanno-tationwasdonebyvemembersofNEteamoftheProjectNextNLP,andcheckedbyfourmembersofit.
1ThispaperisanEnglishversionof(Ichiharaetal.
,2015)[1]withadditionalinformationandsomecorrections.
2http://nlp.
ist.
i.
kyoto-u.
ac.
jp/EN/index.
phpKNP3http://plata.
ar.
media.
kyoto-u.
ac.
jp/mori/research/NLR/JDC/ClassA-1.
list4http://nlp.
cs.
nyu.
edu/irex/index-e.
html5KNPdoesnotextractoptionaltags.
WecomparedKNPoutputswiththemanuallyan-notatedtextsandanalyzederrors.
Table1showstheperformancesofKNP.
Theequa-tionsofrecall,precision,accuracy,andF-measureareasfollows.
"Correct",thenumeratorofrecall,precision,andaccuracy,isthenumberofthecor-rectanswersofKNP.
"Annotated",thedenominatorofrecall,denotesthenumberoftheNEsthatweremanuallyannotated.
"KNPoutputs",thedenomi-natorofprecision,denotesthenumberoftheNEsthatKNPoutput.
Thedenominatorofaccuracyisthelogicalsum(OR)of"Annotated"and"KNPout-puts".
Thedenominatorsofrecall,precision,andac-curacyvarybecauseKNPsometimescannotextractsomeNEsandsometimesextractswronginforma-tion.
Also,anNEthatthesystemoutputsometimesconsistsofmultipleannotatedNEsasillustratedbyanexampleinFigure1andviceversa.
Table1showstherecallislowerthantheprecision.
KNP:PERSON/PERSONAnnotationLOCATION/LOCATIONLOCATION/LOCATIONFigure1:AnexampleofanNEKNPoutputincludesmultipleannotatedNEsRecall=CorrectAnnotated(1)Precision=CorrectKNPoutputs(2)Accuracy=CorrectAnnotated∪KNPoutputs(3)Fmeasure=2Recall·PrecisionRecall+Precision(4)Table1:PerformancesofKNPPerformanceRateCorrectDenominatorRecall61.
79%2641Precision74.
79%16322182Accuracy57.
95%2816F-measure67.
68Theerrorswereclassiedintothefollowingvetypes.
Exampleswereshownwithdescription.
NoextractionTheerrorwhereKNPdidnotex-tracttokensasanNEthoughtheywereanno-tated.
KNP:AnnotationARTIFACT/ARTIFACTNoannotationTheerrorwhereKNPextractedtokensasanNEthoughtheywerenotanno-tated.
KNP:PERSON/PERSONAnnotationWrongrangeTheerrorwhereKNPextractedto-kensasanNEandonlytherangewaswrong.
(Theextractedtokenswerepartiallyannotatedortheywerethepartoftheannotatedtokens.
)KNP1:PERSON/PERSONAnnotation1PERSON/PERSONKNP2:ORGANIZATION/ORGANIZATIONAnnotation2ORGANIZATION/ORGANIZATIONWrongtagTheerrorwhereKNPextractedtokensasanNEandonlythetagtypewaswrong.
KNP:PERSON/PERSONAnnotationLOCATION/LOCATIONWrongrangeandtagTheerrorwhereKNPex-tractedtokensasanNEandboththerangeandthetagtypewerewrong.
KNP:PERSON/PERSONAnnotationLOCATION/LOCATIONTable2:SummaryoferrorsErrortypeNumRateNoextraction61952.
28%Noannotation15913.
43%Wrongrange16213.
68%Wrongtag12710.
73%Wrongrangeandtag1179.
88%Allerrors1184100.
00%Table2showsasummaryoferrors.
Theseerrorswerecountedbythelogicalsum(OR)ofannotatedNEsandKNPoutputs.
Themostfrequenterrorwas"Noextraction"anditaccountedformorethanhalfofthetotalerrors.
Thesecondmostfrequenter-rorwas"Wrongrange"andmostofthemweretheerrorswhereextractedtokenswerethepartoftheannotatedtokens.
Table3showsasummaryoferrorsbytypesofNEs.
Theseerrorswerealsocountedbythelogi-calsum(OR)ofannotatedNEsandKNPoutputs.
"Correct"and"Error"arethenumbersofthecorrectanswersandtheerrorsofKNP.
"Total"isthesumof"Correct"and"Error".
"Noextraction"and"Er-rorswithextraction"inthetablemeanthenumbersof"Noextraction"andtheerrorsotherthan"Noex-traction",respectively.
"Noextractionrate"istheratioof"Noextraction"in"Error".
Table3showsthatnoextractionratesof"ARTI-FACT","PERCENT","TIME",and"OPTIONAL"areespeciallyhigh.
Atthesametime,therearesmallnumberofNEsof"PERCENT"and"TIME"inthecorpora.
Therefore,wecansee"ARTIFACT"isthebigreasonwhythenoextractionrateofalltagsishigh.
Noextractionrateof"OPTIONAL"is100%becauseKNPdoesnotextractOPTIONALsandthisisanotherreason.
Table3alsoshowsthatmostof"TIME","MONEY",and"PRECENT"werecorrectlytaggedbyKNPiftheyweretagged.
Mostoftheerrorswhentheywereextractedarethoseof"ORGANIZA-TION","PERSON",and"LOCATION".
Thesumoferrorsof"ARTIFACT"and"DATE"arelessthan30%ofallerrorswhentheywereextracted.
Table4showstheaccuraciesandtheratesofnoextractionin"Total"accordingtothetagtype.
"Ac-curacy"istheratioofthecorrectanswersin"Total",thesumofcorrectanswersanderrorsofKNP,and"Noextraction/Total"istheratioofnoextractioninit.
Theseerrorswerealsocountedbythelogicalsum(OR)ofannotatedNEsandKNPoutputs.
Table4showsthattheaccuracyof"ARTIFACT"isparticularlylowcomparingwiththeothertags.
Thesametableshowstheratioofnoextractionin"Total"isalsohigh.
Therefore,wecouldseethat"Noextraction"of"ARTIFACT"isthebiggestcauseTable3:SummaryoferrorsbytypesofNEsTagCorrectErrorTotalNoextractionErrorswithextractionNoextractionrateARTIFACT902593491926774.
13%DATE343145488628342.
76%LOCATION4092266357215431.
86%MONEY884922250.
00%ORGANIZATION2362004367712338.
50%PERCENT79129110283.
33%PERSON3642225868813439.
64%TIME2393290100.
00%OPTIONAL01071071070100.
00%AllTags16321184281661956552.
28%Table4:Accuraciesandratesofnoextractionin"Total"accordingtothetagtypeTagAccuracyNoextraction/TotalARTIFACT25.
79%55.
01%DATE70.
29%12.
70%LOCATION64.
41%11.
34%MONEY95.
65%2.
17%ORGANIZATION54.
13%17.
66%PERCENT86.
81%10.
99%PERSON62.
12%15.
02%TIME71.
88%28.
13%OPTIONAL0.
00%100.
00%AllTags57.
95%21.
98%oftheerrorsofKNPandthemainreasonoflowrecall.
3ErrorAnalysisof"NoEx-traction"Thetargetcorporaweusedconsistedofsixgenres,"Q&Asites","whitepapers","blogs","books","magazines",and"newspaperarticles",inBCCWJ.
Table5showsasummaryoferrorsbygenresoftexts.
Theseerrorsexcept"Noextraction"arethosethatKNPoutput.
"Correct"and"Error"arethenumberofthecorrectanswersandtheerrorsofKNP.
"Total"isthesumof"Correct"and"Error".
"Noextraction"and"Errorswithextraction"intheta-blemeanthenumbersof"Noextraction"andtheerrorsotherthan"Noextraction",respectively.
"Noextractionrate"istheratioof"Noextraction"in"Error".
"Docs"isthenumberofdocumentsofthegenre.
Thetotalnumberoferrors(1169)andtotalnum-beroferrorswithextraction(550)aredierentfromthoseinTables2and3(1184and565).
Thisisbe-causesomeNEsthatKNPoutputincludemultipleTable6:Accuraciesandratesofnoextractionin"Total"accordingtothegenreGenreAccuracyNoextraction/TotalQ&A40.
00%44.
21%Whitepaper58.
73%20.
63%Blog50.
74%27.
89%Book50.
35%28.
07%Magazine53.
45%14.
66%Newspaper72.
27%15.
49%All58.
26%22.
10%annotatedNEs.
Inaddition,thenumberofwordsvariesaccordingtothegenre.
WethinkthisisareasonwhythetotalnumberoftheNEswasnotproportionaltothenumberofthedocuments.
Table5showsthatthegenrewhosenoextractionratewasthehighestwas"Q&Asites"andthegenrewiththelowestratewas"magazines".
Table6showstheaccuraciesandtheratesofnoextractionin"Total"accordingtothegenre.
"Accu-racy"istheratioofthecorrectanswersin"Total",thesumofcorrectanswersanderrorsofKNP,and"Noextraction/Total"istheratioofnoextractioninit.
Theseerrorsexcept"Noextraction"arethosethatKNPoutput.
"Accuracy"of"All"(58.
26%)isdierentfrom"Recall"inTable1(61.
79%)becausethenumberoftheNEsKNPoutputwasdierentfromthenumberoftheNEsthatwereannotatedbyhumans.
Table6showsthat"newspaperarticles"isthegenrewhoseaccuracyisthehighest.
WethinkthisisbecauseKNPwastrainedwithnewspaperarticlesofMAINICHISHIMBUN.
Table6alsoshowsthegenrewiththelowestaccuracywas"Q&Asites".
WethinkthisisbecausethewritingstyleofQ&Asiteswasthemostdierentfromthatofnewspaperarticles.
Thesametableshowsthatthegenrewhosenoextractionratewasthehighestwas"Q&Asites"Table5:SummaryoferrorsbygenresoftextsGenreCorrectErrorTotalNoextractionErrorswithextractionNoextractionrateDocsQ&A76114190843073.
68%74Whitepaper42730072715015050.
00%8Blog171166337947256.
63%34Book2172144311219356.
54%5Magazine1861623485111131.
48%2Newspaper5552137681199455.
87%13AllGenres16321169280161955052.
95%136andthegenrewiththelowestratewas"magazines".
3.
1NoExtractionofQ&ASites"Q&Asites"wasthegenrewhoseaccuracywasthelowest.
Theexamplesofnoextractionerrorsin"Q&Asites"areshownasfollows.
iManynamesofproducts,characters,andmedicineswerenotextracted.
(SakuraWars)(SuperNintendoEntertainmentSystem)(ActRaiser)4(Res-identEvil4)(KamenRider)(Ultraman)(Gundam)(Minostacin)(Aspirin)iiAbbreviationswerenotextracted.
Formalnamesarenotedinbrackets.
(MarioWorld)(SuperMarioWorld)GC((NintendoGameCube))JNB((JapanNetBank))LA((LosAngeles))iiiTheunusualdateexpressionswerenotextracted.
(90/11/21)ivHiraganaexpressionsweresometimeswronglyparsed.
"(Satoshi)"in"(CHIEBUKURERSatoshi)"shouldbethenameofpersonbutitiswronglyparsedas"(Satoru)":averb.
vNEswritteninalphabetsandnumberswerenotextracted.
"(JREast)"wereextracted.
3.
2NoExtractionofNewspaperAr-ticles"Newspaperarticles"wasthegenrewhoseaccuracywasthehighest.
Theexamplesofnoextractioner-rorsin"newspaperarticles"areshownasfollows.
iSomeNEswithspecicprexesandsuxeswerenotextracted.
(half**,ex.
halftime)(**region,ex.
(capitalregion)(threemajormetropolitanareas))(**area)(**point)(same**,ex.
(same**year)(sameday)(sameyearautumn))iiOPTIONALswerenotextractedbecauseKNPdoesnotextractoptionaltags.
iiiTheunusualEnglishexpressionsinJapanesesen-tenceswerenotextracted.
KOERAJAPANivBracketssometimescausetheerrors.
(Phoenix(Arizona,US))vNEsthatconsistofgeneralnounswerenotex-tracted.
Thiscouldbethereasonwhythenamesofproductsandcharacterswerenotextracted.
(Hirune,anap)(Zaurus)(FamilyMart)(Sharp)(TheRenaissance)"Softbank"sometimescouldbeextractedandsometimescouldnot.
Theywereparsedasnom-inativecasewhentheywereextractedandas"inclause"whentheywerenot.
4DiscussionAccordingtotheexamplesdescribed,wethinkthatthelackofknowledgeinthedictionaryandtheerrorsoftheparserarethebigreasonsoftheerrorsofthenamedentityrecognition.
Inparticular,thenamesofartifactsincludingthenamesofproductsorchar-actersareoftennewwordsthatwerecoined.
TheseNEsarenotinthedictionaryKNPusesandthere-fore,theyshouldbejudgediftheyweretheNEsornotdependsonthefeaturesofthesurroundingpat-ternsandthesyntacticfeatures.
Asaresult,thecorrectparsingwouldbeimportantfortheNEsthatcannotusedictionaryinformation.
However,theca-sualwritingstylelikeQ&Asitescausestheerrorsinmorphologicalanalysisandparsing.
Wethinkthatifthesentencesoftheseinformalwritingstylescouldbecorrectlyanalyzedandparsed,theerrorswouldbedecreased.
Thetrainingoftextswithinformalwritingstylescouldbethesolutionofthisproblem.
Inaddition,mostoftheNEsthatwerenotextractedbyKNPwerefoundinWikipediaorotherWebsites.
Thisinformationalsocouldhelptherecallimprove.
5ConclusionThispaperreportsanerroranalysisofthenamedentityrecognizerKNPonsixdomainsforrevealingcausesoferrors.
ThetextsofBCCWJweremanu-allyannotatedandcomparedwiththeautomaticallytaggedtexts.
Theanalysisrevealedthatthemostfrequenterrorwas"Noextraction":thecasewherethetokenswerenotextractedbyKNPthoughtheywereannotated.
Italsorevealedthat"Noextrac-tion"of"ARTIFACT"isthebiggestcauseoflowrecalland"Q&Asite"isthegenrewhoseaccuracyisthelowest.
Wefocusedonthenoextractionerrorsandfoundoutthatthelackofdictionaryinformationandthevariouswritingstylescausetheseerrors.
AcknowledgementsThisworkwaspartiallysupportedbyJSPSKAK-ENHIGrantNumber24700138.
WewouldliketothankDr.
RyoheiSasanowhoprovidesusthehelp-fulinformationaboutKNPandteammembersofNEteamofProjectNextNLP.
References[1]MasaakiIchihara,MaikoYamazaki,andKanakoKomiya.
Erroranalysisofnamedentityextrac-tioninbccwj(bccwj).
7,p.
toappear,2015.
[2]RyoheiSasanoandSadaoKurohashi.
Japanesenamedentityrecognitionusingnon-localinfor-mation(injapanese).
IPSJJournal,Vol.
49,No.
11,pp.
3765–3776,2008.
[3]knp.
,19,pp.
110–113,2013.
CloudCone商家我们很多喜欢低价便宜VPS主机的肯定是熟悉的,个人不是特别喜欢他。因为我之前测试过几次,开通的机器IP都是不通的,需要删除且开通好几次才能得到一个可用的IP地址。当然他们家的优势也是有的,就是价格确实便宜,而且还支持删除重新开通,而且机房只有一个洛杉矶MC。实话,如果他们家能多几个机房,保持现在的特点,还是有很多市场的。CloudCone是来自美国的主机销售商,成立于2017...
官方网站:点击访问创梦网络宿迁BGP高防活动方案:机房CPU内存硬盘带宽IP防护流量原价活动价开通方式宿迁BGP4vCPU4G40G+50G20Mbps1个100G不限流量299元/月 209.3元/月点击自助购买成都电信优化线路8vCPU8G40G+50G20Mbps1个100G不限流量399元/月 279.3元/月点击自助购买成都电信优化线路8vCPU16G40G+50G2...
整理一下CloudCone商家之前推送的闪购VPS云服务器产品,数量有限,活动推出可能很快机器就售罄了,有需要美国便宜VPS云服务器的朋友可以关注一下。CloudCone怎么样?CloudCone服务器好不好?CloudCone值不值得购买?CloudCone是一家成立于2017年的美国服务器提供商,国外实力大厂,自己开发的主机系统面板,CloudCone主要销售美国洛杉矶云服务器产品,优势特点是...
softbank官网为你推荐
中国互联网域名注册负责我国境内internet用户域名注册是什么机构免费云主机永久免费的云主机哎或者空间或者vpsasp主机ASP环境是不是所有的主机都默认支持?个人虚拟主机个人网站该购买什么类型虚拟主机?免备案虚拟空间免备案的虚拟主机空间,买了以后会强制备案不?php虚拟空间怎样修改php虚拟空间单个文件上传大小限制asp虚拟空间怎样在一个虚拟空间里放上一个ASP和一个PHP的网站免费网站空间有没有免费的网站空间推荐美国网站空间美国,韩国,香港网站空间免费网站空间申请如何申请到免费的网站空间
怎么申请域名 132邮箱 博客主机 美国主机网 好看的留言 lamp配置 dux 申请个人网站 ntfs格式分区 中国网通测速 免费邮件服务器 web服务器是什么 网页提速 全能空间 西安主机 个人免费邮箱 atom处理器 云服务是什么意思 带宽测试 七牛云存储 更多