TheNewAlgorithmoftheItem-basedonMapReduceZHAOWei1,a1CollegesoftwareTechnologySchool,ZhengzhouUniversityZhengzhou450002,Chinaaiezhaowei@163.
comKeywords:RecommendationsystemparallelcomputingClusteringAbstract.
TraditionalcollaborativefilteringalgorithmbasedonitemandK-meansclusteringalgorithmarestudied,theparallelalgorithmofcollaborativefilteringItem-basedonMapReduceisproposedbyusingMapReduceprogrammingmodel.
Thealgorithmismainlydividedintotwosteps,onestepisK-Meansalgorithmclusteringforusers,anotherstepistheparallelItem-basedalgorithmforclusteringuserrecommendation.
Experimentalresultsshowthatthealgorithmhasobtainedverygoodeffect,improvedtherunningspeedandexecutionefficiency,theimprovedalgorithmismuchsuitableforprocessingbigdata.
IntroductionBigdatausuallyincludesdatasetswithsizesbeyondtheabilityofcommonlyusedsoftwaretoolstocapture,curate,manage,andprocessdatawithinatolerableelapsedtime.
Bigdataishighvolume,highvelocity,and/orhighvarietyinformationassetsthatrequirenewformsofprocessingtoenableenhanceddecisionmaking,insightdiscoveryandprocessoptimization.
Volumemeansbigdatadoesn'tsample;itjustobservesandtrackswhathappens;Velocitymeansbigdataisoftenavailableinreal-time;Varietymeansbigdatadrawsfromtext,images,audio,video;plusitcompletesmissingpiecesthroughdatafusion[1].
Therefore,thebigdatamustbethroughthecomputerstatistics,comparison,analysisofthedatacanbetheobjectiveresults.
Nowelectroniccommercesystemsofeverytransaction,everyinputandeverysearchcanasdata,datathroughthecomputersystemtodothescreening,sorting,analysis,sothattheanalysisresultsisnotonlyanobjectiveconclusion,moreabletohelpbusinessprovidedthedecision-makingofenterprisesandalsocollectedusefuldatacanalsobereasonableplanning,activelyguidethedevelopmentoflargerpowerconsumption,andmoreeffectivemarketingandpromotion.
Withtheincreasingamountofdataintheelectroniccommercesystem,theneedforalargenumberofdatadepthanalysisisincreasinglyurgent.
Therefore,theuseofasimpleandhighscalabilityoftheprogramfortheanalysisofproductrecommendationisparticularlyimportant.
Atpresentdomesticmanyecommercesitesusecollaborativefilteringalgorithm,suchasAmazon,Dangdang,collaborativefilteringalgorithmismainlydividedintobasedontheitemsofthecollaborativefilteringalgorithmanduserbasedcollaborativefilteringalgorithm.
Basedonitemsofcollaborativefilteringalgorithmistomeasurethesimilaritybetweenitemsaccordingtotheuser'spreferences,donotneedtoconsidertheitemspecificcontentfeatures,sothealgorithmismainlyusedine-commercerecommendationandmovierecommendationdomain,thealgorithmwhileinthefieldofelectroniccommercerecommendationhasbeenacertaindegreeofsuccess.
Butinmassivedataarerecommendedwhenthedataisrecommendedperformanceisnothighandthedatainformationlackofsharingandextendedtheleadtothehardwarerequirementscomparedhigherinherentshortcomingsmakeitdidnotreceiveapromotionandsupportofenterpriseelectroniccommerce[2].
SoifweuseMapReducetoachievedistributedparallelcomputing,itwillgreatlyimprovetheefficiencyandperformanceofthealgorithm,andpromotethefurtherdevelopmentofthealgorithm[3-4].
Basedontheitemsofthecollaborativefilteringalgorithmisaccordingtoitemsimilarityanduserhistoryaccessrecordrecommendedtotheusertogeneratealistofitems,buttherearesomesmallproblems,suchasdatasparsityproblemandwhenthemassofusersandthenumberofitems,theuserbehaviorandrecorddatawillgreatly,andthealgorithmforcomputingitemswithsimilarmatrixcostgreatly,algorithmefficiencyandperformancewillgreatlyreduce.
Aimingattheaboveproblems,theclusteringalgorithmhasalsobeenappliedtoacollaborativefilteringalgorithmbasedonitem,themassiveuserclusteringanalysis,soitcanavoidthequestioncarefully,foreachusertorecommendoperation.
Thefirstshoppinguserswithsimilarinterestsintoauserclass,withaclusterofuserrecommendedgoodsarethesame.
Thesecondistoreducethemassiveuserdimensionsbecomedozensofclusteringlimited,thetimecomplexityencounteredabottleneck,andtheparallelclusteringalgorithmusingMapReduceistheeffectivewaytosolvethebottleneck[5].
MapReduceisadistributedprogrammingmodelframeworkonHadoopplatform,intheconditionofnotfamiliarwiththeunderlyingdetailsofthedistributedimplementationoftheimplementationoftheprogram[6].
TheMapReduceasparallelcomputingprogrammingmodel,firstofalltousersofMapReducebasedparallelclusteringandaccordingtotheresultsofuserclustering,ineveryuserclassusingtheMapReduceparallelcollaborativefilteringrecommendation,eventuallygiveusersareasonablepersonalizedcommodityrecommendationlist.
Therunningtimeofdifferentnodesinthequantitativedataiscomparedwiththenewalgorithm.
Theresultsshowthatthedataprocessingperformanceoftheproposedalgorithmisgreatlyimproved.
TheprincipleofMapReduceprogrammingmodelMapReduceisinHadoopplatformbyusingparallelcomputingprogrammingmodel,thetechniqueisproposedbyGoogleforatypicaldistributedparallelprogrammingmodel,theuserintheMapReducemodeldevelopthemapandreducefunctions,canrealizetheparallelprocessing.
Mapwillberesponsiblefordatadispersion,Reduceisresponsiblefordataaggregation.
UsersonlyneedtoachieveMapandReducetwointerface,youcancompletethecalculationofTBleveldata.
BecauseoftheMapReducemodel,thedetailsoftheparallelandfault-tolerantprocessingareencapsulated,whichmakesprogrammingveryeasytoimplement.
MapReduceparallelcalculationisdividedintotwoparts,thefirststepisinitializingtheoriginalinputdatafileandthedatasetisdividedintoapluralityofacertainsizeofdatablock,facilitateparallelcomputing;thesecondstepistostartthemapandreducefunctionsalgorithmofparallelcomputing,finallyproducedthefinalresult.
Figure1ParallelflowchartofMapReduceKeytechnologyresearchandImplementation1.
ThebasicideaofthetraditionalcollaborativefilteringalgorithmbasedonItem-basedThetraditionalbasedonitemsofcollaborativefilteringalgorithmthebasicideaisdividedintothreeparts,thefirstpartistocomputethesimilaritybetweenitems,commonsimilaritycalculationmethodwithcosinesimilarity,Pearsoncorrelationcoefficient,Tanmotocoefficientcorrelationof.
ThispaperselectstheEuclideansimilarityalgorithm,asfollows:TheassumptionisthatthereisavectorXandavectorY:X=(1x,2x,3x),Y=(1y,2y,3y),UsingtheEuclideansimilarityalgorithmtocalculatethesimilaritybetweenXandYSvector(x,y)formulaisasfollows[7]:1(,)1(,)Sxydxy=+(1)Where(,)dxyisthedistancebetweenthevectorXandY,thecalculationformulaisasfollows:222231123(dxyxyyyxx2)Thesecondpartistocalculatetheuserratingsmatrixontheitemsofthegoodsaccordingtothesimilaritymatrix;thethirdpartistheitemsimilaritymatrixWandtheusersoftheitemscorematrixmultiplicationtoobtaintherecommendationresults.
TraditionalItem-Basedcollaborativefilteringrecommendationalgorithmbasedonitemisthestagethataffectstheperformanceofthealgorithm.
Ifthenumberofusersisn,thenumberofcommodityitemsism,thetimecomplexityoffindingalltheitemsinthenprojectisO(2m),thetotalsearchspaceisnusers,sothetimecomplexityofcomputingsimilarityisO(2nm).
Sowhencalculatingthesimilaritymatrixofitems,itisindependentofthesimilaritybetweenthecalculatedandtheotherpairofitemstoaproject,soitispossibletocalculatethesimilaritymatrix.
2.
AnewalgorithmofItem-basedbasedonMapReduceThenewalgorithmismainlydividedintotwosteps;thefirststepistheMapReduceimplementationofK-Meansalgorithmbasedonclusteringofusers.
ThesecondstepistoachievetheparallelrecommendationalgorithmofItem-basedonMapReduce,theproductofuserclusteringrecommendation.
2.
1ThenewalgorithmK-MeansbasedonMapReduceThebasicideaofthetraditionalK-meansclusteringalgorithm:fromMdataobjectsinarbitrarychoiceofKobjectsastheinitialclustercenters;fortherestoftheotherobjects,accordingtotheirdistanceandtheclustercenters,respectively,theyallocatedtoitsmostsimilarclustering;thencalculateeachreceivedanewclusteringalgorithmclusteringcenter;keeprepeatingtheprocessuntilnochangesinacore.
Inthek-meansalgorithmtocalculatethedistancebetweendataobjectsandclustercentersisthemosttime-consumingoperation.
ThedataobjectandKclustercenterdistancecomparisonatthesametime,datafromotherobjectscanalsobecomparedwiththeKdistanceofthecenterofcluster,sotheoperationcanbeparallelized[8]BasedonMapReduceparallelimplementationofK-meansalgorithmcanimprovethespeedoftheclusteringalgorithm,isdividedintothreesteps:thefirststep:themapfunction,foreverypointcalculationrecentlythecenterdistanceandthecorrespondingtothenearestclustercenter.
Thesecondstep:Combinefunction,justcompletedtheMapmachineonthemachinearecompletedwiththesamepointoftheclusterpointofsummation,reducetheamountofcommunicationandcomputationofReduceoperation.
ThisstepisthekeytotheuseofCombinefunctiononthemachineonthefirstofthesameclustermerge,reducedtotheReducefunctionofthetransferandtheamountofcomputation.
Thethirdstep:theReducefunction,theintermediatedataofeachclustercenterwillbeformedandthenewclustercentercanbeobtained.
Eachiterationisrepeatedonthethreestep.
Figure2ParallelFlowChartofK-meansAlgorithmbasedonMapReduce2.
2thecollaborativefilteringalgorithmbasedonMapReduceforparallelimplementationofItem-basedBasedonthesimilaritycalculationformulamentionedabove(1),thispaperpresentsacollaborativefilteringrecommendationalgorithmbasedonMapReduce.
Algorithm1ThecollaborativefilteringrecommendationalgorithmbasedonMapReduceINPUT:Userinformationfile,Iteminformationfile,IntendeduserOUTPUT:IntendeduserrecommendedlistTheprocessisasfollows:Step1:Transformingtheuservectorintoanitemvector;Step2:Parallelcalculationofthesimilaritybetweenitems;thecalculationofthesimilaritybetweenitemsaccordingtotheformula(2)tocalculate;Step3:Similaritymatrixofparallelcomputingobjects;Step4:Parallelcomputinguserratingmatrix;inthecalculationoftheuser'sscoringmatrix,iftheuserisnotontheitemstoomuch,thenthedefaultscoreis1;Step5:Theresultsobtainedbythemultiplicationofthesimilaritymatrixofparallelcomputingobjectsandtheuser'sscorematrixarerecommended.
Experimentalresultanalysis1.
experimentalenvironmentThesimulationexperimentusingVMware_Workstation_10.
0.
3,virtualizationsoftwaretovirtualHadoopcloudplatform.
EightvirtualmachinesareinstalledonthevirtualHadoopcloudplatform,andaHadoopclusterenvironmentisbuiltontheseeightvirtualmachines.
OneofthevirtualmachineasagoodJobTrackernodeNameNode,theothersevenvirtualmachinesdeployedTaskTrackerandDataNode.
Thesemachinesareinthesamelocalareanetwork.
Theexperimentuseseightsetsofvirtualmachinehardwareconfigurationandsoftwareconfigurationasshownintable1:Table1HadoopClusterConfigurationOSCentos6.
4JDKVersion1.
6.
0Hadoop1.
1.
2HardWare2GRAM100GHardDisk2.
ExperimentandanalysisBasedonMapReduceparallelimplementationofItem-basedcollaborativefilteringalgorithminparallelmodeexpansionrateperformancecomparisontest,selectthesizeofthedataset,respectively,intheefficiencyof1-8nodesrunning.
Theexperimentalresultsareshownbelow:Figure3PerformanceTestChartFigure3isbasedonMapReduceparallelimplementationofitembasedcollaborativefilteringalgorithmcantestchart,theXaxisisthenumberofclients,they-axisistheresponsetimeofthesystem.
TheexperimentalresultsshowthatbasedonMapReduceparallelimplementationofitembasedcollaborativefilteringalgorithmperformancecomparedtothetraditionalrecommendationalgorithmissignificantlyimproved.
ConclusionInthispaper,anewalgorithmofcollaborativefilteringalgorithmbasedonMapReduceisproposed.
Theexperimentresultsshowthatthenewalgorithmhashighefficiencyandcanachievehighperformanceatalowcost.
Butinthispaper,theuserclusteringiscompletedonthebasisoftheuserwithasmallnumberofattributes,forhighdimensionalattributesoftheusergroups,butalsotodofurtherresearch.
Inadditiontothenewalgorithminthispaperhasbeenputforward,wewillcontinuetoimprovetheexperimentalmethod,andconstantlyimprovetheaccuracyoftherecommendationalgorithm.
References[1]Chenruming,Challenges,valuesandcopingstrategiesintheeraofbigdata[J].
MobileCommunications.
2012(17):14-15.
[2]SunLingfang,ZhangJing.
ElectronicrecommendationmechanismbasedonRFMmodelandcollaborativefiltering[J].
JournalofJiangsuUniversityofScienceandTechnology(NaturalScienceEdition).
2010,24(3):285-289.
[3]LIGai,PANRong.
etCollaborativefilteringalgorithmparallelizeresearchbasedonlargedatasetsa[J].
ComputerEngineeringandDesign,2012,33(6):2437-2441.
[4]LIWenhai;XUShuren;DesignandimplementationofrecommendationsystemforE-commerceonHadoop[J].
ComputerEngineeringandDesign,2014(35):131-136.
[5]SUNTianhao,LIAnnenget.
ResearchonDistributedCollaborativeFilteringRecommendationAlgorithmBasedonHadoop[J].
ComputerEngineeringandApplications,2014,51(15):124:128[6]XieXuelian,LiLanyou.
ResearchonParallelK-meansAlgorithmBasedonCloundComputingPlatform[J].
ComputerMeasurement&Control,2014,22(5):1510-1512.
[7]YanCun,JiGenlin.
DesignandImplementationofItem-BasedParallelCollaborativeFilteringAlgorithm[J].
JOURNALOFNANJINGNORMALUNIVERSITY(NaturalScienceEdition),2014,37(1):71-75.
[8]WAGNFei,QinXiaolin.
Algorithmfork-meansBasedonDataStreaminCloudComputing[J].
ComputerScience,2015,42(11):235:239.
racknerd怎么样?racknerd最近发布了一些便宜美国服务器促销,包括大硬盘服务器,提供120G SSD+192TB HDD,有AMD和Intel两个选择,默认32G内存,1Gbps带宽,每个月100TB流量,5个IP地址,月付$599。价格非常便宜,需要存储服务器的朋友可以关注一下。RackNerd主要经营美国圣何塞、洛杉矶、达拉斯、芝加哥、亚特兰大、新泽西机房基于KVM虚拟化的VPS、...
日本vps云服务器怎么选择?很多人都会遇到日本vps和日本云服务器怎么选择的问题,日本云服务器具有免备案的特点。小编今天就分析一下日本云服务器价格多少钱,以方便大家选购的时候有个更加合适的取舍。日本云服务器租用前比较选择,高性能、安全、高效、免备案日本云服务器是很关键的因素。那么,日本云服务器该怎么选择呢?日本作为我们的邻国,与其贸易、文化往来是比较多的。日本云服务器价格多少钱一年?一、日本·CN...
柚子互联官网商家介绍柚子互联(www.19vps.cn)本次给大家带来了盛夏促销活动,本次推出的活动是湖北十堰高防产品,这次老板也人狠话不多丢了一个6.5折优惠券而且还是续费同价,稳撸。喜欢的朋友可以看看下面的活动详情介绍,自从站长这么久以来柚子互联从19年开始算是老商家了。六五折优惠码:6kfUGl07活动截止时间:2021年9月30日客服QQ:207781983本次仅推荐部分套餐,更多套餐可进...
centos6.0为你推荐
.cn域名cn是什么域名?急救知识纳入考试在中国急救员可以纳入医护人员吗?百度关键词价格查询如何查到推广关键词的价钱?www.qq530.com谁能给我一个听歌的网站?www.haole012.com阜阳有什么好的正规的招聘网站?www.kanav001.com翻译为日文: 主人,请你收养我一天吧. 带上罗马音标会更好wwwwww.baitu.com我看电影网www.5ken.com为什么百度就不上关键字呢www.5any.comwww.qbo5.com 这个网站要安装播放器抓站工具仿站必备软件有哪些工具?最好好用的仿站工具是那个几个?bbs2.99nets.com这个"风情东南亚"网站有78kg.cn做网址又用bbs.风情东南亚.cn那么多此一举啊!
长沙域名注册 万网域名证书查询 免费动态域名 tier iis安装教程 百兆独享 服务器维护方案 服务器合租 申请免费空间和域名 华为云盘 登陆空间 监控服务器 smtp服务器地址 主机返佣 宿迁服务器 腾讯服务器 web服务器 傲盾代理 以下 赵荣 更多