Hive:简单查询不启用Mapreduce

job而启用Fetchtask写在前面的话,学Hive这么久了,发现目前国内还没有一本完整的介绍Hive的书籍,而且互联网上面的资料很乱,于是我决定写一些关于《Hive的那些事》序列文章,分享给大家.
我会在接下来的时间整理有关Hive的资料,如果对Hive的东西感兴趣,请关注本博客.
https://www.
iteblog.
com/archives/tag/hive-technology/如果你想查询某个表的某一列,Hive默认是会启用MapReduceJob来完成这个任务,如下:hive>SELECTid,moneyFROMmlimit10;TotalMapReducejobs=1LaunchingJob1outof1Numberofreducetasksissetto0sincethere'snoreduceoperatorCannotrunjoblocally:InputSize(=235105473)islargerthanhive.
exec.
mode.
local.
auto.
inputbytes.
max(=134217728)StartingJob=job_1384246387966_0229,TrackingURL=http://l-datalogm1.
data.
cn1:9981/proxy/application_1384246387966_0229/KillCommand=/home/q/hadoop-2.
2.
0/bin/hadoopjob-killjob_1384246387966_0229hadoopjobinformationforStage-1:numberofmappers:1;numberofreducers:02013-11-1311:35:16,167Stage-1map=0%,reduce=0%2013-11-1311:35:21,327Stage-1map=100%,reduce=0%,CumulativeCPU1.
26sec2013-11-1311:35:22,377Stage-1map=100%,reduce=0%,CumulativeCPU1.
26secMapReduceTotalcumulativeCPUtime:1seconds260msecEndedJob=job_1384246387966_0229MapReduceJobsLaunched:Job0:Map:1CumulativeCPU:1.
26secHDFSRead:8388865HDFSWrite:60SUCCESSTotalMapReduceCPUTimeSpent:1seconds260msecOK1122118512311292131613291355135613621/31364Timetaken:16.
802seconds,Fetched:10row(s)我们都知道,启用MapReduceJob是会消耗系统开销的.
对于这个问题,从Hive0.
10.
0版本开始,对于简单的不需要聚合的类似SELECTfromLIMITn语句,不需要起MapReducejob,直接通过Fetchtask获取数据,可以通过下面几种方法实现:方法一:hive>sethive.
fetch.
task.
conversion=more;hive>SELECTid,moneyFROMmlimit10;OK1122118512311292131613291355135613621364Timetaken:0.
138seconds,Fetched:10row(s)上面sethive.
fetch.
task.
conversion=more;开启了Fetch任务,所以对于上述简单的列查询不在启用MapReducejob!
方法二:bin/hive--hiveconfhive.
fetch.
task.
conversion=more方法三:上面的两种方法都可以开启了Fetch任务,但是都是临时起作用的;如果你想一直启用这个功能,可以在${HIVE_HOME}/conf/hive-site.
xml里面加入以下配置:hive.
fetch.
task.
conversionmore2/3SomeselectqueriescanbeconvertedtosingleFETCHtaskminimizinglatency.
Currentlythequeryshouldbesinglesourcednothavinganysubqueryandshouldnothaveanyaggregationsordistincts(whichincurrsRS),lateralviewsandjoins.
1.
minimal:SELECTSTAR,FILTERonpartitioncolumns,LIMITonly2.
more:SELECT,FILTER,LIMITonly(+TABLESAMPLE,virtualcolumns)这样就可以长期启用Fetch任务了,很不错吧,也赶紧去试试吧!
本博客文章除特别声明,全部都是原创!
原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载.
本文链接:【】()PoweredbyTCPDF(www.
tcpdf.
org)3/