originallymimiai.net

D-LinkNet:LinkNetwithPretrainedEncoderandDilatedConvolutionforHighResolutionSatelliteImageryRoadExtractionLichenZhou,ChuangZhang,MingWuBeijingUniversityofPostsandTelecommunications{zhoulichen,zhangchuang,wuming}@bupt.
edu.
cnAbstractRoadextractionisafundamentaltaskintheeldofre-motesensingwhichhasbeenahotresearchtopicinthepastdecade.
Inthispaper,weproposeasemanticsegmentationneuralnetwork,namedD-LinkNet,whichadoptsencoder-decoderstructure,dilatedconvolutionandpretraineden-coderforroadextractiontask.
ThenetworkisbuiltwithLinkNetarchitectureandhasdilatedconvolutionlayersinitscenterpart.
Linknetarchitectureisefcientincomputa-tionandmemory.
Dilationconvolutionisapowerfultoolthatcanenlargethereceptiveeldoffeaturepointswithoutreducingtheresolutionofthefeaturemaps.
IntheCVPRDeepGlobe2018RoadExtractionChallenge,ourbestIoUscoresonthevalidationsetandthetestsetare0.
6466and0.
6342respectively.
1.
IntroductionRoadextractionfromsatelliteimageshasbeenahotre-searchtopicinthepastdecade.
Ithasawiderangeofapplicationssuchasautomatedcrisisresponse,roadmapupdating,cityplanning,geographicinformationupdating,carnavigations,etc.
Intheeldofsatelliteimageroadex-traction,avarietyofmethodshavebeenproposedinrecentyears.
Mostofthesemethodscanbeseperatedintothreecategories:generatingpixel-levellabelingofroads[1,2],detectingskeletonsofroads[3,4]andacombinationofboth[5,6].
IntheDeepGlobeRoadExtractionChallenge[7],thetaskofroadextractionfromsatelliteimageswasformu-latedasabinaryclassicationproblem:tolabeleachpixelasroadornon-road.
Inthispaper,wehandlingtheroadextractiontaskasabinarysemanticsegmentationtasktogeneratepixel-levellabelingofroads,.
Recently,deepconvolutionalneuralnetworks(DCNN)[8,9,10,11]haveshowntheirdominanceonmanyvisualrecognitiontasks.
Intheeldofim-agesemanticsegmentation,fully-convolutionalnetwork(FCN)[12]architecture,whichcanproduceasegmentationmapforanentireinputimagethroughsingleforwardpass,isprevalent.
Mostlatestexcellentsemanticsegmentationnetworks[13,14,15,16]areimprovedversionsofFCN.
Severalpreviousworkshaveapplieddeeplearningtoroadsegmentationtask.
MnihandHinton[17]employedrestrictedBoltzmannmachinestosegmentroadfromhighresolutionaerialimages.
Saitoetal.
[18]usedaclassi-cationnetworktoassigneachpatchextractedfromthewholeimageasroad,buildingorbackground.
Zhangetal.
[1]followedtheFCNarchitectureandemployedaUnetwithresidualconnectionstosegmentroadsfromoneimagethroughsingleforwardpass.
Inthispaper,wefollowthesemethods,usingDCNNtohandleroadsegmentationtask.
Althoughhasbeenextensivelystudiedinthepastyears,roadsegmentationfromhighresolutionsatelliteimagesisstillachallengingtaskduetosomespecialfeaturesofthetask.
First,theinputimagesareofhigh-resolution,sonet-worksforthistaskshouldhavelargereceptiveeldthatcancoverthewholeimage.
Second,roadsinsatelliteimagesareoftenslender,complexandcoverasmallpartofthewholeimage.
Inthiscase,preservingthedetailedspacialinformationissignicant.
Third,roadshavenaturalcon-nectivityandlongspan.
Takingthesenaturalpropertiesofroadsinconsiderationisnecessary.
Basedonthechallengesdiscussedabove,weproposeasemanticsegmentationnet-work,namedD-LinkNet,whichcanproperlyhandlethesechallenges.
D-LinkNetusesLinknet[15]withpretrainedencoderasitsbackboneandhasadditionaldilatedconvolutionlayersinthecenterpart.
Linknetisanefcientsemanticsegmenta-tionneuralnetworkwhichtakestheadvantagesofskipcon-nections,residualblocks[10]andencoder-decoderarchi-tecture.
TheoriginalLinknetusesResNet18asitsencoder,whichisaprettylightbutoutperformingnetwork.
Linknethasshownhighprecisiononseveralbenchmarks[19,20],anditrunsprettyfast.
Dilatedconvolutionisausefulkerneltoadjustrecep-tiveeldsoffeaturepointswithoutdecreasingtheresolu-tionoffeaturemaps.
Itwaswidelyusedrecently,andit182Figure1.
D-LinkNetarchitecture.
Eachbluerectangularblockrepresentsamulti-channelfeaturesmap.
PartAistheencoderofD-LinkNet.
D-LinkNetusesResNet34asencoder.
PartCisthedecoderofD-LinkNet,itissetthesameasLinkNetdecoder.
OriginalLinkNetonlyhasPartAandPartC.
D-LinkNethasanadditionalPartBwhichcanenlargethereceptiveeldandaswellaspreservethedetailedspatialinformation.
EachconvolutionlayerisfollowedbyaReLUactivationexceptthelastconvolutionlayerwhichusesigmoidactivation.
generallyhastwotypes,cascademodelike[21]andparal-lelmodelike[16],bothmodeshaveshownstrongabilitytoincreasethesegmentationaccuracy.
Wetakeadvatagesofbothmodes,usingshortcutconnectiontocombinethesetwomodes.
Transferlearningisausefulmethodthatcandirectlyim-provenetworkpreformanceinmostsituation[22],especiallwhenthetrainingdataislimited.
Insemanticsegmantationeld,initializingencoderswithImageNet[23]pretrainedweightshasshownpromissingresults[16,24].
IntheDeepGlobeRoadExtractionChallenge,ourbestsinglemodelgotIoUscoreof0.
6412onthevalidationset.
2.
Method2.
1.
NetworkArchitectureIntheDeepGlobeRoadExtractionChallenge,theorigi-nalsizeoftheprovidedimagesandmasksis1024*1024,andtheroadsinmostimagesspanthewholeimage.
Still,roadshavesomenaturalpropertiessuchasconnectivity,complexityetal.
Consideringtheseproperties,D-LinkNetisdesignedtoreceive1024*1024imagesasinputandpre-servedetailedspacialinformation.
AsshowninFigure1,D-LinkNetcanbesplitinthreepartsA,B,C,nameden-coder,centerpartanddecoderrespectively.
D-LinkNetusesResNet34[10]pretrainedonIma-geNet[23]datasetasitsencoder.
ResNet34isoriginallydesignedforclassicationtaskonmid-resolutionimagesofsize256*256,butinthischallenge,thetaskistoseg-mentroadsfromhigh-resolutionsatelliteimagesofsize1024*1024.
Consideringthenarrowness,connectivity,complexityandlongspanofroads,itisimportanttoin-creasethereceptiveeldoffeaturepointsinthecenterpartofthenetworkaswellaskeepthedetailedinformation.
Usingpoolinglayerscouldmultiplyincreasethereceptiveeldoffeaturepoints,butmayreducetheresolutionofcen-terfeaturemapsanddropspacialinformation.
Asshownbysomestate-of-the-artdeeplearningmodels[21,25,26,16],183124832*32*51232*32*51232*32*51232*32*51232*32*51212432*32*51232*32*51232*32*51232*32*5121232*32*51232*32*51232*32*512132*32*51232*32*51232*32*51232*32*512Figure2.
ThecenterdilationpartofD-LinkNetcanbeunrolledasthisstructure.
Itcontainsdilatedconvolutionbothincascademodeandparallelmode,andthereceptiveeldofeachpathisdifferent,sothenetworkcancombinefeaturesfromdifferentscales.
Fromtoptobottom,thereceptiveeldsare31,15,7,3,1respectively.
dilatedconvolutionlayercanbedesirablealternativeofpoolinglayer.
D-LinkNetusesseveraldilatedconvolutionlayerswithskipconnectionsinthecenterpart.
Dilatedconvolutioncanbestackedincascademode.
AsshownintheFigure1of[21],ifthedilationratesofthestackeddilatedconvolutionlayersare1,2,4,8,16respec-tively,thenthereceptiveeldofeachlayerwillbe3,7,15,31,63.
Theencoderpart(RseNet34)has5downsamplinglayers,ifanimageofsize1024*1024gothroughtheen-coderpart,theoutputfeaturemapwillbeofsize32*32.
Inthiscase,D-LinkNetusesdilatedconvolutionlayerswithdilationrateof1,2,4,8inthecenterpart,sothefeaturepointsonthelastcenterlayerwillsee31*31pointsontherstcenterfeaturemap,coveringmainpartoftherstcenterfeaturemap.
Still,D-LinkNettakestheadvantageofmulti-resolutionfeatures,andthecenterpartofD-LinkNetcanbeviewedastheparallelmodeasshowninFigure2.
ThedecoderofD-LinkNetremainsthesameastheorig-inalLinkNet[15],whichiscomputationallyefcient.
Thedecoderpartusestransposedconvolution[27]layerstodoupsampling,restoringtheresolutionoffeaturemapfrom32*32to1024*1024.
2.
2.
PretrainedEncoderTransferlearningisanefcientmethodforcomputervi-sion,especiallywhenthenumberoftrainingimagesislim-ited.
UsingImageNet[23]pretrainedmodeltobetheen-coderofthenetworkisamethodwidelyusedinsemanticsegmentationeld[16,24].
IntheDeepGlobeRoadEx-tractionChallenge,wefoundthattransferlearningcanac-celerateournetworkconvergenceandmakeithavebetterperformancewithalmostnoextracost.
3.
ExperimentsIntheDeepGlobeRoadExtractionChallenge.
WeusePyTorch[28]asthedeeplearningframework.
Allmodelsaretrainedon4NVIDIAGTX1080GPUs.
3.
1.
DatasetWetestourmethodonDeepGlobeRoadExtractiondataset[7],whichconsistsof6226trainingimages,1243validationimagesand1101testimages.
Theresolutionofeachimageis1024*1024.
Thedatasetisformulatedasabinarysegmentationproblem,inwhichroadsarelabeledasforegroundandotherobjectsarelabeledasbackground.
3.
2.
ImplementationdetailsInthetrainingphase,wedidnotusecrossvalidation1.
Still,wewantedtomakefulluseoftheprovideddata,sowetrainedourmodelonallofthe6226labeledimages,andonlyusedthe1243validationimagesprovidedbytheorga-nizerforvalidation.
Thismaybeattheriskofovertingonthetrainingset,sowediddataaugmentationinanam-bitiousway,includinghorizontalip,verticalip,diagonalip,ambitiouscolorjittering,imageshifting,scaling.
Forourbestmodel,weusedBCE(binarycrossentropy)+dicecoefcientlossaslossfunctionandchoseAdam[29]asouroptimizer.
Thelearningratewasoriginallyset2e-4,andreducedby5for3timeswhileobservingthetraininglossdecreasingslowly.
Thebatchsizeduringtrainingphasewasxedas4.
Ittookabout160epochsforournetworktoconverge.
Wedidtesttimeaugmentation(TTA)inthepredictingphase,includingimagehorizontalip,imageverticalip,imagediagonalip(predictingeachimage2*2*2=8times),andthenrestoredtheoutputstothematchtheori-ginimages.
Then,weaveragedtheprobofeachprediction,using0.
5asourpredictionthresholdtogeneratebinaryout-puts.
3.
3.
ResultsDuringtheDeepGlobeRoadExtractionChallenge,wetrainedadeepUnetwith7poolinglayers,whichcancoverimagesofsize1024*1024,asourbaselinemodel,andtrainedaLinkNet34withpretrainedencoderbutwithoutdilatedconvolutioninthecenterpart.
TheperformancesofdifferentmodelareshowninTable1.
WefoundthatthepretrainedLinkNet34wasjustalittlebitbetterthantheUnettrainedfromscratch.
WeevaluatedtheIoUofmaskspredictedbyUnetandmaskspredictedbyLinkNet34,and1Ittookabout40hoursforustotrainonemodel,ifwetrainmodelswith5-foldcrossvalidation,itwilltakeus200hourstotryonearchitecture(toolongforus),sowejustdroppedcrossvalidation.
184ModelIoUonvalidationsetUnet(7poolinglayers,no-pretrain)0.
6294LinkNet34(pretrainedencoder)0.
6300EnsembleofUnetandLinkNet340.
6394D-LinkNet(pretrainedencoder)0.
6412Table1.
ResultsonvalidationsetofdifferentmodelsintheDeep-GlobeRoadExtractionChallenge.
LinkNet34withpretraineden-codergotalmostthesamescoreasUnetonthevalidationset.
D-LinkNetgethigherscorethantheEnsemblingofUnetandLinkNet34onthevalidationset.
UnetLinkNet34D-LinkNet34InputFigure3.
Exampleresultsofthreemodels.
ThersttwolinesareexamplesshowingtheroadconnectivityprobleminLinkNet34.
ThereareseveralroadinterruptionsinLinkNet34results.
ThelasttwolinesareexamplesshowingtheincorrectionpredictingofUnet.
Unetismorelikelytowronglyrecognizeroadsasback-groundorrecognizesomethingnon-roadlikeriversasroads.
D-LinkNetavoidsweaknessesinUnetandLinkNet34,andmakesbetterpredictions.
foundthatonthevalidationset,theaveragedIoUofthesetwomodelswas0.
785,whichweconsideredasaprettylowscore.
Wethoughtthesetwomodelsmightgetalmostthesamescoreindifferentways.
OurbaselineUnethadlargerreceptiveeldbuthadnopretrainedencoderandthecenterfeaturemap'sresolutionwas8*8,whichistoosmalltopreservedetailedspacialinformation.
LinkNet34hadpretrainedencoderwhichmadethenetworkhasbet-terrepresentation,butitonlyhad5downsamplinglayers,hardlycoveringthe1024*1024images.
Whilereviewingtheoutputsfromthesetwomodels,wefoundthatalthoughLinkNet34wasbetterthanUnetwhilejudginganobjecttoberoadornot,ithadroadconnectivityproblem.
Someex-amplesareshowninFigure3.
Byaddingdilatedconvolu-tionwithshortcutsinthecenterpart,D-LinkNetcanobtainlargerreceptiveeldthanLinkNetaswellaspreservede-tailedinformationatthesametime,andthusalleviatedtheroadconnectivityproblemoccurredinLinkNet34.
3.
4.
AnalysisWeusedseveralmethodsduringtheDeepGlobeRoadExtractionChallenge,andwehavedoneseveralexperi-mentstondthecontributionofeachmethod.
Themostcontributingmethodistesttimeaugmentation(TTA),itcon-tributesabout0.
029points.
UsingBCE+dicecoefcientlossisbetterthanBCE+IoUlossabout0.
005points.
Pre-trainedencodercontributesabout0.
01points.
Dilatedcon-volutioninthecenterpartcontributesabout0.
011points.
Ambitiousdataaugmentationisbetterthannormaldataaugmentationwithoutcolorjitteringandshapetransfroma-tionabout0.
01points.
4.
ConclusionInthispaper,wehaveproposedasemanticsegmenta-tionnetwork,namedD-LinkNet,forhighresolutionsatel-liteimageryroadextraction.
Byenlargingthereceptiveeldandensemblingmulti-scalefeaturesinthecenterpartwhilekeepingthedetailedinformationatthesametime,D-LinkNetcanhandleroads'propertiessuchasnarrow-ness,connectivity,complexityandlongspantosomeex-tent.
However,D-LinkNetstillhasthewrongrecognitionandroadconnectivityproblems,weplantodomorere-searchontheseproblemsinthefeature.
Inaddition,althoughtheproposedD-LinkNetarchitec-turewasoriginallydesignedfortheroadsegmentationtask,weanticipateitmayalsobeusefulinothersegmentationtasks,andweplantoinvestigatethisinourfutureresearch.
References[1]ZhengxinZhang,QingjieLiu,andYunhongWang.
Roadextractionbydeepresidualu-net.
InIEEEGeoscienceandRemoteSensingLetters.
IEEE,2018.
1[2]RashaAlshehhiandPrashanthReddyMarpu.
Hierarchicalgraph-basedsegmentationforextractingroadnetworksfromhigh-resolutionsatelliteimages.
InISPRSjournalofpho-togrammetryandremotesensing,volume126,pages245–260.
Elsevier,2017.
1[3]BoLiu,HuayiWu,YandongWang,andWenmingLiu.
Mainroadextractionfromzy-3grayscaleimagerybasedondirec-tionalmathematicalmorphologyandvgipriorknowledgeinurbanareas.
InPloSone,volume10,pagee0138071.
PublicLibraryofScience,2015.
1[4]ChinnathevarSujathaandDharmarSelvathi.
Connectedcomponent-basedtechniqueforautomaticextractionofroadcenterlineinhighresolutionsatelliteimages.
InEURASIP185JournalonImageandVideoProcessing,volume2015,page8.
Springer,2015.
1[5]FavyenBastani,SongtaoHe,SoaneAbbar,MohammadAlizadeh,HariBalakrishnan,SanjayChawla,SamMad-den,andDavidDeWitt.
Roadtracer:Automaticextrac-tionofroadnetworksfromaerialimages.
arXivpreprintarXiv:1802.
03680,2018.
1[6]GellertMattyus,WenjieLuo,andRaquelUrtasun.
Deep-roadmapper:Extractingroadtopologyfromaerialimages.
InInternationalConferenceonComputerVision,volume2,2017.
1[7]IlkeDemir,KrzysztofKoperski,DavidLindenbaum,GuanPang,JingHuang,SaikatBasu,ForestHughes,DevisTuia,andRameshRaskar.
Deepglobe2018:Achallengetoparsetheearththroughsatelliteimages.
arXivpreprintarXiv:1805.
06561,2018.
1,3[8]AlexKrizhevsky,IlyaSutskever,andGeoffreyEHinton.
Imagenetclassicationwithdeepconvolutionalneuralnet-works.
InAdvancesinneuralinformationprocessingsys-tems,pages1097–1105,2012.
1[9]KarenSimonyanandAndrewZisserman.
Verydeepconvo-lutionalnetworksforlarge-scaleimagerecognition.
arXivpreprintarXiv:1409.
1556,2014.
1[10]KaimingHe,XiangyuZhang,ShaoqingRen,andJianSun.
Deepresiduallearningforimagerecognition.
InProceed-ingsoftheIEEEconferenceoncomputervisionandpatternrecognition,pages770–778,2016.
1,2[11]ChristianSzegedy,SergeyIoffe,VincentVanhoucke,andAlexanderAAlemi.
Inception-v4,inception-resnetandtheimpactofresidualconnectionsonlearning.
InAAAI,vol-ume4,page12,2017.
1[12]JonathanLong,EvanShelhamer,andTrevorDarrell.
Fullyconvolutionalnetworksforsemanticsegmentation.
InPro-ceedingsoftheIEEEconferenceoncomputervisionandpat-ternrecognition,pages3431–3440,2015.
1[13]OlafRonneberger,PhilippFischer,andThomasBrox.
U-net:Convolutionalnetworksforbiomedicalimagesegmen-tation.
InInternationalConferenceonMedicalimagecom-putingandcomputer-assistedintervention,pages234–241.
Springer,2015.
1[14]VijayBadrinarayanan,AlexKendall,andRobertoCipolla.
Segnet:Adeepconvolutionalencoder-decoderarchitectureforimagesegmentation.
InIEEEtransactionsonpatternanalysisandmachineintelligence,volume39,pages2481–2495.
IEEE,2017.
1[15]AbhishekChaurasiaandEugenioCulurciello.
Linknet:Ex-ploitingencoderrepresentationsforefcientsemanticseg-mentation.
arXivpreprintarXiv:1707.
03718,2017.
1,3[16]Liang-ChiehChen,YukunZhu,GeorgePapandreou,Flo-rianSchroff,andHartwigAdam.
Encoder-decoderwithatrousseparableconvolutionforsemanticimagesegmenta-tion.
arXivpreprintarXiv:1802.
02611,2018.
1,2,3[17]VolodymyrMnihandGeoffreyEHinton.
Learningtodetectroadsinhigh-resolutionaerialimages.
InEuropeanConfer-enceonComputerVision,pages210–223.
Springer,2010.
1[18]ShuntaSaito,TakayoshiYamashita,andYoshimitsuAoki.
Multipleobjectextractionfromaerialimagerywithconvo-lutionalneuralnetworks.
InElectronicImaging,volume2016,pages1–9.
SocietyforImagingScienceandTechnol-ogy,2016.
1[19]MariusCordts,MohamedOmran,SebastianRamos,TimoRehfeld,MarkusEnzweiler,RodrigoBenenson,UweFranke,StefanRoth,andBerntSchiele.
Thecityscapesdatasetforsemanticurbansceneunderstanding.
InProceed-ingsoftheIEEEconferenceoncomputervisionandpatternrecognition,pages3213–3223,2016.
1[20]GabrielJBrostow,JamieShotton,JulienFauqueur,andRobertoCipolla.
Segmentationandrecognitionusingstruc-turefrommotionpointclouds.
InEuropeanconferenceoncomputervision,pages44–57.
Springer,2008.
1[21]FisherYuandVladlenKoltun.
Multi-scalecontextaggregationbydilatedconvolutions.
arXivpreprintarXiv:1511.
07122,2015.
2,3[22]MaximeOquab,LeonBottou,IvanLaptev,andJosefSivic.
Learningandtransferringmid-levelimagerepresentationsusingconvolutionalneuralnetworks.
InComputerVisionandPatternRecognition(CVPR),2014IEEEConferenceon,pages1717–1724.
IEEE,2014.
2[23]JiaDeng,WeiDong,RichardSocher,Li-JiaLi,KaiLi,andLiFei-Fei.
Imagenet:Alarge-scalehierarchicalim-agedatabase.
InComputerVisionandPatternRecognition,2009.
CVPR2009.
IEEEConferenceon,pages248–255.
IEEE,2009.
2,3[24]VladimirIglovikovandAlexeyShvets.
Ternausnet:U-netwithvgg11encoderpre-trainedonimagenetforimageseg-mentation.
arXivpreprintarXiv:1801.
05746,2018.
2,3[25]HengshuangZhao,JianpingShi,XiaojuanQi,XiaogangWang,andJiayaJia.
Pyramidsceneparsingnetwork.
InIEEEConf.
onComputerVisionandPatternRecognition(CVPR),pages2881–2890,2017.
2[26]FisherYu,VladlenKoltun,andThomasFunkhouser.
Dilatedresidualnetworks.
InComputerVisionandPatternRecogni-tion,volume1,2017.
2[27]MatthewDZeiler,GrahamWTaylor,andRobFergus.
Adaptivedeconvolutionalnetworksformidandhighlevelfeaturelearning.
InComputerVision(ICCV),2011IEEEIn-ternationalConferenceon,pages2018–2025.
IEEE,2011.
3[28]AdamPaszke,SamGross,SoumithChintala,GregoryChanan,EdwardYang,ZacharyDeVito,ZemingLin,Al-banDesmaison,LucaAntiga,andAdamLerer.
Automaticdifferentiationinpytorch.
2017.
3[29]DiederikPKingmaandJimmyBa.
Adam:Amethodforstochasticoptimization.
arXivpreprintarXiv:1412.
6980,2014.
3186