modication118123.net

U-Net:ConvolutionalNetworksforBiomedicalImageSegmentationOlafRonneberger,PhilippFischer,andThomasBroxComputerScienceDepartmentandBIOSSCentreforBiologicalSignallingStudies,UniversityofFreiburg,Germanyronneber@informatik.
uni-freiburg.
de,WWWhomepage:http://lmb.
informatik.
uni-freiburg.
de/Abstract.
Thereislargeconsentthatsuccessfultrainingofdeepnet-worksrequiresmanythousandannotatedtrainingsamples.
Inthispa-per,wepresentanetworkandtrainingstrategythatreliesonthestronguseofdataaugmentationtousetheavailableannotatedsamplesmoreeciently.
Thearchitectureconsistsofacontractingpathtocapturecontextandasymmetricexpandingpaththatenablespreciselocaliza-tion.
Weshowthatsuchanetworkcanbetrainedend-to-endfromveryfewimagesandoutperformsthepriorbestmethod(asliding-windowconvolutionalnetwork)ontheISBIchallengeforsegmentationofneu-ronalstructuresinelectronmicroscopicstacks.
Usingthesamenet-worktrainedontransmittedlightmicroscopyimages(phasecontrastandDIC)wewontheISBIcelltrackingchallenge2015inthesecate-goriesbyalargemargin.
Moreover,thenetworkisfast.
Segmentationofa512x512imagetakeslessthanasecondonarecentGPU.
Thefullimplementation(basedonCae)andthetrainednetworksareavailableathttp://lmb.
informatik.
uni-freiburg.
de/people/ronneber/u-net.
1IntroductionInthelasttwoyears,deepconvolutionalnetworkshaveoutperformedthestateoftheartinmanyvisualrecognitiontasks,e.
g.
[7,3].
Whileconvolutionalnetworkshavealreadyexistedforalongtime[8],theirsuccesswaslimitedduetothesizeoftheavailabletrainingsetsandthesizeoftheconsiderednetworks.
ThebreakthroughbyKrizhevskyetal.
[7]wasduetosupervisedtrainingofalargenetworkwith8layersandmillionsofparametersontheImageNetdatasetwith1milliontrainingimages.
Sincethen,evenlargeranddeepernetworkshavebeentrained[12].
Thetypicaluseofconvolutionalnetworksisonclassicationtasks,wheretheoutputtoanimageisasingleclasslabel.
However,inmanyvisualtasks,especiallyinbiomedicalimageprocessing,thedesiredoutputshouldincludelocalization,i.
e.
,aclasslabelissupposedtobeassignedtoeachpixel.
More-over,thousandsoftrainingimagesareusuallybeyondreachinbiomedicaltasks.
Hence,Ciresanetal.
[1]trainedanetworkinasliding-windowsetuptopredicttheclasslabelofeachpixelbyprovidingalocalregion(patch)aroundthatpixelarXiv:1505.
04597v1[cs.
CV]18May20152Fig.
1.
U-netarchitecture(examplefor32x32pixelsinthelowestresolution).
Eachblueboxcorrespondstoamulti-channelfeaturemap.
Thenumberofchannelsisdenotedontopofthebox.
Thex-y-sizeisprovidedatthelowerleftedgeofthebox.
Whiteboxesrepresentcopiedfeaturemaps.
Thearrowsdenotethedierentoperations.
asinput.
First,thisnetworkcanlocalize.
Secondly,thetrainingdataintermsofpatchesismuchlargerthanthenumberoftrainingimages.
TheresultingnetworkwontheEMsegmentationchallengeatISBI2012byalargemargin.
Obviously,thestrategyinCiresanetal.
[1]hastwodrawbacks.
First,itisquiteslowbecausethenetworkmustberunseparatelyforeachpatch,andthereisalotofredundancyduetooverlappingpatches.
Secondly,thereisatrade-obetweenlocalizationaccuracyandtheuseofcontext.
Largerpatchesrequiremoremax-poolinglayersthatreducethelocalizationaccuracy,whilesmallpatchesallowthenetworktoseeonlylittlecontext.
Morerecentapproaches[11,4]proposedaclassieroutputthattakesintoaccountthefeaturesfrommultiplelayers.
Goodlocalizationandtheuseofcontextarepossibleatthesametime.
Inthispaper,webuilduponamoreelegantarchitecture,theso-called"fullyconvolutionalnetwork"[9].
Wemodifyandextendthisarchitecturesuchthatitworkswithveryfewtrainingimagesandyieldsmoreprecisesegmentations;seeFigure1.
Themainideain[9]istosupplementausualcontractingnetworkbysuccessivelayers,wherepoolingoperatorsarereplacedbyupsamplingoperators.
Hence,theselayersincreasetheresolutionoftheoutput.
Inordertolocalize,highresolutionfeaturesfromthecontractingpatharecombinedwiththeupsampled3Fig.
2.
Overlap-tilestrategyforseamlesssegmentationofarbitrarylargeimages(heresegmentationofneuronalstructuresinEMstacks).
Predictionofthesegmentationintheyellowarea,requiresimagedatawithintheblueareaasinput.
Missinginputdataisextrapolatedbymirroringoutput.
Asuccessiveconvolutionlayercanthenlearntoassembleamorepreciseoutputbasedonthisinformation.
Oneimportantmodicationinourarchitectureisthatintheupsamplingpartwehavealsoalargenumberoffeaturechannels,whichallowthenetworktopropagatecontextinformationtohigherresolutionlayers.
Asaconsequence,theexpansivepathismoreorlesssymmetrictothecontractingpath,andyieldsau-shapedarchitecture.
Thenetworkdoesnothaveanyfullyconnectedlayersandonlyusesthevalidpartofeachconvolution,i.
e.
,thesegmentationmaponlycontainsthepixels,forwhichthefullcontextisavailableintheinputimage.
Thisstrategyallowstheseamlesssegmentationofarbitrarilylargeimagesbyanoverlap-tilestrategy(seeFigure2).
Topredictthepixelsintheborderregionoftheimage,themissingcontextisextrapolatedbymirroringtheinputimage.
Thistilingstrategyisimportanttoapplythenetworktolargeimages,sinceotherwisetheresolutionwouldbelimitedbytheGPUmemory.
Asforourtasksthereisverylittletrainingdataavailable,weuseexcessivedataaugmentationbyapplyingelasticdeformationstotheavailabletrainingim-ages.
Thisallowsthenetworktolearninvariancetosuchdeformations,withouttheneedtoseethesetransformationsintheannotatedimagecorpus.
Thisisparticularlyimportantinbiomedicalsegmentation,sincedeformationusedtobethemostcommonvariationintissueandrealisticdeformationscanbesimu-latedeciently.
ThevalueofdataaugmentationforlearninginvariancehasbeenshowninDosovitskiyetal.
[2]inthescopeofunsupervisedfeaturelearning.
Anotherchallengeinmanycellsegmentationtasksistheseparationoftouch-ingobjectsofthesameclass;seeFigure3.
Tothisend,weproposetheuseofaweightedloss,wheretheseparatingbackgroundlabelsbetweentouchingcellsobtainalargeweightinthelossfunction.
Theresultingnetworkisapplicabletovariousbiomedicalsegmentationprob-lems.
Inthispaper,weshowresultsonthesegmentationofneuronalstructuresinEMstacks(anongoingcompetitionstartedatISBI2012),whereweout-4performedthenetworkofCiresanetal.
[1].
Furthermore,weshowresultsforcellsegmentationinlightmicroscopyimagesfromtheISBIcelltrackingchal-lenge2015.
Herewewonwithalargemarginonthetwomostchallenging2Dtransmittedlightdatasets.
2NetworkArchitectureThenetworkarchitectureisillustratedinFigure1.
Itconsistsofacontractingpath(leftside)andanexpansivepath(rightside).
Thecontractingpathfollowsthetypicalarchitectureofaconvolutionalnetwork.
Itconsistsoftherepeatedapplicationoftwo3x3convolutions(unpaddedconvolutions),eachfollowedbyarectiedlinearunit(ReLU)anda2x2maxpoolingoperationwithstride2fordownsampling.
Ateachdownsamplingstepwedoublethenumberoffeaturechannels.
Everystepintheexpansivepathconsistsofanupsamplingofthefeaturemapfollowedbya2x2convolution("up-convolution")thathalvesthenumberoffeaturechannels,aconcatenationwiththecorrespondinglycroppedfeaturemapfromthecontractingpath,andtwo3x3convolutions,eachfol-lowedbyaReLU.
Thecroppingisnecessaryduetothelossofborderpixelsineveryconvolution.
Atthenallayera1x1convolutionisusedtomapeach64-componentfeaturevectortothedesirednumberofclasses.
Intotalthenetworkhas23convolutionallayers.
Toallowaseamlesstilingoftheoutputsegmentationmap(seeFigure2),itisimportanttoselecttheinputtilesizesuchthatall2x2max-poolingoperationsareappliedtoalayerwithanevenx-andy-size.
3TrainingTheinputimagesandtheircorrespondingsegmentationmapsareusedtotrainthenetworkwiththestochasticgradientdescentimplementationofCae[6].
Duetotheunpaddedconvolutions,theoutputimageissmallerthantheinputbyaconstantborderwidth.
TominimizetheoverheadandmakemaximumuseoftheGPUmemory,wefavorlargeinputtilesoveralargebatchsizeandhencereducethebatchtoasingleimage.
Accordinglyweuseahighmomentum(0.
99)suchthatalargenumberofthepreviouslyseentrainingsamplesdeterminetheupdateinthecurrentoptimizationstep.
Theenergyfunctioniscomputedbyapixel-wisesoft-maxoverthenalfeaturemapcombinedwiththecrossentropylossfunction.
Thesoft-maxisdenedaspk(x)=exp(ak(x))/Kk=1exp(ak(x))whereak(x)denotestheactivationinfeaturechannelkatthepixelpositionx∈withZ2.
Kisthenumberofclassesandpk(x)istheapproximatedmaximum-function.
I.
e.
pk(x)≈1forthekthathasthemaximumactivationak(x)andpk(x)≈0forallotherk.
Thecrossentropythenpenalizesateachpositionthedeviationofp(x)(x)from1usingE=x∈w(x)log(p(x)(x))(1)5abcdFig.
3.
HeLacellsonglassrecordedwithDIC(dierentialinterferencecontrast)mi-croscopy.
(a)rawimage.
(b)overlaywithgroundtruthsegmentation.
DierentcolorsindicatedierentinstancesoftheHeLacells.
(c)generatedsegmentationmask(white:foreground,black:background).
(d)mapwithapixel-wiselossweighttoforcethenetworktolearntheborderpixels.
where:→{1,K}isthetruelabelofeachpixelandw:→Risaweightmapthatweintroducedtogivesomepixelsmoreimportanceinthetraining.
Wepre-computetheweightmapforeachgroundtruthsegmentationtocom-pensatethedierentfrequencyofpixelsfromacertainclassinthetrainingdataset,andtoforcethenetworktolearnthesmallseparationbordersthatweintroducebetweentouchingcells(SeeFigure3candd).
Theseparationborderiscomputedusingmorphologicaloperations.
Theweightmapisthencomputedasw(x)=wc(x)+w0·exp(d1(x)+d2(x))22σ2(2)wherewc:→Ristheweightmaptobalancetheclassfrequencies,d1:→Rdenotesthedistancetotheborderofthenearestcellandd2:→Rthedistancetotheborderofthesecondnearestcell.
Inourexperimentswesetw0=10andσ≈5pixels.
Indeepnetworkswithmanyconvolutionallayersanddierentpathsthroughthenetwork,agoodinitializationoftheweightsisextremelyimportant.
Oth-erwise,partsofthenetworkmightgiveexcessiveactivations,whileotherpartsnevercontribute.
Ideallytheinitialweightsshouldbeadaptedsuchthateachfeaturemapinthenetworkhasapproximatelyunitvariance.
Foranetworkwithourarchitecture(alternatingconvolutionandReLUlayers)thiscanbeachievedbydrawingtheinitialweightsfromaGaussiandistributionwithastandarddeviationof2/N,whereNdenotesthenumberofincomingnodesofoneneu-ron[5].
E.
g.
fora3x3convolutionand64featurechannelsinthepreviouslayerN=9·64=576.
3.
1DataAugmentationDataaugmentationisessentialtoteachthenetworkthedesiredinvarianceandrobustnessproperties,whenonlyfewtrainingsamplesareavailable.
Incaseof6microscopicalimagesweprimarilyneedshiftandrotationinvarianceaswellasrobustnesstodeformationsandgrayvaluevariations.
Especiallyrandomelas-ticdeformationsofthetrainingsamplesseemtobethekeyconcepttotrainasegmentationnetworkwithveryfewannotatedimages.
Wegeneratesmoothdeformationsusingrandomdisplacementvectorsonacoarse3by3grid.
ThedisplacementsaresampledfromaGaussiandistributionwith10pixelsstandarddeviation.
Per-pixeldisplacementsarethencomputedusingbicubicinterpola-tion.
Drop-outlayersattheendofthecontractingpathperformfurtherimplicitdataaugmentation.
4ExperimentsWedemonstratetheapplicationoftheu-nettothreedierentsegmentationtasks.
Thersttaskisthesegmentationofneuronalstructuresinelectronmi-croscopicrecordings.
AnexampleofthedatasetandourobtainedsegmentationisdisplayedinFigure2.
WeprovidethefullresultasSupplementaryMaterial.
ThedatasetisprovidedbytheEMsegmentationchallenge[14]thatwasstartedatISBI2012andisstillopenfornewcontributions.
Thetrainingdataisasetof30images(512x512pixels)fromserialsectiontransmissionelectronmicroscopyoftheDrosophilarstinstarlarvaventralnervecord(VNC).
Eachimagecomeswithacorrespondingfullyannotatedgroundtruthsegmentationmapforcells(white)andmembranes(black).
Thetestsetispubliclyavailable,butitsseg-mentationmapsarekeptsecret.
Anevaluationcanbeobtainedbysendingthepredictedmembraneprobabilitymaptotheorganizers.
Theevaluationisdonebythresholdingthemapat10dierentlevelsandcomputationofthe"warpingerror",the"Randerror"andthe"pixelerror"[14].
Theu-net(averagedover7rotatedversionsoftheinputdata)achieveswith-outanyfurtherpre-orpostprocessingawarpingerrorof0.
0003529(thenewbestscore,seeTable1)andarand-errorof0.
0382.
Thisissignicantlybetterthanthesliding-windowconvolutionalnetworkresultbyCiresanetal.
[1],whosebestsubmissionhadawarpingerrorof0.
000420andaranderrorof0.
0504.
IntermsofranderrortheonlybetterperformingTable1.
RankingontheEMsegmentationchallenge[14](march6th,2015),sortedbywarpingerror.
RankGroupnameWarpingErrorRandErrorPixelError**humanvalues**0.
0000050.
00210.
00101.
u-net0.
0003530.
03820.
06112.
DIVE-SCI0.
0003550.
03050.
05843.
IDSIA[1]0.
0004200.
05040.
06134.
DIVE0.
0004300.
05450.
0582.
.
.
10.
IDSIA-SCI0.
0006530.
01890.
10277abcdFig.
4.
ResultontheISBIcelltrackingchallenge.
(a)partofaninputimageofthe"PhC-U373"dataset.
(b)Segmentationresult(cyanmask)withmanualgroundtruth(yellowborder)(c)inputimageofthe"DIC-HeLa"dataset.
(d)Segmentationresult(randomcoloredmasks)withmanualgroundtruth(yellowborder).
Table2.
Segmentationresults(IOU)ontheISBIcelltrackingchallenge2015.
NamePhC-U373DIC-HeLaIMCB-SG(2014)0.
26690.
2935KTH-SE(2014)0.
79530.
4607HOUS-US(2014)0.
5323-second-best20150.
830.
46u-net(2015)0.
92030.
7756algorithmsonthisdatasetusehighlydatasetspecicpost-processingmethods1appliedtotheprobabilitymapofCiresanetal.
[1].
Wealsoappliedtheu-nettoacellsegmentationtaskinlightmicroscopicim-ages.
ThissegmenationtaskispartoftheISBIcelltrackingchallenge2014and2015[10,13].
Therstdataset"PhC-U373"2containsGlioblastoma-astrocytomaU373cellsonapolyacrylimidesubstraterecordedbyphasecontrastmicroscopy(seeFigure4a,bandSupp.
Material).
Itcontains35partiallyannotatedtrain-ingimages.
HereweachieveanaverageIOU("intersectionoverunion")of92%,whichissignicantlybetterthanthesecondbestalgorithmwith83%(seeTa-ble2).
Theseconddataset"DIC-HeLa"3areHeLacellsonaatglassrecordedbydierentialinterferencecontrast(DIC)microscopy(seeFigure3,Figure4c,dandSupp.
Material).
Itcontains20partiallyannotatedtrainingimages.
HereweachieveanaverageIOUof77.
5%whichissignicantlybetterthanthesecondbestalgorithmwith46%.
5ConclusionTheu-netarchitectureachievesverygoodperformanceonverydierentbiomed-icalsegmentationapplications.
Thankstodataaugmentationwithelasticdefor-1Theauthorsofthisalgorithmhavesubmitted78dierentsolutionstoachievethisresult.
2DatasetprovidedbyDr.
SanjayKumar.
DepartmentofBioengineeringUniversityofCaliforniaatBerkeley.
BerkeleyCA(USA)3DatasetprovidedbyDr.
GertvanCappellenErasmusMedicalCenter.
Rotterdam.
TheNetherlands8mations,itonlyneedsveryfewannotatedimagesandhasaveryreasonabletrainingtimeofonly10hoursonaNVidiaTitanGPU(6GB).
WeprovidethefullCae[6]-basedimplementationandthetrainednetworks4.
Wearesurethattheu-netarchitecturecanbeappliedeasilytomanymoretasks.
AcknowlegementsThisstudywassupportedbytheExcellenceInitiativeoftheGermanFederalandStategovernments(EXC294)andbytheBMBF(Fkz0316185B).
References1.
Ciresan,D.
C.
,Gambardella,L.
M.
,Giusti,A.
,Schmidhuber,J.
:Deepneuralnet-workssegmentneuronalmembranesinelectronmicroscopyimages.
In:NIPS.
pp.
2852–2860(2012)2.
Dosovitskiy,A.
,Springenberg,J.
T.
,Riedmiller,M.
,Brox,T.
:Discriminativeun-supervisedfeaturelearningwithconvolutionalneuralnetworks.
In:NIPS(2014)3.
Girshick,R.
,Donahue,J.
,Darrell,T.
,Malik,J.
:Richfeaturehierarchiesforac-curateobjectdetectionandsemanticsegmentation.
In:ProceedingsoftheIEEEConferenceonComputerVisionandPatternRecognition(CVPR)(2014)4.
Hariharan,B.
,Arbelez,P.
,Girshick,R.
,Malik,J.
:Hypercolumnsforobjectseg-mentationandne-grainedlocalization(2014),arXiv:1411.
5752[cs.
CV]5.
He,K.
,Zhang,X.
,Ren,S.
,Sun,J.
:Delvingdeepintorectiers:Surpassinghuman-levelperformanceonimagenetclassication(2015),arXiv:1502.
01852[cs.
CV]6.
Jia,Y.
,Shelhamer,E.
,Donahue,J.
,Karayev,S.
,Long,J.
,Girshick,R.
,Guadar-rama,S.
,Darrell,T.
:Cae:Convolutionalarchitectureforfastfeatureembedding(2014),arXiv:1408.
5093[cs.
CV]7.
Krizhevsky,A.
,Sutskever,I.
,Hinton,G.
E.
:Imagenetclassicationwithdeepcon-volutionalneuralnetworks.
In:NIPS.
pp.
1106–1114(2012)8.
LeCun,Y.
,Boser,B.
,Denker,J.
S.
,Henderson,D.
,Howard,R.
E.
,Hubbard,W.
,Jackel,L.
D.
:Backpropagationappliedtohandwrittenzipcoderecognition.
NeuralComputation1(4),541–551(1989)9.
Long,J.
,Shelhamer,E.
,Darrell,T.
:Fullyconvolutionalnetworksforsemanticsegmentation(2014),arXiv:1411.
4038[cs.
CV]10.
Maska,M.
deSolorzano,C.
O.
:Abenchmarkforcomparisonofcelltrackingalgorithms.
Bioinformatics30,1609–1617(2014)11.
Seyedhosseini,M.
,Sajjadi,M.
,Tasdizen,T.
:Imagesegmentationwithcascadedhierarchicalmodelsandlogisticdisjunctivenormalnetworks.
In:ComputerVision(ICCV),2013IEEEInternationalConferenceon.
pp.
2168–2175(2013)12.
Simonyan,K.
,Zisserman,A.
:Verydeepconvolutionalnetworksforlarge-scaleimagerecognition(2014),arXiv:1409.
1556[cs.
CV]13.
WWW:Webpageofthecelltrackingchallenge,http://www.
codesolorzano.
com/celltrackingchallenge/Cell_Tracking_Challenge/Welcome.
html14.
WWW:Webpageoftheemsegmentationchallenge,http://brainiac2.
mit.
edu/isbi_challenge/4U-netimplementation,trainednetworksandsupplementarymaterialavailableathttp://lmb.
informatik.
uni-freiburg.
de/people/ronneber/u-net