Lengthsoftmod

AlltheImagesofanOutdoorSceneSrinivasaG.
Narasimhan,ChiWang,andShreeK.
NayarColumbiaUniversity,NewYorkNY10027,USA500West120thstreet,Rm450ComputerScience{srinivas,chi,nayar}@cs.
columbia.
eduAbstract.
Theappearanceofanoutdoorscenedependsonavarietyoffactorssuchasviewinggeometry,scenestructureandreectance(BRDForBTF),illumination(sun,moon,stars,streetlamps),atmosphericcon-dition(clearair,fog,rain)andweathering(oraging)ofmaterials.
Overtime,thesefactorschange,alteringthewayasceneappears.
Alargesetofimagesisrequiredtostudytheentirevariabilityinsceneappearance.
Inthispaper,wepresentadatabaseofhighqualityregisteredandcal-ibratedimagesofaxedoutdoorscenecapturedeveryhourforover5months.
Thedatasetcoversawiderangeofdaylightandnightillumi-nationconditions,weatherconditionsandseasons.
Wedescribeindetailtheimageacquisitionandsensorcalibrationprocedures.
Theimagesaretaggedwithavarietyofgroundtruthdatasuchasweatherandillumi-nationconditionsandactualscenedepths.
Thisdatabasehaspotentialimplicationsforvision,graphics,imageprocessingandatmosphericsci-encesandcanbeatestbedformanyalgorithms.
Wedescribeanexampleapplication-imageanalysisinbadweather-andshowhowthismethodcanbeevaluatedusingtheimagesinthedatabase.
Thedatabaseisavail-ableonlineathttp://www.
cs.
columbia.
edu/CAVE/.
Thedatacollectionisongoingandweplantoacquireimagesforoneyear.
1VariabilityinSceneAppearanceTheappearanceofaxedscenedependsonseveralfactors-theviewingge-ometry,illuminationgeometryandspectrum,scenestructureandreectance(BRDForBTF)andthemedium(say,atmosphere)inwhichthesceneisim-mersed.
Theestimationofoneormoreoftheseappearanceparametersfromoneormoreimagesofthescenehasbeenanimportantpartofresearchincomputervision.
Severalresearchershavefocusedonsolvingthisinverseproblemunderspecicconditionsofillumination(constantorsmoothlyvarying),scenestruc-ture(nodiscontinuities),BRDF(lambertian)andtransparentmedia(pureair).
Imagescapturedtoevaluatetheirmethodsadheretothespecicconditions.
Whileunderstandingeachofthesespeciccasesisimportant,modelingsceneappearanceinthemostgeneralsettingisultimatelythegoalofavisionsystem.
Tomodel,developandevaluatesuchageneralvisionsystem,itiscriticaltocollectacomprehensivesetofimagesthatdescribesthecompletevariabilityintheappearanceofascene.
Severalresearchgroupshavecollectedimagesofascene(forexample,faces,textures,objects)undervaryinglightingconditionsA.
Heydenetal.
(Eds.
):ECCV2002,LNCS2352,pp.
148–162,2002.
cSpringer-VerlagBerlinHeidelberg2002AlltheImagesofanOutdoorScene149and/orviewpointsincontrolledlabenvironments.
TheCMU-PIEdatabase[1]has40000facialimagesunderdierentposes,illuminationdirectionsandfa-cialexpressions.
TheFERET[2]databaseconsistsof1196imagesoffaceswithvaryingfacialexpressions.
Similarly,theYaleFacedatabase[3]hasaround165imagestakenunderdierentlighting,poseandocclusioncongurations.
TheSLAMdatabase[4]providesasetof1500imagesoftoyobjectsunderdierentposes.
ThecolorconstancydatasetcollectedbyFuntetal.
[5]providesalargesetofimagesofobjects(boxes,booksandsoon)acquiredunderdierentposesandwithdierentilluminants(uorescent,halogen,etc).
TheCURETdatabase[6]providesasetof12000imagesofrealworldtexturesunder200illuminationandviewingcongurations.
Italsoprovidesanadditionalsetof14000Bi-directionalTextureFunction(BTF)measurementsof61realworldsurfaces.
Severaldatabasesofimagesofoutdoorsceneshavealsobeencollected.
The"naturalstimulicollection"[7]hasaround4000imagesofnaturalscenestakenonclear,foggyandhazydays.
Parragaetal.
[8]provideahyperspectraldatasetof29naturalscenes.
TheMITcityscanningproject[9]providesasetof10000geo-referencedcalibratedimagesacquiredoverawideareaoftheMITcampus.
Thesedatabases,however,donotcoverthecompleteappearancevariability(duetoalloutdoorilluminationandweatherconditions)inanyoneparticularscene.
Finally,web-cams[10]usedforsurveillancecaptureimagesregularlyoverlongperiodsoftime.
However,theyareusuallylowquality,non-calibrated,nottaggedwithgroundtruthdataandfocusonlyonactivityinthescene.
Notethatthereferenceswehaveprovidedforvariousdatabasesarebynomeanscomplete.
Wereferthereaderto[11]foramorecomprehensivelisting.
Inthispaper,wepresentasetofveryhighqualityregisteredimagesofanoutdoorscene,capturedregularlyforaperiodof5months.
Theviewpoint(orsensor)andthescenearexedovertime.
Suchadatasetisacomprehensivecollectionofimagesunderawidevarietyofseasons,weatherandilluminationconditions.
Thisdatabaseservesadualpurpose;itprovidesanextensivetestbedfortheevaluationofexistingappearancemodels,andatthesametimecanprovideinsightneededtodevelopnewappearancemodels.
Toourknowledge,thisisthersteorttocollectsuchdatainaprincipledmanner,foranextendedtimeperiod.
Thedatacollectionisongoingandweplantoacquireimagesforoneyear.
Webeginbydescribingtheimageacquisitionmethod,thesensorcalibrationprocedures,andthegroundtruthdatacollectedwitheachimage.
Next,weillustratethevariousfactorsthateectsceneappearanceusingimagesfromourdatabasecapturedover5months.
Wedemonstratethoroughevaluationofanexistingmodelforoutdoorweatheranalysis,usingtheimagedatabase.
2DataAcquisition2.
1SceneandSensorThesceneweimageisanurbanscenewithbuildings,treesandsky.
Thedistancesofthesebuildingsrangefromabout20meterstoabout5kilometers.
Thelarge150S.
G.
Narasimhan,C.
Wang,andS.
K.
NayarAnti-reflectionGlassKodakDCS31510-bitCamera24-70mmZoomLensPan/TiltStageWeatherProofEnclosureFireWire(IEEE1394)Fig.
1.
Acquisitionsetupdistancerangefacilitatestheobservationofweathereectsonsceneappearance.
Seegure5fortheentireeldofview.
ThedigitalcameraweuseforimagecaptureisasingleCCDKODAKProfessionalDCS315(seegure1).
Asusual,irradianceismeasuredusing3broadbandR,G,andBcolorlters.
AnAFNikkor24mm-70mmzoomlensisattachedtothecamera.
2.
2AcquisitionSetupThesetupforacquiringimagesisshowningure1.
Thecameraisrigidlymountedoverapan-tiltheadwhichisxedrigidlytoaweather-proofbox(seeblackboxingure1).
Theweather-proofboxiscoatedontheinsidewithtwocoatsofblackpainttopreventinter-reectionswithinthebox.
Ananti-reectionglassplateisattachedtothefrontofthisboxthroughwhichthecameraviewsthescene.
Betweenthecameraandtheanti-reectionplate,isalterholder(for,say,narrowbandspectrallters).
Theentireboxwiththecameraandtheanti-reectionglassplateismountedonapanelrigidlyattachedtoawindow.
2.
3ImageQualityandQuantityImagesarecapturedautomaticallyeveryhourfor20hourseachday(onanaverage).
Thespatialresolutionofeachimageis1520*1008pixelsandtheintensityresolutionis10bitsperpixelpercolorchannel.
Currently,wehaveacquiredimagesforover150days.
Intotal,thedatabasehasaround3000images.
Duetomaintainanceissuesthatarisefromprolongedcamerausage(camerapowerfailuresandmechanicalproblemsincontrollingcamerashutter),wehavehadtoremovethecameratwicefromtheenclosure.
Webelievetheresultinglossoffewdaysinthedatabasecanbetoleratedsincethedatasethasenormousredundancy.
Thenewimagesetsareregisteredwithexistingonesusingthematlabimageregistrationutility.
Tocapturebothsubtleandlargechangesinilluminationandweather,highdynamicrangeimagesarerequired.
So,weacquireimageswithmultipleexpo-sures(bychangingthecamerashutterspeedwhilekeepingtheaperturecon-stant)andapplypriortechniquestocomputeahighdynamicrangeimage(≈12bitsperpixel)ofthescene.
Sincetheilluminationintensityisexpectedtovarywithtime,thesetofexposuresarechosenadaptively.
First,anauto-exposureimageistakenanditsshutterspeedisnoted.
Then4moreimagesarecapturedwithexposuresaroundthisauto-exposurevalue.
Thistypeofadaptiveexposureselectioniscommonlyusedbyphotographersandiscalledexposurebracketing.
AlltheImagesofanOutdoorScene151FocalLengthReprojectionErr2845mm7.
2,9.
5pixels0.
070.
18pixelsu,v00RadialDist.
(C)1Fig.
2.
Geometriccalibrationusingplanarcheckerboardpatterns.
Tableshowsesti-matedintrinsicparameters.
Thedistortionparametersnotshownaresettozero.
3SensorCalibration3.
1GeometricCalibrationGeometriccalibrationconstitutestheestimationofthegeometricmappingbe-tween3Dscenepointsandtheirimageprojections.
Sincethecalibrationwasdoneinalocationdierentfromthatusedforimageacquisition,weestimateonlytheintrinsicparametersofthecamera.
Intrinsicparametersincludetheeectivefocallength,f,skew,s,centerofprojection(u0,v0)anddistortionpa-rameters,C1.
.
.
Cn(radial)andP1,P2(tangential).
Then,therelationbetweenobservedimagecoordinatesandthe3Dscenecoordinatesofascenepointis:ui=Dus(xifzi+δu(r)i+δu(t)i)+u0vi=Dv(yifzi+δv(r)i+δv(t)i)+v0,(1)where(δu(r)i,δv(r)i)arefunctionsoftheradialdistortionparameters,and(δu(t)i,δv(t)i)arefunctionsofthetangentialdistortionparameters.
Du,Dvarethecon-versionfactorsfrompixelstomillimeters.
See[12]formoredetails.
Wecapturedtheimagesofaplanarcheckerboardpatternundervariousorientations(seegure2).
Thecorrespondingcornersofthecheckerboardpatternsintheseimagesweremarked.
Thesecorrespondingcornerpointswereinputtoacalibrationroutine[13]toobtaintheintrinsicparameters.
Figure2showstheestimatedintrinsicparameters.
TheCCDpixelsaresquareandhenceskewisassumedtobe1.
Thedeviationoftheprincipalpointfromtheimagecenterisgivenbyu0,v0.
Onlytherstradialdistortionparameter,C1,isshown.
Theremainingdistortionparametersaresettozero.
3.
2RadiometricCalibrationAnalysisofimageirradianceusingmeasuredpixelbrightnessrequirestheradio-metricresponseofthesensor.
Theradiometricresponseofasensoristhemap-ping,g,fromimageirradiance,I,tomeasuredpixelbrightness,M:M=g(I).
Then,theprocessofobtainingIfromM:I=g1(M),uptoaglobalscalefactor,istermedasradiometriccalibration.
152S.
G.
Narasimhan,C.
Wang,andS.
K.
Nayar(a)HighExposure(b)AutoExposure(c)LowExposure(d)RadiometricResponse(e)ComputedHighDynamicRangeImage(HistogramEqualized)RGBIM010.
5010.
5ImageIrradianceMeasuredIntensity1231231Fig.
3.
RadiometricSelf-Calibration.
(a)-(c)Threeimages(10bitsperpixelperRGBchannel)capturedwithdierentcameraexposures.
(d)Computedradiometricresponsefunctionsofthe3RGBchannels.
Theresponsefunctionsarelinearwithslopes1.
5923,1.
005and0.
982forR,G,andBrespectively.
Thecolorscanbebalancedbynormalizingtheslopeofeachresponsefunction.
(e)Histogramequalizedhighdynamicrangeimageirradiancemapobtainedbycombiningimagestakenwithmultipleexposures.
Insetsindicatethatthedynamicrangeinthisimageismuchhigherthanthedynamicrangeinanyimagecapturedusingasingleexposure.
TheresponsefunctionsofCCDcameras(withoutconsideringthegammaorcolorcorrectionsappliedtotheCCDreadouts)areclosetolinear.
Wecomputedtheresponsefunctionsofthe3RGBchannelsseparatelyusingMitsunagaandNayar's[14]radiometricself-calibrationmethod.
Inthismethod,imagescap-turedwithmultipleexposuresandthetheirrelativeexposurevaluesareusedtoestimatetheinverseresponsefunctioninpolynomialform.
Theresultsofthecalibrationareshownintheplotsofgure3(d).
Noticethattheresponsefunc-tionsofR,G,andBarelinearandtheyhavedierentslopes-1.
5923,1.
005and0.
982respectively.
Tobalancethecolors,wenormalizetheresponsefunc-tionsbytherespectiveslopes.
Theimagestakenwithdierentexposuresarelinearizedusingthecomputedresponsefunction.
Ahighdynamicrangeimageirradiancemap(seegure3)isobtainedbyusingaweightedcombinationofthelinearizedimages.
Thisimagehassignicantlymoredynamicrangethenanyoftheoriginalimagestakenwithsingleexposures[15].
Thehighdynamicrangecanproveveryusefulwhenanalyzingbothsubtleandlargechangesinweatherandillumination.
4GroundTruthDataAnydatabaseisincompletewithouttheaccompanyinggroundtruth.
Wehavetaggedourimageswithavarietyofgroundtruthinformation.
Mostimportantcategoriesofthegroundtruthwecollectedarescenedepthandweatherinfor-mation.
ThedepthsofscenepointsaremainlyobtainedusingsatellitedigitalAlltheImagesofanOutdoorScene153Conditionsat2001.
03.
0611:51amWindNNW(340)10MPHVisibility11/4mile(s)SkyconditionsOvercastWeatherLightsnow,MistPrecipitationlasthourAtraceTemperature32.
0F(0.
0C)DewPoint32.
0F(0.
0C)RelativeHumidity100%OFOVFig.
4.
Samplegroundtruthdata.
[Left]Asatellitedigitalorthophotoofaportionofthescene.
Theredspotindicatesthepositionofthesensorandbrightregionindicatestheeldofview.
Arcview[17]isusedtomeasuretheorthographicdistancebetweenanytwoscenepoints(seenintopview)withanaccuracyof1meter.
[Right]TheweatherdataobtainedfromNationalWeatherServicewebsites[18].
orthophotossuppliedbytheUnitedStatesGeologicalSurvey[16].
Arcview[17]isamappingsoftwarethatisusedtomeasuretheorthographicdistancebetweentwoscenepoints(visibleintheorthophotos)uptoanaccuracyof1meter.
Seegure4foranexampleofasatelliteorthophoto.
Notethataccuratedepthisnotavailableatallpixels.
However,sincetheeldofviewconsistsofmainlyverticalbuildings,roughplanarmodelscanbeused.
Theposition(longitude,latitudeandaltitude)ofthesensorisincludedinthedatabase.
Thisinformationalongwiththedateandtimeofday,canbeusedtoaccuratelycomputesunandmoonorientationrelativetothesensor.
Forexactequations,see[19,20].
EveryhourweautomaticallycollectstandardweatherinformationfromtheNationalWeatherServicewebsites[18].
Thisincludesinformationaboutskycondition(sunny,cloudy),weathercondition(clear,fog,haze,rain),visibility,temperature,pressure,humidityandwind(seegure4).
Suchinformationcanbeusedtoestimatethescatteringcoecientoftheatmosphere[21].
5WILD:WeatherandILluminationDatabaseWeillustratethevariabilityinsceneappearanceduetoweather,illumination,seasonchanges,andsurfaceweatheringusingimagesfromourdatasetcapturedovervemonths.
5.
1VariationinIlluminationThedistributionofenvironmentalilluminationonasceneproducesawideva-rietyofsceneappearances.
Commonlynoticedeectsincludeshadows,colorsofsunriseandsunset,andilluminationfromstarsandmoonatnight.
Thehumanvisualsystemreliesonilluminationinthescenetoperceivescenereectance(retinexandcolorconstancy[22])andshape[23]correctly.
Asaresult,ren-deringascenewithconsistentilluminationiscriticalforrealismingraphics.
Considerableamountofeorthasbeenputintomodelingoutdoorillumination.
154S.
G.
Narasimhan,C.
Wang,andS.
K.
NayarThebook"DaylightanditsSpectrum"[24]providesacompendiumofcolorandintensitydistributionsofskylightformanyyearsofthe20thcentury.
Daylightspectrumhasalsobeenrepresentedusingasetoflinearbases[25].
Worksthatmodelcloudsandtheireectontheambientilluminationalsoexistinliterature[26,27].
Ingraphics,sceneshavebeenrenderedunderdierentdaylight[28]andnightilluminations[29].
Shadowsareapowerfulcueforshapeandilluminationperception.
Renderingshadowsandextractingshapeinformationfromshadows[30,31]arealsoimportantproblems.
Letusconsiderthevarioussourcesofilluminationinanyoutdoorscene.
Theprimarysources(self-luminous)includethesunduringtheday,thestarsandlampsduringnight.
Therearenumerousothersecondaryilluminationsourcessuchasskylight,groundlight,moonlight,airlight[32](duetoscatteringoflightbytheatmosphere),andscenepointsthemselves(inter-reections[33]).
Ourgoalistoincludetheeectsofallthesesourcesinonecomprehensivedatabase.
Figure5shows6imagesfromourdatabaseillustratingthevariousshadowcongura-tionsonasunnyday.
Figure6showsdierentilluminationcolorsandintensitiesatsunrise,noon,sunsetandnight.
Figure7depictsthevariationsinambientlightingduetovaryingcloudcovers.
Whenviewedup-close,roughsurfacesappeartohave3Dtextures(duetosurfaceheightvariations)ratherthan2Dtextures.
Theappearanceof3Dtex-tureshasbeenmodeledin[6].
Figure8showstheappearanceofarooftopwithridgesatdierenttimesoftheday.
Noticethechangeinappearanceofcastshadowsduetosurfaceheightvariations.
Theabovevariabilityinasingledatabasefacilitatesresearchgroupstostudytheilluminationeectsindividuallyaswellassimultaneously.
Forinstance,onemayusejustsunnydayimagesatoneparticulartimeofday,whenthesunpositionremainsconstantandshadowsdonotchange.
Inanotherinstance,onecanconsiderimagescapturedonlyoncloudydaystomodelscenesunderambientlighting.
5.
2VariationinWeatherConditionsMostvisionalgorithmsassumethatlighttravelsfromascenepointtoanob-serverunaltered.
Thisistrueonlyoncleardays.
Inbadweather,however,theradiancefromascenepointisseverelyscatteredandthus,theappearanceofascenechangesdramatically.
Theexactnatureofscatteringdependsonavarietyoffactorssuchasshapes,orientationsanddensitiesofatmosphericparticlesandthecolors,polarizations,andintensitiesofincidentlight[34].
Recently,therehasbeenworkoncomputingscenedepthsfrombadweatherimages[35]aswellastoremoveweathereectsfromimages.
Someworksextractstructureandcleardayscenecolorsusingimagesofascenetakenunderdierentweatherconditions[36]orthroughdierentpolarizerorientations[37].
Otherworksusepre-computedormeasureddistancestorestorescenecontrast[38,39].
Ingraphics,scenes(in-cludingskies)havebeenrenderedconsideringmultiplescatteringoflightintheatmosphere[28,40].
AlltheImagesofanOutdoorScene15509/07/2001,3PMClearandSunny09/07/2001,10AMClearandSunny09/07/2001,12NoonClearandSunny09/07/2001,11AMClearandSunny09/07/2001,1PMClearandSunny09/07/2001,2PMClearandSunnyFig.
5.
Imagesillustratingdierentshadowcongurationsonaclearandsunnyday.
Shadowsprovidecuesforilluminationdirectionandthescenestructure.
Noticethepositionsofthesharpshadowsonthebuildings.
09/05/2001,6AMSunRise09/05/2001,10PMNight09/05/2001,7PMSunSet09/05/2001,12noonNoonFig.
6.
Imagesillustratingthevariouscolorsandintensitiesofilluminationatsunrise,noon,sunsetandnight.
Noticethesignicantchangeinthecolorsofthesky.
7AM,PartlyCloudy,PartlySunny7AM,IncreasedCloudCover7AM,OvercastSkyFig.
7.
Imagesshowingvariouslevelsofcloudcover.
Theimageontheleftshowstheappearanceofthescenewithafewscatteredclouds.
Thetwoimagesontherightweretakenundermostlycloudyandcompletelyovercastconditions.
Noticethesoftshadowsduetopredominantambientlighting.
156S.
G.
Narasimhan,C.
Wang,andS.
K.
Nayar9AM12Noon3PMFig.
8.
Whenviewedatcloseproximity(nescale),theappearancesofsurfacesshouldbemodeledusingthebi-directionaltexturefunction(BTF)insteadoftheBRDF.
Noticethechangeincastshadowsduetotheridgesontherooftop.
Allimagesarehistogramequalizedtoaidvisualization.
06/08/2001,1PMClearandSunny09/14/2001,1PMFoggy06/14/2001,1PMHazy06/02/2001,1PMLightMistandRainFig.
9.
Imagestakenatthesametimeofdaybutunderdierentweatherconditions.
Noticethedegradationinvisibility,especially,offarawayscenepointsinbadweather.
05/28/01,11PMClearNight05/26/01,11PMMistyNightFig.
10.
Nightimagesshowinglightsourcesunderclearandmistyconditions.
Thesourcesappearlikespecksoflightonaclearnight.
Noticethelightspreadingduetomultiplescatteringsonamistynight.
HowcantheabovemodelsforweatheranalysisbeevaluatedUnderwhatconditionsdothesemodelsfailorperformwellTosatisfactorilyanswersuchquestions,weneedtoevaluatetheperformanceofsuchmodelsandalgorithmsonimagesofascenecapturedunderawidevarietyofilluminationandweatherAlltheImagesofanOutdoorScene157conditions.
Ourdatabaseincludesimagesofthesceneundermanyatmosphericconditionsincludingclearandsunny,fog,haze,mist,rainandsnow.
Figure9shows4imagesofthesamescenecapturedunderdierentweatherconditions.
Noticethesignicantreductionincontrast(andincreaseinblurring)infarawaybuildings.
Broadeningoflightbeamsduetomultiplescatteringsintheatmosphereisclearlyillustratedbythelampsimagedatnight(seegure10).
Considertheeventofmildfogsettinginbeforesunrise,becomingdenseastimeprogressesandnallyclearingbynoon.
Webelievethatsuchlengthy,timevaryingprocessescanbestudiedbetterusingourdatabase.
Studyofsuchprocesseshavepotentialimplicationsforimagebasedrendering.
5.
3ExampleEvaluation:WeatherAnalysisConsiderimagestakenunderdierentweatherconditions.
Theobservedcolor,E,ofascenepointinbadweatherislinearlyrelatedtoitscleardaycolordirection,D,andthecoloroftheweathercondition(say,fogorhaze),A,bythedichromaticmodelofatmosphericscattering[35]:E=mD+nA.
(2)So,E,A,andDlieonthesame"dichromaticplane"(seegure11).
Here,m=E∞ρeβd,n=E∞(1eβd),disthedepthofthescenepointfromtheobserver,βiscalledthescatteringcoecientoftheatmosphere[41],E∞isthebrightnessatthehorizon,andρisafunctionofthescenepointBRDF[36].
UnderwhatweatherconditionsisthedichromaticmodelvalidHowwelldoesthedichromaticmodeldescribethecolorsofscenepointsunderaparticularweathercondition(say,mist)Figure11showstheresultsofevaluatingthemodelforfog,haze,mistandrainusingmultiple(5inthiscase)imagestakenundereachweathercondition.
Basedonthedichromaticmodel,NarasimhanandNayar[36]developedcon-straintsonchangesinscenecolorsunderdierentatmosphericconditions.
Usingtheseconstraints,theydevelopedalgorithmstocompute3Dstructureandcleardaycolorsofascenefromtwoormoreimagestakenunderdierentbutunknownweatherconditions.
HowdoweevaluatetheperformanceofsuchanalgorithmFigure12showsthecomparisonofthedefoggedimagewiththeactualcleardayimageofthesceneundersimilarilluminationspectra(overcastskies).
Theaccuracyofthecomputedscaleddepth,βd,iscomparedagainstthegroundtruthrelativedepthvaluesobtainedfromthesatelliteorthophotos.
Thisdemon-stratesthatmodelsandalgorithmspertainingtoweathercanbeevaluatedmorethoroughlyusingimagesfromthisdatabase.
NarasimhanandNayar'salgorithmdescribedaboveshowsthatcleardayimagescanbeobtainedusingtwoimagestakenunderdierentweather(say,foggy)conditions.
Giventwofoggyimages,canwegeneratenovelfoggyimagesWeshowthisispossibleusingthedefoggedcolorρD,fogcolorA,andopticaldepthβd,ateachpixel.
Notethatthesequantitiescanbecomputedusingtheabovealgorithm.
Scatteringcoecientβisameasureofthedensityoffog.
Thus,thedensityoffogcaneitherbeincreasedordecreasedbyappropriatelyscaling158S.
G.
Narasimhan,C.
Wang,andS.
K.
NayarD∧A∧E1E2ODichromaticPlaneRGBOD∧A∧EmnWeatherSkyAvgErrErr<3FogMistRainMildHazeDenseHazeOvercastOvercastOvercastOvercastSunny0.
581.
251.
132.
273.
61ooooo9588917644o%%%%%(a)(b)(c)Fig.
11.
Dichromaticplanegeometryanditsevaluation.
(a)DichromaticModel[35].
(b)TheobservedcolorvectorsEiofascenepointunderdierent(twointhiscase)weatherconditions(say,mildfoganddensefog)lieonaplanecalledthedichromaticplane.
(c)Experimentalvericationofthedichromaticmodelwiththesceneimaged5timesundereachofthedierentfoggy,misty,rainyandhazyconditions.
Thethirdcolumnisthemeanangulardeviation(indegrees)oftheobservedscenecolorvectorsfromtheestimateddichromaticplanes,over1.
5megapixelsintheimages.
Thefourthcolumnprovidesthepercentageofpixelswhosecolorvectorswerewithin3degreesoftheestimateddichromaticplane.
Notethatthedichromaticmodelworkswellforfog,mist,rainanddensehazeunderovercastskies.
Formildhazeconditionsundersunnyskies,themodeldoesnotperformwell.
Suchevaluationispossibleonlysinceourdatabasehasseveralimagesundereachweathercondition.
theopticaldepthβd.
Substitutingthescaledopticaldepthkβd,cleardaycolorρD,andfogcolorA,intoequation2,wecomputethecolorsofscenepointsundernovelfogconditions.
ThehorizonbrightnessE∞iskeptconstantsinceitisjustaglobalscalefactor.
Figure13shows4novelimagesgeneratedwithincreasingfogdensities.
5.
4SeasonalVariationsThetypesofoutdoorilluminationandweatherconditionschangewithseasons.
Forinstance,theintensitydistributionofsunlightandskylightdierfromsum-mertowinter[24].
Similarly,theatmosphericconditionsthatmanifestinfallaresignicantlydierentfromthosethatoccurinwinter.
Modelsoftheatmosphereindierentseasonscanbefoundin[21]andotherrelatedpapers[42].
Sinceweacquiretheimagesforoverayear,changesinsceneappearanceduetochangesinseasonscanbestudied.
Forinstance,onemighteasilycomparetheimagestakenonacleardayinspringwithimagestakenonacleardayinwinterunderidenticalsensorsettings.
Figure14shows2imagestakenatthesametimeofdayinsummerandfall.
5.
5SurfaceWeatheringOversubstantialperiodsoftime,wecommonlyseeoxidationofmaterials(rust-ing),depositionofdirtonmaterialsandmaterialsbecomingwetordry.
TheseeectsareimportantforrealisticscenerenderingandhavebeenmodeledbyDorseyandHanrahan[43].
Sinceourimageshavehighspatialresolution,por-tionsoftheimagecorrespondingtosmallregionsinthescene(say,aportionofAlltheImagesofanOutdoorScene159dddddd1145325PM,Fog6PM,FogComputedDepthMap(Brightened)RelativeDepthVerificationusingSatelliteOrthophotoDataComputedDefoggedImageActualcleardayimage(undermostlycloudyskies)d3/d1d2/d1d4/d1d5/d19.
5510.
4576.
987.
726.
416.
498.
829.
30RelativeDepthGroundTruthComputedValueFig.
12.
Computingstructureanddefoggingfromtwofoggyimages.
Tablecomparingthecomputedrelativedepthswithgroundtruthrelativedepths(obtainedusingsatel-liteorthophotos)of5dierentregions,d1d5,inthescene.
Therelativedepthsareaveragedoversmallneighborhoods.
Thewindowregionsdonotremainconstantandthusproduceerroneousdepthvalues.
Alltheimagesarecontraststretchedfordisplaypurposes.
160S.
G.
Narasimhan,C.
Wang,andS.
K.
NayarFig.
13.
Generationofnovelimageswithincreasingfogdensities(orscatteringcoef-cients).
Therelativescatteringcoecientsusedinthiscaseareβ,2β,3βand4βrespectively.
SummerSeason,06/15/2001,11AMFallSeason,09/15/2001,11AMFig.
14.
Imagestakenatthesametimeofdaybutondaysinsummerandfall.
Boththeimagesweretakenonclearandsunnydays.
Noticethesubtledierencesincolorsandthepositionsofshadows.
Fig.
15.
Portionsofarooftopinthescenewhenitisdry,partlyandcompletelywet.
awall)canbeanalyzed.
Figure15showsasmallpatchinthescenewhenitisdryandwet.
AlltheImagesofanOutdoorScene1616SummaryThegeneralappearanceofascenedependsonavarietyoffactorssuchasillu-mination,scenereectanceandstructure,andthemediuminwhichthesceneisimmersed.
Severalresearchgroupshavecollectedandanalyzedimagesofscenesunderdierentcongurationsofilluminations(bothspectrumanddirection),andviewpoints,incontrolledlabenvironments.
However,theprocessesthatef-fectoutdoorsceneappearancesuchasclimate,weatherandilluminationareverydierentfromindoorsituations.
Ultimately,visionalgorithmsareexpectedtoworkrobustlyinoutdoorenvironments.
Thisnecessitatesaprincipledcollec-tionandstudyofimagesofanoutdoorsceneunderallilluminationandweatherconditions.
Wehavecollectedalargesetofhighqualityregisteredimagesofanoutdoorurbanscenecapturedperiodicallyforvemonths.
Wedescribedtheacquisitionprocess,calibrationprocessesandgroundtruthdatacollected.
Theutilityofthedatabasewasdemonstratedbyevaluatinganexistingmodelandalgorithmforimageanalysisinbadweather.
Thisdatasethaspotentialusedinvision,graphicsandatmosphericsciences.
Acknowledgements.
ThisworkwassupportedinpartsbyaDARPAHu-manIDContract(N00014-00-1-0916)andanNSFAward(IIS-99-87979).
TheauthorsthankE.
Rodasforbuildingtheweather-proofcameraenclosure.
References1.
Sim,T.
,Baker,S.
,Bsat,M.
:TheCMUpose,illuminationandexpression(PIE)databaseoffaces.
TechReportCMU-RI-TR-01-02(2001)2.
Phillips,P.
,Wechsler,H.
,Huang,J.
,Rauss,P.
:Theferetdatabaseandevaluationprocedureforface-recognitionalgorithms.
ImageandVisionComputing16(1998)3.
Belhumeur,P.
N.
,Hespanha,J.
P.
,Kriegman,D.
J.
:Eigenfacesvs.
Fisherfaces:Recognitionusingclassspeciclinearprojection.
PAMI19(1997)711–7204.
Murase,H.
,Nayar,S.
,Nene,S.
:Softwarelibraryforappearancematching(slam).
In:ARPA94.
(1994)I:733–7375.
Funt,B.
,Barnard,K.
,Martin,L.
:IsmachinecolourconstancygoodenoughIn:ProcECCV.
(1998)6.
Dana,K.
,Nayar,S.
,vanGinneken,B.
,Koenderink,J.
:Reectanceandtextureofreal-worldsurfaces.
In:ProcCVPR.
(1997)151–1577.
vanHateren,J.
H.
,vanderSchaaf,A.
:Independentcomponentltersofnaturalimagescomparedwithsimplecellsinprimaryvisualcortex.
InProc.
RoyalSocietyofLondonB265(1998)359–3668.
Parraga,C.
A.
,Brelsta,G.
J.
,Troscianko,T.
,Moorhead,I.
:Colorandilluminationinformationinnaturalscenes.
JOSAA15(1998)563–5699.
Teller,S.
,Antone,M.
,Bodnar,Z.
,Bosse,M.
,Coorg,S.
,Jethwa,M.
,Master,N.
:Calibratedregisteredimagesofanextendedurbanarea.
In:Proc.
CVPR.
(2001)10.
Hazecam:Alivewebcam.
(In:http://www.
hazecam.
net)11.
CMUPage:Thecomputervisionhomepage.
(In:http://www.
cs.
cmu.
edu/cil/vision.
html)12.
Heikkila,J.
,Silven,O.
:Afour-stepcameracalibrationprocedurewithimplicitimagecorrection.
In:Proc.
CVPR.
(1997)1106–1112162S.
G.
Narasimhan,C.
Wang,andS.
K.
Nayar13.
Bouguet,J.
Y.
:Cameracalibrationtoolbox.
(In:http://www.
vision.
caltech.
edu/bouguetj/calibdoc)14.
Mitsunaga,T.
,Nayar,S.
:Radiometricselfcalibration.
In:CVPR.
(1999)I:374–38015.
Nayar,S.
,Mitsunaga,T.
:Highdynamicrangeimaging:Spatiallyvaryingpixelexposures.
In:Proc.
CVPR.
(2000)I:472–47916.
USGS:U.
S.
GeologicalSurveyMappinghomepage.
(In:http://mapping.
usgs.
gov)17.
ESRI:TheESRIhomepage.
(In:http://www.
esri.
com)18.
NWS:Thenationalweatherservicehomepage.
(In:http://www.
nws.
noaa.
gov)19.
Aurora:Sunanglebasics.
(In:http://aurora.
crest.
org/basics/solar/angle/)20.
Naval-Observatory:TheAstronomicalAlamanacfortheYear2001.
USR.
G.
OGovernmentPrintingOce(2001)21.
Acharya,P.
K.
,Berk,A.
,Anderson,G.
P.
,Larsen,N.
F.
,Tsay,S.
C.
,Stamnes,K.
H.
:Modtran4:MultiplescatteringandBRDFupgradestomodtran.
SPIEProc.
Op-ticalSpectroscopicTechniquesandInstrumentationforAtmosphericandSpaceResearchIII3756(1999)22.
Land,E.
,McCann,J.
:Lightnessandretinextheory.
JOSA61(1971)23.
Ramachandran,V.
:Perceivingshapefromshading.
SciAmer259(1988)76–8324.
Henderson,S.
T.
:DaylightanditsSpectrum.
NewYork:Wiley(1977)25.
Slater,D.
,Healey,G.
:WhatisthespectraldimensionalityofilluminationfunctionsinoutdoorscenesIn:Proc.
CVPR.
(1998)105–11026.
Moon,P.
,Spencer,D.
:Illuminationfromanon-uniformsky.
IllumEngg37(1942)27.
Gordon,J.
,Church,P.
:Overcastskyluminancesanddirectionalluminousre-ectancesofobjectsandbackgroundsunderovercastskies.
App.
Optics5(1966)28.
Tadamura,K.
,Nakamae,E.
,Kaneda,K.
,Baba,M.
,Yamashita,H.
,Nishita,T.
:Modelingofskylightandrenderingofoutdoorscenes.
In:Eurographics.
(1993)29.
Jensen,H.
W.
,Durand,F.
,Stark,M.
M.
,Premoze,S.
,Dorsey,J.
,Shirley,P.
:Aphysically-basednightskymodel.
InProc.
SIGGRAPH(2001)30.
Kender,J.
,Smith,E.
:Shapefromdarkness.
In:Proc.
ICCV.
(1987)539–54631.
Kriegman,D.
,Belhumeur,P.
:Whatshadowsrevealaboutobjectstructure.
JOSA-A18(2001)1804–181332.
Koschmieder,H.
:Theoriederhorizontalensichtweite.
Beitr.
Phys.
freienAtm.
,12(1924)33–53,171–18133.
Nayar,S.
,Ikeuchi,K.
,Kanade,T.
:Shapefrominterreections.
IJCV6(1991)34.
Hulst,V.
D.
:LightScatteringbysmallParticles.
JohnWileyandSons(1957)35.
Nayar,S.
,Narasimhan,S.
:Visioninbadweather.
InProc.
ICCV(1999)36.
Narasimhan,S.
,Nayar,S.
:Chromaticframeworkforvisioninbadweather.
InProc.
CVPR(2000)37.
Schechner,Y.
,Narasimhan,S.
,Nayar,S.
:Instantdehazingofimagesusingpolar-ization.
InProc.
CVPR(2001)38.
Oakley,J.
,Satherley,B.
:Improvingimagequalityinpoorvisibilityconditionsusingaphysicalmodelfordegradation.
IEEETrans.
onImageProcessing7(1998)39.
Yitzhaky,Y.
,Dror,I.
,Kopeika,N.
:Restorationofaltmosphericallyblurredimagesaccordingtoweather-predictedatmosphericmodulationtransferfunction.
OpticalEngineering36(1998)40.
T.
Nishita,Y.
Dobashi,E.
N.
:Displayofcloudstakingintoaccountmultipleanisotropicscatteringandskylight.
SIGGRAPH(1996)41.
McCartney,E.
:OpticsoftheAtmosphere:Scatteringbymoleculesandparticles.
JohnWileyandSons(1975)42.
AFRL/VSBM:Modtran.
(In:http://www.
vsbm.
plh.
af.
mil/soft/modtran.
html)43.
Dorsey,J.
,Hanrahan,P.
:Digitalmaterialsandvirtualweathering.
ScienticAmerican282(2000)