Acrealplayersp

EURASIPJournalonAppliedSignalProcessing2004:2,192–206c2004HindawiPublishingCorporationSMART:AnEfcient,Scalable,andRobustStreamingVideoSystemFengWuMicrosoftResearchAsia,3FSigmaCenter,No.
49ZhichunRoad,Haidian,Beijing100080,ChinaEmail:fengwu@microsoft.
comHonghuiSunMicrosoftResearchAsia,3FSigmaCenter,No.
49ZhichunRoad,Haidian,Beijing100080,ChinaEmail:hongsun@microsoft.
comGuobinShenMicrosoftResearchAsia,3FSigmaCenter,No.
49ZhichunRoad,Haidian,Beijing100080,ChinaEmail:jackysh@microsoft.
comShipengLiMicrosoftResearchAsia,3FSigmaCenter,No.
49ZhichunRoad,Haidian,Beijing100080,ChinaEmail:spli@microsoft.
comYa-QinZhangMicrosoftResearchAsia,3FSigmaCenter,No.
49ZhichunRoad,Haidian,Beijing100080,ChinaEmail:yzhang@microsoft.
comBruceLinMicrosoftCorporation,OneMicrosoftWay,Redmond,WA98052-6399,USAEmail:blin@microsoft.
comMing-ChiehLeeMicrosoftCorporation,OneMicrosoftWay,Redmond,WA98052-6399,USAEmail:mingcl@microsoft.
comReceived9December2002;Revised12September2003SMART,theacronymofscalablemediaadaptationandrobusttransport,isasuiteofcompressionandtransmissiontechnologiesforecient,scalable,adaptive,androbustvideostreamingoverthebest-eortInternet.
Itconsistsoftwoindispensableparts:SMARTvideocodingandSMARTvideostreaming.
TheSMARTvideocodingpartisanecientDCT-baseduniversalnegran-ularityscalablecodingscheme.
SincetheSMARTvideocodingschemeadoptsmultiple-looppredictionanddriftingreductiontechniquesatthemacroblocklevel,itcanachievehighcodingeciencyatawiderangeofbitrates.
Moreimportantly,itprovidesallsortsofscalabilities,thatis,quality,temporal,spatial,andcomplexityscalabilities,inordertoaccommodateheterogeneoustime-variantnetworksanddierentdevices.
TheSMARTvideostreamingpartisatransportschemethatfullytakesadvantagesofthespecialfeaturesofthescalablebitstreams.
Anaccuratebandwidthestimationmethodisrstdiscussedastheprerequi-siteofnetworkadaptation.
Then,exibleerrorresiliencetechniqueandunequalerrorprotectionstrategyareinvestigatedtoenhancetherobustnessofstreamingSMARTbitstream.
TheSMARTsystemshowsexcellentperformanceswithregardtohighcodingeciency,exiblechannelbandwidthadaptation,smoothplayback,andsuperiorerrorrobustnessinstaticanddynamicexperiments.
Keywordsandphrases:videostreaming,negranularityscalability,videotransmission,bandwidthestimation,errorresilience,unequalerrorprotection.
SMART:AnEcient,Scalable,andRobustStreamingVideoSystem1931.
INTRODUCTIONWiththerecentdevelopmentsincomputingtechnology,compressionandtransmissiontechnologies,high-capacitystoragedevices,andhigh-speedwiredandwirelessnetworks,moreandmoreusersexpecttoenjoyhigh-qualitymultime-diaservicesovertheInternet[1,2,3].
Ingeneral,therearetwoapproachestoprovidemultimediaservicesondemand:oinedownloadingandonlinestreaming.
Sincethestreamingapproachenablesuserstoexperienceamultimediapresenta-tionontheywhileitisbeingdownloadedfromtheInter-net,ithasprevailedinboththeacademiaandtheindustry.
Invirtueofthestreamingtechniques,usersnolongerhavetosuerfromlongandevenunacceptabletransporttimeforfulldownload.
Figure1exempliesatypicalscenarioforstreamingthesamecontenttousers.
Rawvideosequencesareusuallycom-pressedinadvanceandthensavedinthestoragedevice.
Upontheclient'srequest,thestreamingserverretrievescom-pressedbitstreamfromthestoragedeviceanddeliversitthroughtheInternetthatconsistsofmanyheterogeneoussubnetworks.
Receiversmayusedierentdevicesfordecod-ing,presentingthereceivedvideodataatdierentresolu-tions,dierentframerates,anddierentqualitiesdependingontheirconnectionspeedsanddevicecapabilities.
Infact,suchmultimediastreamingservicescreatesev-eralchallengeswhichmaylieintechnicaleldsevenbeyondvideocompression.
Thesechallengesmainlyincludebutarenotlimitedtothefollowing.
(1)ContentsMultimediacontentsarehugeandgrowingrapidly.
Forex-ample,onlyfromRealNetworksCompany'sstatisticsin2001[4],over350000hoursoflivesports,music,news,anden-tertainmentcontentsweretransmittedovertheInternetev-eryweek.
Furthermore,thereareseveralhundredthousandhoursofcontentsavailableondemand.
Toecientlyandef-fectivelydeliversuchhugemultimediacontents,advancedcompressionandtransmissiontechnologiesarecrucial.
(2)NetworksThenetworksusedtodelivermultimediacontentsarebe-comingmoreandmorecomplicatedandheterogeneous.
Ad-ditionally,unliketraditionaldedicatednetworks,sincethegeneralbest-eortInternetlacksqualityofservice(QOS)guarantee,networkconditionsthemselvesmaybechangingfromtimetotime.
Thisrequiresthatcompressedmultime-diacontentsaredeliverableoverdierentnetworksfromnar-rowbandtobroadbandandfromwiredtowirelessnetworks.
Italsorequiresthatthedeliverymechanismisabletoadapttonetworkvariationswhileprovidingaconsistentuserex-perience.
Inaddition,sincepacketlossorchannelerrorisinevitableduringtransmission,advancederrorcontroltech-nologiesarerequiredtoprotectthetransmitteddata.
(3)DevicesEnd-userdevicesarealsobecomingverydierentinprocess-ingpower,memory,displayresolution,andbandwidth.
Thisrequirestailoringmultimediacontentsanddeliveryschemestobestteachdeviceinordertoprovidethebestpossiblemultimediauserexperience.
Astraightforwardsolutionwouldbetoindependentlycompressthesamevideosequenceintomanynonscalablebitstreamsforeverypossiblebitrate,framerate,resolution,anddevicecomplexity.
Actually,thissolutionhasbeenexten-sivelyappliedtomostofthecommercialstreamingproducts,suchasWindowsMediaPlayersystemandRealPlayersys-tem[4,5].
Whenavideosequenceisretrieved,thestreamingserverchoosesanappropriateversionofbitstreamaccordingtoactualconnectionspeedanddevicecapability,andthentransmitsittotheuser.
Obviously,videostreamingsystemsbasedonthenon-scalablecompressiontechniqueshaveseveralproblemsintakingtheabovechallenges.
Firstly,nonscalablevideobit-streamsarenotabletoadapttotime-variantnetworks.
Eventhoughswitchingamongmultiplenonscalablebitstreamsisallowedatsomekeyframesthatareeithercompressedwith-outpredictionorcodedwithanextralosslesscodedswitch-ingbitstream,suchstreamingsystemsonlyprovidecoarseandsluggishcapabilityinadaptingtobandwidthvariationsduetolimitationinboththenumberofbitstreamsandthenumberofkeyframes.
Somestudieshavetriedtosolvethisproblembyswitchingataspecialpredictiveframe,forexample,Sframein[6],SPframein[7],andSFframein[8],whichcanreduceswitchingoverheadandprovidemoreswitchingpointsatthesamecost.
Secondly,nonscal-ablevideobitstreamisverysensitivetotransmittederrorsbecausealmosteverybitinthebitstreamisveryimportantandindispensablefordecodingagroupofpictures(GOP).
Ontheotherhand,thescalablemediaadaptationandrobusttransport(SMART)systemproposedinthispaperisbasedonscalablecompressiontechniquesandisabletopro-videecient,adaptive,androbustvideostreamingovertheInternet.
Thecoreofthesystemisanecientanduniversalnegranularityscalable(FGS)videocodec.
Itusesmultipleversionsofreferenceswithincreasingqualitytomakemotionpredictionmoreaccurateforimprovedcodingeciency.
Atthesametime,adriftingreductiontechniqueisproposedtopreventpossibleerrorpropagationduetocorruptedhigh-qualityreferences.
Whenthetwotechniquesareappliedatthemacroblocklevel,theSMARTsystemcanachieveagoodtrade-obetweenlowdriftingerrorsandhighcodinge-ciency.
Besidesecientnegranularityqualityscalability,theSMARTsystemsupportsecienttemporalandspatialscalabilitiesbyutilizingsimilartechniques.
Furthermore,thenegranularityscalabilityoncomplexityisalsoachievedbyadjustingthedecodingresolution,framerate,andbitrate.
Infact,theSMARTsystemprovidesauniversalscalablecod-ingframework.
Forasequence,thegeneratedbitstreamscanbeservedtoavastrangeofapplicationsfromlowbitratetohighbitrateandfromaPCdevicetoanon-PCdevicewith-outcomplicatedtranscoding.
TheSMARTvideostreamingpartisatransportschemethatfullytakesadvantageofthespecialfeaturesofSMARTvideobitstream.
Itrstestimatestheavailablechannelbandwidththroughahybridmodel-basedandprobe-based194EURASIPJournalonAppliedSignalProcessingCISCOSYSTEMSCISCOSYSTEMSCISCOSYSTEMSCISCOSYSTEMSCISCOSYSTEMSCISCOSYSTEMSCISCOSYSTEMSCISCOSYSTEMSReceiver2CableDigitalTVAccessSWReceiver3LaptopGatewayDomainADomainBDomainCInternetAccessSWEthernetStreamingserverReceiver1SourceDesktopPCGatewayWirelessnetworksReceiver4MobilephoneFigure1:Anexempliedscenarioforstreamingvideo.
method.
Afterward,thetransmittedvideobitstreamsaretruncatedtoabitratethattswellintheestimatedchan-nelbandwidth.
Sincepacketlossesareinevitableinthegen-eralInternet,errorcontrolmechanismisakeycomponentinthispart.
AexibleerrorresiliencetechniqueisproposedtoadaptivelyenhancetherobustnessofSMARTvideobit-stream.
Inaddition,theSMARTsystemprovidesalayeredbitstreamstructurewithamoreimportantbaselayerandlessimportantenhancementlayers.
Forwarderrorcorrection(FEC)andautomaticretransmissionrequest(ARQ)tech-niquesareappliedtothebaselayersoastoreducepacketlossratioandretransmissiondelay.
Therestofthispaperisarrangedasfollows.
Section2givesabriefoverviewoftheSMARTsystem.
TheSMARTvideocodingtechniquesarediscussedinSection3.
Section4introducesthechannelestimationmethodusedintheSMARTsystem.
TheexibleerrorresiliencetechniqueandunequalerrorprotectionaredescribedinSection5.
Theex-perimentalresultspresentedinSection6demonstratetheadvantagesoftheSMARTsystem.
Finally,Section7con-cludesthispaper.
2.
OVERVIEWOFTHESMARTSYSTEMThissectiongivesanoverviewoftheSMARTcodingandstreamingsystem.
Atpresent,therearetwomodesforastreamingservertodelivervideodatatousers:multicastandunicast.
Inthemulticastmode,theserverneedstosendonlyonebitstreamtoagroupofusers,whichisautomaticallyreplicatedtoallgroupmembers[9,10],butthisrequeststhatthenetworkhastobeequippedwithmulticast-enablerouters.
Intheunicastmode,theserverdeliversvideobit-streamtoeachuserindividually.
Theconnectionconditionsbetweentheserverandeachusercanbeestimatedandmon-itoredduringtransmission.
SincemanyroutesinthecurrentInternetdonotenablethemulticastmode,theSMARTsystemdiscussedinthispa-perwillfocusontheunicastapplications.
Figure2illustratestheblockdiagramoftheSMARTsystem.
SourcevideoisrstinputintotheSMARTencodermoduletogenerateabaselayerbitstreamandoneortwoenhancementlayerbit-streams.
Besidesbitstreams,theSMARTencodergeneratesadescriptionleforeachenhancementbitstreamthatcon-tainsallinformationforexibleerrorresilienceandpacke-tization.
ThedetailedcodingtechniqueswillbediscussedinSection3,andthedescriptionleisintroducedinSection5.
IftheSMARTencoderispowerfulenoughforreal-timecom-pression,thegeneratedbitstreamscanbedirectlypackedanddeliveredjustasinthelivestreamingapplications.
Fortheon-demandstreamingapplications,boththegeneratedbit-streamsanddescriptionlesaresavedinthestoragedeviceforfutureretrieval.
WhentheusersubmitsarequesttotheSMARTstream-ingserver,likethereal-timestreamingprotocol(RTSP)[11],theretrievedcontent,destinationaddress,anduserdevicecapabilityarersttransmittedbythetransmissioncontrolprotocol(TCP).
AfterthecontrolmoduleintheSMARTserverreceivestherequest,oneuserdatagramprotocol(UDP)connectionisestablishedimmediatelybetweentheserverandtheuser.
BoththevideodataandthefeedbackfromtheSMARTclientaretransmittedbythisUDPconnec-tion.
Atthesametime,thecontrolmoduleinformstheservertoretrievetherequestedcontentfromthestoragedevice.
Intheinitialstage,theSMARTsystemdoesnotknowthecurrentchannelconditionsbetweentheserverandtheclient.
Thusthebaselayerbitstreamispackedwiththereal-timetransportprotocol(RTTP)[12]formatusingdefaultchannelparameters.
Atthesametime,aprespeciedFECstrategyisusedinthebaselayerbitstreamtogeneratepar-itypackets.
Ingeneral,sincethebaselayerbitrateisverylowintheSMARTsystem,severalsecondsofsourceandparitypacketscanberapidlydeliveredtotheclientasprebuering.
Bytransmittingthesepackets,thestatisticchannelparame-ters,suchaspacketlossratioandlatency,arepackedwiththeSMART:AnEcient,Scalable,andRobustStreamingVideoSystem195SMARTsystemSourceSMARTencoderRealtimeStreamingStorageErrorresilienceControlRequestFECNetworkmonitorFeedbackTransportDataInternetControlowDataowUserdeviceSMARTdecoderSMARTclientFigure2:TheblockdiagramoftheSMARTsystem.
real-timecontrolprotocol(RTCP)format[12]andsentbacktothenetworkmonitormoduleintheSMARTserver.
Ac-cordingly,theSMARTservercanestimatethecurrentavail-ablechannelbandwidth.
Withtheobtainedchannelparameters,theSMARTserverstartstooptimallypackthebaselayerandenhance-mentlayerbitstreamswithRTTPformat.
FECprotectiondepthtothebaselayercanbealsoadaptivetothechannelconditions.
Inordertoavoidnetworkcongestion,theac-tualbandwidthfortheenhancementlayeristheremainingpartoftheestimatedchannelbandwidthafterdeliveringthebaselayerandFECpackets.
Sincetheenhancementlayerbit-streamprovidesbitlevelscalability,itcanbereadilyandpre-ciselytruncatedtotinthegivenbandwidth.
Consequently,theSMARTsystemcanfullyutilizeavailablechannelband-widthandprovidetheuserwithbetterquality.
Packetlossratioandlatencyareperiodicallysentbackbytheclient.
TheSMARTservercantimelyadjustdatatransmissionaccordingtothefeedbacksandtheestimatedchannelbandwidth.
IntheSMARTsystem,anotherimportantfeedbackfromtheclientisthenegativeacknowledgement(NACK)tono-tifytheSMARTserverinwhichbaselayerpacketsarelostduringtransmission.
Sincethebaselayerisstillanonscal-ablebitstream,anylostpacketwouldmakethequalityoftheframesfollowedinthesameGOPdegraderapidly.
Therefore,theARQtechniqueisalsousedtoprotectthebaselayerintheSMARTsystem.
Oncetheclientdetectslostpacketsatthebaselayer,afeedbackisimmediatelysentout.
Theserverwillrapidlyretransmitthelostpackets.
Atthesametime,anyARQrequestreceivedbytheserverwillaectthesend-ingratetopreventfurthercongestioninthechannel.
SincethebaselayerbitrateisverylowintheSMARTsystem,theycanbestronglyprotectedwithsmalloverheadbits.
Inaddi-tion,SMARTvideocodingalsoprovidestheenhancementlayerwithaninherenterrorrecoveryfeature.
Anylostpacketdoesnotcauseobviousvisualartifacts.
Moreover,itcanbegracefullyrecoveredinthefollowingframes.
Therefore,thecurrentSMARTsystemdoesnothaveanyprotectiontotheenhancementlayerbitstreams.
Inthefollowingsections,thekeytechniquesusedintheSMARTsystem,suchasSMARTvideocoding,bandwidthes-timation,errorresilience,andunequalerrorprotection,willbediscussedindetail.
3.
SMARTVIDEOCODINGHowtoecientlycompressvideodatawithvariousscala-bilitiesofrate,quality,temporal,spatial,andcomplexityisanactiveresearchtopicinvideocodingeld.
Scalablevideocodingtechniqueshavebeendevelopedrapidlyinthepastdecade.
Amongthem,spatialandtemporalscalablecod-ingtechniquesthatprovidevideopresentationatdierentresolutions,andframerateshavebeenacceptedinsomemainvideocodingstandardssuchasMPEG-2,MPEG-4,andH.
263++[13,14,15].
Inaddition,FGSvideocodingtechniqueshavebeenex-tensivelystudiedinrecentyears.
MPEG-4standardalreadyacceptedthebitplanecodingtechniqueinthestreamingvideoprole(SVP)[16,17].
InMPEG-4FGS,anencoderusingthemotion-compensateddiscretecosinetransforma-tion(DCT)transformcodinggeneratesabaselayervideoasthelowestqualitylayer.
Theresiduebetweentheoriginalimageandthereconstructedbaselayerimageformstheen-hancementlayerwiththebitplanecodingtechnique,whichprovidesanembeddedbitstreamandnegranularityqualityandtemporalscalabilities.
OnemajorfeatureinMPEG-4FGSisthatthebaselayerandallthebitplanesattheenhancementlayerinapredictedframearealwayscompensatedfromthereconstructedver-sionofthebaselayerinthereference.
Therefore,itprovidesaremarkablecapabilityinbothbandwidthadaptationander-rorrecovery.
Bypredictingtheenhancementlayerfromthebaselayer,anybitstreamtruncationandlostpacketsattheenhancementlayerhavenoeectontheframesfollowed.
However,thisalsomakesMPEG-4FGSsuerfromseveredegradationincodingeciencyduetothelowestqualityref-erence.
Furthermore,itisdicultforMPEG-4FGStocom-pressdierent-resolutionvideoatdierentlayers;otherwise,196EURASIPJournalonAppliedSignalProcessingBaselayer1stbitplane2ndbitplane3rdbitplane4thbitplane12345FramesFigure3:Theproposedmultiple-looppredictiontechniquewithtworeferencescase.
thecodingeciencyattheenhancementlayerwouldbefur-therdegraded.
Therefore,theSMARTvideocodingisproposedbasedonourpreviousworks[18,19].
Themultiple-looppredic-tionanddriftingreductiontechniquesarerstusedatthequalityenhancementlayertoachieveagoodtrade-obe-tweenhighcodingeciencyandlowdriftingerrors.
Then,thesetechniquesareextendedtothetemporalandspatialscalabilities,consequently,forminganecientanduniversalscalablevideocodingframework.
3.
1.
Multiple-looppredictionThemultiple-looppredictiontechniquewasrstproposedin[18,19]toimprovethecodingeciencyofMPEG-4FGS.
Thebasicideaistouseasmanypredictionsfromtheen-hancementlayeraspossibleinsteadofalwaysusingthebaselayerasinMPEG-4FGS.
Becausethequalityofaframeishigherattheenhancementlayerthanatthebaselayer,thiswillmakemotionpredictionmoreaccurate,thusimprov-ingthecodingeciency.
Consideringthecostbyintroduc-ingmultiplereferencesattheenhancementlayer,Figure3il-lustratesatypicalmultiple-looppredictionschemewithoneadditionalreferenceusedintheenhancementlayercoding.
InFigure3,thegrayrectangularboxesdenotetherecon-structedbaselayerorthereconstructedenhancementlayeratacertainbitplaneasreferencesforthenextframecoding.
Solidarrowswithsolidlinesbetweentwoadjacentframesarefortemporalprediction,solidarrowswithdashedlinesareforpredictioninthetransformdomain,andhollowar-rowswithsolidlinesareforreconstructionofhigh-qualityreferencefromthepreviousbaselayer.
Eachframeatthebaselayerisalwayspredictedfromthepreviousframeatthebaselayer(low-qualityreference)soastoavoidanyeectfromthelostenhancementdata.
Eachframeattheenhancementlayerispredictedfromthepreviousframeattheenhancementlayer(high-qualityreference)forhighcodingeciency.
IntheFGSvideocodingschemes,thebaselayerbitrateisusuallyverylow.
Itisreasonabletoassumethatthebaselayerbitstreamcanbecompletelytransmittedtotheclient.
Sincethebaselayerisstillpredictedfromthepreviousbaselayer,anybitstreamtruncationandlostpacketsattheenhance-mentlayerhavenoeectonthebaselayervideo.
However,whenthosebitplanesusedtoreconstructthehigh-qualityreferencearetruncatedorcorruptedduringtransmission,thiswouldinevitablycausedriftingerrorsattheenhance-mentlayer.
Asaresult,thedecodedenhancementlayervideomaybedeterioratedrapidly.
3.
2.
DriftingreductionInordertoeectivelyreducedriftingerrorsattheenhance-mentlayer,thebasicideaistomakesurethattheencoderandthedecoderhavethesamereconstructedreferenceforanyfu-tureframeprediction,althoughthereconstructedreferencemaynothavethebestqualityitcouldgetifreconstructedusingthehigh-qualityreference.
WewillshowthisideathroughanexampleinFigure3.
Inthedecoderend,ifthethirdbitplaneinFrame1istrun-catedordroppedwhichisusedintheencoderendtogetthehigh-qualityreference,theenhancementlayerinFrame2willhavetousethepreviouslow-qualityreferenceinstead.
Ofcourse,somequalitylosseswouldbeintroducedbydoingso.
However,aslongasinboththeencoderendandthede-coderendthereconstructionofthehigh-qualityreferenceofFrame2alwaysusesthebaselayerofFrame1asthereference,thentheerrorsinFrame1couldnotfurtherpropagatetoanyframesfollowed.
Inotherwords,thereferenceusedforpre-dictioncouldbedierentfromthatusedforreconstruction.
ThisfeaturewillpreventtheerrorsdriftingandpreserveallthebandwidthadaptationanderrorrecoveryfeaturesasinMPEG-4FGS.
AsshownbyhollowarrowswithsolidlinesinFigure3,someframes,suchasFrames2and4,reconstructthehigh-qualityreferencesfromthepreviouslow-qualityreferenceatboththeencoderandthedecodertopreventtheerrorsprop-agatingintofutureframes.
However,ifthethirdbitplaneofFrame1isavailableatthedecoderend,abettersecondbitplanequalityofFrame2canstillbereconstructedfromthehigh-qualityreferencefordisplaypurposeonly.
Inotherwords,thereconstructionofdisplayimagecanbedierentfromthatofreferenceimage.
Althoughtheproposedtechniquesignicantlyreducesthedriftingerrorsfromthepreviousframes,itstillhasanegativeeectoncodingeciencybecausethehigh-qualityreferencedoesnotalwaysgetthebestqualityitcouldget.
Ifthereferenceforpredictionandreconstructionischosenasframe-based,thatis,allenhancementlayermacroblocksinaframewiththesamereference,itisverydicultfortheSMARTvideocodingtoprovideagoodtrade-obetweenhighcodingeciencyandlowdriftingerrors.
3.
3.
Macroblock-basedmodeselectionThetechniquechoosingtheproperreferenceforpredictionandreconstructionateachenhancementlayermacroblockisrstproposedin[20].
DerivedfromMPEG-4FGSandFigure3,threeintercodingmodesasshowninFigure4aredenedforcodingtheenhancementintermacroblock.
TherectangularboxesintherstrowdenotethebaselayerandtherectangularboxesinotherrowsdenotebitplanesattheSMART:AnEcient,Scalable,andRobustStreamingVideoSystem197Low-qualityreferenceHigh-qualityreferenceMode1Mode2Mode3Figure4:Threeintercodingmodesforthequalityenhancementlayer.
enhancementlayer.
Grayrectangularboxesindicatethosetobereconstructedasreferences.
Solidarrowswithsolidlinesbetweentwoadjacentframesarefortemporalpredictions,solidarrowswithdashedlinesareforpredictioninthetrans-formdomain,andhollowarrowswithsolidlinesareforreconstructionofhigh-qualityreferencefromthepreviousbaselayer.
InMode1,thebaselayerandtheenhancementlayerarebothpredictedandreconstructedfromthepreviouslow-qualityreference.
Sincethelow-qualityreferenceisalwaysavailableatthedecoder,thereisnodriftingerrorinthismode.
Thecodingeciencyofthismodeislowduetolow-qualitytemporalprediction.
Ifallenhancementlayermac-roblocksareencodedwiththismode,theproposedschemeissimilartoMPEG-4FGS.
InMode2,thebaselayerispredictedandreconstructedfromthepreviouslow-qualityreference,buttheenhance-mentlayerispredictedandreconstructedfromtheprevi-oushigh-qualityreference.
Itcansignicantlyimprovethecodingeciencyatmoderateandhighbitrates.
Thereisnodriftingerroratthebaselayer.
Whenthechannelbandwidthisnothighenoughtotransmitthehigh-qualityreference,thismodewouldcausedriftingerrorsattheenhancementlayer.
InMode3,theenhancementlayerispredictedfromtheprevioushigh-qualityreferencewhilereconstructedfromthepreviouslow-qualityreferenceatboththeencoderandthedecoder.
Thismodewasforthepurposeofdriftingreduc-tion.
Sincethelow-qualityreferenceisalwaysconsistentatboththeencoderandthedecoder,thedriftingerrorsprop-agatedfromprevioushigh-qualityreferencescanbeelimi-natedwithMode3.
MoreintercodingmodescouldbereadilyaddedintheSMARTcodingaslongastheyhavethevirtueinimprov-ingcodingeciencyorreducingerrorpropagation.
Inor-dertoachieveagoodtrade-obetweenlowdriftingerrorsandhighcodingeciency,amodeselectionalgorithmisproposedtochoosethepropercodingmodeforeachmac-roblock.
Besidestheabovethreeintermodes,intramodeisal-lowedintheenhancementlayercoding.
Intramodeorinter-modeisdeterminedbymotionestimation.
Ifamacroblockisencodedwiththeintramodeatthebaselayer,thecor-respondingenhancementmacroblockisalsoencodedwiththeintramodewithouttemporalprediction.
Ifamacroblockatthebaselayerisencodedwithtemporalprediction,theproposedmodeselectionalgorithmhastodeterminewhichintercodingmodeshouldbeusedatthecorrespondingen-hancementmacroblock.
ThereferenceforpredictioninMode1isoflowqualitybutthereferenceusedinMode2andMode3isofhighqual-ity.
IftheabsolutemeanofthepredictedDCTresiduespro-ducedinMode1islessthanthatinModes2and3,thecur-rentmacroblockiscodedusingMode1;otherwise,themodeselectionalgorithmfurtherdeterminesthecodingmodebe-tweenMode2andMode3.
BothModes2and3arepredictedfromthehigh-qualityreference,thedierencebetweenthemliesinthereferenceforreconstruction.
Ingeneral,mostoftheenhancementmacroblocksshouldbecodedwithMode2forhighcodingeciency.
Mode3isusedonlywhenthedriftingerrorsaremorethanagiventhreshold.
Inordertoestimatethepotentialdriftingerrorsattheencoder,theiter-ativedriftingmodelproposedin[21]isgivenasfollows:y(n)=0,n=1,MCny(n1)+DCT1X(n1),N≥n>1.
(1)Here,NisthetotalnumberofframesinaGOP,MC(·)andDCT1denotemotioncompensationandIDCT,respectively,y(n1)istheaccumulativeerrorpropagatedtothe(n1)thframe,andX(n1)isDCTcoecientsencodedinthosebitplanesforreconstructionofthehigh-qualityreferenceinthe(n1)thframe.
Withmotioncompensation,theirsumformsthenextdriftingerrorsinthenthframe.
Ifthees-timateddriftingerrory(n)ismorethanthegiventhresh-old,thismacroblockisencodedwithMode3;otherwise,thismacroblockisencodedwithMode2.
Fortheconvenienceofabetterunderstandingoftheproposedmultiple-loopprediction,driftingreduction,andmacroblock-basedmodeselection,Figure5illustratesanex-empliedblockdiagramoftheSMARTdecoderwithqualityscalability.
Therearetworeferenceframesinthedecoder.
Therstoneislocatedinthebaselayerdecoderandstoredintheframebuer0asalow-qualityreference,whilethesecondoneislocatedintheenhancementlayerdecoderandstoredintheframebuerasahigh-qualityreference.
Onlythelow-qualityreferenceisallowedintherecon-structionofthebaselayerinordertoassurethatnodrift-ingerrorexistsatthislayer.
Theenhancementlayercanusetwodierentqualityreferencesforreconstruction.
Theen-hancementbitstreamisrstdecodedusingbitplanevariablelengthdecoding(VLD)andmodeVLD.
Thebitplanesattheenhancementlayerarecategorizedintoalowerenhancementlayerandahigherenhancementlayer.
Onlythebitplanesatthelowerenhancementlayerareusedtoreconstructthehigh-qualityreference.
InFigure5,n(t)isthenumberofbitplanesatthelowerenhancementlayerandm(t)isthenum-berofadditionalbitplanesforthereconstructionofthedis-playframe.
Thedecodedblock-basedbitplanesareusedtorecon-structtheDCTcoecientsofthelowerandhigherenhance-mentlayersusingthebitplaneshiftmodules.
Afterinverse198EURASIPJournalonAppliedSignalProcessingBitplaneshift+IDCT+ClippingVideoS2EnhancementbitstreamBitplaneVLDm(t)n(t)Bitplaneshift+IDCTModeVLDMCFramebuer1MVs+ClippingVideo(optional)S1EnhancementlayerdecoderBaselayerbitstreamVLDQ1IDCT+ClippingVideo(optional)MVsMCFramebuer0BaselayerdecoderFigure5:TheexempliedSMARTdecoderwithqualityscalability.
DCT,thelowerenhancementDCTcoecientsplusthere-constructedbaselayerDCTcoecientsgeneratetheerrorimageforreference,andallDCTcoecientsincludingthehigherenhancementlayergeneratetheerrorimagefordis-play.
Furthermore,therearetwoswitchesS1andS2attheSMARTdecoderthatcontrolwhichtemporalpredictionisusedateachenhancementmacroblock.
Thedecodedmac-roblockcodingmodedecidestheactionsofthetwoswitches.
WhenonemacroblockiscodedasMode1,theswitchesS1andS2connecttothelow-qualityprediction.
WhenitiscodedasMode2,bothoftheswitchesS1andS2connecttothehigh-qualityprediction.
WhenitiscodedasMode3,theswitchS1connectstothelow-qualityprediction.
How-ever,theswitchS2stillconnectstothehigh-qualitypredic-tion.
Sincethedisplayframedoesnotcauseanyerrorprop-agation,thedisplayframeisalwaysreconstructedfromthehigh-qualitypredictioninMode3.
3.
4.
UniversalscalablecodingframeworkThetechniquesdiscussedinSections3.
1,3.
2,and3.
3canbereadilyextendedtothetemporalandspatialscalablevideocoding.
Thebasicideaistousemorethanoneenhance-mentlayerbasedonacommonbaselayertoimplementnegranularityquality,temporal,andspatialscalabilitieswithinthesameframework.
Inordertoachievehighcodingef-ciencyforvariousscalabilities,multiplepredictionloopswithdierentqualityreferencesareemployedinthepro-posedframework.
Forexample,byutilizingthehigh-qualityreferenceinthespatialenhancementlayercoding,thepro-posedframeworkcanlikewisefulllecientspatialscala-bility.
ThecomplexityscalabilityisinseparablewithotherscalabilitiesintheSMARTcodec.
Itisachievedbyincreas-ing/decreasingthebitrate,framerate,andresolution.
Thechangesintheframerateandresolutionprovidecoarsescal-abilityoncomplexity.
Becauseofthepropertyofnegran-ularityofeachlayeronbitrate,theSMARTcodecalsoprovidesnescalabilityoncomplexitybyadjustingthebitrateofeachlayer.
Thelowestcomplexityboundisthelow-resolutionbaselayerdecoding,whichshouldbesucientlylowformanyapplications.
Figure6illustratestheproposeduniversalscalablecod-ingframework.
Sourcevideowithtworesolutionsiscom-pressedintheproposedframework.
Narrowrectanglesde-notelow-resolutionvideoandwiderectanglesdenotehigh-resolutionvideo.
Therearetwodierentenhancementlayerssharingacommonbaselayer,andtwooptionalenhancementones.
Thebottomlayeristhebaselayer.
Itisusuallygener-atedasthelowestquality,lowestresolution,leastsmooth-ness,andleastcomplexity.
Thequalityenhancementlayercompressesthesameresolutionvideoasthatatthebaselayer.
Itwillimprovethedecodedqualityofthebaselayer.
Thetem-poralenhancementlayerimprovesthebaselayerframerateandmakesthedecodedvideolooksmooth.
Theresttwoen-hancementlayersimprovethevideoqualityandframerateathighresolution.
Thesetwoenhancementlayersareoptionalintheproposedframeworkandappearonlyifthevideowithtwodierentresolutionsisencoded.
Thesameresolutionen-hancementlayersarestoredinthesamebitstreamle.
There-fore,theSMARTcodingschemegeneratesatmostthreebit-streams:onebaselayerbitstreamandtwoenhancementlayerbitstreams.
Exceptthatthebaselayerisencodedwiththeconven-tionalDCTtransformplusVLCtechnique,alloftheen-hancementlayersareencodedwiththebitplanecodingtech-nique.
Inotherwords,everyenhancementlayerbitstreamcanbearbitrarilytruncatedintheproposedframework.
InSMART:AnEcient,Scalable,andRobustStreamingVideoSystem199Spatial-qualityenhancementlayerSpatial-temporalenhancementlayerQualityenhancementlayerTemporalenhancementlayerBaselayerIPPFigure6:TheproposedSMARTcodingframework.
Figure6,eachrectangledenotesthewholeframebitstreamatoneenhancementlayer.
Theshadowregionistheactualtransmittedpart,whereastheblankregionisthetruncatedpart.
HencetheproposedSMARTvideocodingprovidesthemostexiblebitratescalability.
Sincethemultiple-looppredictiontechniqueisusedintheproposedframework,everylayer,excludingthebaselayer,canselectthepredictionfromtwodierentreferences.
AsshownbysolidarrowswithsolidlinesinFigure6,thequalityenhancementlayerusethereconstructedbaselayerandthereconstructedqualityenhancementlayeratacer-tainbitplaneasreferences.
Asshownbyhollowarrowswithsolidlines,thetemporalenhancementlayerisbidirectionallypredictedfromthebaselayerandthequalityenhancementlayer.
Thepredictionsforthetwohigh-resolutionenhance-mentlayersaredenotedbysolidarrowswithdashedlinesandhollowarrowswithdashedlines,respectively.
Similarly,someintercodingmodesaredenedatthetemporalandspatialenhancementlayers,whichcanbefoundin[22,23,24].
Eachcodingmodehasitsuniqueref-erencesforpredictionandreconstruction.
ThesimilarmodeselectionalgorithmdiscussedinSection3.
3canbealsoap-pliedtothetemporalandspatialenhancementlayers.
Infact,someothertechniquesproposedin[25,26,27,28]canbeeasilyincorporatedintotheframeworkbydeningseveralnewcodingmodes.
4.
CHANNELESTIMATIONInthestreamingapplications,oneimportantcomponentiscongestioncontrol.
Congestioncontrolmechanismsusuallycontaintwoaspects:estimatingchannelbandwidthandreg-ulatingtherateoftransmittedbitstream.
SincetheSMARTvideocodingprovidesasetofembeddedandfullscalablebitstreams,rateregulationintheSMARTsystemisessen-tiallyequaltotruncatingbitstreamstoagivenbitrate.
Thereisnotanycomplicatedtranscodingneeded.
Theremainingproblemishowtoestimatethechannelbandwidth.
Typically,channelestimationtechniquesaredividedintotwocategories:probe-basedandmodel-based.
Theprobe-basedtechniquesestimatethechannelbandwidthbottleneckbyadjustingthesendingrateinawaythatcouldmaintainpacketlossratiobelowacertainthreshold[29].
Themodel-basedtechniquesarebasedonaTCPthroughputmodelthatexplicitlyestimatesthesendingrateasafunctionofrecentpacketlossratioandlatency.
Specically,theTCPthrough-putmodelisgivenbythefollowingformula[30]:λ=1.
22*MTURTT*√p,(2)whereλisthethroughputofaTCPconnection(inB/s),MTUisthepacketsizeusedbytheconnection(inbytes),RTTistheround-triptimeoftheconnection(inseconds),andpisthepacketlossratiooftheconnection.
Withformula(2),theservercanestimatetheavailablebandwidthbyreceivingfeedbackparametersRTTandpfromtheclient.
Amongallexistingmodel-basedapproaches,TCP-friendlyratecontrol(TCP-FRC)[31]isthemostdeployableandsuccessfulone.
Thesendingrateformula,byconsideringtheinuenceoftimeout,isgivenasfollows:λ=MTURTT2p/3+RTO33p/8p1+32p2,(3)whereRTOistheTCPretransmissiontime-outvalue(insec-onds).
However,TCP-FRChasoneobviousdrawbackundesir-ablefortheSMARTsystem,thatis,theestimatedbandwidthalwaysuctuatesperiodicallyevenifthechannelbandwidthisverystable.
ThereasonisthatTCP-FRCistryingtoin-creasethesendingratewhenthereisnolostpacket.
Thisun-fortunatelyleadstoashort-termcongestion.
SinceTCP-FRCisverysensitiveinthelowpacketlossratiocase,thesendingrateisgreatlyreducedagaintoavoidfurthercongestion.
Therefore,theSMARTsystemadoptsahybridmodel-basedandprobe-basedmethodtoestimatetheavailablechannelbandwidth.
TCP-FRCisrstusedtocalculateaninitialestimatedbandwidthbypacketlossratioandRTT.
Ifthereisnolostpacket,theestimatedbandwidthshouldbemorethanthepreviousestimation.
Ontheotherhand,somepacketsthatcontainlessimportantenhancementdataaretransmittedwiththeprobingmethod.
ThisisafeatureoftheSMARTbitstream.
Eventhoughthosepacketsarelostforprobing,theydonotaectotherdatapackets.
Ingeneral,theestimatedbandwidthbytheprobingmethodisviewedasthebottleneckbetweentheserverandtheclient.
Thees-timatedbandwidthinTCP-FRCshouldbenotmorethanthatestimatedbytheprobingmethod.
Therefore,theprob-ingmethodprovidesanupperboundforTCP-FRCsoastoreduceuctuationsinbandwidthestimation.
VideopacketsintheSMARTsystemarecategorizedintothreeprioritiesforbandwidthallocation.
Theretransmittedandbaselayerpacketshavethehighestpriority.
Estimatedbandwidthisrstusedtodeliverthemtotheclient.
TheFECpacketsofthebaselayerhavethesecondpriority.
Ifthees-timatedbandwidthismorethanthatneededbythehigh-estprioritypackets,theyaredeliveredpriortotheenhance-mentpackets.
Finally,theremainingchannelbandwidthis200EURASIPJournalonAppliedSignalProcessingusedtodeliverthetruncatedenhancementbitstreams.
Infact,theenhancementpacketsalsoimplicatesseveraldier-entpriorities,Forexample,thebitplanesforreconstruc-tionofthehigh-qualityreferencearemoreimportantthanotherbitplanes,andatlowbitrates,thequalityenhance-mentlayermaybemoreimportantthanthetemporalen-hancementlayer,andsoon.
Becauseoflimitationinpages,thispapernolongerfurtherdiscussesthisissue.
5.
ERRORCONTROLInthestreamingapplications,errorcontrolmechanismisanotherimportantcomponenttoensurereceivedbitstreamsdecodable,whichoftenincludeserrorresilience,FEC,ARQ,andevenerrorconcealment[32,33].
Inthissection,wewilldiscusstheerrorresiliencetechniqueandunequalerrorpro-tectionusedintheSMARTsystem.
5.
1.
FlexibleerrorresiliencePacketlossesareofteninevitablewhiletransmittingcom-pressedbitstreamsovertheInternet.
Besidesthenecessaryframeheader,someresynchronizationmarkersandrelatedheaderinformationhavetobeinsertedinthebitstreamgen-erationsothatthelostpacketsdonotaectotherdatapack-ets.
Thisisthemostsimpleerrorresiliencetechnique,butveryuseful.
Theresynchronizationmarkerplustheheaderanddatafollowedisknownasaslice.
InMPEG-4,theresyn-chronizationmarkerisavariablelengthsymbolfrom17bitsto23bits[14].
Thesliceheaderonlycontainstheindexoftherstmacroblockinthisslice.
Ingeneral,theresynchro-nizationmarkerandthesliceheaderareinsertedatagivenlengthornumberofmacroblocks.
However,thismethodhastwoobviousproblemswhenitisappliedtotheenhancementlayerbitstreamintheSMARTsystem.
Firstly,althoughtheSMARTenhancementlayerbit-streamprovidesbitlevelscalability,theactualminimumunitinthepacketizationprocessisaslice.
Thiswouldgreatlyre-ducethegranularityofscalability.
Secondly,theslicelengthisdecidedintheencodingprocessandxedinthegener-atedbitstream.
Forthestreamingapplications,itisimpossi-bletoadjusttheslicelengthagaintoadapttochannelcon-ditions.
Ingeneral,longerslicemeansloweroverheadbitsandbiggereectsoflostpacket.
Onthecontrary,shorterslicemeanshigheroverheadbitsandlowereectsoflostpacket.
Adaptivelyadjustingtheslicelengthisaverydesirablefea-tureinthestreamingapplications.
Therefore,aexibleerrorresiliencetechniqueisproposedintheSMARTenhancementlayerbitstream.
IntheSMARTsystem,therearenoresynchronizationmarkersandsliceheadersintheenhancementlayerbit-stream.
Thus,thegeneratedbitstreamisexactlythesameasthatwithouterrorresilience.
Butthepositionsofsomemac-roblocksandtheirrelatedinformationneededinthesliceheaderarerecordedinadescriptionle.
Besidestheindexoftherstmacroblock,thesliceheaderattheenhancementlayeralsocontainsthelocatedbitplaneoftherstmac-roblock.
Wecallthesemacroblocksresynchronizationpoints.
NotethateachresynchronizationpointisalwaysmacroblockFrame:17302Bits:0Type:2Time0:19Maxlayer:9VPstart:17808Bits:5BPnum:0isGLL:0MBnum:0VPstart:17822Bits:3BPnum:0isGLL:0MBnum:1VPstart:18324Bits:0BPnum:2isGLL:0MBnum:81Figure7:Theexemplieddescriptionle.
aligned.
Inthisstage,resynchronizationpointsdonotcauseactualoverheadbitsinthegeneratedbitstreams.
Thus,thedescriptionlecouldevenrecordeverymacroblock.
Figure7exempliesthestructureofthedescriptionle.
TheeldsFrameandBitsinthesamerowareusedtolocatethestartpositionofaframeinthebitstream.
Theunitsofthesetwoeldsarebyteandbit,respectively.
TheeldBitsisalwayszerointherstrowofeveryframeduetobyte-aligned.
TheeldTypeindicatestheframetype:0forIframe,1forPframe,and2forBframe.
Theeldtimeistherel-ativetimeofthecurrentframe.
Therstdigitinthiselddenotesthenumberofseconds,andtheseconddigitdenotestheframeindexinasecond.
TheeldMaxLayeristhemax-imumnumberofbitplanesinaframe.
TheeldsVPstartandBitsareusedtolocatethestartpositionofamacroblock.
TheeldBPnumisthelocatedbitplaneofthecurrentmac-roblock.
TheeldisGLLindicateswhetherthismacroblockisusedtoreconstructthehigh-qualityreferenceornot.
Itpro-videsaprioritytotransmittheenhancementbitstreams.
TheeldMBnumistherstmacroblockindexinaslice.
TheproposedexibleerrorresilienceisusedonlyattheenhancementDCTdata.
Ifthemotionvectorsexistattheenhancementlayer,forexample,intemporalframes,theyaredierentiallycodedtogetherbeforeDCTcoecients.
TheVOPheaderandcodedmotionvectorsareprocessedasaslice.
Thereisnotanyresynchronizationpointwithinthemincasethatthelostmotionvectorsinasliceaectothermo-tionvectorsdecodedinanothersliceduetomotionvectorprediction.
SimilartotheentropycodingusedinMPEG-4FGS,thereisnotanyDCand/orACcoecientpredic-tionamongneighboringblocks.
Therefore,theslicesinaframehavenodependencyexceptfortheinherentrelation-shipamongbitplanes.
Withthedescriptionle,theproposederrorresiliencetechniqueintheSMARTsystemcanchooseanyresynchro-nizationpointstochopanenhancementlayerbitstreamintoslices.
However,sincethepositionoftheresynchronizationpointmaybenotbyte-alignedinthebitstream,onelostpacketprobablymakesmanypacketsfollowedundecodable.
AsshowedinFigure8,macroblockNisaresynchroniza-tionpoint.
ItsharesbyteminthebitstreamwithmacroblockN1.
IfthemacroblockNisselectedasthestartofaslice,thesetwomacroblocksmaynotlocateinthesametransportpacket.
Ifbytembelongstothepreviouspacket,thepacketofmacroblockNisevenreceivedundecodablewhenthepacketofmacroblockN1islostduringtransmission.
AsimpletechniqueisproposedtosolvethisproblemasshowninFigure8.
Whenaresynchronizationpointisse-lectedasthestartofoneslice,therstbyteofthismacroblockSMART:AnEcient,Scalable,andRobustStreamingVideoSystem201ResynchronizationpointMacroblockN1MacroblockNBytem1BytemBytem+1MacroblockN1MacroblockNBytem1BytemBytemBytem+1Figure8:TheerrorresilienceintheSMARTsystem.
isduplicatedintotwoslicessothatthelostpacketcannotaf-fecteachother.
Thisleadstotheprobabilitythattheheadandtailofeachslicemayhaveseveraluselessbits.
Thede-coderhastoknowhowmanyuselessbitsshouldbeskipped.
Therefore,thenumbersofuselessbitsintheheadandtailgeneratedfromthedescriptionleneedtobeencapsulatedintothetransportpacketandtransmittedtotheclient.
TheeldsMBnumandBPnumatthesliceheaderalsoneedtobeencapsulatedintothetransportpacketandtransmittedtotheclient.
Weevaluatetheproposederrorresiliencetechniquecom-paredwiththatinMPEG-4.
Intheproposedtechnique,abytehastobeduplicatedforeveryselectedresynchronizationpoint.
Inaddition,thecorrespondingnumbersofuselessbitsarealsocontainedinthepacket.
But,bitsfortheresynchro-nizationmarkerinMPEG-4bitstreamcanbesaved.
There-fore,theproposedtechniquehasthesimilaroverheadbitsineachslice.
However,itenablestheSMARTsystemtoadap-tivelyadjusttheslicelengthaccordingtorate-distortionop-timizationandchannelconditions.
Thisisaverydesirablefeatureinthestreamingapplications.
5.
2.
UnequalerrorprotectionSincetheSMARTvideocodingprovidesalayeredbitstreamstructurewithamoreimportantbaselayerandlessimpor-tantenhancementlayers,errorprotectiontechniquessuchasFECandARQareunevenlyappliedtothebaselayerandtheenhancementlayer.
Ingeneral,ifthestreamingsystemshavenorequestondelay,FECwouldnotplayanimportantrolebecausethelostpacketscanberecoveredbyARQ.
IntheSMARTsystem,thebitrateofthebaselayerisverylowanditmayonlyoccupyasmallpartofthetotalbitrate(usuallylessthan20%).
WhenfourdatapacketsareprotectedbyoneFECpacket,theover-headforFECisonlyabout5%.
Inreturn,ifthelostpack-etstakeplacerandomly,mostofthemmayberecoveredbyFEC.
ItwillconsiderablyreducethesystemdelayduetoARQ.
Basedontheseconsiderations,theSMARTsystemusesFECasanoptionatthebaselayeriflowdelayisrequestedinsomeapplications.
Italsoprovidesaspacetoachieveabettertrade-obetweenARQdelayandFECoverhead.
WhenFECisenabled,thebaselayerpacketsaredividedintomanygroupscontainingKsourcepacketspergroup.
AssumethatNKparitypacketswillbeproducedwithaReed-Solomoncodec.
WhentheseNpacketsaretransmittedoverthebest-eortInternet,anyreceivedsubsetofKsourceorparitypacketscanbeusedtoreconstructtheoriginalKsourcepackets.
IntheSMARTsystem,KisoftensetasN1inordertoavoidtoomuchoverheadintroducedbyFEC.
ThetargetusingFECismainlytorecoveroccasionallostpacketandreducethedelaycausedbyARQ.
ThebaselayerbitstreamintheSMARTsystemisanon-scalableone.
Furthermore,themotioncompensationtech-niqueisusedinthebaselayercoding.
AnylostpacketwillmakethequalityoftheframesfollowedinaGOPdegraderapidly.
Therefore,theARQtechniqueisalsoappliedtothebaselayertohandleburstpacketlosses.
IfthelostpacketsthatcannotberecoveredfromFECaredetectedatthebaselayer,aNACKfeedbackisimmediatelysenttotheserver.
Ifnoacknowledgementfeedbackisreceived,thetransmittedbaselayerpacketsaresavedinaspecialbuer.
TheSMARTwillgetthelostbaselayerpacketsfromthespecialbuerandretransmitthemtotheclientuntiltimeout.
IfthebaselayerpacketsarrivetoolateorarenotabletoberecoveredbyFECandARQ,theSMARTsystemwillskiptothenextGOP.
Inaddition,theclientperiodicallysendstheacknowledgementfeedbacksothattheserverdiscardsthereceivedbaselayerpacketsfromthespecialbuer.
FromthediscussionsinSection3,weknowthattheSMARTvideocodingprovidestheembeddedenhancementbitstreams.
Anytruncationandlostpacketsattheenhance-mentbitstreamareallowed.
Itcanbegracefullyrecoveredbythedriftingreductiontechnique.
Therefore,noerrorpro-tectiontechniquesareappliedtotheenhancementpacketsinthecurrentSMARTsystem.
Infact,considerthatthelostpacketsinlowbitplanesusedtoreconstructthehigh-qualityreferencemaystillhaveabigeectonmaintaininghighde-codedquality.
Thetechniquesforpartlyprotectingtheen-hancementlayerpacketsshouldbefurtherinvestigated.
6.
EXPERIMENTSBothstaticanddynamicexperimentsaredesignedtoevalu-atetheperformancesoftheSMARTsystemoncodinge-ciency,channelestimation,bandwidthadaptation,errorro-bustness,andsoon.
6.
1.
StatictestsThreedierentcodingschemes,namely,MPEG-4FGSwith-outglobalmotioncompensation,SMARTcodingwithoutmultiple-loopprediction,andSMARTcoding,arecomparedintermsofcodingeciency.
MPEG-4FGSprovidestheref-erenceofscalablecodingschemeforcomparisons.
Thenaldriftamendment(FDAM)softwareofMPEG-4FGSreleasedinJune2002isusedtocreatetheresultsofMPEG-4FGS[34].
TheSMARTsystemusesWindowsMediaVideoEncoder8.
0(WMV8)asthebaselayercodec.
TheMPEG-4testingse-quencesForemanandCoastguardwithcommonintermedi-ateformat(CIF)areusedinthisexperiment.
Intherstsetofexperiments,thetestingsequencesarecodedat10Hzencodingframerate.
OnlytherstframeisencodedasIframeandtherestoftheframesareencodedasPframes.
ThemainparametersintheMPEG-4FGSbaselayeraregivenasfollows:202EURASIPJournalonAppliedSignalProcessing38373635343332313029dB96160224288352416KbpsSMARTMPEG-4FGSSMARTFGS(a)3332313029282726dB96160224288352416KbpsSMARTMPEG-4FGSSMARTFGS(b)Figure9:ThecurvesofaveragePSNRversusbitrateat10fpswith-outBframeandtemporalscalability.
(a)ForemanCIFY(10Hz).
(b)CoastguardCIFY(10Hz).
(i)motionestimation:±32pixels,(ii)motioncompensation:quarter-pixelprecision,(iii)quantization:MPEG,(iv)directsearchrange:2(half-pixelunit),(v)advancedprediction:Enable,(vi)skippedmacroblock:Enable.
TheresultsoftherstsetofexperimentsaredepictedinFigure9.
InthecurvesofMPEG-4FGS,thebaselayeriscodedwithquantizationparameter31,andthequal-ityenhancementlayerbitstreamistruncatedataninter-valof32kbps.
Byadjustingthequantizationparameter,theSMARTcurvehasabitrateatthebaselayersimilartoMPEG-4FGS.
ThecurvesofSMARTFGSareobtainedwiththeSMARTsystembyonlyusingMode1.
ThecurvesofSMARTareobtainedwithallthediscussedcodingtech-niquesinthispaper.
38373635343332313029dB256356456556656756856956KbpsSMARTMPEG-4FGSSMARTFGS(a)343332313029282726dB256356456556656756856956KbpsSMARTMPEG-4FGSSMARTFGS(b)Figure10:ThecurvesofaveragePSNRversusbitrateat30fpswithBframeandtemporalscalability.
(a)ForemanCIFY(30Hz).
(b)CoastguardCIFY(30Hz).
SMARTFGSandSMARTusethesamecodingtechniqueatthebaselayer.
SinceonlyMode1isusedinSMARTFGS,theenhancementlayercodingisessentiallythesameasthatinMPEG-4FGS.
WMV8providesaverygoodbaselayercomparedwithMPEG-4;thecodingeciencygainatthebaselayeriscloseto2.
8dBinForemanand1.
6dBinCoast-guardcomparedwithMPEG-4FGS.
Butwithoutthepro-posedenhancementpredictiontechnique,thecodinge-ciencygainisbecomingsmallerandsmallerwithbitratesincreasing.
ThecodingeciencygainofSMARTFGSisonly1.
6dBinForemanand0.
44dBinCoastguardatthehighestbitrate.
However,theSMARTcurveswiththeproposedtech-niquespresenttheconsistentperformanceinawiderangeofbitrates.
Thebitrateforthehigh-qualityreferenceisabout346kbpsinForemanand322kbpsinCoastguard.
Thecodingeciencygain,whenthehigh-qualityreferenceisavailable,is2.
9dBinForemanand1.
7dBinCoastguard.
SMART:AnEcient,Scalable,andRobustStreamingVideoSystem2031152896640384128kbps157711531729230528813457FrameActualEstimate(a)1152896640384128kbps157711531729230528813457FrameActualEstimate(b)Figure11:TheestimatedchannelbandwidthintheSMARTsys-tem.
(a)Estimatedbandwidthinbsonesequence.
(b)Estimatedbandwidthinbstwosequence.
Inaddition,althoughthehigh-qualityreferencesareusedintheenhancementlayercoding,theSMARTcurvesstillhavethesimilarperformanceastheSMARTFGScurvesatlowbitrates.
TheSMARTcurvehasonlyabout0.
15dBlossat150kbps.
Thisprovesthattheproposeddriftingreductiontechniquecaneectivelycontrolthedriftingerrors.
Inthesecondsetofexperiments,thetestingsequencesarecodedat30Hzencodingframerate.
OnlytherstframeiscodedasIframe.
TherearetwotemporalframesinthescalablecodingschemebetweenapairofIandPortwoPframes.
Otherexperimentalconditionsarethesameasintherstsetofexperiments.
ThesameexperimentalresultsgiveninFigure10areobservedasintherstsetofexperiments.
SinceneitherMPEG-4FGSnortheSMARTcodeccon-tainsoneoftheswitchingtechniques,forexample,Sframe,SPframe,orSFframe,thereaderswhoareinterestedinthecomparisonsbetweenthescalablevideocodingandtheSPframeonH.
26LTMLcanreadtheMPEGproposalin[35].
6.
2.
DynamictestsThedynamicexperimentstrytotesttheSMARTsystemun-derthedynamicchannel,suchasstreamingvideoovertheInternet,wherethechannelbandwidthvariesinawiderangeofbitrates.
TheconditionsofMPEG-4FGSvericationtest3433323130292827262524dB125497397121145SDynamic1024kbps(a)353331292725dB125497397121145SDynamic1024kbps(b)Figure12:Thedecodedqualityoverthedynamicchannel:(a)bsoneY.
(b)bstwoY.
areusedinthisexperiment[36].
TwoCIFsequences,bsoneandbstwo,eachwith4032frames(168secondsat24fps)areused.
Thechannelbandwidthvariesfrom1024kbpsto256kbpsandthenrecoversto1024kbpsagainwithastepof256kbps.
Everybitratelasts24seconds.
Thedynamicchan-nelsimulationisdonebythecommercesimulator,theCloudsoftware(http://www.
shunra.
com).
Byusingthehybridmodel-basedandprobe-basedband-widthestimationscheme,whenthesequencesbsoneandbstwoaretransmittedoverthesimulateddynamicchan-nel,theestimatedbandwidthisrecordedandplottedinFigure11.
Thedashed-linecurvesaretheactualchannelbandwidthlimitedbytheCloudsimulator.
Whenthechan-nelbandwidthswitchesfromhighbitratetolowbitrate,theestimatedbandwidthwithTCP-FRCcanrapidlyde-creaseinordertoavoidnetworkcongestion.
Whenthechan-nelbandwidthincreases,theestimatedbandwidthcanalsocatchthisvariationatashorttime.
Furthermore,thecurvesinFigure11fullydemonstratetheadvantageofthehybridbandwidthestimationmethod,wheretheprobingmethodgivesanupperboundtopreventTCP-FRCfromraisingthesendingrateoverthenetworkbottleneck.
Therefore,theSMARTsystemhasastableestimationwhenthechannelbandwidthstaysinaconstant.
204EURASIPJournalonAppliedSignalProcessingThedecodedqualityofsequencesbsoneandbstwoarealsorecordedandplottedinFigure12.
Eachsampleistheav-eragePSNRinasecond.
Twofactors,channelbandwidthandvideocontent,willaectthenaldecodedquality.
Some-times,evenifthechannelbandwidthishigh,thedecodedPSNRmaynotbehighduetoactivecontent.
Inordertoeliminatethevideocontentfactorinevaluatingtheperfor-manceoftheSMARTsystemonbandwidthadaptation,thePSNRcurvesdecodedat1024kbpsaredrawninFigure12asreference.
Thedistancesbetweenthedynamiccurveandthe1024kbpscurvereectthebandwidthadaptationcapabilityoftheSMARTsystem.
AsshowninFigure12,thedecodedPSNRislessthanthatat1024kbpsupto4.
4dBfrom73to96secondsbecausetheestimatedbandwidthisonly240kbpsaround.
From49to72secondsandfrom97to120seconds,theestimatedchannelbandwidthisabout480kbps.
ThedecodedPSNRissigni-cantlyimprovedcomparedwiththatat240kbps.
From25to48secondsandfrom121to144seconds,theestimatedband-widthisabout720kbps.
ThedecodedPSNRisonlyslightlylessthanthatat1024kbps.
TheSMARTsystemprovidesal-mostthesamequalityasthatat1024kbpsfrom1to24sec-ondsandfrom145to168seconds.
Theestimatedbandwidthinthesetwoperiodsisabout950kbps.
Thus,theSMARTsys-temshowsexcellentperformanceonbandwidthadaptation.
Althoughtherearealotofpacketlosseswhileswitchingthechannelbandwidthfromhighbitratetolowbitrate,withtheproposederrorresiliencetechniqueandunequalerrorprotection,allpacketlossesatthebaselayerarerecoveredinthesimulation.
Nogreenblocksappearedinthedecodedvideo.
Fortheenhancementbitstreams,thereisnotanyerrorprotection.
Theeectsofpacketlossesattheenhancementlayeraregraduallyrecoveredbythedriftingreductiontech-nique.
TherearealsonoobviousvisualartifactsandqualitydegradationintheaveragePSNRcurves.
Atlast,theSMARTvideoplayerisgiveninFigure13.
Itcanreal-timedecodetheCIFsequenceat1024kbpswithPIII800MHz.
Thedecodedvideoispresentedinthebiggestwin-dow.
Theright-upperwindowshowsthecurveoftheesti-matedchannelbandwidthandtheright-bottomwindowisfortheprogramlist.
Thepacketlossratioisdrawninthewindowbetweenthem.
Aprogressbarisusedtoindicatethestatusofthereceivedbuer.
TheproposedSMARTsystemisalsousedtorunthere-sultsofMPEG-4FGSvericationtests,wheretheSMARTcodecisreplacedbyMPEG-4FGScodec.
Theexperimentalresultshavebeenreleasedin[37].
7.
CONCLUSIONSANDFUTUREWORKSTheSMARTsystempresentsanecient,adaptive,andro-bustschemeforstreamingvideoovertheInternet.
Firstly,sincethemultiple-looppredictionanddriftingreductiontechniquesareappliedatthemacroblocklevel,theSMARTsystemcanoutperformMPEG-4FGSupto3.
0dB.
Secondly,theSMARTsystemhasexcellentcapabilityinnetworkband-widthanddeviceadaptationduetotheembeddedenhance-Figure13:TheinterfaceoftheSMARTvideoplayer.
mentbitstreamsandtheuniversalscalabilities.
Thirdly,withtheproposedbandwidthestimationmethod,theSMARTsystemcanrapidlyandstablycatchbandwidthvariations.
Atlast,sincealayeredbitstreamstructurewithamoreimpor-tantbaselayerandlessimportantenhancementlayersispro-videdintheSMARTsystem,thebaselayerbitstreamishighlyprotectedbytheproposederrorresilienceandunequalerrorprotectiontechniqueswithsmalloverhead.
TheSMARTsys-temcanprovideuserswithmuchsmoothplaybackexperi-enceandmuchbettervisualqualityinthebest-eortInter-net.
AlthoughtheSMARTsystemshowsgoodperformancesoncodingeciency,bandwidthadaptation,channelestima-tion,anderrorrobustness,therearestillseveralproblemsneededtobefurtherstudiedinthefuture,suchashowtofur-therimprovethecodingeciencytocoveranevenwiderbitraterange;howtooptimallyallocatetheavailablebandwidthtodierentenhancementlayerssothattheperceptionqual-itylooksbetter;howtooptimallypacketizethebaselayerandtheenhancementlayerbitstreamssothatthepacketlosseshavelesseects;howtooptimallydecidetheparametersinFECandARQtoachieveabettertrade-obetweenARQde-layandFECoverhead;andhowtoprotectthosebitplanesforreconstructionofthehigh-qualityreferenceattheen-hancementlayerswithsmalloverhead.
Inaddition,howtoeectivelyutilizethefeaturesandtechniquesoftheSMARTsysteminthemulticastapplicationsisanothertopicworthyoffurtherstudy.
ACKNOWLEDGMENTSManycolleaguesandvisitingstudentsinMicrosoftResearchAsiaalsotookpartintheSMARTsystem.
TheauthorswouldliketothankDr.
W.
Zhu,Dr.
Q.
Zhang,andL.
Wangfortheircontributioninthebandwidthestimationpart;X.
Sunforne-grainqualityandtemporalscalability;Dr.
Q.
WangandDr.
R.
Yangforne-grainspatialscalability;andProf.
Z.
XiongandS.
ChengforsettinguptheSMARTserverinTexasA&MUniversity.
SMART:AnEcient,Scalable,andRobustStreamingVideoSystem205REFERENCES[1]J.
Lu,"SignalprocessingforInternetvideostreaming:are-view,"inProc.
SPIE:ImageandVideoCommunicationsandProcessing2000,vol.
3974,pp.
246–259,SanJose,Calif,USA,January2000.
[2]A.
Luthra,Needforsimplestreamingvideoprole,ISO/IECJTC1/SC29/WG11,M5800,Noordwijkerhout,TheNether-lands,March2000.
[3]D.
Wu,Y.
T.
Hou,W.
Zhu,Y.
-Q.
Zhang,andJ.
M.
Peha,"StreamingvideoovertheInternet:approachesanddirec-tions,"IEEETrans.
CircuitsandSystemsforVideoTechnology,vol.
11,no.
3,pp.
282–300,2001.
[4]RealNetworksfacts,2001,http://www.
realnetworks.
com/company.
[5]WindowsMediatechnologies,http://www.
microsoft.
com/windows/windowsmedia.
[6]N.
FarberandB.
Girod,"RobustH.
263compatiblevideotransmissionformobileaccesstovideoservers,"inProc.
In-ternationalConferenceonImageProcessing,vol.
2,pp.
73–76,SantaBarbara,Calif,USA,October1997.
[7]M.
JarczewiczandR.
Kurceren,AproposalforSP-frames,ITU-TQ.
6/SG16,VCEG-L27,Elysee,Germany,January2001.
[8]X.
Sun,S.
Li,F.
Wu,G.
B.
Shen,andW.
Gao,"Ecientandexibledrift-freevideobitstreamswitchingatpredictiveframes,"inProc.
IEEEInternationalConferenceonMultimediaandExpo,vol.
1,pp.
541–544,Lausanne,Switzerland,August2002.
[9]S.
McCanne,V.
Jacobson,andM.
Vetterli,"Receiver-drivenlayeredmulticast,"inProc.
ConferenceonApplications,Tech-nologies,Architectures,andProtocolsforComputerCommuni-cation(ACMSIGCOMM'96),pp.
117–130,Stanford,Calif,USA,August1996.
[10]D.
-N.
Yang,W.
Liao,andY.
-T.
Lin,"MQ:anintegratedmech-anismformultimediamulticasting,"IEEETrans.
Multimedia,vol.
3,no.
1,pp.
82–97,2001.
[11]H.
Schulzrinne,A.
Rao,andR.
Lanphier,Realtimestreamingprotocol(RTSP),InternetEngineeringTaskForce,Internetdraft,RFC2326,April1998.
[12]H.
Schulzrinne,S.
Casner,R.
Frederick,andV.
Jacobson,RTP:Atransportprotocolforreal-timeapplications,InternetEngi-neeringTaskForce,Internetdraft,RFC1889,January1996.
[13]MPEGvideogroup,Informationtechnology—Genericcodingofmovingpicturesandassociatedaudio,ISO/IEC13818-2,In-ternationalstandard,1995.
[14]MPEGvideogroup,Genericcodingofaudio-visualobjects:part2,ISO/IEC14496-2,Internationalstandard,1998.
[15]ITU-TRecommendationH.
263,Videocodingforlowbitratecommunication,Version2,1998.
[16]W.
Li,"OverviewofnegranularityscalabilityinMPEG-4videostandard,"IEEETrans.
CircuitsandSystemsforVideoTechnology,vol.
11,no.
3,pp.
301–317,2001.
[17]M.
vanderSchaarandH.
Radha,"Ahybridtemporal-SNRne-granularscalabilityforInternetvideo,"IEEETrans.
Cir-cuitsandSystemsforVideoTechnology,vol.
11,no.
3,pp.
318–331,2001.
[18]F.
Wu,S.
Li,andY.
-Q.
Zhang,"DCT-predictionbasedpro-gressivenegranularityScalabilitycoding,"inProc.
Interna-tionalConferenceonImageProcessing(ICIP'00),vol.
3,pp.
566–569,Vancouver,BritishColumbia,Canada,September2000.
[19]F.
Wu,S.
Li,andY.
-Q.
Zhang,"Aframeworkforecientprogressivenegranularityscalablevideocoding,"IEEETrans.
CircuitsandSystemsforVideoTechnology,vol.
11,no.
3,pp.
332–344,2001.
[20]X.
Sun,F.
Wu,S.
Li,W.
Gao,andY.
-Q.
Zhang,"Macroblock-basedprogressivenegranularityscalablevideocoding,"inProc.
IEEEInternationalConferenceonMultimediaandExpo(ICME'01),pp.
461–464,Tokyo,Japan,August2001.
[21]F.
Wu,S.
Li,B.
Zeng,andY.
-Q.
Zhang,"Driftingreductioninprogressivenegranularscalablevideocoding,"inProc.
PictureCodingSymposium,Seoul,Korea,April2001.
[22]X.
Sun,F.
Wu,S.
Li,W.
Gao,andY.
-Q.
Zhang,"Macroblock-basedprogressivenegranularityscalable(PFGS)videocod-ingwithexibletemporal-SNRscalablilities,"inProc.
Inter-nationalConferenceonImageProcessing,pp.
1025–1028,Thes-saloniki,Greece,October2001.
[23]Q.
Wang,F.
Wu,S.
Li,Y.
Zhong,andY.
-Q.
Zhang,"Fine-granularityspatiallyscalablevideocoding,"inProc.
IEEEInt.
Conf.
Acoustics,Speech,SignalProcessing,pp.
1801–1804,SaltLakeCity,Utah,USA,May2001.
[24]R.
Yan,F.
Wu,S.
Li,R.
Tao,andY.
Wang,"Ecientvideocodingwithhybridspatialandne-grainSNRscalabilities,"inProc.
SPIE:VisualCommunicationsandImageProcessing2002,vol.
4671,pp.
850–859,SanJose,Calif,USA,January2002.
[25]R.
KalluriandM.
vanderSchaar,Single-loopmotion-compensatedbasedne-granularscalability(MC-FGS)withcross-checkedresults,ISO/IECJTC1/SC29/WG11,M6831,Pisa,Italy,2001.
[26]A.
Reibman,L.
Bottou,andA.
Basso,"DCT-basedscalablevideocodingwithdrift,"inProc.
InternationalConferenceonImageProcessing,pp.
989–992,Thessaloniki,Greece,October2001.
[27]A.
ReibmanandL.
Bottou,"ManagingdriftinDCT-basedscalablevideocoding,"inProc.
IEEEDataCompressionCon-ference,pp.
351–360,SaltLakeCity,Utah,USA,April2001.
[28]W.
-H.
PengandY.
K.
Chen,"Mode-adaptivenegranularityscalability,"inProc.
InternationalConferenceonImageProcess-ing,pp.
993–996,Thessaloniki,Greece,October2001.
[29]D.
Wu,Y.
T.
Hou,W.
Zhu,etal.
,"Onend-to-endarchitec-turefortransportingMPEG-4videoovertheInternet,"IEEETrans.
CircuitsandSystemsforVideoTechnology,vol.
10,no.
6,pp.
923–941,2000.
[30]S.
FloydandK.
Fall,"Promotingtheuseofend-to-endcon-gestioncontrolintheInternet,"IEEE/ACMTransactionsonNetworking,vol.
7,no.
4,pp.
458–472,1999.
[31]M.
Handley,S.
Floyd,J.
Padhye,andJ.
Widmer,TCPfriendlyratecontrol(TFRC):Protocolspecication,InternetEngineer-ingTaskForce,Internetdraft,RFC3448,January2003.
[32]A.
E.
Mohr,E.
A.
Riskin,andR.
E.
Ladner,"Unequallosspro-tection:gracefuldegradationofimagequalityoverpacketera-surechannelsthroughforwarderrorcorrection,"IEEEJour-nalonSelectedAreasinCommunications,vol.
18,no.
6,pp.
819–828,2000.
[33]P.
A.
ChouandZ.
Miao,"Rate-distortionoptimizedsender-drivenstreamingoverbest-eortnetworks,"inProc.
IEEE4thWorkshoponMultimediaSignalProcessing,pp.
587–592,Cannes,France,October2001.
[34]Videogroup,Informationtechnology—Codingofaudio-visualobjectspart5,Amendment1:ReferencesoftwareforMPEG-4,ISO/IECJTC1/SC29/WG11,MPEGM4711,Jeju,March2002.
[35]F.
Wu,X.
Sun,andS.
Li,ComparisonsbetweenPFGSandJVTSP,ISO/IECJTC1/SC29/WG11,MPEGm8426,Fairfax,2002.
[36]Testgroup,MPEG-4visualnegranularityscalabilitytoolsver-icationtestplan,ISO/IECJTC1/SC29/WG11,MPEGN4456,Pattaya,Thailand,2001.
[37]Testgroup,ReportonMPEG-4visualnegranularityscalabil-itytoolsvericationtest,ISO/IECJTC1/SC29/WG11,MPEGN4791,Fairfax,2002.
206EURASIPJournalonAppliedSignalProcessingFengWureceivedhisB.
S.
degreeinelectri-calengineeringfromtheUniversityofXi'anElectricalScienceandTechnology,Xi'an,China,in1992,andhisM.
S.
andPh.
D.
degreesincomputersciencefromHarbinInstituteofTechnology,Harbin,China,in1996and1999,respectively.
HejoinedMi-crosoftResearchAsia,Beijing,China,asanAssociateResearcherin1999andwaspromotedtoaResearcherin2001.
HehasplayedamajorroleinInternetMediaGroupindevelopingscal-ablevideocodingandstreamingtechnologies.
Hehasauthoredandcoauthoredover60papersinvideocompressionandcontributedsometechnologiestoMPEG-4andH.
264.
Hisresearchinterestsin-cludevideoandaudiocompression,multimediatransmission,andvideosegmentation.
HonghuiSunreceivedhisB.
S.
degreefromZhejiangUniversity,HangZhou,China,in1992,andhisM.
S.
degreeincomputergraphicsfromBeijingUniversity,Beijing,China,in1995,allincomputerscience.
HewasaLecturerinComputerScienceDepart-ment,BeijingUniversity,Beijing,China,from1995to1999.
HejoinedMicrosoftRe-searchAsia,Beijing,ChinaasaResearchSoftwareDesignEngineerin1999andwaspromotedtoSeniorResearchSoftwareDesignEngineerin2001.
Hisworkmainlyfocusesonvideocompression,multimediatrans-mission,andnetworktechnology.
GuobinShenreceivedhisB.
S.
degreefromHarbinUniversityofEngineering,Harbin,China,in1994,hisM.
S.
degreefromSoutheastUniversity,Nanjing,China,in1997,andhisPh.
D.
degreefromtheHongKongUniversityofScienceandTechnol-ogy(HKUST)in2001,allinelectricalen-gineering.
HewasaResearchAssistantatHKUSTfrom1997to2001.
Sincethen,hehasbeenwithMicrosoftResearchAsia.
Hisresearchinterestsincludedigitalimageandvideosignalprocessing,videocodingandstreaming,peer-to-peernetworking,andparallelcomputing.
ShipengLireceivedhisB.
S.
andM.
S.
de-greesfromtheUniversityofScienceandTechnologyofChina(USTC)in1988and1991,respectively,andthePh.
D.
degreefromLehighUniversity,Bethlehem,PA,in1996,allinelectricalengineering.
HewaswiththeElectricalEngineeringDepart-ment,UniversityofScienceandTechnol-ogyofChina,Hefei,China,from1991to1992.
HewasamemberofthetechnicalstaatSarnoCorporation,Princeton,NJ,from1996to1999.
HehasbeenaResearcherwithMicrosoftResearchChina,Beijing,sinceMay1999.
Hisresearchinterestsincludeimage/videocom-pressionandcommunications,digitaltelevision,multimedia,andwirelesscommunication.
HehascontributedsometechnologiestoMPEG-4andH.
264.
Ya-QinZhangreceivedtheB.
S.
andM.
S.
degreesinelectricalengineeringfromtheUniversityofScienceandTechnologyofChina(USTC),Hefei,Anhui,China,in1983and1985,andthePh.
D.
degreeinelec-tricalengineeringfromGeorgeWashing-tonUniversity,Washington,DC,in1989.
HeiscurrentlytheManagingDirectorofMicrosoftResearchAsia,Beijing,China,in1999.
Hehasauthoredandcoauthoredover200refereedpapersinleadinginternationalconferencesandjour-nals,andhasbeengrantedover40USpatentsindigitalvideo,Internet,multimedia,wireless,andsatellitecommunications.
Dr.
ZhangservedasEditor-in-ChieffortheIEEETrans.
onCircuitsandSystemsforVideoTechnologyfromJuly1997toJuly1999.
HewastheChairmanoftheVisualSignalProcessingandCommunicationsTechnicalCommitteeoftheIEEECircuitsandSystems(CAS)So-ciety.
Hehasreceivednumerousawards,includingseveralindustrytechnicalachievementawardsandIEEEawards,suchastheCASJubileeGoldenMedal.
HerecentlyreceivedtheOutstandingYoungElectricalEngineerof1998Award.
BruceLinreceivedhisB.
S.
degreefromNa-tionalTaiwanUniversityin1988andhisM.
S.
andPh.
D.
degreesfromtheUniver-sityofMaryland,CollegePark,in1994and1996,respectively,allincomputerscience.
HewasaResearchAssistantatthecenterforautomaticresearchattheUniversityofMarylandfrom1992to1995.
Since1995,hehasbeenworkingwithMicrosoftonvideocompression.
Currently,heisaDevelop-mentManagerofMediaProcessingTechnologygroupinMicrosoftDigitalMediaDivision.
HisfocusisonWindowsmediavideoandvariousimage/videoprocessingcomponentsforWindows.
Ming-ChiehLeewasborninTaiwan.
HereceivedhisB.
S.
degreeinelectricalengi-neeringfromtheNationalTaiwanUniver-sity,Taiwan,in1988,andhisM.
S.
andPh.
D.
degreesinelectricalengineeringfromCali-forniaInstituteofTechnology,Pasadena,in1991and1993,respectively.
HisPh.
D.
re-searchtopicwasonstillandmovingimagecompressionusingmultiscaletechniques.
FromJanuary1993toDecember1993,hewaswiththeJetPropulsionLaboratoryasamemberofthetech-nicalstaandwasworkingonmultiresolutionimagetransmis-sionandenhancement.
InDecember1993,hejoinedtheadvancedvideocompressiongroupofMicrosoftCorporation,Redmond,Wash,asaSoftwareDesignEngineer.
HeisnowtheProductUnitManagerinchargeoftheCoreMediaProcessingTechnologygroupinMicrosoftDigitalMediaDivision.
Hisgrouphasproducedtech-nologiesincludingWindowsmediavideo,Windowsmediaaudio,WindowsmediaaudioProfessional,andWindowsmediaaudiovoice.