making5555tk

M.
ReedLittleandL.
Nigay(Eds.
):EHCI2001,LNCS2254,pp.
73–90,2001.
Springer-VerlagBerlinHeidelberg2001ExploringNewUsesofVideowithVideoSpaceNicolasRousselDIUF,UniversitédeFribourgCheminduMusée,31700Fribourg,Switzerlandnicolas.
roussel@unifr.
chAbstract.
ThispaperdescribesvideoSpace,asoftwaretoolkitdesignedtofacilitatetheintegrationofimagestreamsintoexistingornewdocumentsandapplicationstosupportnewformsofhuman-computerinteractionandcollaborativeactivities.
Inthisperspective,videoSpaceisnotfocusedonperformanceorreliabilityissues,butratherontheabilitytosupportrapidprototypingandincrementaldevelopmentofvideoapplications.
Thetoolkitisdescribedinextensivedetails,byshowingthearchitectureandfunctionalitiesofitsclasslibraryandbasictools.
SeveralprojectsdevelopedwithvideoSpacearealsopresented,illustratingitspotentialandthenewusesofvideoitwillallowinthefuture.
1IntroductionAlthoughalmostfortyyearshavepassedsinceAT&T'sfirstPicturePhone,mostcommercialsystemsandapplicationsdealingwithvideocommunicationtodayarestillbasedonaphoneparadigm.
Thisparadigmusuallyimpliesaformalandexplicitsetupofthelinkbybothpartiesandthedisplayoftheimages"as-is"onamonitororinawindowonacomputerscreen.
Overthelasttwentyyearshowever,theHCIandCSCWresearchcommunitieshaveproposedanumberofinnovativeusesofvideoincludingsupportforinformalcommunication[1]aswellashighlyfocusedcollaborativeactivities[2],orimageprocessingforhuman-computerinteraction[3].
Inthispaper,IpresentvideoSpace,asoftwaretoolkitdesignedtofacilitatetheuseofimagestreamstosupportsuchnewformsofhuman-computerinteractionandcomputer-supportedcollaborativeactivities.
VideoSpaceismotivatedbythedesiretofocusontheusesofvideo,ratherthanthetechnologiesitrequires.
Inthisperspective,thetoolkitisnotfocusedonperformanceorreliabilityissues,butratherontheabilitytosupportrapidprototypingandincrementaldevelopmentofvideoapplications.
ThisapproachcontrastswithmanyoftheresearchthemesusuallyassociatedtovideointheMultimediaorNetworkcommunitiessuchascompression,transportorsynchronization.
VideoSpaceisnotaimedatthesetopics.
ItisratherintendedtohelpHCIandCSCWresearcherswhowanttoexplorenewusesoftheimages.
VideoSpaceisdesignedafterA.
Kay'sfamoussaying:"simplethingsshouldbesimple,complexthingsshouldbepossible".
Itprovidesusersanddeveloperswithasetofbasictoolsandaclasslibrarythatmakeiteasytointegrateimagestreamswithinexistingornewdocumentsandapplications.
Thetools,forexample,allow74NicolasRousseluserstodisplayimagestreamsinHTMLdocumentsinplaceofordinarystaticimagesortoembedthesestreamsintoexistingXWindowapplications.
Creatingavideolinkwiththelibraryrequiresonlyafewlinesofcode;managingmultiplesourcesandincludingvideoprocessingisnotmuchmorecomplicated.
SincetheimagestreamsmanagedbyvideoSpaceofteninvolvelivevideoofpeople,thetoolkitalsoprovidesaflexiblemechanismthatallowsuserstomonitorandcontrolaccesstotheirownimage.
Thepaperisorganizedasfollows.
Afterintroducingsomerelatedwork,IdescribevideoSpacebyshowingthearchitectureandfunctionalitiesofitslibraryandbasictools.
Ithenpresentseveralprojectsbasedonthetoolkitthatillustrateitspotentialandthenewusesofvideoitwillallowinthefuture.
Finally,Idiscusssomelessonslearnedfromthisworkandconcludewithdirectionsforfutureresearch.
2RelatedWorkPrototypinghasbeenrecognizedasanefficientmeansofdevelopinginteractiveapplicationsforsometime[4]:iterativedesignpromotestherefinementandoptimizationoftheenvisionedinteractiontechniquesthroughdiscussion,exploration,testinganditerativerevision.
However,exploringnewusesofvideothroughprototypingishard.
Researchersarefacedwithmultipledifficultproblemssuchastheneedfordigitizinghardwareaswellasspecializedencoding/decodingalgorithmsandcommunicationprotocols.
Moreover,evaluatingasolutiontoanyoftheseproblemsusuallymeanshavingsomesolutiontoallofthem.
Manyinnovativeworksontheuseofimagestreamsovercometheseproblemsbyusingspecificvideohardware.
EarlyMediaSpaces,forexample,werebasedonanalogaudio/videonetworks[5,6,7].
ClearBoard[2]alsousesananalogvideolinkanddedicatedvideooverlayboardstosuperimposetwoimagestreamsinreal-time.
Likewise,Videoplace[8]reliesondedicatedhardwareforimageprocessing.
Althoughthesespecifichardwaresolutionsallowresearcherstofocusontheinteractionsandnotthetechnologyrequiredtoimplementthem,theyareusuallyexpensive,hardtosetup,hardtomaintainandsometimesevenhardtoreproduce.
Specificvideohardwareallowstocreatefullyfunctional,high-fidelityprototypes.
Buthigh-fidelityprototypingisnotgoodforidentifyingconceptualapproaches,unlessthealternativeshavealreadybeennarroweddowntotwoorthree,orsometimesevenone[4].
Low-fidelityprototypeshaveprovedusefultonarrowthesealternatives.
Penandpaper,paintingprogramsorothersimpleelectronictoolscanbeusedtogetfeedbackfrompotentialusersfromtheveryearlystagesofproductdevelopment.
SoftwaretoolkitssuchasTcl/TkorGroupKit[9]havelongbeenusedforrapidprototypinganditerativedevelopmentofgraphicalinterfacesandgroupwareapplications.
ThemorerecentadventofWeb-basedapplicationsalsopromotestheuseofquicklyhackedHTMLinterfacesthatcanbeeasilymodifiedtoexplorealternativedesigns.
ThemotivationforcreatingvideoSpaceresidesinthelackofsuchflexiblesoftwaretoolsforexploringnewusesofvideothroughthecreationofhigh-fidelityaswellaslow-fidelityprototypes.
Mostmodernoperatingsystemsprovidesoftwarelibrariestomanipulateimagestreams,suchasAppleQuickTime,MicrosoftDirectXorSGIExploringNewUsesofVideowithVideoSpace75DigitalMediaLibraries.
Theselibrarieshavealltheirownadvantagesanddisadvantages,buttheyareusuallyplatform-dependentandincompatiblewitheachother.
Thesecharacteristicsmakethemdifficulttouseforbuildingthecomplex-andusuallydistributed-applicationsrequiredtoexplorenewusesofvideo.
AlthoughIrealizetheimportanceofthelow-levelservicesofferedbytheselibraries,IbelievethattheHCIandCSCWcommunitieswouldbenefitfromahigher-levelsoftwareplatformforimagestreamsmanipulation.
TheMashstreamingmediatoolkit[10]andsomeotherplatformsdevelopedbytheMultimediaandNetworkresearchcommunitiesofferhigh-levelvideodigitizingandtransmissionservices.
However,theseplatformsnaturallytendtofocusonthetransmissiontechniquesandusuallyrelyontheideathatimagesaretobedisplayed"as-is",asbigandasfastaspossible.
Althoughthisconceptioniswellsuitedtoapplicationssuchasvideoconferencing,tele-teachingorvideo-on-demand,itistoorestrictiveformoreinnovativeapplicationsthatmightinvolveimageprocessingorcompositionatanypointbetweenitsproductionandfinaluse.
TheJavaMediaFramework(JMF)isprobablytheexistingplatformclosesttovideoSpace.
Itprovidesprogrammerswithasetofhigh-levelclassestodigitize,store,transmit,processanddisplayimages.
However,itisaclosedproductofacommercialorganization.
Atthetimeofthiswriting,forexample,JMFsupportsvideodigitizingonMicrosoftWindowsandSunSolarisplatformsonly.
Sinceitssourcecodeisnotpubliclyavailable,implementingthisfeatureonotherplatformsrequireswritingseparateextensionsandpossiblyrewritingsomeexistingcode.
Moreover,correctingbugsorimplementingnewfeaturesisaprivilegeofafewpeoplewhoseinterestsmightdifferfromthoseoftheHCIandCSCWresearchcommunities.
VideoSpacegrewoutofmynumerousexperiencesandfrustrationsinwritingapplicationsthatdigitize,synthesize,transmit,store,retrieve,display,modifyoranalyzeimagestreams.
Bymovingthecommonelementsoftheseapplicationsintoalibraryandprovidingtoolssuchasanetworkvideoserver,Icannowquicklycreatevideoapplicationswhosecomplexityshrunkfromseveralthousandlinesofcodetoonlyafewhundredlines.
This,inturn,facilitatestheexplorationofawiderrangeofusesoftheimagestreamstosupporthuman-computerinteractionanddistantcollaboration.
3TheVideoSpaceLibraryThevideoSpacelibrarywasinitiallydevelopedinC++onSGIworkstations.
Initscurrentstate,itconsistsoflessthanahundredclassesand15000linesofcode.
Thehardwareandsystemdependentcodeisclearlyseparatedfromtherestofthecode,whichallowedustoeasilyportittoLinuxansSunSolarisandshouldfacilitateporttoothersystems.
ThelibraryandseveralvideoSpaceapplicationswereindeedsuccessfullyportedtoAppleMacOSinSeptember1999,althoughthisdevelopmentbranchwaslaterabandoned.
InadditiontotheprimaryC++library,someofthecoreservicesofvideoSpacearealsoavailablefromPythonscripts,throughadynamicextension,andsomeothershavebeenre-implementedinpureJava.
IntegrationwithTcl/TkisalsoprovidedthroughaspecificityoftheXWindowsystemthatwillbedetailedinthenextsection.
76NicolasRoussel3.
1GeneralOverviewThevideoSpacelibraryisbuiltaroundtheconceptofimage:itprovidesdeveloperswithclassesandfunctionstoproduce,process,transmit,displayandrecordimages(Fig.
1)andtomultiplexthesebasicoperations.
Fig.
1.
Typicallife-cycleofavideoSpaceimageVideoSpaceimagesareproducedbylocalornetwork-accessibleimagesources.
Animagesourcemaydigitizeorsynthesizeinreal-timetheimagesitproduces.
Itmayalsoretrievethemfromapre-recordedstream.
AllimagesourcesaredescribedbyaURLthatdefinestheminauniqueway.
TheoperationmodeofaparticularsourcecanbeadjustedthroughtheadditionofquerystringparameterstoitsURL.
Suchparameterscanbeused,forexample,tospecifytheframe-rate,thesizeorthequalityoftheimagesproduced.
VideoSpacesupportsreal-timeprocessingofimagesthroughfiltersthattransformoranalyzethem.
Transformationfiltersmodifythedimensions,thedataortheencodingoftheimageswhereasanalysisfiltersonlyextractvaluableinformationfromthem.
Thelibraryprovidesanumberoffiltersforbothtransformingandanalyzingimages.
Someofthemallowtoconvertimagesbetweenthedifferentencodingsavailableandtoresizethem.
Othersareusedforgammacorrectionorconvolutionbya3x3kernel.
Otherstopaintrectangularregions,toinsertanimageinanotheroneortosuperimposetwoimages.
Morecomplexfilterssuchaschroma-keying,imagedifferenceandbasicmotiondetectionarealsoprovided.
Imagesinksareusedtotransmitimagesonthenetwork,todisplaythemonthecomputerscreenoronananalogvideooutput(e.
g.
,aseparatemonitor)andtorecordthemintofiles.
Likeimagesources,imagesinksandtheiroperationmodearedescribedbyURLs.
Aswewillseeinthefollowingsections,thisuseoftext-baseddescriptionsforsourcesandsinksallowstospecifythematrun-time,makingitpossibletocreatesimpleyetpowerfulapplicationsformanipulatingtheimagestreams.
Combininganimagesourcewithasinkmayresultincomplexexecutionflows.
Forexample,ifanimagecomingfromaremotesourcehastobedisplayedonananalogvideooutput,theapplicationhastolistentothenetworkconnectiontogettheimagedataand,atthesametime,ithastomakesurethatthehardwareisreadytodisplayitwhenitiscompletelydecoded.
Theuseofmultipleimagesourcesorsinksmakesthisexecutionflowmorecomplex.
Interactiveuserinterfacesaddevenmoreeventsourcestomonitor,increasingthecomplexityfurther.
InordertoreducethisExploringNewUsesofVideowithVideoSpace77complexity,thevideoSpacelibraryprovidessimplemechanismsbasedonclassinheritanceandmethodoverridingformultiplexingobjectsthatdealwithfiles,networkconnectionsorhardwaredevices.
3.
2ImplementationDetailsImage,pixelencodingsandmemorymanagement.
TheImageclassimplementssimpledatastructuresthatdescribethewidth,heightandpixelencodingofarectangularbitmapandpointtothememorylocationofthecorrespondingdata.
Thefollowingencodingsaresupported:L-forluminance-,RGB,ABGR,RGBA,Y'CbCr4:2:0andJPEG.
Inordertofacilitatememorymanagementbetweensuccessivemanipulations,thememorylocationofanImagecannotbemodifiedwithoutexplicitlystatingwhatshouldhappentothenewbufferwhenitislatelyreplaced.
Thisinformation,storedalongwiththepointer,allowsasingleImageobjecttosuccessivelyusedifferentmemoryareasthatareautomaticallyde-allocatedwhennotneededanymore.
Networksupport.
ThevideoSpacelibraryprovidesseveralclassesforsendingandreceivingUDPdatagrams-unicastormulticast-,creatingTCPserversandclientsanddecodingHTTPmessages.
TheseclassesallowvideoSpaceapplicationstoexchangeanykindofdata,includingimages.
Fournetworkprotocolsforimagestreaminghavebeenimplementedontopofthem:Netscape'sHTTPServer-Pushextension[11]appliedtoaseriesofJPEGimages,theclientsideoftheRFBprotocol1,andVSMPandVSTP,twoproprietaryprotocols.
VSMPisbasedonUDPandcanbeusedforone-to-one(unicast)orone-to-many(multicast)videotransmissions.
ImagesareJPEG-encodedsotheycanfitinasingledatagramanda"besteffort"strategyisused:lostdatagramsarenotretransmitted.
VSTPwasdesignedforclient-typeapplicationsthatrequestvideofromaserver.
ItcombinesaVSMPtransmissionofimageswithanHTTPconnectionusedasasignalingchannel:theclientsendsontheHTTPconnectiontheinformationrequiredtostarttheVSMPtransmission,i.
e.
thelocalhostnameandaUDPportnumber,andclosesittostopthetransmission.
AlthoughVSMPandVSTPweredesignedforsimplicityandnotperformanceorreliability,theirperformancelevelisquiteacceptable:overthelastthreeyears,theyhavebeenusedtotransmitvideostreamsbetweenFrance,TheNetherlands,Denmark,Germany,SwitzerlandandAustriaatupto20QCIFimagespersecond(176x144pixels)and10CIFimagespersecond(320x240pixels)withalatencyoflessthanhalfasecondandnoperceivableimageloss.
1TheRemoteFrameBuffer(RFB)protocolwasdevelopedbyAT&TfortheirVirtualNetworkComputing(VNC)project[12].
Itallowsthinclientapplicationstodisplayreal-timeimagesofaremoteXWindow,AppleMacOSorMicrosoftWindowsdesktopserverandtosendmouseandkeyboardeventstothisserver.
78NicolasRousselImageproduction.
ThevideoSpacelibrarydefinesanabstractImageSourceclasswithmethodsforstartingtheimageproduction,gettingthenextorthemostrecentimageavailable2andstoppingtheimageproduction.
SeveralclassesderivedfromImageSourceimplementtheactualsourcetypessupportedbythelibrary.
Afactoryfunctionallowstherun-timecreationofagenericImageSourceobjectfromtheappropriatederivedclass,givenaURLandthedesiredencoding.
TheURLdescribingahardwarevideoinputonthelocalmachinespecifiesthedeviceandtheactualinputusedonthisdevice(e.
g.
,analoginputnumbertwoondigitizingboardone).
Twogenericnames,anydevandanynode,canbeusedasdefaultvaluesforportability.
Threeoptionalquerystringparameters(zoom,lengthandpause)controlthesizeoftheimages,theirnumberandthetimebetweentwosubsequentones.
Afourthoneallowsuserstoindicatewhetherthehardwareresourceshouldbelocked(locked=1)orwhetherotherapplicationscanpreemptitwhenneeded.
URLscorrespondingtonetworksourcesspecifytheremotehostnameoraddress(orgroupaddress,inthecaseofmulticast),possiblyfollowedbyaTCPorUDPportnumber,somepathinformationandsomequerystringparameters.
Inadditiontothefourimagestreamingprotocolswementioned,videoSpacesupportstosomeextentthereal-timecaptureofawindowonthecomputerscreenthroughtheXprotocol.
Italsosupportstwotypesofpre-recordedimagesources,oneforsingleimageJPEGfiles,andtheotherforfilescontainingJPEGimagestreamsproducedbytheServer-Pushprotocol.
ThefollowingURLsillustrateandsummarizetheimagesourcescurrentlyimplemented:videoin:/anydev/camerazoom=3localdigitizinghardwarehttp://host:5555/push/videoHTTPserverusingthePushextensionrfb://host:1VNCservervsmp://host:9823videoSpaceapp.
usingVSMPunicastvsmp://225.
0.
0.
252:5557videoSpaceapp.
usingVSMPmulticastvstp://host/videopause=1videoSpaceapp.
usingVSTPxwindow://localhost:0/0x1c0000eXwindow(experimental)file:/tmp/test.
jpgsingleJPEGimagefile:/tmp/demo.
vssJPEGstreamintheServer-PushformatImageprocessing.
Imagefilterscanbeimplementedasfunctions,orasclassesthatassociatedataandpossiblystate-basedtransitionstotheprocessingalgorithm.
Inthelattercase,aSimpleFilterclasscanbeusedasabaseclasstoshareacommonsyntaxandallowtherun-timespecificationoffilters.
Processingalgorithmsareusuallyimplementedforasubsetoftheavailableencodings(e.
g.
,RGB,RGBAandABGR).
Consequently,encodingconversionmightberequiredbeforeand/orafterapplyingafilter.
2Themostrecentimagemightnotbethenextoneifthesourceimplementssomebufferingalgorithm.
ExploringNewUsesofVideowithVideoSpace79Imagetransmission,displayandrecording.
ApointertothememorylocationofanImagedatacanbeobtainedthroughagetDatamethod.
SincetheencodingssupportedbyvideoSpacearealsosupportedbymanylow-levelgraphicalorvideolibraries,itsimagescanbeeasilymanipulatedwiththem.
OpenGL,GTKortheLinuxSVGAlibrary,forexample,canbeusedtodisplaytheimagesonthecomputerscreen.
OnSGIO2andOctaneworkstationswithproperhardware,theDigitalMediaLibrariescanalsobeusedtosendthemtoanexternalanalogvideodevicesuchasamonitororaVCR.
RecentadditionstothelibraryalsoprovidesomepreliminarysupportforcreatingMPEG-1andRealPlayercompatiblevideostreams.
ThevideoSpacelibrarydefinesanabstractImageSinkclassandseveralderivedclassestofacilitatethetransmissionofimagesusingVSMP,theirdisplayusingseveralgraphicaltoolkits,andtheirrecordinginseveralformats.
Asforimagesources,afactoryfunctionallowstherun-timecreationofagenericImageSinkobjectfromgivenaURL.
ThefollowingURLsillustrateandsummarizetheimagesinkscurrentlyimplemented:vsmp://localhost:9823videoSpaceapp.
usingVSMPunicastvsmp://225.
0.
0.
252:5557videoSpaceapp.
usingVSMPmulticastglxwindow://localhost:0OpenGLdisplayinanXWindowgtkwindow://localhost:0GTKwindowsvga:/640x480centered=1SVGAfull-screendisplay(Linuxonly)videoout:/anydev/anynodeanalogvideooutput(SGIonly)file:/tmp/capture.
jpegsingleJPEGimagefile:/tmp/capture.
vssJPEGstreamintheServer-Pushformatfile:/tmp/capture.
mpegMPEG-1stream(experimental,Linuxonly)file:/tmp/capture.
rmRealPlayerstream(experimental,Linuxonly)Multiplexing.
ThevideoSpacelibraryprovidesaMultiplexNodeclassformultiplexinglow-leveloperationsonfiles,networkconnectionsandhardwaredevices.
Themultiplexmethodofthisclass,basedontheUNIXpollsystemcall,suspendstheexecutionoftheapplicationuntilatimerexpiresorsomedescriptorassociatedtoafile,connectionordevicebecomesreadableorwritable.
multiplexreliesontwoothermethods:prepare,thatspecifiesthesetofdescriptorstowatchandthetimelimit,andcheck,thatdefineshowtheobjectreactstolow-levelstatechanges.
Manyclassesofthelibrary,includingallimagesourcesandsinks,derivefromMultiplexNodeandoverridethesetwomethodstoimplementhigh-levelreactiveobjects.
EveryMultiplexNodeobjectmaintainsalistofotherMultiplexNodeinstancesassociatedtoitbytheapplicationdeveloper.
Callingthemultiplexmethodofoneobjectautomaticallycallstheprepareandcheckmethodsofallassociatedinstancesinadditiontothoseoftheprimaryobject.
Thisallowsdeveloperstodescribehierarchicalstructuresofhigh-levelreactiveobjectsandtomultiplextheminasinglecall.
Thefollowingcodeexampleillustratesthisbyshowinghowtoblendalocalandaremoteimagestreamanddisplaytheresultingimagesonananalogvideooutput:80NicolasRousselImageSource*src1=createImageSource(Image::RGB,"videoin:/anydev/camerazoom=2");ImageSource*src2=createImageSource(Image::RGB,"vstp://remotehost/video");ImageSink*dst=createImageSink("videoout:/anydev/anynode");dst->addNode(src1);//Associatethetwoimagesourcesdst->addNode(src2);//totheimagesinksrc1->start();src2->start();dst->start();Imageimg1,img2,composite;boolnewComposite=false;while(dst->isActive()){dst.
multiplex();//Multiplexthesinkandthesourcesif(src1->getNextImage(&img1)||src2->getNextImage(&img2)){//Atleastoneimagehaschanged//updatethecompositeblendImages(&img1,&img2,&composite);newComposite=true;}if(newComposite){//Thecompositestays"new"untilhandlednewComposite=!
dst->handle(&composite);}}4VideoSpaceBasicToolsInadditiontotheC++library,aimedatdevelopers,thevideoSpacetoolkitprovidesend-userswithanumberoftoolsthatcanbeusedoff-the-shelf,withnoorlittleprogramming,andserveasbuildingblocksformorecomplexapplications.
ExploringNewUsesofVideowithVideoSpace814.
1VideoServerThemaintoolprovidedbyvideoSpaceisvideoServer[13].
VideoServerisapersonalWebserverrunbytheuserofaworkstationanddedicatedtovideo:itistheuniquepointofaccesstothatperson'svideosources.
Thethreeservicesitprovidesare:1.
creatingaone-waylivevideoconnection;2.
retrievingapre-recordedvideofile;3.
actingasarelayforanotherimagesource.
VideoServerservicesaremappedtoresourcenamesthatareaccessiblethroughtheHTTPprotocolandcanbedescribedbysimpleURLs.
VideodataitselfcanbetransmittedwiththeHTTPServer-PushorVSTP.
ThefollowingURL,forexample,requestsalivevideostreamwith1frameevery60secondstobesentwithHTTPServer-Push(5555isvideoServer'sdefaultportnumber):http://host:5555/push/videopause=60ThisURLrequestsapre-recordedvideofiletobesentwithVSTPtoaclientlisteningondesthostonport9257:http://host:5555/vstp/movie/demo.
vsshost=desthost&port=9257ThisthirdexampleillustratestheuseofvideoServerasarelayforanothersource,inthiscaseanRFBserver.
TheURLdescribingthesourcerelayed,rfb://srchost:1,hastobeencodedinordertobeusedinthequerystringoftherequest:http://host:5555/push/relaysrc=rfb%3A%2F%2Fsrchost%3A1UsingcustomHTTPserverstoprovidevideoservicesisnotnew.
AnumberofWebcamsavailableontheInternetworkthisway,andinfactvideoSpaceusersoftenusethemasimagesources.
However,videoServerdiffersfromtheseWebcamsonamajorissue:itprovidesitsuser(i.
e.
,thepersonwhorunsit)withaccesscontrolandnotificationmechanismstosupportprivacy.
Foreveryrequestitreceives,videoServerexecutesanexternalprogram,thenotifier,withargumentsindicatingthenameoftheremotemachine,possiblytheremoteuser'sloginname,theresourcethatledtotheserver-theHTTPreferrer-andtherequestedservice.
Inresponse,thenotifiersendsbacktotheserverthedescriptionoftheservicetoexecute,whichcandifferfromtheonetheclientrequested.
Thedefaultnotifier,aUNIXshellscript,allowsuserstoeasilydefineaccesspolicies.
Lowqualityorpre-recordedvideo,forexample,canbesenttounidentifieduserswhileknownpeoplegetahighqualitylivevideostream.
Inadditiontocomputingtheservicetoexecute,thenotifiercantriggerseveralauditoryorgraphicalnotificationstoreflectsomeoftheinformationavailable,suchastheidentityoftheremoteuserortheservicerequested.
Whenalivevideorequestdoesnotspecifythenumberofimages,videoServerlimitsitto5000,thatis,uptothreeminutes.
Thisensuresthatconstantmonitoringcannottakeplacewithoutperiodicallyaskingpermissionandthustriggeringnotifications.
AlltheseelementsfacilitatetheacceptanceofvideoServerbytheusersandhelpsfindingatrade-offbetweenaccessibilityandprivacy.
82NicolasRoussel4.
2VideoClientVideoClientstartedasasimplelightweightapplicationdesignedtodisplayvideoSpacestreamsonthecomputerscreen,thevideobeingscaledtomatchthewindowsize[13].
Astheclasslibraryevolved,itbecameamoregenerictoolforeasilyfilteringvideostreamsanddisplaying,recording,orsendingthemonthenetwork.
ThecurrentimplementationofvideoClientisabout100linesofC++andallowstospecifyatrun-timeanImageSource,aSimpleFilterandanImageSink.
Forexample,videoClientcanbeusedfor:1.
recordingimagesfromlocalhardwareintoafilevideoClient-ivideoin:/anydev/anynode\-ofile:demo.
vss2.
multicastingtherecordedsequenceafterapplyingadifferencefiltervideoClient-ifile:demo.
vss\-fdifference\-ovsmp://225.
0.
0.
252:55573.
displayingthemulticastedstreaminanXwindowusingOpenGLvideoClient-ivsmp://225.
0.
0.
252:5557\-oglxwindow://localhost:0Inadditiontotop-levelwindowswhichcanbemanipulatedthroughthewindowmanager,theglxwindowimagesinkcantakeadvantageofthearchitectureoftheXWindowsystemtodisplayvideoinanewsubwindowofanexistingone.
Thiscanbedonebysimplyspecifyingtheidoftheparentwindow:videoClient-oglxwindow://localhost:0parent=0x120000eThisfeatureofglxwindowcanbeusedto"augment"existingXWindowapplicationswithvideostreams.
Sincethevideowindowisachildofthehostwindow,itismoved,raised,loweredandiconifiedwithit.
Thiswayforexample,onecaneasilyaddvideotoatext-basedchatapplication.
Thisapproachcanalsobeusedwithuserinterfacetoolkitsthatexplicitlysupportwidgetsthathostexternalapplications.
Forexample,theframewidgetoftheTcl/Tktoolkitcanhostaseparateapplicationbysettingthewidget'scontainerpropertytotrue.
ThesamplecodebelowillustratestheuseofvideoClienttoembedavideostreaminaTkframe:#createaframewidgetforthevideoframe.
video-containertrue-width160-height120setid[winfoid.
video]#launchavideoClientinsidetheframeexecvideoClient\ExploringNewUsesofVideowithVideoSpace83-ivstp://host/video\-oglxwindow://localhost:0parent=$id&#maketheframevisiblepack.
video4.
3OtherVideoToolsInadditiontovideoServerandvideoClient,videoSpaceprovidesuserswithanumberofothertoolstomanipulate,displayorconvertvideostreams.
Oneofthesetools,forexample,allowstochangethewidthandheightofastream.
Anotheronecancombineseveralstreamsinasingleoneusingamosaicplacement.
AsimpleUNIXShellscriptalsoallowsusersto"stick"avideoClientonanyexistingXWindowusingtheparentparameteroftheglxwindowimagesink.
5ExploringNewUsesofVideoInthissection,IpresentseveralprojectsthatweredevelopedwithvideoSpaceandbenefittedfromitsflexibilityandextensibility.
Thedetailsoftheseprojectsunfortunatelyfallbeyondthescopeofthispaper.
Somehavebeenpublishedandsomeotherswillbe.
WhatIwanttoillustratehereishowthevarietyofimagesourcesprovidedbyvideoSpacecombinedwithitsfacilitiesfortransmitting,displayingandrecordingvideostreamssupporttherapidprototypingandincrementaldevelopmentofvideoapplications.
HowvideoSpacefacilitatestheexplorationofnewusesofvideo.
5.
1UsingDocumentsasInterfacestoAwarenessandCoordinationServicesOverthelastfewyears,IhavebeenexploringtheuseofWeb-basedvideoenvironmentstoprovideawarenessandcoordinationservicestodistributedgroupsofpeople[13].
InspiredbypreviousworkonMediaSpaces,IhaveusedvideoSpacetopromotethedevelopmentofcollaborativeenvironmentsinwhichvideocommunicationfacilitiesareembeddedintotheexistingenvironmentoftheusers,i.
e.
theirdocumentsandapplications,ratherthanprovidedasseparateapplications.
MostWebbrowserscandisplayJPEGimagesandsomeofthemcanalsodisplayanHTTPServer-PushedJPEGstreaminplaceofanordinaryimagewithoutanyplug-in.
Combinedwiththesebrowsers,videoServerallowstoincludeliveorpre-recordedvideostreamsinHTMLdocumentsbyusingcodesuchas:ByincludingsuchreferencestovideoServers,onecaneasilycreatededicatedinterfacestothevideocommunicationspace,suchastheawarenessviewofFig.
2,butalso"augment"existingdocumentsbyembeddingvideoservicesintheexistingcontent.
Forexample,onecanincludeavideolinkinane-mailmessagesothe84NicolasRousselreceiverwillseethesender'sofficewhenhereadsthemessage.
WhencooperativelyeditinganHTMLdocument,theauthorscanalsoincludeliveviewsoftheirofficesinthedocumentsotheycanseeiftheirco-authorsarepresentwhentheyworkonit.
Fig.
2.
HTML-basedawarenessviewshowingseveralvideoServersHTML-basedinterfacescanbeeasilysharedandexchanged.
AsmoreandmorepeoplearegettingfamiliarwithHTMLauthoring,theyarealsoeasilytailorable.
UsingHTMLandvideoServerstocreateaWeb-basedvideocommunicationenvironmentemphasizesthedesignprinciplethat"simplethingsshouldbesimple".
VideoServerisindeedindailyuseinseveralplacesaroundtheworld.
EndusershavecreatedHTMLinterfacestoitandhavedevelopedusagepatternswithoutanyknowledgeofthearchitectureofthesystem.
5.
2DesigningaNewCommunicationDevicetoSupportTeleconvivialityVideoconferencingsystemsaregettingclosertotechnicalperfectioneveryyear,makingthemmoreandmorepleasanttouse.
However,theusualsettingofthesesystemsfavorsafacetofacebetweenalltheparticipantsofeachsite,whichmakescross-siteinformalconversationsdifficult:beforeandafteraformalmeetingorduringpauses,peopletendtodiscusswithsomeoftheirco-locatedpartnersandoftenignoretheremotepeople.
Istronglybelievethatoneofthekeystoinformalcommunicationinthesesituationsisintheabilityforpeopletomoveawayfromthecentralfocuspointoftheformalmeetingandgatherinsmallcross-sitegroupsisolatedfromeachother.
Startingfromthisidea,severalresearchpartnersandIdesignedanewcommunicationdevice,lepuits[14](thewell),thatcombinesanSGIO2workstationwithmicrophones,cameras,speakersandahorizontalvideoprojectionsystemtoestablishanaudioandvideolinkbetweendistantgroupsofupto6people(Fig.
3).
ExploringNewUsesofVideowithVideoSpace85Fig.
3.
Lepuits:hardwareprototypeandsampledisplayconfigurations.
Thetworightmostconfigurationsaddpre-recordedsimulatedviewstotherealviewsfromtheprototypeThedesignoflepuitsrequiredseveraliterationsfromtheinitialsketchestofullyfunctionalprototypes.
WhatisinterestingtomentionhereisthatvideoSpaceallowedtodevelopthesoftwareusedtocomposetheprojectedimagesseveralhundredkilometersawayfromtheplacewherethehardwarepartswereactuallyassembled.
Pre-recordedvideostreamsshowingaroughviewofwhatthecameraswouldseewereusedtoexperimentwithdifferentcompositionmethodsevenbeforethefirstprototypewasbuilt.
Networksourceswerelaterusedtosimulateanalogvideocablesandproprietarycodecsbetweentwoprototypes.
Themosaiccomposerapplicationwasalsousedtosimulateitsanalogequivalentwhenitwasdecidedtousesuchadevicetoputallthecamerasofaprototypeonasinglevideostream.
Inadditiontosupportingrapidprototypingofthesoftwareandsimulationofvarioushardwareconfigurations,videoSpaceallowedustodevelopthesoftwarerequiredforlepuitsonalaptoprunningLinuxandtolaterrecompileitwithoutasinglemodificationandrunitontheSGIO2softheprototypes.
5.
3UsingtheHandasaTelepointerGesturingisanaturalmeansofcommunicationforhumans.
Handgesturesinparticularareoftenusedtoexpressideas,torefertoobjects,toattractattentionortosignalturntaking.
Torecreatethiscommunicationchanneloverdistance,real-timegroupwaresystemsusuallydisplaytelepointersthatparticipantscanmoveoverthesharedview.
However,standardtelepointersusuallylacksemanticinformation.
Atbest,theyarechosenamongapredefinedsetofshapesand/orcolors,whichmakesithardtodrawattention,todesignateseveralobjectsatthesametimeortoexpressanidea.
Amousecursorisaverypoorsubstituteforthehandforgesturecommunication,andsomecolleaguesandIthoughtthattheimageofthehanditself,capturedinreal-time,woulddoabettertelepointer.
Wesetupacameraaboveadeskcoveredbyalargebluesheetofpaperandrecordedseveralvideosequencesshowingsomehandgesturesoverthissolid-colorbackground.
Wethendevelopedachroma-86NicolasRousselkeyingfiltertoextracttheimageofthehandsfromthesesequencesandseveralprototypestodisplaythemoverotherimagestreamsorrunningapplications[15].
Thefirstprototypesuperimposedthechroma-keyedvideostreamoverascreen-shotofasupposedlysharedapplication.
Thisfirstimplementationgaveussomeinterestinghintsforfurthertests.
Werealized,forexample,thatthechroma-keyingprocessallowedustoannotatethesharedviewwithrealworldobjects.
Weimplementedasecondprototypeinwhichtheusercouldcontrolthesize,thepositionandthetransparencyofthechroma-keyedoverlaywiththemouseandthekeyboard.
Thissecondprototypeusedthesame"parentwindow"trickasvideoClienttooverlaythechroma-keyedvideostreamonarunningapplication.
Thissolution,however,couldn'tbeappliedtomorethanoneapplicationatthesametimeandwastoospecifictotheXWindowsystem.
Insteadoftryingothercomplexwaystosuperimposeourchroma-keyedvideostreamoveratraditionalcomputerdesktop,wesimplyusedanRFBimagestreamasthebackgroundofathirdprototype(Fig.
4).
Fig.
4.
UsingthehandasatelepointeroveranRFBimagestreamItisstilltooearlytoclaimthattheimageofthehandsareindeedbetterthanatraditionaltelepointer.
However,again,thisprojectemphasizestherapidprototypinganditerativedesignparadigmsupportedbyvideoSpace,fromthedevelopmentofthechroma-keyingfilterusingpre-recordedsequencestotheimplementationofseveralalternativedesignsusingdifferentimagesources.
Asthechroma-keyingfilterwasintegratedintothevideoSpacelibrary,thecomplexityoftheprototypeswasreducedtotheminimum:eachofthemisonly200linesofC++code.
5.
4ExploringNewDesktopInteractionsAsillustratedbythepreviousexample,usingtheimagesofcomputerdesktopsinvideoSpaceapplicationsofferssomeinterestingperspectivesforexploringnewinteractionsormetaphorsforthesedesktops.
IamcurrentlydevelopingvideoDesktop,ExploringNewUsesofVideowithVideoSpace87anewvideoSpaceapplicationthatactsasawindowmanagerforanXWindowserveraccessiblethroughRFB(anXvncserver).
Asawindowmanagerapplication,videoDesktopknowsthedimensionsofalltheapplicationwindows.
Byusingasimpletilingplacementalgorithm(Fig.
5),itisabletoextracttheimagesofindividualapplicationsfromtheRFBimagestreamandusethemastexturestocomposeanewdesktopview.
Fig.
5.
SampleviewofthetileddesktopimagestreamFigure6showsanexampleofgraphicalre-compositionoftheapplicationsshowninFig.
5.
Thiscompositionscales,rotatesandtranslateseveryapplicationandaddssometransparencyandasimpleshadow.
Initscurrentstate,videoDesktopallowstosendkeyboardeventstotheapplicationsbasedonaclick-to-focuspolicy.
MouseeventcoordinateswouldhavetobetransformedbeforesendingtheeventtotheXvncserver.
However,thistransformationisnotcurrentlyimplemented.
Fig.
6.
ViewcomposedbyvideoDesktopfromthetileddesktopimagestreamofFig.
588NicolasRousselThetechniqueusedbyvideoDesktopisquitesimilartotheredirectionmechanismoftheTaskGallery[16].
However,whereasTaskGalleryrequiresamodifiedversionofWindows2000,videoDesktopisonly800linesofcodeandrequiresonlyastandardXvncserver.
Althoughthisprojectisstillinaveryearlystage,italreadyallowstouseamoderngraphicaltoolkitsuchasOpenGLtoexperimentnewwindowmanagingtechniquesthatwereuntilnowrelyingongraphicalmodelsfromthe80sor90s.
6DiscussionThelibraryandtoolscomposingvideoSpacearetheresultofaniteratingprocessofdesignanduse.
Asthelibrarywasgettingmorecomplex,thetoolsweregettingsimpler.
Asthelevelofabstractionoftheclasseswasincreasing,thenumberoflinesofcodeofeachtoolwasdecreasing.
Inthissection,Idiscussseveralaspectsofthetoolkitthatcontributedtothisevolutionanditssuccess.
OneoftheearlykeychoiceswastheseparationbetweentheImageclass,usedasasimpledatacontainer,andthesources,filtersandsinksthatoperateonthedata.
TheimplementationoftheseimageoperatorsasindependentclassesinsteadofmethodsoftheImageclassincreasestheflexibilityofthetoolkitandfacilitatesitsextension.
Everyapplicationcaneasilydefineitsownoperatorsthatcanbecreatedandmodifiedatrun-time.
Ifthesenewoperatorsproveusefulforotherapplications,theycanthenbeintegratedintothecorelibrary.
TheJPEGencodingandtheHTTPServer-Pushprotocolwereinitiallychosenfortheireaseofuseandimplementation,nottheirperformance.
ThesamedesireofsimplicityinspiredthedesignofVSMPandVSTP.
However,videoSpaceisnotrestrictedtothesesimplechoices.
Newencodingorprotocolscanbeeasilyaddedtothelibrarytoovercomeperformanceproblemsortocommunicatewithotherapplications.
AddinganencodingsimplyconsistsinaddingaconstanttothelistofsupportedencodingsintheImageclassandprovidingafewconversionfilters.
Similarly,asillustratedbytherfbimagesource,addingaprotocolsimplyconsistsinaddingclassesderivedfromImageSourceand/orImageSinkandchangingtheassociatedfactoryfunctions.
AnotherearlykeychoicewastheuseofURLstonamevideosources.
Whenanewsourcetypeisaddedtothetoolkit,thefactoryfunctionismodifiedandallapplicationsbenefitfromitwithoutanyotherchange.
Asexplainedintheprevioussection,localsourcesorpre-recordedstreamsareoftenusedinsteadofnetworksourcesduringdevelopmenttotestanapplication,tocomparealternativedesignsortoevaluateperformance.
TheuseofURLsmakeitpossibletoswitchbetweensourceswithoutevenrecompilingtheapplication;therefore,applicationscanbedevelopedonmachinesthatdonothaveanyvideohardwareorevenanetworkconnection.
AnotheradvantageofusingURLsistheseamlessintegrationwithWebapplications,mostnotablyWebbrowsers.
ExploringNewUsesofVideowithVideoSpace897ConclusionandFutureWorkVideoSpaceisasoftwaretoolkitdesignedwithonespecificgoalinmind:facilitatetheexplorationofnewusesofvideo.
Inthispaper,Ihavedescribedtheclasslibraryandbasictoolsitprovides.
Ihavealsopresentedseveralprojectsthatillustrateitsflexibilityanditsabilitytosupportrapidprototypingandincrementaldevelopment.
VideoSpacehasnowreachedastablestateandsomeofitsapplicationsareindailyuseinseveralresearchlabs.
Itssourcecodeispubliclyavailablefordownload3.
Short-termdevelopmentplansincludetheintegrationofadditionalvideoencodingsandprotocolssuchasMPEG-4.
Iwouldalsoliketoexperimentwithfiltersfordetecting,identifyingandtrackingobjectsorpeople,basedonthetechniquespresentedin[3].
Anotherdirectionforfutureworkistheadditionofhighlevelfunctionalitiesrelatedtosessionmanagement,suchasexplicitsupportforbi-directionalormultipleconnections.
AcknowledgmentsVideoSpacegrewoutofthemanyvideoapplicationsIwrotewhileIwasworkingatLRI-UniversitéParis-SudontheTelemediaproject,fundedbyFranceTélécomR&D.
ThanksareduetoMichelBeaudouin-Lafon,whocontributedtomanyideaspresentedinthispaper.
IamalsogratefultoJacquesMartin,Jean-DominiqueGascuelandtheotherpeoplefromCSTBandiMAGISwhocontributedtothedesignandprototypingoflepuits.
CurrentworkonvideoSpaceissupportedbyERCIM,theEuropeanResearchConsortiumforInformaticsandMathematics,andtheSwissNationalScienceFoundation.
References1.
W.
Mackay.
MediaSpaces:EnvironmentsforInformalMultimediaInteraction.
InM.
Beaudouin-Lafon,editor,Computer-SupportedCo-operativeWork,TrendsinSoftwareSeries.
JohnWiley&SonsLtd,1999.
2.
H.
Ishii,M.
Kobayashi,andK.
Arita.
IterativeDesignofSeamlessCollaborationMedia.
CommunicationsoftheACM,37(8):83–97,August1994.
3.
J.
L.
Crowley,J.
Coutaz,andF.
Bérard.
Perceptualuserinterfaces:thingsthatsee.
CommunicationsoftheACM,43(3):54–64,March2000.
4.
J.
Rudd,K.
Stern,andS.
Isensee.
Lowvs.
high-fidelityprototypingdebate.
ACMinteractions,3(1):76–85,1996.
5.
M.
OlsonandS.
Bly.
ThePortlandExperience:areportonadistributedresearchgroup.
InternationalJournalofMan-MachineStudies,34:211–228,1991.
6.
C.
Cool,R.
S.
Fish,R.
E.
Kraut,andC.
M.
Lowery.
IterativeDesignofVideoCommunicationSystems.
InProc.
ofACMCSCW'92ConferenceonComputer-SupportedCooperativeWork,Toronto,Ontario,pages25–32.
ACM,NewYork,November1992.
7.
W.
BuxtonandT.
Moran.
EuroPARC'sIntegratedInteractiveIntermediaFacility(IIIF):EarlyExperiences.
InMulti-UserInterfacesandApplications,pages11–34.
S.
Gibbsand3Seehttp://www-iiuf.
unifr.
ch/~rousseln/projects/videoSpace/90NicolasRousselA.
A.
Verrijn-Stuart,North-Holland,September1990.
ProceedingsofIFIPWG8.
4Conference,Heraklion,Greece.
8.
M.
Krueger.
ArtificialRealityII.
Addison-Wesley,1991.
9.
M.
RosemanandS.
Greenberg.
BuildingRealTimeGroupwarewithGroupKit,AGroupwareToolkit.
ACMTransactionsonComputer-HumanInteraction,3(1):66–106,March1996.
10.
S.
McCanne,E.
Brewer,R.
Katz,L.
Rowe,E.
Amir,Y.
Chawathe,A.
Coopersmith,K.
Mayer-Patel,S.
Raman,A.
Schuett,D.
Simpson,A.
Swan,T-L.
Tung,D.
Wu,andB.
Smith.
TowardaCommonInfrastructureforMultimedia-NetworkingMiddleware.
InProc.
of7thIntl.
WorkshoponNetworkandOperatingSystemsSupportforDigitalAudioandVideo(NOSSDAV97),May1997.
11.
AnExplorationofDynamicDocuments.
Technicalreport,NetscapeCommunications,1995.
12.
T.
Richardson,Q.
Stafford-Fraser,K.
R.
Wood,andA.
Hopper.
VirtualNetworkComputing.
IEEEInternetComputing,2(1):33–38,Jan-Feb1998.
13.
N.
Roussel.
Mediascape:aWeb-basedMediaspace.
IEEEMultimedia,6(2):64–74,April-June1999.
14.
N.
Roussel,M.
Beaudouin-Lafon,J.
Martin,andG.
Buchner.
Terminaletsystèmedecommunication.
PatentsubmittedforapprovalbyFranceTélécomR&D(INPIn°00-08670),July2000.
15.
N.
RousselandG.
Nouvel.
Lamaincommetélépointeur.
InTome2desactesdesonzièmesjournéesfrancophonessurl'InteractionHommeMachine(IHM'99),Montpellier,pages33–36,Novembre1999.
16.
G.
Robertson,M.
vanDantzich,D.
Robbins,M.
Czerwinski,K.
Hinckley,K.
Risden,D.
Thiel,andV.
Gorokhovsky.
Thetaskgallery:a3Dwindowmanager.
InProc.
ofACMCHI2000ConferenceonHumanFactorsinComputingSystems,pages494–501.
ACM,NewYork,April2000.
DiscussionD.
Salber:Thisismoreofacommentthanaquestion-youmentionedtheprivacyprovidedforthepersoninfrontofthecamera,butwhataboutthepersonconnectingtothecameratheiraddressisrevealed.
Thereisprivacy,butitisonlyoneway.
N.
Roussel:Theuseridentityishidden,butonecannothideconnectingmachineidentity.
I'mnothighlyconcernedwiththislevelofprivacy,Idonotthinkitisamajorissue-ifsomeonewantstoconnecttome,thentheyshouldnotbeconcernedwithmeknowingwhoitis.
ThatlevelofprivacyisOK.
J.
Coutaz:Aremark:Becauseofthelackofreciprocity,Istoppedlookingatyourwebpage.
IfeltIwasa"voyeur",thatis,lookingatyouwhileyouwerenotawareofme.
N.
Roussel:Experiencewiththesystemsofarhasnotshownthistobeacommonreaction.
N.
Graham:WheredidyouuseVNCprotocolintheexampleshowingahandoveradesktop.
Ididnotrealizethiswaspossible.
N.
Roussel:VNCwasusedbecausetheexampleisnotshowingscreensharing,itisshowingaremotedisplay,thewindowshownisthatofdifferentsystem.