selected22pipi.com

HALId:hal-01437054https://hal.
inria.
fr/hal-01437054Submittedon17Jan2017HALisamulti-disciplinaryopenaccessarchiveforthedepositanddisseminationofsci-entificresearchdocuments,whethertheyarepub-lishedornot.
ThedocumentsmaycomefromteachingandresearchinstitutionsinFranceorabroad,orfrompublicorprivateresearchcenters.
L'archiveouvertepluridisciplinaireHAL,estdestinéeaudéptetàladiffusiondedocumentsscientifiquesdeniveaurecherche,publiésounon,émanantdesétablissementsd'enseignementetderecherchefranaisouétrangers,deslaboratoirespublicsouprivés.
ExpressiveKeyboards:EnrichingGesture-TypingonMobileDevicesJessalynAlvina,JosephMalloch,WendyMackayTocitethisversion:JessalynAlvina,JosephMalloch,WendyMackay.
ExpressiveKeyboards:EnrichingGesture-TypingonMobileDevices.
Proceedingsofthe29thACMSymposiumonUserInterfaceSoftwareandTech-nology(UIST2016),ACM,Oct2016,Tokyo,Japan.
pp.
583-593,10.
1145/2984511.
2984560.
hal-01437054ExpressiveKeyboards:EnrichingGesture-TypingonMobileDevicesJessalynAlvina1,2JosephMalloch1,2WendyE.
Mackay1,21Inria,2LRI,UniversiteParis-Sud,CNRS,UniversiteParis-SaclayUniversiteParis-SaclayF-91405Orsay,FranceF-91405Orsay,France{alvina,malloch,mackay}@lri.
frABSTRACTGesture-typingisanefcient,easy-to-learn,anderror-toleranttechniqueforenteringtextonsoftwarekeyboards.
Ourgoalisto"recycle"users'otherwise-unusedgesturevari-ationtocreaterichoutputundertheusers'control,withoutsacricingaccuracy.
Experiment1revealsahighlevelofexistinggesturevariation,evenforaccuratetext,andshowsthatuserscanconsciouslyvarytheirgesturesunderdifferentconditions.
WedesignedanExpressiveKeyboardforasmartphonewhichmapsinputgesturefeaturesidentiedinExper-iment1toacontinuousoutputparameterspace,i.
e.
RGBcolor.
Experiment2showsthatuserscanconsciouslymod-ifytheirgestures,whileretainingaccuracy,togeneratespe-ciccolorsastheygesture-type.
Usersaremoresuccessfulwhentheyfocusonoutputcharacteristics(suchasred)ratherthaninputcharacteristics(suchascurviness).
Wedesignedanappwithadynamicfontenginethatcontinuouslyinterpolatesbetweenseveraltypefaces,aswellascontrollingweightandrandomvariation.
Experiment3showsthat,inthecontextofamoreecologically-validconversationtask,usersenjoygeneratingmultipleformsofrichoutput.
WeconcludewithsuggestionsforhowtheExpressiveKeyboardapproachcanenhanceawidevarietyofgesturerecognitionapplications.
AuthorKeywordsContinuousInteraction;ExpressiveCommunication;GestureInput;GestureKeyboard;Mobile;TextInput.
ACMClassicationKeywordsH.
5.
2.
UserInterfaces:Inputdevicesandstrategies.
:Miscel-laneousINTRODUCTIONPeoplehavebeenwritingforthousandsofyears,usingawidevarietyoftechniques,includingcuneiformonclaytablets,carvedrunes,heiroglyphics,Chinesecaligraphy,andillumi-natedmanuscripts.
Thedevelopmentofmoveable-typeprint-ingpressesbroughtameasureofstandardizationtotext,sinceeachletterwasnolongerdirectlyproducedbyaperson.
OnceJessalynAlvina,JosephMalloch,WendyE.
Mackay.
ExpressiveKeyboard:EnrichingGesture-TypingonMobileDevices.
InProceedingsofthe29thACMSymposiumonUserInterfaceSoftwareandTechnology(UIST2016),ACM,October2016.
pp583–593.
cACM,2016.
Thisistheauthor'sversionofthework.
ItispostedherebypermissionofACMforyourpersonaluse.
Notforredistribution.
ThedenitiveversionispublishedinUIST2016,October16–19,2016,Tokyo,Japan.
ISBN978-1-4503-4189-9/16/10.
http://dx.
doi.
org/10.
1145/2984511.
2984560digitalcomputersarrived,thisuniformitybecameperfect–digitalsymbolicvaluesforeachletteraredenedaccordingtospecicschemes,e.
g.
ASCIIandUnicode,removingtheneedforreinterpretingthepossiblyambiguousvisualappear-anceofinked,carved,orotherwiserenderedtext.
Thisvaluablereductioninambiguityresultedinacorre-spondingreductioninpersonalizationthathadbeenpresentinearlierwritingsystems.
Lettersarerecordedperfectlywithstylisticinformationstoredseparately,andappliedtolarge,heavilyquantizedblocksoftext.
Forexample,thesubtleemphasisencodedimplicitlyinacontinuouslyvaryingpenstrokeisnowsimplyrenderedasastandarditalictypeface.
Alongwithaseverereductioninthegranularityofcontrol,thisapproachalsodiscardspotentiallyvaluablechannelsforimplicitcommunicationofpersonalstyle,moodoremotionalstate,andtemporalorsituationalcontexts.
Userscanofcourseeditthefont,typefacesizeandcoloroftherenderedtext,butthisisnecessarilyseparatefromtheactualtextinput.
Computerkeyboardsareusuallyconstructedasanarrayofla-beledmomentaryswitches(buttons),butinterestingly,mostmobiledevicescapturetextinputvia"soft"keyboardsdis-playedonhigh-resolution2Dtouchscreens.
Thus,althoughtheoutputissymbolic,theinputishighlyoversampledinbothspaceandtime,givingustheopportunitytoexploremorecontinuousformsofcontrol.
Forexample,dynamickey-targetresizingbasedonmodelsoflikelywordsorlettersequencesincreasestheapparentaccuracyofsoftkeyboardsandpartiallyresolvesthe"fat-nger"problem[13].
Gesture-typing[24]isamoreinterestingalternativethatof-fersanefcient,easy-to-learn,anderror-tolerantapproachforproducingtypedtext.
Insteadoftappingkeys,usersdrawtheshapeofeachword,beginningwiththerstletterandcontin-uingthroughtheremainingletters.
Typically,arecognitionenginecompareseachwordgesturetoapre-designed"tem-plate"representingtheidealwordshape.
Word-gesturesarenotuniqueforeachword,butcanberobustlymatchedusingacombinationofkinematicmodels,multidimensionaldistancemetrics,andlanguagemodelstoresolveambiguities.
Ges-turesthatvarysignicantlymaythusstillregisterascorrect.
Aswithothersoftkeyboards,thegoalofgesture-typingkey-boardsistoproducethesingle,"correct"typedwordintendedbytheuser;itiseithercorrectorincorrect,andinputvari-ationisofinterestonlyforthepurposeofdesigningtoler-antrecognitionsystems.
Gesturevariationistreatedessen-tiallyasadeformationofthecorrectshapeanddiscarded1asunwantednoise.
Tobesure,asmallpartofthevaria-tionismotor-systemordigitizernoise,andcannotbeconsid-eredmeaningful.
Howeverhumanexperiencewithhandwrit-ingclearlyshowsthepotentialforpersonalandstylistically-communicativevariationofoutputmediathroughperformedhumangestures.
Whatifwecouldleverageatleastpartofthenaturalvari-ationingesture-typingtoincreasetherichnessandnuanceoftext-basedcommunicationchannelsMobiledevicesal-readyincludehigh-resolutionsensorscapableofmeasuringthevariation,andcommercializedgesturetypingsystemsarewidelyinstalledandarealreadydesignedtotoleratedeforma-tionsofthe"ideal"gesturetemplate.
Capturingcontinuousfeaturesofthevariationandmappingittopropertiesoftherenderedtextcouldre-enablesomeofthebenetsofhand-writing,suchasrecognizablepersonalstyles,implicitcom-municationofmood,activity,orcontext;andexplicitcom-municationofemphasis,sarcasm,humor,andexcitement.
ExpressiveKeyboardsWeintroduceExpressiveKeyboards,anapproachthattakesadvantageofrichvariationingesture-typedinputtopro-duceexpressiveoutput.
Ourgoalistoincreaseinformationtransferintextualcommunicationwithaninstrumentthaten-ablesuserstoexpressthemselvesthroughpersonalstyleandthroughintentionalcontrol.
Thisapproachaddsalayerofgestureanalysis,separatefromtherecognitionprocess,thatquantiesthedifferencesbetweenthegesturetemplateandthegestureactuallydrawnonthekeyboard.
Thesefeaturescanthenbemappedtooutputpropertiesandrenderedasrichoutput.
BeforewecanbuildanExpressiveKeyboard,wemustrstaddressfourresearchquestions:1.
Doesgesture-typingperformanceactuallyvarysubstan-tiallyacrossusers(duetobiomechanicsorpersonality),orcontext(activityorenvironment)2.
Canthisvariationbequantiedasdetectiblefeatures3.
Canusersdeliberatelycontroltheseadditionalfeaturesoftheirgestureswhilegesturetypingrealtext4.
HowdousersappropriateExpressiveKeyboardsinamorerealisticsettingThispaperpresentsrelatedwork,andthenattemptstoanswertheaboveresearchquestionsthroughaseriesofexperimentsandsoftwareprototypes.
Experiment1isdesignedtover-ifywhethergesture-typingperformancevariesacrosspartici-pantsandexperimentalconditions.
Wereporttheresultsandhowtheyledtotheselectionofthreefeaturesthatformalow-dimensionalrepresentationofgesturevariation.
Experiment2isdesignedtotestwhetherornotuserscandeliberatelyvaryboththeselectedfeaturesandtheparametersoftherenderedoutputtextusingasimpliedcontrolmapping,whilesimul-taneouslytypingtherequiredtext.
Wereportontheresultsandhowtheyinuencedthedesignofasecondprototype,whichmapsusers'gesturestoadynamicfont.
Experiment3isdesignedtocollectecologicallyvalidin-the-wilddatacon-sistingofreal-worldconversationsbetweenpairsoffriends.
Wereporttheresultofthisstudy,aswellasusers'perceptionsofthedynamicfont.
Weconcludewithdirectionsforfutureresearch,includingadditionalmappingsandapplications.
RELATEDWORKMuchoftheresearchondigitalwritingusesmachinelearningtoimprovecontentrecognition,e.
g.
,bypredictingthemostlikelywordfromthecontext(auto-completion)[12]orbyimprovingspellingorgrammar(auto-correction)[10].
Thesesystemsseektopredicttheuser'sintention,atsomelevelofprobability,toproducethe"correct"outcome.
Ineachcase,theoutputisxed:typingonbothhardandsoftkeyboardsproducesstandardoutputthatlacksnon-verbalcues[23].
Userssometimesuseboldoritalictypefaces,orALLCAPStoemphasizeablockoftext.
Toconveymoresubtleexpression,usersmayalsoinsertemoticons,eitherbyselectingthemfromamenu;typingaparticularkeyword,e.
g.
,'sad'toproduce,drawingagesture[21],orthroughanemoticonrecommendationsystem[22].
However,theactofselectinganemoticonisnotintegraltotheproductionofthetextandcaneasilydistracttheuserfromtheactofwrit-ing[1,21].
Thedegreeofexpressionisalsolimitedtothepre-denedsetofemoticons.
EnhancingText-basedCommunicationSomeresearchershaveexploredhowtosupportsubtleexpressionintext-basedcommunication.
Forexample,EmoteMail[1]annotatesemailparagraphswiththesender'scompositiontimeandfacialexpression.
KeyStrokes[19]usesshapesandcolorstovisualizetypingstyleandtextcontent.
Iwasakietal.
[15]addedsensorstoaphysicalkeyboardtocapturetypingpressureandspeed.
Mobiledevicesoffernewpossibilitiesforgeneratingrichtext,giventheirtouchscreensandmultiplesensorscapableofcapturingtemporal,spatialandcontextualfeatures.
Forexample,Azenkot&Zhai[3]investigatedhowuserstypeonsoftkeyboardsandfoundthattouchoffsetsvaryaccordingtohowtheyholdthedevice.
Buscheketal.
[6]combinedtouchoffset,key-holdtime,anddeviceorientationtodynamicallypersonalizethefont.
GestureasanExpressiveInstrumentAthirdalternativeistousegestures.
Researcherswhostudygestureformusicordanceoftentakeacompletelydiffer-entperspective,emphasizingthecontinuousqualitiesofhu-mangesturesoverrecognition:individualvariationisvaluedratherthanignoredorrejected.
Theseresearcherscharac-terizegesturevariationintermsofqualitiesofmovement:spatialfeatures[6,7]);temporalfeatures;continuity;power;pressure;activation;andrepetitions[9].
Thisapproachtostudyingandusinggesturecontrastswithdenitionsofthetermfromlinguisticsandcognitivepsychol-ogy.
SeeMcNeill[18]foramorein-depthdiscussionofthecompetingconceptualunderstandingsoftheterm'gesture'.
Intheartisticdomain,therichnessofgesturecanbetrans-formedintocontinuousoutput,e.
g.
,[11],ortoinvokeacom-mand[16]inamoreintegratedinteraction.
Ifthegoalistomakethesystem'fun'andchallenging,thesystemshouldencouragecuriosity[19].
Huntetal.
[14]foundthatcontinu-ous,multi-parametricmappingsencouragepeopletointerpretandexploregestures,althoughlearningthesemappingstakestime.
Humangesturevariationcanalsobeaffectedbymove-mentcost[20],interactionmetaphorsandsystembehavior.
2QUANTIFYINGVARIATIONINGESTURE-TYPINGWeconductedawithin-participantsexperimentwiththreetypesofINSTRUCTIONastheprimaryfactor:Participantsgesturetypespeciedwords"asaccuratelyaspossible";"asquicklyaspossiblewhilestillbeingaccurate";and"ascre-ativelyaspossible,havefun!
"Theaccuratelyconditionshouldprovidetheminimumlevelofvariabilityfornovicegesture-typistsastheytrytomatchthewordshapeascloselytothetemplateaspossible.
Thequicklyconditionmightreal-isticallybefoundinreal-lifegesture-typingundertimecon-straints,andpresumablyresultsingreatervariabilityanddi-vergencefromthetemplate.
Thecreativelyconditionwasde-signedtoprovokemoreextremevariation,andisnotintendedtomatchareal-worldgesture-typingscenario.
Wechosethreesetsof12wordsthatvarysystematicallyac-cordingtothreedimensions:length(SHORT4characters);angle(ZERO,ACUTE,orOBTUSE);andletterrepetition(SINGLE,e.
g.
,lose,orDOUBLE,e.
g.
,loose).
Weconsideranglebetweenstrokesegmentsbecauseitmayaffectperformance[20].
Forexample,thewordpureeislong,withadoubleletter'e',andazerodrawingangle,i.
e.
astraightlineonthekeyboard;taxiisshort,withasinglelet-terandatleastoneobtuseangle:thechunkaxi.
Eachletterappearsatleastonceineachset.
ParticipantsWerecruitedsevenmenandvewomen,allright-handed,meanage26.
Allusemobilephonesdaily,butnonehadusedgesture-typingpriortothisexperiment.
ApparatusWedevelopedacustomAndroidapplicationrunningonanLGNexus5(Android5.
1)smartphone.
Itdisplaysanon-interactiveWizard-of-Oz(WOZ)keyboardthatmatchesthepositionanddimensionsofastandardQWERTYEnglishkeyboard.
Weusethekeyboardevaluationtechniquede-scribedin[4];theWOZkeyboardcollectsgesturecoordinatesthatarelaterfedtoaword-gesturerecognizer(keyboard).
ProcedureSessionslastapproximately50minutes.
Participantssitinachairandholdthephonecomfortablyintheirlefthands,sotheycanperformallgestureswiththeirrightindexnger.
Participantsareencouragedtotalkaloudastheydraweachword.
Duringinitialtraining,participantsmaypracticeuntiltheyfeelcomfortableusingthegesture-typingtechnique.
Eachtrialintheexperimentbeginswithaninstructiondis-playedatthetopofthescreen,e.
g.
,"Drawasaccuratelyaspossible",withawordcenteredbelow,e.
g.
,queue,andasoftkeyboardatthebottomofthescreen(seeFig.
1a).
Thetrialendswhenparticipantslifttheirnger,afterwhichtheyansweramultiple-choicequestionastotheirlevelofcon-dence:"Doyouthinkyouwrotevein"(Yes,No,orNotsure).
Eachwordispresentedasasub-blockwith10replications.
Theexperimentconsistsof360trials(12wordsx3instruc-tionsx10replications).
Allparticipantsbeginwiththeaccu-ratelyinstruction;quicklyandcreativelyarecounterbalancedfororderacrossparticipants.
The12wordsarechosenfromFigure1.
Gesturevariations:a)Accuratelyisstraight,b)quicklyissmooth,andc)creativelyisinatedandhighlyvaried.
thethreewordsets;counter-balancedwithinandacrosspar-ticipants.
DataCollectionWerecordthetouchcoordinatesinordertoextractspatialandtemporalcharacteristicsofeachgesture.
Welatersim-ulatethegesturedataongesture-typingrecognizers,KB-1andKB-2,toderiveACCURACY,i.
e.
therecognizerscorefortheintendedword(True=1,False=0).
Wealsorecordtheparticipant'sCONFIDENCERATE–anordinalmeasureofthepost-trialanswers(Yes=1,NotSure=0.
5,No=0).
Thepost-questionnaireasksparticipantstodescribehowtheyvariedtheirgesturesaccordingtoeachinstruction.
Wealsorecordakinematiclogofeachgesture,usingscreencapture,andaudiorecordtheparticipant'sverbalcomments.
ResultsandDiscussionTherstresearchquestionconcernstheextenttowhichtheparticipant'sgesturesvaryastheygesture-type.
Werstex-aminedthesubjectivemeasuresobtainedthroughthepost-questionnaireandlookedattheexistingvariabilityingesturedatatoidentifycandidatesforgesturefeatures.
Wecollected4320uniquegestures.
Weremoved22outliers(0.
5%),denedaswhen1)aparticipantsaidtheymadeamistake,e.
g.
accidentallyliftingthengerbeforenishingthegesture;2)theyanswerednotothepost-trialquestion;and3)gesturelengthwas12;whilecurvesandloops(obtusecorners)are100/255),andatleastoneotherRGBcolorcomponentinRGBthatdiffersfromtheotherphrases(difference≥20/255).
FulllingbothconditionsresultsinCORRECTRATE=2,whereasfulllingonlyoneresultsinCORRECTRATE=1.
Wecountnumberoferrorsbasedonhowmanytimestheparticipanterasedawordbeforesubmittingaresults.
Werecordthescreenandaudiothroughouttocaptureverbalcomments.
ResultsParticipantsweregenerallyabletocontrolthevariationintheirgestures(overallCORRECTRATEis1.
3outof2.
0),butwereunabletofullymeetthegoalsofeachcondition.
AnANOVAshowedthatBothCONTROLandINSTRUCTIONsig-nicantlyaffectCORRECTRATE(F1,11=18.
5andF2,22=28.
7respectively,allp2.
0,F1,11=27,p2.
9,F2,22=4.
3,p<.
0001).
ConveyingEmotion&WritingforDifferentRecipientsParticipantsvariedtheirgestureswhenexpressingcertainemotionsorwhenwritingtodifferentrecipients.
Participantsuseddifferentstrategies:sixdeliberatelyvariedtheirgestureinput;vevariedtheirgestureoutput;andonlyonepartici-pantvariedboth.
Whenthehypotheticalrecipientwastheirbossortheirpar-ent,veparticipantsreportedtheywantedtomakethetextcolordarker,andthuswrotemoreslowlyandaccurately.
Incontrast,whenwritingtoaclosefriendorchildrelative,sevenparticipantssaidtheydrewmoreslowly,withcurviergesturesanddetours,resultinginbrightercolors(Fig.
5).
Threeparticipantschosepinkorredtowritetotheirpartners;P9drewaheartshapethatleftthekeyboardarea.
Participantsexpressednegativeemotionsusingslower,straightergestures,resultingindarkercolors.
Fourparticipantsassociated'an-gry'withgreaterspeedandmostexpressedbeingbusybydrawingfaster,curviergestures.
Onlyoneparticipantstatedthattheydidnotchangetheirstyleofgesture-typing.
DiscussionParticipantswereabletocontrolaspectsoftheirgesturestoproducetheirintendedoutcomes,althoughperhapsnotasconsistentlyastheywouldlike.
Participantsfounditeasiertocontroltheoutputoftheirgestures,i.
e.
toexplicitlycon-trolthecolorofaphrase,thantocontrolthecharacteristicsoftheirinput,i.
e.
tocontrolthecurviness,sizeandspeedoftheirgestures.
Surprisingly,only25%(3/12)wereabletocorrectlyguessthemappingbetweencolorandtheirgestures.
Someparticipantstookadvantageofthevariationsinthecolorfeedbacktoreectuponandmodifytheirperformance,whichwasnotpossiblefortheinputconditions.
Thissuggeststhatparticipantsfacedatrade-offduringthersttimeusingExpressiveKeyboard:theyexpendedmoreeffortrewritingthephraseuntiltheygotthedesirablecolors,butintheendweremoresuccessfulinfulllingtheinstructions.
Clearly,theparticipantsneedmoretimetopractice,sincemostparticipantswereusinggesture-typingforthersttime.
Whilebothtypesofaccuracymustbeimproved,webelievethisisapromisingstart:noviceusersareunlikelytohaveaclearunderstandingofhowgesturerecognitionalgorithmswork,butthisshouldnotpreventthemfromgeneratingrichoutputbyvaryingtheirgestures.
Participantsdemonstratedthattheyarecapableofdrawingcertaintypesofgestures,e.
g.
extremelycurvyorlargeges-tures,eveniftheydonotchoosetodosowhentypingwithoutrichoutput.
Bycontrast,participantshavedifcultymain-tainingaconstantdrawingspeed.
Gesturestypicallystartquicklyandthenslowdown,whichwithourmappingpro-ducesacolorgradientintheVaried-Outputcondition.
Thisshouldbeaparticularlysimplegesturetocontrol,butinter-estingly,onlythreefounditeasyandnonecouldarticulatehowitworks.
Figure4.
Participantscanintentionallycontrolgesturesizewhenasked(BoundingBoxRatio),butdonotvaryitotherwise.
7Figure5.
Successfulcontrolofcolorthroughgesture:a)[instruction:Different-Output]bright-greenindicatescurviness,darkgreenindicatesstraightlines;b)[instruction:Varied-Input]inatingthegestureincreasedredvalues;c)[instruction:recipients]P8changedthecolordeliberatelyfordifferentrecipients;d)[instruction:expressemotion]P4madecurviergestureswithmoredetoursfor'happy',andslowandlesscurvygesturesfor'sad'.
Weobservedlargeindividualdifferencesacrossparticipantswithrespecttotheabovefeatures.
Perhapsnotsurprisingly,thissuggeststhateachindividualislikelytoappropriateex-pressivegesturesintheirownway,andgeneratedistinctive,personalgesturestyles,justastheydowiththeirhandwrit-ing.
EXPERIMENT3:ECOLOGICALVALIDITYWedesignedathirdexperimenttoexplorehowExpressiveKeyboardsareusedinamoreecologically-validsetting,andwhenmappedtomorecomplexfeaturesthancolor.
Theex-perimentisa[2x2]repeatedmeasureswithtwofactors:KEY-BOARDTYPE{baselinekeyboard,ExpressiveKeyboard}andTEXTTYPE{user-generated,prescribed}.
Thebaselinekey-boardisastandardgesture-typingkeyboard.
TheExpressiveKeyboardgeneratesadynamicfontwhichshapeandcolorschangesdependingonthegesturefeatures.
ImplementationWeimplementedanExpressiveKeyboardthatmapsgesturefeaturestoauser-deneabledynamicfont.
Thedynamicfontiscreatedthroughasimpleapplicationwedevelopedthatletsusersdraweachletterbydeningcontrolpoints.
Userscancreateseveraltypefacesanddynamicallyinterpolatebe-tweenthemtogeneratenewintermediatefontscontinuously.
Theinterpolationbetweenntypefaceschangesthepositionofcomponentcontrolpointsbasedonaweightingfunction:typefaceinterpolated=nj=1typefacejweightjnj=1weightj(5)whereeachtypefaceisavectorofcontrolpointsforeachlet-ter.
ForExperiment3,wepredenedafontwithn=2;onetype-faceismoreskewed(italic)thantheother.
WeusedthesamegesturefeaturesasforExperiment2,withthespeedconsis-tencyratiomappedtotheweightratioforfontinterpolation;inationratiomappedtostrokethickness(bold);andcurvi-nessmappedtothemagnitudeofrandomoffsetsappliedtoeachcontrolpoint.
WealsousedthecolormappingusedinExperiment2.
ParticipantsWerecruitedsixpairsoffriends,sevenmenandvewomen(agerange19-40,mean25.
4);allusemobilephonesdaily.
Halfofthemusegesture-typingdaily,theothersarenon-users.
Noparticipantshadparticipatedinthetwopreviousexperiments.
ApparatusFortherstsession,wecustomizedanAndroidchatapplica-tion3tocapturethegesturedataaswellasthetypedwords.
Forthesecondsession,wedevelopedacustomAndroidap-plicationthatpresentseitherthebaselinekeyboardoranEx-pressiveKeyboardthatrendersthedynamicfont.
Inaddition,wedevelopedasimplefontenginethatletsusersdenestatictypefacesbyconnectingcontrolpointstoformeachletter;thissoftwarewasusedtodesignthefontsetsusedintheex-perimentbutnotintheexperimentitself.
Forrecognition,weusedKB-2onthesameLGNexus5(Android5.
1)smart-phoneasinthepreviousexperiments.
ProcedureParticipantssitcomfortablyinachairwhilegesture-typing.
Theexperimentisdividedintothreesessions.
Intherstses-sion,wesetupliveconversationsbetweenpairsofpartici-pantstocollectgesture-typedtextsinanaturalsetting.
Eachpairofparticipantschatfor15minuteswithoutanyrestrictiononhowtogesture-type.
Forthesecondsession,weselectvesentencesfromthechatastheuser-generatedtext.
Thesecondsessionconsistsofthreeblocksofvetrialseach.
Thisisanindividualtaskwheretheparticipanthastowriteveuser-generatedandveprescribedsentences(fromnews,blogs,etc).
Participantsareinstructedtogesture-typeasifwritingtotheirpeerandtoassumethepeerwillseethesameoutput.
Thesessionalwaysstartswiththeprescribedtextwithbaselinekeyboard,followedbyanintroductiontotheExpressiveKeyboardandthemappingused.
Participantsareencouragedtopracticetounderstandhowthesystemworks;noparticipantpracticedlongerthanveminutes.
Forthenexttwoblocksparticipantsareaskedtowriteboththepre-scribedanduser-generatedtexts(counter-balancedacrossparticipants).
WedonotspecicallytellthemhowtouseEx-pressiveKeyboardandletthemuseitastheylike.
Through-outthesession,weasktheparticipantstodescribealoudwhattheywanttodoandwhattheyarethinking.
Thethirdsessionisaquiz(threeblocksofthreetrials).
Weaskthemtogesture-type"hello"threetimeswithspecicoutputgoals:1)boldandred,2)italicandcontainingblue,3)green.
Finally,weinterviewthemregardingtheirpreferenceandholdaminibrainstormingsessiononhowtheymightdenetheirownfeaturesandmappingiftheycould.
Anexperimentsessionlastfor40minutes.
3AndroidHive:http://www.
androidhive.
info/8KEYBOARDTYPETEXTTYPEUser-generatedPrescribedWORDACCURACYBaselineKeyboard95%75.
4%ExpressiveKeyboard68.
5%79.
5%FEATUREACCURACYExpressiveKeyboard55.
9%61.
7%Table3.
AccuracyinExperiment3.
Participantschangedthewaytheygesture-typewithdifferentkeyboards,andaremorelikelytoex-plorewithExpressiveKeyboard.
AlthoughusingExpressiveKeyboarddoesnotnecessarilydecreaseword-accuracy,participantserasedcorrectwordstomodifytheoutputpropertiesleadingtolowvaluesforfeature-accuracy.
DataCollectionWecalculatethethreegesturefeaturesaswellasWORDAC-CURACYandFEATUREACCURACYasinExperiment2.
Welogthetimestampand2Dcoordinateofeachtouchevent,andrecordthescreenandaudiotocaptureverbalcomments.
ResultsandDiscussionOutof2312performedgestures,weremoved183thatwerenotgesture-typed,mostofwhichweresingle-letterwords.
Forthequiz,wecollected108gesturesandremoved2out-lierswheretheparticipantshadliftedtheirngeratthestartofthegesture.
StandardGesture-Keyboardvs.
ExpressiveKeyboardWerananANOVAtesttocompareperformanceofKEYBOARDTYPEandTEXTTYPE.
WORDACCURACYissignicantlyaffectedbyKEYBOARDTYPE(F1,11=30.
7,p<0.
0001).
ThereisasignicantinteractionbetweenKEY-BOARDTYPEandTEXTTYPE(F1,11=15,p=.
0001).
How-ever,furtheranalysiswithTukeyHSDshowedthattheonlysignicantdifferenceisfoundwhentheparticipantswroteuser-generatedtextusingbaselinekeyboardcomparedtoEx-pressiveKeyboard.
Therewasnosignicantdifferencebe-tweenbaselineandExpressiveKeyboardwhenwritingapre-scribedtext.
ThissuggeststhattheuseoftheExpressiveKeyboarddoesnotnecessarilydecreaseWORDACCURACY;insteadthecontextwhenchattingmayhelptheparticipantsbettermatchthedesignparametersoftherecognizer.
However,withExpressiveKeyboard,therewerecasesinwhichtheparticipantserasedacorrectly-recognizedwordtomodifytheoutputproperties:18.
9%foruser-generatedand23.
3%forprescribed.
TheoverallFEATUREACCU-RACY,whichrepresentsaccuracyofExpressiveKeyboardisshowninTable3.
ThetimespenttodrawagesturealsosignicantlyincreasedwhenusingExpressiveKeyboard(F1,11=64.
9,p<.
0001),mean2secondsperword,whiletheyspenttheleasttimewhenchatting(0.
7spw).
ThereisasignicanteffectofKEYBOARDTYPEoninationandcurviness(F1,11=30.
6,p<.
0001andF1,11=5.
6,p<.
0177respectively).
Theparticipantssignicantlyinatedtheirges-tureswhenusingExpressiveKeyboard(mean1.
6)ascom-paredtobaselinekeyboard(mean1.
2).
Thereisnosignicantdifferencewithregardsofthespeedconsistency.
Figure6.
UsingExpressiveKeyboardinExperiment3:a)P2naturallymadecurvygesture(green)butmadestraightgestures(darkgreen)toemphasizesomewords,b)P6deliberatelymadetwowords("burgershop")thesameshapeandcolor,c)P3emphasizedtherstword,butthendeliberatelymadehisgesturemoreprecise(darkcolor)insteadofcurvy.
Fromthepost-questionnaire,welearnedthattheparticipantstookadvantageofthefactthattheycouldchangetheoutputproperties.
Allparticipantsstatedthattheychangedthewaytheygesture-typedwithExpressiveKeyboard.
Mostusedittohighlightaspecicwordorphraseinthesentence,ratherthantryingtocontroltheappearanceofeveryword.
Twoofthemmentionedtheychangedthepropertiestomatchtheirintonationwhenreadingthesentence.
Threeofthemstatedthattheyexpressedtheirfeelingsormoodwhenwriting,e.
g.
"whenit'ssomethinghappyItriedtowritefaster[so]thattextbecomesgreenandblue"(P6).
AninterestingappropriationofExpressiveKeyboardistore-ectontheirowngesture-typinghabits.
P1,P10,andP11realizedthattheytendtomakecurvygestures,andtheyde-liberatelylettheoutputchangeaccordingtotheirnaturalin-putstyle.
Onthecontrary,P3andP8madeaspecialef-forttomaketheoutputtextassimilaraspossible(e.
g.
allblack).
Thissuggeststhatcontinuouschangestooutputprop-ertiescanprovideimportantfeedbackfortheparticipantsandmaychangetheirbehaviourwhengesture-typing–notonlytocustomizetheoutput,butalsototrytogesture-typemoreprecisely.
ControlofGestureFeaturesInthequizsection,theaverageWORDACCURACYwas86%.
Participantsweremostaccuratewhencontrollingspeedcon-sistency(33outof36trials,or92%)andcurviness(32/36or89%),butlessaccuratewhencontrollingination(26/34or76%).
Furtheranalysisrevealedthattherecognitionerrorwascausedby1)toomuchdeformation(6outof15errors),2)faultystartorendposition(6/15),and3)removingthen-gertooearly(3/15).
Thissuggeststhatwhilemanyfactorsaffectrecognitionrate,certaintypesofintentionalvariationmayincreaseerrortoasmallextent.
Theinstructionhadasignicanteffectonallthegesturefea-tures(allp<.
0001).
Apost-hoctestwithTukeyHSDshowedasignicantdifferencebetweentheinstruction"italic"andtheothers.
InFig.
7,wecanseethatingeneralthepartici-pantsspeed-up(meanrate=1.
5)withhighervariabilitywhenaskedtomaketheoutputitalic;howeverboththevalueandthevariabilitydroppedforinstructionswhichrequiredthemtokeepthespeedconstant(0.
86and0.
84for"bold"and"green"respectively).
Similarsignicantdifferencesalsoappearforinationrate.
Pairwiset-testswithTukeyHSDshowedthatthereisasig-9Figure7.
Participantssuccessfullyvariedorkeptconstanttheinationandspeedconsistencywheninstructed.
Theynaturallymadecurvyges-tures,butcouldmakethemmorecurvywheninstructed.
nicantdifferencebetweentheinstruction"bold"andtheoth-ers.
However,for"green",theinationrateisabithigher(mean=1.
6).
Basedonourobservation,thisisbecausetheparticipantsovershotwhentryingtomakeacurvygesture.
Meanwhile,thegesturesarequitecurvyforallinstructions:withmeans10.
9,9.
9,and13.
8for"bold","green",and"italic",respectively(avalueof12indicatesminimumcurvi-ness).
Gesturesaremostcurvywheninstructedimplicitly,however,itisnotsignicantlydifferentfrom"bold".
Thissuggeststhatinatingthegesturealsoincreasedcurviness.
Overall,participantsnaturallymadecurvygestures,butwereabletoincreasecurvinesswhennecessary.
GENERALDISCUSSIONWhenprovidedthetools,participantstookadvantageofthevariationintheirgesturestoenhancecommunication.
Experi-ment3alsoconrmsthatparticipantswereabletoselectivelycontrolspeedandination,andnaturallymadecurvyges-tures.
NineparticipantsfeltthatusingExpressiveKeyboardwasmoreenjoyablethanusingastandardgesture-typingkey-board(1neutral,2disagree).
Thismeansthatdespitethefactthatitismoretime-consumingandlesserror-prone,thepar-ticipantsweremorewillingtoengageingesture-typingitself.
Sinceallparticipantswerenecessarilynoviceusers,amorelongitudinalstudyofExpressiveKeyboardsisneededtode-terminewhetheraccuracyimproveswithexperience.
Addi-tionalfactorsmayalsoaffectuserperformancewhenusingExpressiveKeyboard,forexample,inExperiment3sixpar-ticipantsmentionedthattheysometimeschangedtheirhandpositiontoincreasetheprecisionoftheirngeringesturing,e.
g.
fromusingthumbtoindexnger.
Whileourinvestigationmainlyfocusedonintentionalvaria-tion,webelieveunintentionalvariationisasinterestingandcanenrichinter-personalcommunication.
Simplymappingthisvariationtosmalldifferencesintherenderedoutputwouldgenerateadegreeofexpressionthatcouldberecog-nizableaspersonalstyles.
Ausermaynotspecicallycon-troltheirgesturewhenwritingonabumpybus,yettheoutputcanreectthewritingprocess–somethingthathandwritingcancaptureeasily.
Consciouscontrolofthisvariationwouldnotberequiredtoimplicitlycommunicatestyle,personality,context,ormood.
Wearenotinterestedin'identifying'ex-pressionoremotionbasedongestures,sincewebelievericherinterpretationswillberesultfromasysteminwhichuserscandeveloptheirowncommunicationcontextsandrelatedmean-ings.
Inrealusage,usersshouldbeabletovarythesensitivityoftheoutputvariationorturnitoffcompletely[2],especiallyincaseswheretheuserpreferstogeneratemoreformaloutput.
Usersshouldalsobeableto(re)designtheirownfeaturede-tectors,text-renderingproperties,andmappingslinkingthetwo.
Finally,wearenotseekingaslavishrecreationoftradi-tionalhandwriting,norareplacementforothercommunica-tionchannelssuchasemoticons.
Instead,weaimfornewformsofnuancedtextualcommunicationsupportedbyandinteractingwithnewtechnologies:touchscreens,probabilis-ticlanguagemodels,vectorgraphicfonts,etc.
Theprecisewaysinwhichthesenewopportunitieswouldbeusedforcommunicatingstyleoremotionalexpressionlikelycannotbedesigned,butmustemergethroughappropriationofthesystembycommunitiesofusers.
CONCLUSIONANDFUTUREWORKThispaperproposesanovelapproachforgesture-typingthatmapsgesturevariationtooneormorecontinuouspropertiesoftherenderedtext,producingrichoutputthatisbothac-curateandunderusercontrol.
UsingExpressiveKeyboards,userscansimultaneouslycontrolbothtextcontentandstyle–capturedaschangesinshape,weight,colororotheroutputparameters.
Throughaseriesofexperiments,weestablishedthatusersdoindeedvarytheirgesture-typinginquantiableways,thattheycancontrolsuchvariationsintentionally,andthattheyndasystemthatusesthevariationtoenrichtherenderedtextinterestingandenjoyable.
Thisworkvalidatessomeoftheconceptsbehindleverag-ingvariationingesture-typingbehavior,openingupawiderangeofresearchdirections.
Inadditiontoimprovementsinthevarietyandqualityofcalculatedgesturefeaturesandourdynamicfontengine,weareinvestigatinghowExpressiveKeyboardsmightbeusedtoimprovegesture-typingaccuracythroughprogressivefeedback.
Wealsoplantoexpandthedifferentformsofoutputthatcanbemappedtousergesture,includingparametricemoji,hand-writing,andnuancedspeechsynthesis.
WeareparticularlyinterestedinstudyingtheuseofExpressiveKeyboardsinecologicalsettingstoseehowitisappropriatedforprovidingnewtypesofdigitalcommunication.
ACKNOWLEDGMENTSThisresearchwassupportedbyERCgrantnumber:321135,CREATIV:CreatingCo-AdaptiveHuman-ComputerPartner-ships.
WewouldliketothankShuminZhaiforhishelpfulcommentsonearlierdrafts,aswellasmembersofExSitufortheirvaluablesuggestions.
10REFERENCES1.
¨Angeslev¨a,J.
,Reynolds,C.
,andOModhrain,S.
Emotemail.
InACMSIGGRAPH2004Posters,R.
Barzel,Ed.
,ACMPress(2004).
2.
Arif,A.
S.
,Kim,S.
,Stuerzlinger,W.
,Lee,G.
,andMazalek,A.
Evaluationofasmart-restorablebackspacetechniquetofacilitatetextentryerrorcorrection.
InProceedingsofthe2016CHIConferenceonHumanFactorsinComputingSystems,CHI'16,ACM(NewYork,NY,USA,2016),5151–5162.
3.
Azenkot,S.
,andZhai,S.
Touchbehaviorwithdifferentposturesonsoftsmartphonekeyboards.
InProc.
ACMMobileHCI2012,ACMPress(2012),251–260.
4.
Bi,X.
,Azenkot,S.
,Partridge,K.
,andZhai,S.
Octopus:evaluatingtouchscreenkeyboardcorrectionandrecognitionalgorithmsvia"remulation".
InProc.
ACMCHI2013,ACMPress(2013),543–552.
5.
Burgbacher,U.
,andHinrichs,K.
Modellinghumanperformanceofstroke-basedtextentry.
InProc.
ACMCHI2015,ACMPress(2015),137–143.
6.
Buschek,D.
,DeLuca,A.
,andAlt,F.
Thereismoretotypingthanspeed:Expressivemobiletouchkeyboardsviadynamicfontpersonalisation.
InProc.
ACMMobileHCI'15,ACMPress(2015),125–130.
7.
Cao,X.
,andZhai,S.
Modellinghumanperformanceofpenstrokegestures.
InProc.
ACMCHI2007,ACMPress(2007),1495–1504.
8.
Caramiaux,B.
,Bevilacqua,F.
,andTanaka,A.
Beyondrecognition:usinggesturevariationforcontinuousinteraction.
InACMCHIEA2013,ACMPress(2013),2109–2118.
9.
Caramiaux,B.
,Donnarumma,M.
,andTanaka,A.
Understandinggestureexpressivitythroughmusclesensing.
ACMTrans.
Comput.
-Hum.
Interact.
21,6(Jan.
2015),31:1–31:26.
10.
Dunlop,M.
,andLevine,J.
Multidimensionalparetooptimizationoftouchscreenkeyboardsforspeed,familiarityandimprovedspellchecking.
InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems,CHI'12,ACM(NewYork,NY,USA,2012),2669–2678.
11.
Gody,R.
I.
,andLeman,M.
,Eds.
MusicalGestures:Sound,Movement,andMeaning.
Routledge,2009.
12.
Goodman,J.
,Venolia,G.
,Steury,K.
,andParker,C.
Languagemodelingforsoftkeyboards.
InEighteenthNationalConferenceonArticialIntelligence,AmericanAssociationforArticialIntelligence(MenloPark,CA,USA,2002),419–424.
13.
Gunawardana,A.
,Paek,T.
,andMeek,C.
Usabilityguidedkey-targetresizingforsoftkeyboards.
InProceedingsofthe15thInternationalConferenceonIntelligentUserInterfaces,IUI'10,ACM(NewYork,NY,USA,2010),111–118.
14.
Hunt,A.
,Wanderley,M.
M.
,andParadis,M.
Theimportanceofparametermappinginelectronicinstrumentdesign.
JournalofNewMusicResearch32,4(December2003),429–440.
15.
Iwasaki,K.
,Miyaki,T.
,andRekimoto,J.
Expressivetyping:Anewwaytosensetypingpressureanditsapplications.
InCHI'09ExtendedAbstractsonHumanFactorsinComputingSystems,CHIEA'09,ACM(NewYork,NY,USA,2009),4369–4374.
16.
Kristensson,P.
O.
,andZhai,S.
Commandstrokeswithandwithoutpreview:Usingpengesturesonkeyboardforcommandselection.
InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems,CHI'07,ACM(NewYork,NY,USA,2007),1137–1146.
17.
MacKenzie,I.
S.
,andSoukoreff,R.
W.
Phrasesetsforevaluatingtextentrytechniques.
InCHI'03ExtendedAbstractsonHumanFactorsinComputingSystems,CHIEA'03,ACM(NewYork,NY,USA,2003),754–755.
18.
McNeill,D.
GestureandThought.
UniversityofChicagoPress,2005.
19.
Neumann,P.
,Tat,A.
,Zuk,T.
,andCarpendale,S.
Keystrokes:Personalizingtypedtextwithvisualization.
InProceedingsofthe9thJointEurographics/IEEEVGTCConferenceonVisualization,EUROVIS'07,EurographicsAssociation(Aire-la-Ville,Switzerland,Switzerland,2007),43–50.
20.
Pastel,R.
Measuringthedifcultyofsteeringthroughcorners.
InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems,CHI'06,ACM(NewYork,NY,USA,2006),1087–1096.
21.
Pirzadeh,A.
,Wu,H.
-W.
,Bharali,R.
,Kim,B.
M.
,Wada,T.
,andPfaff,M.
S.
Designingmulti-touchgesturestosupportemotionalexpressioninim.
InCHI'14ExtendedAbstractsonHumanFactorsinComputingSystems,CHIEA'14,ACM(NewYork,NY,USA,2014),2515–2520.
22.
Tu,H.
,Ren,X.
,andZhai,S.
Differencesandsimilaritiesbetweenngerandpenstrokegesturesonstationaryandmobiledevices.
ACMTrans.
Comput.
-Hum.
Interact.
22,5(Aug.
2015),22:1–22:39.
23.
Walther,J.
B.
,Loh,T.
,andGranka,L.
Letmecounttheways:Theinterchangeofverbalandnonverbalcuesincomputer-mediatedandface-to-faceafnity.
JournalofLanguageandSocialPsychology24,1(2015),36–65.
24.
Zhai,S.
,andKristensson,P.
O.
Theword-gesturekeyboard:Reimaginingkeyboardinteraction.
Commun.
ACM55,9(Sept.
2012),91–101.
11