Cesatichrome17

MetronomeLKM:AnopensourcevirtualkeyboarddrivertomeasureexperimentsoftwarelatenciesPabloGaraizar1&MiguelA.
Vadillo2Publishedonline:15September2017#PsychonomicSociety,Inc.
2017AbstractExperimentsoftwareisoftenusedtomeasurereac-tiontimesgatheredwithkeyboardsorotherinputdevices.
Inpreviousstudies,theaccuracyandprecisionoftimestampshasbeenassessedthroughseveralmeans:(a)generatingac-curatesquarewavesignalsfromanexternaldeviceconnectedtotheparallelportofthecomputerrunningtheexperimentsoftware,(b)triggeringthetypematicrepeatfeatureofsomekeyboardstogetanevenlyseparatedseriesofkeypressevents,or(c)usingasolenoidhandledbyamicrocontrollertopresstheinputdevice(keyboard,mousebutton,touchscreen)thatwillbeusedintheexperimentalsetup.
Despitetheadvantagesoftheseapproachesinsomecontexts,noneofthemcanisolatethemeasurementerrorcausedbytheexper-imentsoftwareitself.
MetronomeLKMprovidesavirtualkeyboardtoassessanexperiment'ssoftware.
Usingthisopensourcedriver,researcherscangeneratekeypresseventsusinghigh-resolutiontimersandcomparethetimestampscollectedbytheexperimentsoftwarewiththosegatheredbyMetronomeLKM(withnanosecondresolution).
Oursoftwareishighlyconfigurable(intermsofkeyspressed,intervals,SysRqactivation)andrunson2.
6–4.
8Linuxkernels.
KeywordsVirtualkeyboard.
Experimentalsoftware.
OpensourcesoftwareOverthelastdecades,computershavebecomeanessentialtooltoconductpsychologicalexperiments.
Intheseexperi-mentalsetups,participantstypicallyinteractwithseveralinputdevicesconnectedtoacomputerrunninganoperatingsystemthatenablesusingtheexperimentsoftware.
Ascanbeseen,severallayersseparateparticipants'interactionsfromtheex-perimentsoftwarecontrollingthoseinteractions.
Researchersmustbearinmindeachoftheseabstractionlevelswhende-signingexperimentalparadigmswithhightimingrequire-ments,asallofthemcanbetheoriginofmeasurementerror.
Thequalityofthemeasuresreturnedbytheexperimentsoftwareisassessedinbenchmarkingstudies,inwhichtheperformanceofthesoftwareistestedunderwell-knowncon-ditionsanditsmeasurementsarematchedagainsttheexpectedresults.
Forthespecificcaseofreaction-timemeasures,benchmarkingstudiesrequireasourceofusereventsthatcanbecontrolledexperimentally.
Severalmethodscanbeusedtogenerateuserinteractioneventsunderwell-controlledexperimentalconditions.
Noneofthemisfreefromlimitations,buttheyalsohaveinterestingadvantagesthatdifferentiatethemfromeachotherandmakethemparticularlysuitableforspecificsituations.
Inthefollow-ingsections,weprovideabriefoverviewofextantmethodstogenerateuserinputandweexplainwhygeneratingkeyboardeventsthroughavirtualkeyboarddrivercanbeanadequatesolutiontotestexperimentsoftwareunderawiderangeofconditions.
ExperimentalparticipantsAnobviousmeanstogatherrealisticuserinteractiondataistorecruitparticipantsandmeasuretheirperformanceundercontrolledconditions.
Thisisacommonapproachinbehavioralsciences.
Forinstance,ReimersandStewart*PabloGaraizargaraizar@deusto.
es1FacultyofEngineering,UniversityofDeusto,Avda.
Universidades24,48007Bilbao,Spain2UniversidadAutónomadeMadrid,Madrid,SpainBehavRes(2017)49:1686–1695DOI10.
3758/s13428-017-0958-7(2007)usedthismethodtoanalyzetheaccuracyandprecisionofreactiontimedatacollectedwithaFlashprograminasampleofundergraduatestudentswhoparticipatedinexchangeforcoursecredit.
Similarly,Keller,Gunasekharan,Mayo,andCorley(2009)recruitedvoluntaryparticipantstotesttheaccuracyoftheirJava-basedWebExppackageforthedesignofonlineexperiments.
Themainadvantageofthisapproachisitsexternalvalidity.
Theresultscanbeeasilygeneralizedtorealstud-iesbecausethedataaregatheredundernaturalconditions.
However,animportantshortcomingisthatparticipants'responsesmightnotbereliable,duetopracticeeffectsortotheuseofunrepresentativesamples.
Inmanycases,itisimpossibletocalibrateproperlytheexperimentalde-vicesortoassesstheimpactof(systematicorunsystem-atic)errorsarisingfromthehardware.
Becauseofthis,wethinkthatthisstrategymustbeconsideredcomplementarytoapproachesbasedonlowerlevels.
SpecializedhardwareInlightoftheselimitations,someresearchershaveresortedtospecializedhardwaretosimulateuserinteraction,whichgrantsmorecontroloverthegenerateduserdata.
TheBlackBoxToolkit(Plant,Hammond,&Turner,2004)isanexcellentexample.
Thishardwarecansendsignalsthroughtheparallelport(Schneider,Eschman,&Zuccolotto,2002)or,alternatively,activateabuttonlocat-edinaninputdevicethatisconnectedtothecomputerrunningtheexperimentalsoftware(Plant&Turner,2009).
Otherresearchershaveconnectedphoto-sensorstoBartificialrespondents^(Husler,Sommer,&Chroust,2007;Rorden&Hanayik,2014;Schubert,D'Ausilio,&Canto,2013)orsolenoidsthatgeneratekeyboardpresses(Neath,Earle,Hallett,&Surprenant,2011)tosimulateanautomatedexperimentalsubjectcapableofinteractinginaconsistentandreproduciblemannerwiththeapplicationortechnologyunderstudy.
Themostinterestingfeatureofthisalternativeapproach,ascomparedtousingrealparticipants,isthatitallowsre-searcherstoreplicatethestudyasmanytimesasnecessary,manipulatinganyvariablethatcouldberelevantandassessingitsimpactonthefinalresult.
Althoughthisispossiblythebeststrategyforbenchmarkingthewholeexperimentalsetup,itisnotfreefromproblems,either.
Thesignalgeneratedbyanexternaldevicemightnotreplicateexactlytheinteractionofrealusers.
Potentially,sourcesofbiasintheinputdeviceemployedbyusersmightgounnoticedinthesystemthatsendsandcollectssignalsfromtheexternalgenerator(e.
g.
,sendingkeyboardeventsthroughtheparallelporttotestanexperimentalsetupinwhichUSBkeyboardswillbeused).
Theoppositesituationisalsopossible—thatis,sourcesofbiasmightaffecthowsignalsaresentorreceivedfromthegeneratorwithoutproducingasimilareffectontheinputdevicesemployedbyusers(e.
g.
,usinganArduino-baseddevicetogeneratekeyeventsandsendthemthroughtheUSBporttotestanexperimentalsetupinwhicharesponsepadconnectedtoaPIO12cardwillbeused).
Consequently,themeasurementscollectedwiththismeth-odmightnotalwaysconvergewiththoseofthepreviousmethod.
Thisproblemisavoidedwhentheexternalgener-atorcanbedirectlyconnectedtotheinputdeviceitself(e.
g.
,hackingaUSBkeyboardtoconnectaspecifickeytotheexternalgenerator),butthisisnotalwayspossible.
Off-the-shelfhardwareGiventhattheuseofspecializedhardwareisexpensive,bothintermsoftimeandmoney,someresearchershaveusedgeneral-purposehardware,readilyavailableinmostequip-ment,togenerateuser-interactiondata.
Forinstance,Eichstaedt(2001),amongothers,assessedthetemporalpreci-sionofJavaappletsusingthetypematicrepeatrateofacon-ventionalkeyboard.
Typematicrepeatallowsresearcherstogeneratealargenumberofkeypressespersecondautomati-cally.
Itcanbeactivatedeasilybykeepingakeypressedforatimelongerthanthetypematicdelay.
Oncethisintervalissurpassed,key-presseswillbegeneratedautomaticallyac-cordingtothetypematicrepeatrate,untilthekeyisreleasedoradifferentkeyispressed.
Althoughthismightlooklikeasimpleandvalidmethodtogenerateconstantuserinteractiondatawithoutsophisticatedequipmentandwithoutacomplexsetup,thisapproachisun-satisfactoryforseveralreasons.
Firstly,typematicrepeatisnotimplementedconsistentlyindifferentkeyboards.
InthecaseofAT-PS/2keyboards,itisdefinedinthecommunicationprotocol(Chapweske,2003),butinUSBkeyboardsitisim-plementedbytheoperatingsystem,usingthepacketsendrateandthenumberofreceivedpacketstoestimatethedurationofkeypresses.
Therefore,typematicrepeatisnotalwaysgener-atedbytheexternaldeviceitself,butbytheoperatingsystem.
Secondly,general-purposekeyboardsareusuallymanufacturedwithlowqualitystandards.
Incontrasttootherinputdeviceslikejoysticksorclickers,keyboardsareexpect-edtorequirealowresponserate(below500pressespermin-ute)and,hence,manufacturershavelittlemotivationtobuildthemwithoptimalcomponents.
Finally,thepopularizationofwirelesskeyboardsaddsanotherlayerofcomplexity,giventhattheelectromagneticfieldismoreexposedtointerferencethanaregularcableconnection.
Becauseofallthesereasons,wethinkthatthetypematicrepeatrateshouldnotbeusedtoassesstheaccuracyandprecisionofanapplication.
BehavRes(2017)49:1686–16951687SpecializedsoftwareBearinginmindtheshortcomingsofalltheapproachesbasedonhardware,severalresearchershavedevelopedsoftwareal-ternativesfortheautomaticgenerationofuserinteractiondata.
Togetherwithsimplesystemsforthegenerationofkeypressesormouseevents,suchasuinputorKeypresser,therearecom-pletesuitesfortheautomatizationofuserinputspecificallydesignedfortheWeb,suchasSelenium(Badleetal.
,2012).
Thefactthatthegenerationofuserinteractiondoesnotdependonhardwareallowsthisapproachtocircumventallthebiasesandlatenciesinherenttohardware.
However,themainlimitationofthisstrategyisthatthesoftwareusedtogenerateuserinteractiondatacaninfluencetheperformanceoftheapplicationunderstudy,andviceversa.
Ideally,thiskindofmechanismsshouldbelocatedatasoftware-architecturelevelthatavoidsanyinfluencefromothersoft-warecomponents(e.
g.
,atthekernellevel)andmustrelyontimingmechanismsthatremainimmunetooverloadattheuserapplicationlevel.
MetronomeLKMToovercomethelimitationsoftheapproachespresentedintheprevioussection,wedecidedtodevelopakeyboarddriverrelyingonhigh-resolutiontimersthatcangenerateconfigurablekeyboardevents.
Thereasonsforthischoicearethat:(1)noneoftheconventionalinputdevices(e.
g.
,key-board,mouse,touchscreen)canworkwithsubmillisecondtimedelays(Bhalla&Bhalla,2010;Crosbie,1990;Damian,2010;Plant,Hammond,&Whitehouse,2003;Plant&Turner,2009;Segalowitz&Graves,1990),whichimposesalowerlimitontheprecisionoftimemeasurements(Rorden&Hanayik,2014);(2)theuseofuncommoncommunicationports,liketheparallelport,requiresadaptingtheapplicationtothisinputdevice,whichisnotalwayspossible;(3)thefactthatthisdriverrunsatthekernellevelavoidstheimpactofoverloadattheuserapplicationlevel;(4)thisapproachallowedustousetimers(i.
e.
,high-resolutiontimers)andtimingfunctions(i.
e.
,ktime)withprecisionandaccuracybelowthemillisecond;and(5)fromtheperspectiveofsoft-ware,theexperimentalprogramwillreceivetheautomaticallygeneratedkeyboardeventsasiftheyhadbeengeneratedbyarealuser.
WechoseGNU/LinuxasadevelopmentplatformduetothelargevolumeofpubliclyavailableinformationforthedevelopmentofdriversfortheLinuxkernel(Bovet&Cesati,2005),thesimplicityoftheCcodeneededtodevelopaloadablekernelmodule(LKM),andthehighlyflexiblemeansofferedtodeployandtestseveraltypesofvirtualma-chines,kerneldebuggers,andregistrysystems.
GNU/Linuxisalsoausefulplatformforbehavioralresearchers,giventhatsomeofthemostpopularsoftwarepackagesforthedeploy-mentofpsychologicalexperiments,suchasPsychoPy(Peirce,2007,2009),OpenSesame(Matht,Schreij,&Theeuwes,2012),orPsychophysicsToolbox(Brainard,1997;Pelli,1997),areexplicitlydesignedtorunonthisoperatingsystem.
JavaScriptlibrariesforInternet-basedpsychologicalresearch(e.
g.
,jsPsych,lab.
js;deLeeuw,2015;Henninger,Mertens,Shevchenko,&Hilbig,2017)canalsorununderGNU/Linux.
Ingeneral,GNU/Linuxandopensourcesoftwarearebecomingincreasinglypopularamongscientistsfrommanyfields,becausetheyreducesubstantiallythecostsofresearch(Pearce,2012).
MetronomeLKMispublishedunderanopensourceli-cense(GPLv3),whichgrantsdevelopersandresearchersper-missiontouse,adapt,auditorcorrectitfreeofcharge.
Thecoderepositoryispubliclyavailableathttps://github.
com/txipi/MetronomeLKM.
ConfigurationparametersMetronomeLKMisadriverdesignedforthekernelofLinux2.
6.
21(orhigher)thatreliesonhigh-resolutiontimerstogen-erateaccurateandprecisekeyboardevents.
Theparametersofthedrivercanbedefinedwhenitisloadedorthroughthesysvirtualfilesystem(i.
e.
,/sys/module/metronome/pa-rameters/)anditcanbeenabledordisabledusingasys-temrequestwiththeSysRqkey.
Inadditiontogeneratingtherequiredkeyboardevents,thedriveralsologseachofthemwithns-resolutiontimingfunctions(i.
e.
,ktime)inthekernellogfile.
SeveralparametersofMetronomeLKMcanbedefinedduringruntime.
SeeTable1foracomprehensivelist.
Oncethedriverisloaded,itcanbeactivatedwiththecombinationofkeyspreviouslydefined(SysRq+Dbydefault)orbymeansofthecommandecho1>/sys/module/metronome/parameters/metronome_status.
MetronomeLKMcanbestoppedbyfollowingthesameprocedure:eitherpress-ingthepredefinedcombinationofkeys(SysRq+D)orthroughthecommandecho0>/sys/module/metro-nome/parameters/metronome_status.
HowMetronomeLKMworksWhenMetronomeLKMisloadedintothekerneloftheoperatingsystem,functionmetronome_initrequeststhenecessarysystemresourcestoregisteranewinputdevicewithinput_allocate_deviceandinput_register_device.
Iftheregistrationissuccess-ful,MetronomeLKMsetsthesystemrequestthatenablesordisablesthegenerationofkeyboardeventsandconfiguresthemonotonichigh-resolutiontimer(withthehrtimer_initfunctionandthekernel'sCLOCK_MONOTONICpolicy,whichtakesintoaccounttimedriftsinmultiprocessorsystems).
1688BehavRes(2017)49:1686–1695Oncethisprocessiscomplete,metronome_hrt_callbackconfirmsthattherehasbeennodelayinthemanagementofthetimer(usingthehrtimer_forwardfunction),generatestherequestedkeyboardevents(usinginput_report_keyandinput_sync),logsthetimewhentheyweresubmitted(withktime_get)andresetsthehigh-resolutiontimer(returningthevalueHRTIMER_RESTART).
Iftheuserentersthecombi-nationofkeysdefinedtochangethestatusofMetronomeLKMatanytime,thiswilltriggersysrq_handle_metronome,whichtogglesbetweenactiveandinactivestatus.
HowtouseMetronomeLKMMetronomeLKMisaLoadableKernelModuleforLinux-basedsystems.
Therefore,aLinuxdistribution(e.
g.
,Debian,Ubuntu,orGentoo)witha2.
6.
21(orhigher)kernelisaprerequisitetouseit.
Inaddition,werecommendinstallingseveralsoftwarepack-agesbeforedownloading,compilingandinstallingMetronomeLKM.
Inthissection,weexplainbrieflythestepsthatshouldbefollowedinaDebian-orUbuntu-basedLinuxdistribution,buttheseguidelinescanbeadaptedeasilytootherdistributions.
Toinstalltherequiredpackages,theusermusthaverootac-cesspermission.
AllLinuxsystemsallowtheusertogetrootaccessusingthesu(i.
e.
,superuser)command.
Somerecentsystemsusethesudo(i.
e.
,super-userdo)commandasthede-faultmethodtoperformadministrationtasks.
InUbuntu,forexample,anycommandprecededbythewordsudowillbeexecutedwithrootaccesspermission.
Therefore,toinstallthesoftwarepackagesrequiredbyMetronomeLKM,theusershouldrunsudoapt-getinstallgitbuild-essen-tiallinux-headers-genericdkms.
Oncethesepack-ageshavebeeninstalled,MetronomeLKMcanbedownloadedfromtheGitHubrepositoryrunninggitclonehttps://github.
com/txipi/metronome.
git–depth1.
Tocompileandinstallit,theusermustaccessthemetronomedirectory(usingcdmetronome)andlaunchthecompilationandinstallationprocesswithsudomakeinstall.
Oncetheprocessiscomplete,MetronomeLKMcanbeinstalledintotheoperatingsystemkernelwiththecommandinsmodmetronome.
ko.
Asweexplainedabove,MetronomeLKMenablesanddisablestheautomaticgenerationofkeyboardeventsusingthespecialkeySysRq(inmanykeyboards,thisfunctionissharedwiththePrintScreenkeyandisactivatedwiththekeycombinationAlt+PrintScreen).
However,manyLinuxdistri-butionspreventtheuseoftheSysRqkeyonthedefaultsetup,asitgrantspermissiontoperformadministrationtaskslikeresettingthesystem.
Inthesecases,SysRqshouldbeactivatedwiththecommandsudosysctl-wkernel.
sysrq=1.
Alternatively,thischangecanbedonepermanentlycreatingafilecalled/etc.
/sysctl.
d/90-sysrq.
confwiththeinstructionkernel.
sysrq=1.
Finally,iftheuserwantstokeeprecordofmanykeyboardeventsgeneratedbyMetronomeLKM,thekernelringbufferwherethisinformationisstored(whichcanbecheckedwithdmesg)mightnotbelargeenough.
Toexpandthesizeofthebuffer,theusermustaddthelog_buf_lenmodifiertothesystemstartupmanager.
InUbuntu,thiscanbedonechangingthefile/etc.
/default/grubinthespecificlinewherethevariableGRUB_CMDLINE_LINUX_DEFAULTisde-fined,settingittoBquietsplashlog_buf_len=16M^,where16Mreferstothedesiredbuffersize.
Study1ThegoalofthepresentstudyistotestthevalidityofMetronomeLKMtogeneratekeyboardeventscollectedbyasimpleX-Window-System(i.
e.
,theLinuxGraphicalUserInterface)application.
InStudy2,wecomparethosemeasure-mentswiththeresultsobtainedbyaJavaScriptWebapplica-tionbasedonstandardDOMeventstimestampsrunningonGoogleChromeandMozillaFirefox.
MethodApparatusandmaterialsWeinstalledMetronomeLKMonanUbuntuLinuxvanilladistribution,unconnectedtotheInternetandisolatedfromexternalsourcesofasynchronousevents.
Onthissetting,weruntheLogkeysX-Window,anTable1ConfigurationparametersforMetronomeLKMNameDescriptionmetronome_delayDelay(innanoseconds)betweenkeyinputevents.
Severaldelayscanbedefinedfordifferentkeyevents(metronome_delay,metronome_delay_2,metronome_delay_3).
Initiallysetto1E09(i.
e.
,109ns=1s).
metronome_keyKeyeventthatwillbegeneratedbyMetronomeLKM.
Severalkeyeventscanbedefined(metronome_key,metronome_key_2,metronome_key_3).
InitiallysettoKEY_SPACE(i.
e.
,spacebar).
metronome_sysrqCombinationofkeys(inadditiontoSysRq)thattriggersthesystemrequesttoenableordisablethegenerationofkeyboardevents.
InitiallysettoBd,^whichmeansthattherequestistriggeredbythecombinationSysRq+D.
metronome_statusStatus(activeorinactive)ofthekeyeventgenerationprocess.
Initiallysetto0(inactive).
BehavRes(2017)49:1686–16951689applicationfortheX-Window-Systemgraphicalinterfaceex-plicitlydevelopedtologkeyboardevents.
ThisisachievedthroughthefunctionsXGrabKeyboard(togetkeyeventsonaX-Window-Systemenvironment)andgettimeofday(togettimestampswithmicrosecondaccuracy).
ProcedureThespecificstepswefollowedwere(1)weloadedMetronomeLKMwiththeappropriateparameters(inthede-faultconfiguration,thedriverremainsinactiveinitially),(2)launchedtheapplicationunderstudy,(3)activatedthedriverthroughasystemrequest(SysRq),(4)loggedtheinputeventswiththeapplicationunderstudyandwiththekeyboarddriveritself,and(5)analyzedthedata.
Thesameprocedurewasfollowedwithdifferentkeypressintervals(i.
e.
,onekeypressevery1,000,500,100,50,10,5,and1ms)until10,000ormoresampleshadbeencollected.
ResultsanddiscussionTheresultsofthetestsconductedwithMetronomeLKMandLogkeysX-WindowareshowninTable2,whereallvaluesrefertomilliseconds.
Ascanbeseen,themeanerrorsobtainedwithMetronomeLKMarewellbelowamillisecond,aroundnanosecondsintheworstcases,withstandarddeviationsof4.
499μsintheworstcase.
ThemeanerrorsobtainedwithLogkeysX-Windowarealsobelowamillisecond(between4.
599and0.
178μs),butthestandarddeviationsarenotice-ablylargerthanthoseobtainedatthekernellevel(between4.
214μsand0.
112ms).
Thesevaluescanbeusedasabase-linefortheassessmentofperformanceunderWebapplica-tionsrunningonGoogleChrome17andMozillaFirefox10inStudy2.
Study2ThegoalofStudy2wastobenchmarkaJavaScriptWebapplicationbasedonstandardDOMeventstimestampsrunningonGoogleChrome17andMozillaFirefox10usingMetronomeLKM.
MethodApparatus,materials,andprocedureAsinStudy1,weinstalledMetronomeLKMonanUbuntuLinuxvanilladistri-butionunconnectedtotheInternetandisolatedfromexternalsourcesofasynchronousevents.
Onthissetting,werunaJavaScriptWebapplicationthatreliesonthetimestampsofstandardDOMEvents(Pixley,2000)tologthetimeelapsedfromtheprevioustimestamptothemomentwhenthekey-boardeventisregistered(i.
e.
,thekeypressevent).
Theproce-durewasotherwiseidenticaltoStudy1.
ResultsanddiscussionTable3summarizestheresultsobtainedwithGoogleChrome17.
AlthoughthemeanerrorsobtainedbyMetronomeLKMaregenerallyclosetoananosecond,theirstandarddeviationsaresystematicallylargerthantheerrorsobservedinStudy1.
ThispatternisalsoobservedforthetestsconductedonMozillaFirefox10,whichsuggeststhatthesmalllossofac-curacyinthegenerationofkeyboardeventsmightbeduetothehigherloadimposedbyonlineapplicationsrunningonauseragent,ascomparedtoasimpleLogkeysX-Windowap-plication.
Inanycase,alltheerrorsarebelowamillisecond,wellbelowtheresolutionofstandardDOMevents(i.
e.
,thetimestamppropertyofDOMTimeStamptype,anun-signedlonglongaimedtostoreavalueinmilliseconds).
ThemeanerrorsandstandarddeviationsobservedinthetimestampsofDOMeventsarealsobelowamillisecondinallcases.
Theseresultsconfirmthevalidityofthissettingtocol-lectuserinteractiondatafromonlineapplicationswithhightimingrequirements.
Table4showstheresults,inmilliseconds,ofthetestscon-ductedwithMozillaFirefox.
Ascanbeseen,theresultsareverysimilartothosefromtheprevioustest,suggestingthattheTable2TimingerrorsinStudy1(inmilliseconds)IntervalMetronomeLKMLogkeysX-WindowMSDMSD1–0.
000000070.
0046608470.
000001200.
0588757595–0.
000009950.
0044993700.
004108200.
112822616100.
000000030.
000192422–0.
000838200.
00421433250–0.
000000930.
0010328600.
000546800.
025032313100–0.
000005550.
0009300280.
004598500.
094976506500–0.
000012390.
001294279–0.
003493700.
0091271181,000–0.
000001290.
001002863–0.
000177700.
046649194Table3TimingerrorsinStudy2withGoogleChrome17(inmilliseconds)IntervalMetronomeLKMGoogleChromeMSDMSD1–0.
001399850.
0058273000.
000000000.
03464274850.
000008030.
030747087–0.
000700000.
04122717510–0.
000006740.
0146778910.
170500000.
409690779500.
000006950.
0155690020.
402500000.
6443099131000.
000000470.
013899963–0.
000100000.
156532553500–0.
000000020.
0385883170.
011800000.
2219587501,0000.
000005720.
0125744200.
000000000.
3455603291690BehavRes(2017)49:1686–1695systemloadsimposedbyGoogleChromeandMozillaFirefoxdonotdiffersubstantially.
RegardingthedatacollectedwiththeWebapplication,thereseemstobenomeasurementerrorforintervalslongerthan10ms,giventhelowresolution(milliseconds)ofstandardDOMeventstimestamps.
Below10ms,weobservedisolatedtimingerrorswithmeansbelowamicrosecondandstandarddeviationsbelowamillisecond.
ThisconfirmstheresultsobtainedwithGoogleChrome17andsupportsthevalidityofstandardDOMeventstimestampstocollectuserinteractiondatawithstandardWebapplications.
Study3TheresultsofStudies1and2allowustoconcludethat,inprinciple(andsettingasidethepotentiallimitationsofhardware;seePlant,2016,andvanSteenbergen&Bocanegra,2016,foradiscussionofthisissue),itispossibletocollectuserinteractiondatawithstandardWebtechnologieswithprecisionandaccuracybelowamillisecond.
However,theseresultsalsorevealalimitationinthecollectionofreac-tiontimeswithstandardDOMevents.
Theirlow(millisecond)resolutioncanmasksmallvariationsatlowerlevels.
Itwouldbeadvisabletoextendtheirresolutiontomicrosecondsorevennanoseconds.
Infact,somedevelopersofJavaScriptpro-jectsrelatedtotimemeasurement(e.
g.
,jsPerf,Benchmark.
js)havedecidedtoleveragethetimingfeaturesofaJavaappletprovidedwiththenanosecond-resolutionnanoTimefunc-tionor,alternatively,torelyonproprietaryextensionslikechrome.
Interval,whichreturnsthenumberofmicro-secondssinceJanuary1,1970.
Fortunately,theHighResolutionTimeAPIrendersthesealternativesolutionsunnecessary(Mann,2012).
ThisAPIimplementsamonotonicallygrowingtimefunc-tionwithaccuraciesbelowamillisecond(thankstotheDOMHighResTimeStamptype,whichrepresentsvaluesinmillisecondswithaprecisionofmicroseconds)andisimmunetoproblemsarisingfromclockdriftsinmultipro-cessorsystemsorchangesinthesystemclock.
Giventheadvantagesofthismethod,wethoughtthatitwouldbeadvisablethatDOMeventsregisteredtheirtimestampswiththeDOMHighResTimeStamptype,sothatthesevaluescouldbecomparedtothoseobtainedwithnewerAPIsthatrelyonthistypeoftimestamps,suchastheHighResolutionAPI(Mann,2012)ortheAPIforthetimecontrolofproceduralanimations(Robinson&McCormack,2015),amongothers.
Becauseofthis,wesubmittedarequesttoimplementthisimprovementinversion4ofDOMEventstotheW3C'sDOMEventsWorkingGroup(seehttp://lists.
w3.
org/Archives/Public/www-dom/2012AprJun/0077.
htmlandhttp://lists.
w3.
org/Archives/Public/www-dom/2012AprJun/0092.
html).
OurrequestisstillpendingapprovalbutithasalreadybeenimplementedinlatestversionsofGoogleChromeandMozillaFirefox.
ThegoalofStudy3wastobenchmarkaJavaScriptWebapplicationrunningonmorerecentbrowserswithhigh-resolutiontimestampsforevents(GoogleChrome58andMozillaFirefox54)usingMetronomeLKM.
MethodApparatus,materials,andprocedureAsinStudies1and2,weinstalledMetronomeLKMonanUbuntuLinuxvanilladistributionunconnectedtotheInternetandisolatedfromex-ternalsourcesofasynchronousevents.
Onthissetting,werunaJavaScriptWebapplicationthatreliesonthehigh-resolutiontimestampsofDOMEventstologthetimeelapsedfromthebeginningoftheexecutionoftheWebapplication(navigationStart)tothemomentwhenthekeyboardeventisregistered(i.
e.
,thekey-pressevent).
TheprocedurewasotherwiseidenticaltoStudies1and2.
ResultsanddiscussionTable5summarizestheresultsobtainedwithGoogleChrome58.
Allthevaluesrefertomeasurementerrorsinmilliseconds.
Asinthepreviousstudies,themeanerrorsobtainedbyMetronomeLKMwereclosetoananosecond,andtheirstan-darddeviationswerebelowamillisecond.
Despitethenumer-ousupgradesinGoogleChromebetweenStudies2and3(i.
e.
,fromversion17to58),MetronomeLKMbehavedsimilarly.
Themeanerrorsandstandarddeviationsinthetimestampsofhigh-resolutionDOMeventswereagainbelowthemillisec-ondinallcases,confirmingthevalidityofthissettingtocol-lectuserinteractiondatafromonlineapplicationswithhightimingrequirements.
ToassesshowsystemloadcanaffecttheaccuracyandprecisionofMetronomeLKM,werepeatedthisstudyen-ablingtheCPUthrottlingfeature.
GoogleChrome'sCPUthrottlingallowsWebdeveloperstosimulatetheexecutionTable4TimingerrorsinStudy2withMozillaFirefox10(inmilliseconds)IntervalMetronomeLKMMozillaFirefoxMSDMSD1–0.
00301390.
005213274–0.
000500000.
0842642285–0.
000002090.
0113121490.
000100000.
010000000100.
000000350.
0245555960.
000500000.
110909784500.
000005280.
0284436450.
000000000.
0000000001000.
000004950.
0142436730.
000000000.
000000000500–0.
000001140.
0468713840.
000000000.
0000000001,0000.
000001140.
0189162810.
000000000.
000000000BehavRes(2017)49:1686–16951691ofaWebapplicationinlow-resourceoroverloadedcom-puters.
Thisfeaturecanbesettoarangeofvaluesfrom2xto20x,representingincreasinglevelsofoverload.
Werepeat-edtheprocedureofthepresentstudywhilesettingthisvari-ableto20x.
Table6showstheresultsofthetestsconductedwith20xCPUthrottlingonGoogleChrome58.
Interestingly,theseresultsdonotdiffersubstantiallyfromthosegatheredwithoutCPUthrottling.
(Ifanything,theyareslightlybetter,becauseinthe10-msintervalconditiononGoogleChrome58withoutthrottling,7outof10,000keyeventswereprocessed1–3mslate,whereasthisdidnothappeninthetestswiththrottling.
)ThisledustothinkthatMetronomeLKMcanbeusedtogeneratekeyeventsaccuratelyandpreciselyevenunderconsiderableCPUoverload.
Table7showstheresultsofthetestsconductedwithMozillaFirefox54.
Ascanbeseen,theresultsfromMetronomeLKMareverysimilartothoseintheprevioustests.
RegardingthedatacollectedwiththeWebapplication,thisMozillaFirefoxversionbehavessignificantlyworsethantheothers.
Itcannotproperlyregisterallkeyeventsinthemostdemandingcase(1-msinterval).
Intermsofprecision,wefoundsubmillisecondprecisioninthe1-,5-,10-,50-,and1,000-msintervaltests.
However,theprecisionsoftestswith100-and500-msintervalswere1.
666and1.
988ms,respec-tively.
Thesesuboptimalresultsmightbeexplainedbecauseversion54isthefirstMozillaFirefoxversion(andthelastavailablestableversionatthismoment)toprovidehigh-resolutioneventtimestamps,andthisfeaturemightnothavebeenthoroughlytestedyet.
Study4InStudies1–3,weusedMetronomeLKMtoexploretheprecisionofthenewDOMeventsinthelatestversionsofGoogleChromeandMozillaFirefox.
Assessingtheiraccura-cywouldrequireananalysisoftheabsolutevaluesofthetimestamps,insteadoftherelativedifferencebetweenconsecutivekeyboardevents.
Thisisadauntingchallengeinnewsystems,duetothevarietyoftimingsourcesusedbydifferenttimingfunctions.
BothMetronomeLKManddmesgrelyonamonotonictimingsourcenotsynchronizedwiththesystemclock.
Thistimingsourcemonotonicallyupdatesthevalueofatimecounterrelativetothesystemuptime.
Therefore,thesystemclockcanpotentiallydriftawaywithrespecttothismonotonictimingfunction.
Infact,itiseasytocheckthisbyrunningthefollowingcommandsinLinuxwithrootpermis-sion:echotest>/dev/kmsg&&dmesg-T|tail-1&&date.
Ifthereisnodriftbetweenthetwotimingsources,bothdatesshouldbeidentical.
However,ifwesuspendtheTable5TimingerrorsinStudy3withGoogleChrome58(inmilliseconds)IntervalMetronomeLKMGoogleChromeMSDMSD1–0.
00000023150.
0102970746976920.
0000000000000000.
0000000000008355–0.
00000305690.
0290495386861140.
0000000000000000.
07483688967342710–0.
00000085630.
0300503504615190.
0000000000000020.
27314366319960250–0.
00000094610.
0204212226088760.
0000000000000080.
0141428427831381000.
00000667830.
0195889737766520.
0000000000000140.
000000000052873500–0.
00000022790.
0167935194329440.
0000000000001010.
0000000000110671,000–0.
00000037350.
0105975829931980.
0000000000002160.
000000000013466Table6TimingerrorsinStudy3withGoogleChrome58at20xCPUthrottling(inmilliseconds)IntervalMetronomeLKMGoogleChrome20xMSDMSD1–0.
00000008050.
0015027033932110.
0000000000000000.
0000000000052425–0.
00000010810.
0162422226843000.
0000000000000010.
00000000000496110–0.
00000115120.
0142421285059420.
0000000000000000.
000000000007990500.
00000298020.
0155673341934380.
0000000000000060.
0000000000295231000.
00000304320.
0162854375203760.
0000000000000050.
004472359574564500–0.
00000000950.
0156825905184550.
0000000000000740.
0000000000054701,0000.
00000000640.
014487621290040–0.
0000000000000300.
0000000000112671692BehavRes(2017)49:1686–1695computerforseveralminutesandrepeatthetest,thedriftwouldbeevident.
InanysystemthatupdatesthesystemclockautomaticallythroughNTPorsimilarprotocols,thereisagoodchancethatthesystemclockwilldriftawayfromthemonotonictimingsourceofthekernel.
RegardingDOMeventstimestamps,thestandardeventsusethesystemclockasthetimingsourceand,consequently,theirvaluescanbedirectlycomparedtothesystemclockandaresubjecttothesamedrift.
Onthecontrary,DOMeventswithhigh-resolutiontimestampsreturnavaluerelativetothebeginningoftheexecutionoftheWebapplication(navigationStart)andcanonlybecomparedtothesys-temclockintheabsenceofdrift(i.
e.
,navigationStart+event.
timeStampshouldbeequivalenttothevaluereturnedbyDate).
Thisbeingsaid,woulditbepossibletocomparethetimestampsgatheredbyMetronomeLKMwiththeDOMevents'high-resolutiontimestampsAscanbeseeninFig.
1,thetimestampsofMetronomeLKMareoffsetrelativetothesystemboottime(btime).
Thecorrespondingtimeinthesystemclockcanbeobtainedwiththecommandgrepbtime/proc/stat.
Regrettably,thisvaluehasapreci-sionofseconds,whichlimitstheaccuracyofanycompari-sonbetweenbtimeandthesystemclock.
Thesameproblemappliestothestartingpointofthebrowsertimestamps.
ThenavigationStartpropertystoresthevalueofthewallclockinmillisecondsatthebeginningofexecutionoftheWebapplication.
However,thisvaluecannotbedirectlycomparedtobtime,duetodifferencesinresolutionandtothepotentialdriftofthesystemclock.
Additionally,thefactthatnavigationStartreturnsvaluesinmilliseconds,whereashigh-resolutiontimestampsworkinmicroseconds,impliesthatanycomparisonbetweenthesetimestampsandothereventsstoringtimestampsfromthesystemclockcanbemadeonlyatthemillisecondlevel.
Inspiteofthesehindrances,wedecidedtoconductafinalstudytoanalyzetheaccuracyofthekeyboardeventsgeneratedbyMetronomeLKMandregisteredbyGoogleChrome58.
Table7TimingerrorsinStudy3withMozillaFirefox54(inmilliseconds)IntervalMetronomeLKMMozillaFirefoxMSDMSD10.
00000226040.
0194282392333831.
9332677734000000.
53680423123034250.
00000658500.
036059731861193–0.
0005760635000000.
642599968616566100.
00000256900.
043567016249743–0.
0006591145999990.
832928759893024500.
00000582740.
012179655831148–0.
0002125102999960.
5919397543466511000.
00000045610.
011726603144952–0.
0003238468000071.
6657409628722405000.
00000003430.
012328960555666–0.
0003970353000011.
9877850526285401,0000.
00000060200.
013258050157191–0.
0003139788000200.
490275936869454Fig.
1Thethreetimelinesinvolvedinthesetofstudiespresentedinthisarticle.
Themiddletimelinerepresentsthesystemclock(oftenknownastheBwallclock^),subjecttodynamicadjustmentsandtimedrift.
ToptimelinerepresentsthemonotonicclockoftheLinuxkernel:Itstartsatbtime,andallsubsequenttimestampsareoffsetsrelativetothisstartingpoint.
ThebottomlinerepresentsthemonotonicclockoftheHighResolutionTimingAPIofthebrowser:ItstartsatnavigationStart,andallsubsequenttimestampsareoffsetsrelativetothisstartingpoint.
KeyeventsgeneratedbyMetronomearemarkedaskn,andthecorrespondinginputevents(en)areregisteredslightlylaterbythebrowser.
BehavRes(2017)49:1686–16951693MethodApparatus,materials,andprocedureWemodifiedthesourcecodeofMetronomeLKMtoprovidenotonlythetimestampgatheredfromLinuxkernel'smonotonictimesource,butalsofromthesystemclock(viathegetnstimeofdaytimingfunction).
Then,weinstalledthismodifiedversionofMetronomeLKMonanUbuntuLinuxvanilladistributionunconnectedtotheInternetandisolatedfromexternalsourcesofasynchronousevents.
Onthissetting,werunaJavaScriptWebapplicationthatre-liesonthehigh-resolutiontimestampsofDOMEventstologthetimeelapsedfromthebeginningoftheexecutionoftheWebapplication(navigationStart)tothemo-mentwhenthekeyboardeventisregistered(i.
e.
,thekeypressevent).
TheprocedurewassimilartoStudies1–3,butwereducedthenumberofsamplescollectedfrom10,000to50becausetheprecisionofthesoftwarehadalreadybeenassessedinthepreviousstudies,andthisstudywasaimedatdeterminingthedelaybetweenthegenerationofthekeyeventbyMetronomeLKMandthetimestampgatheredbythebrowser.
ResultsanddiscussionForeverykeyboardeventgenerated,wecalculateditsUNIXtimestamp(thetimeelapsedfromthe1stofJanuary,1970),bothinthekernelloggeneratedbyMetronomeLKMandinthedatafilegeneratedbytheWebapplicationrunningonGoogleChrome58.
Thenwecalculatedthedifferencebe-tweenthebrowserandkerneltimestampsandfounda0.
779±0.
343-msdifferenceforallintervals:(a)1,000ms(M=0.
770191,SD=0.
031244);(b)500ms(M=1.
091819,SD=0.
010752);(c)100ms(M=0.
838900,SD=0.
022353);(d)50ms(M=1.
109455,SD=0.
024969);(e)10ms(M=1.
087820,SD=0.
023810);(f)5ms(M=0.
202062,SD=0.
031899);and(g)1ms(M=0.
354179,SD=0.
010953).
Toassesstheaccuracyofreactiontimemeasures,oneneedstocollectthetimestampofthemomentwhentheoriginaleventwascreated,notthetimestampofthemo-mentwhenitwaslogged.
Ascanbeseenintherawdatafromourstudies,availableathttps://osf.
io/6dv3e,thereisanoticeabledelaybetweenthetimestamprecordedbyMetronomeLKMandtheonerecordedatthekernel'sringbuffer(e.
g.
,in[6287.
072289]metronome:Keyevent(57,6287064130600ns,thereisa8.
1584-msdifferencebetweenthetwotimestamps).
ThiscaveatappliestotheprocessingofDOMeventtimestampsinWebapplicationsorothertypesofsoftware(i.
e.
,avoidingcallingtheperformance.
nowfunctioninsidetheeventhandler,andinsteadusingtheevent'stimestampdefinedatthetimeofcreatingtheeventitself).
ConclusionsandoutlookInthepresentarticlewehaveintroducedMetronomeLKM,anewtooltobenchmarktheaccuracyandprecisionofreactiontimemeasurestakeninexperimentalsoftwarerunningunderGNU/Linux.
Thistoolallowsresearcherstogeneratekey-presseventsusinghigh-resolutiontimersandcomparethetimestampscollectedbytheexperimentsoftwarewiththosegath-eredbyMetronomeLKM(withnanosecondresolution).
AftertestingtheperformanceofLKM(Study1),wehaveusedittobenchmarktheprecision(Studies2and3)andaccuracy(Study4)ofaJavaScriptapplicationrunningunderseveralversionsofGoogleChromeandMozillaFirefox.
OurresultsconfirmthattheDOMeventtimestampscanbeusedtocollectuserinter-actiondatawithstandardWebapplications.
Furthermore,theyalsohighlightthevalueofMetronomeLKMasabenchmarkingtool.
Allthedatasetsusedinthepresentstudiesareavailableunderafreelicenseathttps://osf.
io/6dv3e.
ToconfirmwhetherthepresentresultscanbegeneralizedtootherversionsofGoogleChromeandMozillaFirefoxandoperatingsystemsotherthanLinux,wehaverunthemainJavaScriptbenchmarkingsuitesinGoogleChromeandMozillaFirefoxrunningunderUbuntuLinux16.
04.
2andMicrosoftWindows10Pro.
BothforGoogle(Octane2.
0:25068inLinuxvs.
25102inWindows;JetStream:140.
30±7.
0520inLinuxvs.
133.
72±2.
7159inWindows;Kraken:1251.
1±0.
7%inLinuxvs.
1310.
8±1.
5%inWindows)andMozillaFirefox(Octane2.
0:25266inLinuxvs.
25167inWindows;JetStream:143.
60±9.
8593inLinuxvs.
133.
35±7.
0495inWindows;Kraken:1257.
7±4.
7%inLinuxvs.
1351.
5±2.
4%inWindows),weobtainedverysimilarresultsinLinuxandWindows.
Theseconvergingresultsareconsis-tentwiththedevelopmentmodeladoptedforthesetwobrowsers,asbothofthemrelyonacommonbaseofsourcecodefortheirimplementationineachoperatingsystem.
ItisworthnotingthatMetronomeLKMcanalsobeusedincombinationwithotherbenchmarkingmethods.
Forinstance,asexplainedintheIntroduction,somestudieshaveassessedthereliabilityofreactiontimemeasuresusingdevicesthatcande-tectchangesinthescreenandrespondtothemsendingasignalbacktotheexperimentalcomputer(Husleretal.
,2007;Neathetal.
,2011;Rorden&Hanayik,2014;Schubertetal.
,2013).
Thisapproachisanexcellentmeanstoevaluatewholeexperi-mentalsettings,butitsresultscannotinformusaboutthespe-cificsourcesofnoisethatmightaffectthedata.
Anyproblemintheaccuracyorprecisionofreactiontimemeasurescouldbedueeithertotheexperimentalsoftware,hardware,oraninter-actionbetweenboth.
MetronomeLKMcanbeusedincombi-nationwiththesemethodstoisolatethespecificcontributionofexperimentalsoftwaretotheoverallamountofnoiseinreactiontimes,providingausefulbaselinefortheassessmentofanyadditionalnoiseinducedbyhardware.
1694BehavRes(2017)49:1686–1695Tosumup,MetronomeLKMprovidesasimplemeanstosimulateuserkeystrokespreciselyandaccurately.
Itsopensourcelicenseenablesresearchersanddeveloperstoaudit,correct,andadaptitastheyseefit.
However,itisimportanttorememberthatsoftwarebenchmarkingisaimedtofindunforeseenerrorsandimprovesoftware,nottovalidateexper-imentalsetups.
Researchersmusttesttheirwholesetups(in-cludinghardwareandsoftware)beforeconductingexperi-ments,usingmorecomprehensivemethods,suchaspilotex-perimentswithparticipantsorusingspecializedhardware.
AuthornoteSupportforthisresearchwasprovidedbyDepartamentodeEducación,UniversidadeseInvestigaciónoftheBasqueGovernment(GrantNo.
IT1078-16).
M.
A.
V.
wassupportedbyProgramadeAtraccióndeTalentoInvestigadordelaComunidaddeMadrid(Grant2016-T1/SOC-1395).
Theauthorsdeclarenoconflictofinterestinthepublicationofthisstudy.
ReferencesBadle,S.
,Bakken,J.
,Barantsev,A.
,Beans,E.
,Berrada,D.
,Bevan,J.
,.
.
.
Wagner-Hall,D.
(2012).
Selenium—WebBrowserAutomation.
Retrievedfromhttp://seleniumhq.
org.
Bhalla,M.
,&Bhalla,A.
(2010).
Comparativestudyofvarioustouchscreentechnologies.
InternationalJournalofComputerApplications,6,12–18.
Bovet,D.
P.
,&Cesati,M.
(2005).
UnderstandingtheLinuxkernel(3rded.
).
O'Reilly.
Brainard,D.
H.
(1997).
ThePsychophysicsToolbox.
SpatialVision,10,433–436.
doi:https://doi.
org/10.
1163/156856897X00357Chapweske,A.
(2003).
ThePS/2mouse/keyboardprotocol.
Retrievedfromwww.
computer-engineering.
org/ps2protocol.
Crosbie,J.
(1990).
TheMicrosoftmouseasamultipurposeresponsedevicefortheIBMPC/XT/AT.
BehaviorResearchMethods,22,305–316.
Damian,M.
F.
(2010).
DoesvariabilityinhumanperformanceoutweighimprecisioninresponsedevicessuchascomputerkeyboardsBehaviorResearchMethods,42,205–211.
doi:https://doi.
org/10.
3758/BRM.
42.
1.
205deLeeuw,J.
R.
(2015).
jsPsych:AJavaScriptlibraryforcreatingbehav-ioralexperimentsinawebbrowser.
BehaviorResearchMethods,47,1–12.
doi:https://doi.
org/10.
3758/s13428-014-0458-yEichstaedt,J.
(2001).
Aninaccurate-timingfilterforreactiontimemea-surementbyJAVAappletsimplementingInternet-basedexperi-ments.
BehaviorResearchMethods,Instruments,&Computers,33,179–186.
Husler,J.
,Sommer,M.
,&Chroust,S.
(2007).
Optimizingtechnicalprecisionofmeasurementincomputerizedpsychologicalassess-mentonWindowsplatforms.
PsychologyScience,49,116–131.
Henninger,F.
,Mertens,U.
K.
,Shevchenko,Y.
,&Hilbig,B.
E.
(2017).
lab.
js:Browser-basedbehavioralresearch.
doi:https://doi.
org/10.
5281/zenodo.
597045Keller,F.
,Gunasekharan,S.
,Mayo,N.
,&Corley,M.
(2009).
Timingaccuracyofwebexperiments:AcasestudyusingtheWebExpsoft-warepackage.
BehaviorResearchMethods,41,1–12.
doi:https://doi.
org/10.
3758/BRM.
41.
1.
12Matht,S.
,Schreij,D.
,&Theeuwes,J.
(2012).
OpenSesame:Anopen-source,graphicalexperimentbuilderforthesocialsciences.
BehaviorResearchMethods,44,314–324.
doi:https://doi.
org/10.
3758/s13428-011-0168-7Mann,J.
(2012).
HighResolutionTime(W3Ccandidaterecommenda-tion22May2012).
Retrievedfromwww.
w3.
org/TR/2012/CR-hr-time-20120522/.
Neath,I.
,Earle,A.
,Hallett,D.
,&Surprenant,A.
M.
(2011).
ResponsetimeaccuracyinAppleMacintoshcomputers.
BehaviorResearchMethods,43,353–362.
doi:https://doi.
org/10.
3758/s13428-011-0069-9Pearce,J.
M.
(2012).
Buildingresearchequipmentwithfree,open-sourcehardware.
Science,337,1303–1304.
Peirce,J.
W.
(2007).
PsychoPy—PsychophysicssoftwareinPython.
JournalofNeuroscienceMethods,162,8–13.
doi:https://doi.
org/10.
1016/j.
jneumeth.
2006.
11.
017Peirce,J.
W.
(2009).
GeneratingstimuliforneuroscienceusingPsychoPy.
FrontiersinNeuroinformatics.
2,10.
doi:https://doi.
org/10.
3389/neuro.
11.
010.
2008Pelli,D.
G.
(1997).
TheVideoToolboxsoftwareforvisualpsychophysics:Transformingnumbersintomovies.
SpatialVision,10,437–442.
doi:https://doi.
org/10.
1163/156856897X00366Pixley,T.
(2000).
Documentobjectmodelevents.
Retrievedfromwww.
w3.
org/TR/DOM-Level-2-Events/events.
htmlPlant,R.
R.
(2016).
Areminderonmillisecondtimingaccuracyandpotentialreplicationfailureincomputer-basedpsychologyexperi-ments:Anopenletter.
BehaviorResearchMethods,48,408–411.
doi:https://doi.
org/10.
3758/s13428-015-0577-0Plant,R.
R.
,Hammond,N.
,&Turner,G.
(2004).
Self-validatingpresen-tationandresponsetimingincognitiveparadigms:HowandwhyBehaviorResearchMethods,Instruments,&Computers,36,291–303.
doi:https://doi.
org/10.
3758/BF03195575Plant,R.
R.
,Hammond,N.
,&Whitehouse,T.
(2003).
Howchoiceofmousemayaffectresponsetiminginpsychologicalstudies.
BehaviorResearchMethods,Instruments,&Computers,35,276–284.
doi:https://doi.
org/10.
3758/BF03202553Plant,R.
R.
,&Turner,G.
(2009).
Millisecondprecisionpsychologicalresearchinaworldofcommoditycomputers:Newhardware,newproblemsBehaviorResearchMethods,41,598–614.
doi:https://doi.
org/10.
3758/BRM.
41.
3.
598Reimers,S.
,&Stewart,N.
(2007).
AdobeFlashasamediumforonlineexperimentation:Atestofreactiontimemeasurementcapabilities.
BehaviorResearchMethods,39,365–370.
doi:https://doi.
org/10.
3758/BF03193004Robinson,J.
,&McCormack,C.
(2015).
Timingcontrolforscript-basedanimations(W3CWorkingGroupNote,22September,2015).
Retrievedfromwww.
w3.
org/TR/2015/NOTE-animation-timing-20150922/Rorden,C.
,&Hanayik,T.
(2014).
StimSync:Open-sourcehardwareforbehavioralandMRIexperiments.
JournalofNeuroscienceMethods,227,90–99.
Schneider,W.
,Eschman,A.
,&Zuccolotto,A.
(2002).
E-Primeuser'sguide.
Pittsburgh,PA:PsychologySoftwareTools.
Schubert,T.
W.
,D'Ausilio,A.
,&Canto,R.
(2013).
UsingArduinomi-crocontrollerboardstomeasureresponselatencies.
BehaviorResearchMethods,45,1332–1346.
doi:https://doi.
org/10.
3758/s13428-013-0336-zSegalowitz,S.
J.
,&Graves,R.
E.
(1990).
SuitabilityoftheIBMXT,AT,andPS/2keyboard,mouse,andgameportasresponsedevicesinreactiontimeparadigms.
BehaviorResearchMethods,Instruments,&Computers,22,283–289.
vanSteenbergen,H.
,&Bocanegra,B.
R.
(2016).
PromisesandpitfallsofWeb-basedexperimentationintheadvanceofreplicablepsycholog-icalscience:AreplytoPlant(2015).
BehaviorResearchMethods,48,1713–1717.
doi:https://doi.
org/10.
3758/s13428-015-0677-xBehavRes(2017)49:1686–16951695