deepbaiduspider

GeorgiaSouthernUniversityDigitalCommons@GeorgiaSouthernElectronicThesesandDissertationsGraduateStudies,JackN.
AverittCollegeofSpring2016DeterminingUniqueAgentsbyEvaluatingWebFormInteractionBenCooleyFollowthisandadditionalworksat:https://digitalcommons.
georgiasouthern.
edu/etdPartoftheDigitalCommunicationsandNetworkingCommons,andtheOtherComputerEngineeringCommonsRecommendedCitationCooley,Ben.
DETERMININGUNIQUEAGENTSBYEVALUATINGWEBFORMINTERACTIONThisthesis(openaccess)isbroughttoyouforfreeandopenaccessbytheGraduateStudies,JackN.
AverittCollegeofatDigitalCommons@GeorgiaSouthern.
IthasbeenacceptedforinclusioninElectronicThesesandDissertationsbyanauthorizedadministratorofDigitalCommons@GeorgiaSouthern.
Formoreinformation,pleasecontactdigitalcommons@georgiasouthern.
edu.
DETERMININGUNIQUEAGENTSBYEVALUATINGWEBFORMINTERACTIONbyBENJAMINCOOLEY(UndertheDirectionofJamesHarris)ABSTRACTBecauseoftheinherentrisksintoday'sonlineactivities,itbecomesimperativetoidentifyamalicioususermasqueradingassomeoneelse.
IncorporatingbiometricanalysisenhancestheconfidenceofauthenticatingvalidusersovertheInternetwhileprovidingadditionallayersofsecuritywithnohindrancetotheenduser.
ThroughtheanalysisoftrafficpatternsandHTTPHeaderanalysis,thedetectionandearlyrefusalofrobotagentsplaysagreatroleinreducingfraudulentloginattempts.
INDEXWORDS:Web,Security,Biometrics,Computerscience,Robot,Machinelearning,Keystrokedynamics,Network,Website,AutomationDETERMININGUNIQUEAGENTSBYEVALUATINGWEBFORMINTERACTIONbyBENJAMINCOOLEYB.
S.
AugustaStateUniversity,2010AThesissubmittedtotheGraduateFacultyofGeorgiaSouthernUniversityinPartialfulfillmentoftherequirementsforthedegreeMASTEROFCOMPUTERSCIENCESTATESBORO,GEORGIA2016BENJAMINCOOLEYAllRightsReserved1DETERMININGUNIQUEAGENTSBYEVALUATINGWEBFORMINTERACTIONbyBENJAMINCOOLEYMajorProfessor:Committee:JamesHarrisLixinLiYoumingLiElectronicVersionApprovedMay20162ACKNOWLEDGEMENTSIwishtothankmyadvisorJamesHarrisforhisinterestandsupportofmywork.
ForthehelpandsupportofmyfatherStuartCooleyandmotherHeatherTew,Igivemyutmostgratitude.
Additionally,IwouldliketothankmyuncleSteveCampbellandhiswifeRachael,KatieHotchkiss,BrianMykietka,andJoshGarwoodfortheirhelpinproofingandtheirsupportthroughoutthecompletionofthisthesis.
3TABLEOFCONTENTSACKNOWLEDGEMENTS.
2LISTOFFIGURES5LISTOFTABLES.
6LISTOFEQUATIONS.
7CHAPTERS1INTRODUCTION82LITERATUREREVIEW9WEBROBOTS9DETERMININGHUMANNESS12BEHAVIORSOFAGENTS.
13DETERMININGUNIQUENESSBETWEENHUMANAGENTS.
213PURPOSE.
284METHODOLOGY29WEBSITETOTRACKINPUTS29WEBROBOTTRAFFICGENERATION34COMPARISONMETHODS.
375ANALYSISOFRESULTS42TRAFFICANALYSIS42TIMINGVECTORANALYSIS49SUMMARY546FUTUREWORK.
567CONCLUSION.
57REFERENCES58APPENDICES61ADISSIMILARITYALGORITHM.
61BDATACAPTURELAYOUTTEMPLATEMARKUP63CLOGINPAGEMARKUP.
64DNON-INTRUSIVEDATACAPTURINGJAVASCRIPT67EINTERACTIONFORMCONTROLLER.
694FOBJECTMODELENTITIES.
72GBACKPROPAGATIONCLASSIFIER.
75HSUPPORTVECTORMACHINECLASSIFIER76IDECISIONTREECLASSIFIER77JDISSIMILARITYCOMPARISONCLASSIFIER78KCOMMONEXPERIMENTVARIABLESANDFUNCTIONS.
80LBACKPROPAGATIONTEST83MSUPPORTVECTORMACHINETESTS84NDECISIONTREETESTS86ODISSIMILARITYCOMPARISONTESTS.
89PDATABASESCRIPT935LISTOFFIGURESFigure1:PercentageofRobotsbyHTTPRequests12Figure2:Visualrepresentationoftypedcharacters25Figure3:Patternofthetransformeddatathathasbeencaptured.
26Figure4:Imageofthedatacaptureapplication31Figure5:DataCaptureDatabaseSchema33Figure6:KeyHoldTimeswithFakeLogin37Figure7:KeyPressTimeswithFakeLogin376LISTOFTABLESTable1:HumanHeadersduringasessionontheDataCaptureWebSite15Table2:"MJ12bot"RobotheadersduringasessionontheDataCaptureWebSite16Table3:Humanclickpath16Table4:Passwordtimesbetweenkeystrokes24Table5:Differencesintimebasedonwhenakeyispressedandthepreviouskeyreleased25Table6:SampleRequestHeaders30Table7:Samplerequestinformation30Table8:Sampleinputcapture(Keyboard)30Table9:Sampleinputcapture(Mouse)31Table10:ValidLoginAttempts36Table11:AdditionalKnownAttempts36Table12:Asampleofrandomlygeneratedloginattempts36Table13:User-AgentHeaders43Table14:ResourcesAccessedbyRobots43Table15:FastResourceAccessHeaders44Table16:UsersubmittedtimingvectorforEmailField45Table17:SeleniumsubmittedtimingvectorforEmailField47Table18:SahisubmittedtimingvectorforEmailField48Table19:ResultsofSupportVectorMachineTraining51Table20:DecisionTreeResults(valuesareper10,000)527LISTOFEQUATIONSEquation1:DissimilarityEquation41Equation2:ValidloginFunctionusingDissimilaritymethod418CHAPTER1INTRODUCTIONIdentifyingthehumannessofaparticularuserofaWebpageiscrucial.
Webrobotsconsumevaluablebandwidthandperformautomatedactions,suchasrepeatedattemptstologintouseraccounts.
StudiessuggesttheWebrobottohumanratioisashighas10:1regardingthenumberofsessionsopenedonaWebserver;andashighas4:1concerningbandwidthconsumedonaparticularWebserver[1].
ThisthesisexploresthedistinctionsbetweenhumansandrobotsandtheuniqueinteractionsbetweenhumansandWebforms.
TheexplorationdefinesWebrobotsandtheiruses,thebehaviorsofdifferentagents,trafficanalysisbetweenhumanandnon-humanrobots,andvariousidentificationmechanisms.
Thisresearchalsoevaluatesthedifferencesbetweenhumansandrobots,anddifferencesbetweenmultiplehumanactors.
9CHAPTER2LITERATUREREVIEW2.
1WebRobotsTherearetwoclassificationsofWebrobots:non-maliciousandmalicious.
MostcurrentresearchidentifiesWebrobotsbasedonWeb-browsingpatternsandresourcerequests.
TherehasbeenlittleresearchconcerninghowrobotsinteractwithWebforms.
AlatersectionofthisthesisanalyzescollecteddatathatdetailstheinteractionsbetweenWebrobotsandWebforms.
2.
1.
1Non-MaliciousRobotsNon-maliciousWebrobotsnormallytaketheformofaWebcrawler;arobotthattraversesWebpagessearchingforpiecesofinformation.
Thereareseveralsub-classificationsofWebcrawlers.
Theseclassificationsincludeindexer,verifier,harvester,scraper,RSScrawler,experimental,andunknowntypes[2].
2.
1.
1.
1IndexerIndexerrobotscreateamapoftheInternet.
TheserobotsfollowobjectsdefinedinaWebsite'srobots.
txtandlinkstocreatethemap.
2.
1.
1.
2VerifierVerifierrobotsverifyorvalidateasetoffunctionalityorrulesofWebpages.
SeveralexamplesofverificationareHTMLverification,brokenlinkdetection,accessibilitycompliance,andqualityassurancetesting.
102.
1.
1.
3HarvesterHarvesterrobotsdowhattheirnameimplies;harvestcontentonWebpages.
Thesetypesofcrawlersrequestresourcessuchasimagesanddocuments.
2.
1.
1.
4ScraperScraperrobotssaverequestedHTMLfiles.
Maliciousandnon-malicioususerspurposethesaveddatafortheirownmeans.
MaliciouspurposesincludethegatheringofemailaddressesforanattempttobreakintoaWebpage.
Non-maliciouspurposesincludegatheringdataforlanguageanalysisanddeeplearning.
2.
1.
1.
5RSSCrawlerRSScrawlerskeepup-to-dateinformationfromvariousRSSfeeds.
ThiscrawlerisoftenonascheduleandrefreshestheRSSfeedsperiodically.
2.
1.
1.
6ExperimentalExperimentalcrawlersuseasetofcombinedtechniquestomap,verify,orharvestdataontheWeb.
Thesearenaturallyusedforresearchorexperimentalpurposes,hencethenameExperimentalcrawler.
2.
1.
1.
7UnknownOtherWebcrawlersexist,howeverthefunctionoftheseareoutsideofthescopeofthisthesis.
112.
1.
2MaliciousRobotsVariousrobottypesexhibitmaliciousintent.
Malicioususersautomatetasksthattakesignificanttimeforahumantoperform,increasingtheefficacyofanattack.
Someexamplesofmaliciousrobotsincludethosethatdestroyordisruptanonlineservice.
2.
1.
2.
1DestructiveDestructiverobotscancauseharmtoaWebserverbyperformingoperationssuchasfillinguptheharddisk,registeringmanyinvaliduserstotakeovertheuser-base,orexploitmechanismsofabusinessforfinancialgain.
2.
1.
2.
2DisruptiveSomedisruptiverobotspreventWebserversfromservingpagestolegitimateusers.
ADenialofServiceattackpreventsaccesstoWebcontentbycausingaWebservertoexecutetoomanyI/Orequests,orfloodsanetworkwithrequests.
Spamrobotsareanotherformofdisruptiverobot.
TheserobotsgenerateinformationonanonlineWebapplication,orsendunwantedemailstoanenduser.
Thepercentageoftrafficthatspamrobotsgeneratehasdecreasedoverthepreviousyears[3].
2.
1.
3DistributionofRobotsWebrobotsaccountformostWebtraffic,accountingforupto61.
5%ofallWebtrafficontheInternet[3].
WhileWebcrawlersareaccountablefor20%to31%ofglobalWebtraffic,independentWebsitesmaysee50%ormoreoftheirowntrafficgeneratedbyWebcrawlers[3][4][5].
Maliciousrobotsaccountforasmuchas5%ofInternettraffic.
Figure1showsthedistributionoftrafficbyrobots.
Therearefewresourcesthatspecifythedistributionofrobotcategories,however,twopapersbyDorenprovidedetailsonpotential12distributionsofWebrobots[2][6].
DorenandhiscolleaguesattheUniversityofConnecticutcollectedthedatabetween2007and2009.
Figure1-PercentageofRobotsbyHTTPRequests2.
2DeterminingHumannessByassessinglogfilesandforminteractions,itispossibletoidentifynon-humanrequests.
Currentresearchsuggeststhatvariousmeasuresofinternet-browsingpatternscanidentifyrobots.
Forexample,indexersoftenvisitarobots.
txtfileontheWebserver,wherehumansseldomvisitthisresource.
ItispossibletodeterminepotentialrobotsbyweighingthetimebetweenrequestsandthetimearobotinteractswithaWebpage.
EvaluationofrequestheadersandothercommonWeb-browserbehaviorstogaugewhetherauserisaWebrobotispromising.
Severalattributesusedasheuristicdetectionmarkers[7]are:1)Didtheuserrequestthe"robots.
txt"file2)Whatisthetimebetweenrequests3)Whatisthevariancebetweenrequesttimes4)Arethere404responsecodes40776392176013000234010203040506070IndexerVerifierHarvesterScraperRSSCrawlerExperimentalUnknownOtherPercentageofRobotsbyHTTPRequests20072009135)Arethere400responsecodes6)Doestherequesthaveabotuser-agentheader7)Doestherequesthaveauser-agentheaderthatdoesnotmatchacommonbrowser8)Dorequestshavethesameorsimilarreferrerforeachpagevisited9)DoestheURLresolutionhashighnumberofunresolvedrequests10)IstheHTTPversionnumberincludedComparingtherequestingIPaddresstoalistofIPaddressesknowntogeneratetrafficviaautonomousagentscanidentifyrobotagents.
Forexample,GooglestatestheirrobotsoriginatefromaDNSentrywhosedomainisgooglebot.
comorgoogle.
com.
Theaddressreturnedbydoingahostnamelookupforapotentialrobotisavaluewhosedomainwasasspecifiedabove.
Duringthiswriting,IPAddress:66.
249.
66.
1givesahostnameof"crawl-66-249-66-1.
googlebot.
com"andisdefinedasagooglecrawler.
WhenmachinelearningalgorithmsaretrainedusingthepatternsofknownWebrobots,autonomousagentscanbeidentified.
Classificationtechniquessuchasneuralnetworks,supportvectormachines,anddecisiontreesareexamplesofthesealgorithms.
Thisthesislaterintroducesthesealgorithmsandtheresultsoftheseclassifiers.
2.
3BehaviorsofAgents2.
3.
1HumanBehaviorSeveralmechanismsidentifyhumanbehaviorontheWeb.
ThesemechanismsassessWebpageaccesstime,sessioncosts,sessionlength,andlinkfollowingbehavior[1][7][8][9].
HumansseldomaccessmultipledifferentWebpageswithinasinglesecond.
Humanbrowsingrequiresvisualorauditoryprocessingwhichresultsinmuchslowerrequesttimesthanautomated14agents.
ItisevenrarerforhumanstoaccessdifferentWebpageswithinasinglehalfsecond.
Webbrowsersusedbyhumansloadadditionalresourcessuchasimages,scriptfiles,andothermultimediacontent.
Byrequestingmultimediaandscriptfiles,atypicalhumansessioncostsmorenetworkbandwidthandmemorythanarobotagentdoes.
Robotagentstendtoforgotheseadditionalresources,astheydonotprovidetheresourcesthatmostrobotsrequire.
Sessionlength,thenumberofWebpagerequestsmadebyanagent,isanimportantcriterionwhendeterminingifanagentishuman.
HumanstypicallyrequestfewerWebpagesthanrobotsinanygivensession[1].
HumansregularlyrequestfewerWebpagesoverasessionbecausehumanstendtofollowlinksofinterest,insteadoftryingtoindex,orharvestdataasrobotsdo.
Becausehumansalsoprocessinformationmoreslowlythanrobots,thesessiontimesarelongerforhumansthanrobots.
Humansrepresentfeweraccessespertimeintervalthanrobots,whichmeansthatthenumberofrequestspertimeintervalisagoodmeasureofhumanness.
Thewayhumansfollowlinksisanothercriteriontoconsider.
Humanbehaviorregularlyfollowsadepth-firstapproachinfollowinglinks,whereasrobotsusuallyperformabreadthfirstsearch.
RobotstendtoqueuelinkslocatedonaWebpage,whicharguesforwhyrobotsperformbreadth-firstsearches.
Moreover,robotsfollowhiddenlinksmoreoftenthandohumans.
2.
3.
1.
1PredictabilityandStandardBehaviorHumansdemonstrateseveralpredictablebehaviorswhenitcomestointeractingwithWebpages.
ThissectionexploresHTTPheaders,networkbehaviorpatterns,andforminteractionbehaviorsastheypertaintohumanagents.
15HTTPheadersthatcomefromhumanagentsarepredictable,meaningtheytendnottodifferentiateinthenumberortypeofheaderssent.
WildHTTPheaderswerecapturedusingadata-capturingwebsite.
ThemethodologyofthisresearchexploresthewaythatthisWebsitecapturesdataanddiscusseshowthedataisanalyzed.
DatacollectedbythisWebsiteshowsthatmanyWebcrawlersdonotsubmitacookieheaderifpromptedtosendonebacktotheserver;mosthumanagentsreturnthisheader.
Thetablesbelowshowthisbehaviorforhumanandrobotagents(cellsareshadedtoaidreading).
Onlythehumanrespondswiththerequiredcookie.
TheDataCaptureWebSitesendsaresponseheader"Set-Cookie",whichonlyhumanagentsseemtorespectwhenperformingadditionalrequestsduringasession.
SeeTable1below.
IPAddressURLHeaderNameDateandTime76.
26.
211.
37http://www.
trytologin.
com/X-REWRITE-URL2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/Upgrade-Insecure-Requests2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/User-Agent2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/Host2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/Accept-Language2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/Accept-Encoding2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/Accept2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/Connection2015-10-2519:39:40.
31076.
26.
211.
37http://www.
trytologin.
com/X-REWRITE-URL2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/Upgrade-Insecure-Requests2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/User-Agent2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/Referrer2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/Host2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/Cookie2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/Accept-Language2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/Accept-Encoding2015-10-2519:39:53.
64076.
26.
211.
37http://www.
trytologin.
com/Accept2015-10-2519:39:53.
640Table1-HumanHeadersduringasessionontheDataCaptureWebSiteTable1contrastswithTable2inthatTable1displaysthehumanreturningaheader,whereasTable2exhibitsrobotbehaviorthatdoesnotresendtheheader.
16IPAddressUrlNameDateandTime91.
121.
169.
194http://www.
trytologin.
com/Connection2015-09-2920:00:43.
09791.
121.
169.
194http://www.
trytologin.
com/Accept2015-09-2920:00:43.
09791.
121.
169.
194http://www.
trytologin.
com/Accept-Language2015-09-2920:00:43.
09791.
121.
169.
194http://www.
trytologin.
com/Host2015-09-2920:00:43.
09791.
121.
169.
194http://www.
trytologin.
com/User-Agent2015-09-2920:00:43.
09791.
121.
169.
194http://www.
trytologin.
com/X-REWRITE-URL2015-09-2920:00:43.
09791.
121.
169.
194http://www.
trytologin.
com/Connection2015-09-2920:00:46.
53791.
121.
169.
194http://www.
trytologin.
com/Accept2015-09-2920:00:46.
53791.
121.
169.
194http://www.
trytologin.
com/Accept-Encoding2015-09-2920:00:46.
53791.
121.
169.
194http://www.
trytologin.
com/Accept-Language2015-09-2920:00:46.
53791.
121.
169.
194http://www.
trytologin.
com/Host2015-09-2920:00:46.
53791.
121.
169.
194http://www.
trytologin.
com/User-Agent2015-09-2920:00:46.
53791.
121.
169.
194http://www.
trytologin.
com/X-REWRITE-URL2015-09-2920:00:46.
537Table2-"MJ12bot"RobotheadersduringasessionontheDataCaptureWebSiteHumanstendtorequestHTTPresourcesfromthemainWebpageandsubsequentvisitedpages.
Webpagemarkupexposesresourcesmeantforobservation.
StandardWebbrowserrequestssubmitreferrerheaders,whichcreatealogicalclickpathastheusernavigatesaWebsite.
Thisisduetotheagentfollowinglinksreadilyavailableonthewebpageandbyusingnormalbrowsercontrols,suchasforwardandback.
Thefollowingtablecontainsasubsetofacapturedhumanclickpath.
IPAddressReferrerURLRequestTypeDateandTime202.
67.
36.
228//Home/ContactGET2015-09-1123:06:44.
863202.
67.
36.
228/Home/Contact/GET2015-09-1123:06:59.
940202.
67.
36.
228//POST2015-09-1123:07:53.
200202.
67.
36.
228//POST2015-09-1123:08:02.
497202.
67.
36.
228//Home/AboutGET2015-09-1123:08:15.
857202.
67.
36.
228//Home/AboutGET2015-09-1123:08:16.
247202.
67.
36.
228/Home/About/GET2015-09-1123:08:21.
093Table3-HumanclickpathPatternshaveidentifiabledifferencesinhumanandroboticnetworkbehavior[6][10][11].
Forexample,whenhumanagentsaccessWebpagestheytendtocreateasingleHTTPrequest.
Iftheserverloadistoohightohandlerequestsinatimelymanner,theuseroftendoesnotcontinueto17makeHTTPrequests.
Conversely,arobotdoesnotcareifresponsetimesarelongandcontinuestoattempttoaccesstheresourcefromtheWebserver[10].
2.
3.
2Robotbehavior2.
3.
2.
1AccessMaterialsnotgenerallyseenbyhumansSomecrawlersattempttoaccessanyresourceavailabletothem[11].
ResourceavailabilityisassessedthroughdifferentmechanismssuchasiftheresourceappearingwithadirectlinkfromtheWebpage,fromanotherresource(suchasrobots.
txt),throughtryingtoaccessresourcesfromasearchengineorothersite'slinks,orbyrandomlytryingtoaccessresourcesknowntoexist(loginpages,configurationfiles,etc.
)[6][11].
BasedonthelogsfromtheDataCaptureWebSite,itwasdeterminedthatarobotattemptstoaccessdeniedresourcesasdefinedintherobots.
txt.
Onlyoneagentaccesseddeniedresources;itwaslikelytryingtoenumerateresources,whichcouldprovideadministrativeaccess.
Mostethicalbots,suchastheGooglebotandBingbot,didnotattempttoaccesstheresourcesmarkedasdeny.
2.
3.
2.
2ResourceAccessTimesvsInteractionTimeAspresentedearlier,Webpageaccesstimesallowfortheidentificationofrobots.
RobotstendtoaccessseveralWebpagesveryquickly,usuallywithinafractionofasecond.
Humanusers,whiletheyareabletoaccessresourcesatsuchspeeds,donotusuallydoso.
Humanbrowsingtimetendstobeoverone-halfsecondperpagerequest,whereasrobotswouldhavemadenumerousrequestsforresourcesinthistime[1].
AnotheritemofinterestconcerningresourceaccesstimesistheidentificationofrobotsviatheiraccessofImagefilescomparedtoHTMLfiles.
Mostcrawlers,excludingharvesters,tendnottograbimages.
Instead,theserobotsoftenpreferitemsofasmallersizesuchastextfilesandWeb18pages[6].
Normalhumanbrowsingrequestseveryimageonthepage,ifimagesexist.
Robotstendtoexcludetheseimages.
AlNoamany,etal.
,considertheratioof10Webpagerequeststo=0.
Theclassificationofaresultisinvalidiftheresultwerelessthan0.
394.
3.
2SupportVectorMachinesSupportvectormachinesaddreliableclassificationstoadatamodel.
Whileexploringvariousmethodologiestoperformclassifications,thisclassificationsystemseemedpromisingformanytypesofanalysis.
Theimplementationofasupportvectormachinewasusedtogaugetheeffectivenessofthismodelofinput.
UsingAccord.
Neuroframework[28],thesupportvectormachineevaluatedwasakernelsupportvectormachine,whichhadthefollowingparameters:Numberofinputs:10Kernel:Gaussianwithasigmaof100LearningAlgorithm:sequentialminimaloptimizationThetestsrunfortheSVMran1000timestogainanaveragebaseline.
ThesetestsusedthesetofvalidinputsandtheknowninvalidinputfortrainingtheSVM.
Mosttestsusedasetofrandomlygenerateddatatotrainasinvalidloginattempts.
Thetestsincludedeither0,10,100,or1000randomlygeneratedinputstouseduringtraining.
Oncetrainingfinished,10,000loginattemptswereclassified.
TheclassificationofaresultisvalidiftheSVMgaveanoutput>=0.
Theclassificationofaresultisinvalidiftheresultwaslessthan0.
4.
3.
3DecisionTreesWhentrainingthedecisiontrees,severaldifferentparametersregardinginvalidtestingdatawereevaluated.
ThedecisiontreeengineusedtheC45algorithm,teninputs,andcontinuousvariablesrangingfrom-10000to10000.
Trainingoftheengineincludedsixvalidlogins,oneinvalidlogin,and10,000generatedinvalidloginattempts.
Resultsobtainedfromthisinitialmodelencouragedtheuseofadditionaltrainingusingrandomlygeneratedinvalidlogins.
Experimentswerethen40performedusingtrainingby1,10,100,1000,and5000randomlygeneratedinvalidlogins.
TheclassificationofaresultisvalidiftheSVMgaveanoutputofone.
Theclassificationofaresultisinvalidiftheresultwereequaltozero.
4.
3.
4DissimilarityComparisonAdissimilaritycomparisonfunctioncomparesknowninputstobuildageneralizedmodelwithvariousstatisticsandcomparesthesevaluesagainstanattemptedvalidation.
Thegeneralapproachofthiscomparisonhasitsbaseinstatisticalmodeling.
Severalvalidloginattemptswerecapturedtocreateamastermodel,alsoknownasamastertrajectory.
Fiveknownvalidinputswereusedtocreatetheinitialmodelduringtheinitialimplementationofthealgorithm.
Theaveragevalueforallknownfeatureswascalculatedandstoredasifitwasasinglevalidloginwhencreatingthismastertrajectory.
Comparisonofeachvalidloginattempttothemastertrajectorywasusedtocalculateaveragedissimilarityandstandarddeviation.
Comparisonofaveragedissimilarityagainstanyadditionalinputtoperformverificationtakesplaceoncethismodelwascreated.
Thesetofinputsforanattemptedlogincomprisesthetimeakeyispressed,andthepreviouskeyisreleased.
Thevaluesarestoredinanarrayofnumericvaluessuchas[80,0,134,-31,55,210…].
ThesedatapointsareplottedonaCartesianmaptocreatetheknowndissimilarityprofile;forthesamplejustprovided:[{X=80,Y=0},{X=134,Y=-31},{X=55,Y=210}…].
Equation1showsthegeneralequationforcalculatingthedissimilaritybetweentwoprofiles(MandA).
41(,)=∑√()2+()2=1Equation1-DissimilarityEquationTheequationsshowninEquation2showtheevaluationcriterianeededinordertovalidateifauserhastypedinthesamemannerashisorherpreviousattempts.
=+()()=(,)≤Equation2–ValidloginFunctionusingDissimilaritymethodWhenanewloginattempttakesplace,thedissimilaritybetweenthenewattemptandthemastertrajectoryarecalculated.
Thiscalculationisthencomparedagainstanevaluationthreshold.
Ifthedissimilarityoftheattemptislessthantheevaluationthreshold,theuserisauthenticated.
42CHAPTER5ANALYSISOFRESULTSAlldatausedforanalysiscanbefoundathttps://goo.
gl/zkP3TE.
NotethatthislinkrequiresaGeorgiaSouthernUniversitylogintoaccesstheinformation.
5.
1TrafficAnalysisCluesexisttoidentifyagentsashumanorroboticwhenevaluatingthetrafficgeneratedfromtheWebapplication.
MostoftheindexerrobotsthataccessedtheapplicationhadsomeidentifierintheHTTPrequestheader.
However,otherfactorsshouldbeevaluated,includingtheinter-pageaccessesandforminformationgatheredbyformsubmissions.
5.
1.
1HTTPHeadersandAccessTimesItispossibletoidentifyrobotagentsthroughtheevaluationofHTTPheaders.
However,withmoresophisticatedrobotagents,theseheadersappearasvaliduserrequests.
Asdescribedintheintroduction,thereissomedisparitybetweenwhatWebbrowsersandrobotagentssubmitasHTTPheaders.
ByutilizingtheDataCaptureWebSite,theexclusionorinclusionofcertainfieldheadersquicklyidentifiespotentialroboticagents.
However,itismoredifficulttoidentifyhumanagentsfromoneanotheriftheyweretousethesameWebbrowser.
Table13showsthemostcommonUser-AgentheadersthatweresubmittedtotheDataCaptureWebSite.
Notetheoccurrenceofvaluessuchas"bot","agent",or"spider"(theserowsarehighlighted).
43User-AgentHeadersMozilla/5.
0(compatible;MSIE10.
0;WindowsNT6.
1;Trident/6.
0)Mozilla/5.
0(compatible;Googlebot/2.
1;+http://www.
google.
com/bot.
html)Mozilla/5.
0(compatible;MJ12bot/v1.
4.
5;http://www.
majestic12.
co.
uk/bot.
php+)Mozilla/5.
0(compatible;Baiduspider/2.
0;+http://www.
baidu.
com/search/spider.
html)Mozilla/5.
0(WindowsNT5.
1;rv:6.
0.
2)Gecko/20100101Firefox/6.
0.
2Mozilla/5.
0(compatible;bingbot/2.
0;+http://www.
bing.
com/bingbot.
htm)Mozilla/5.
0(WindowsNT6.
1;WOW64)AppleWebKit/537.
36(KHTML,likeGecko)Chrome/36.
0.
1985.
143Safari/537.
36Mozilla/5.
0(iPhone;CPUiPhoneOS8_3likeMacOSX)AppleWebKit/600.
1.
4(KHTML,likeGecko)Version/8.
0Mobile/12F70Safari/600.
1.
4(compatible;Googlebot/2.
1;+http://www.
google.
com/bot.
html)Mozilla/5.
0(compatible;NetcraftSurveyAgent/1.
0;+info@netcraft.
com)Mozilla/5.
0(X11;Linuxx86_64)AppleWebKit/537.
36(KHTML,likeGecko)Chrome/37.
0.
2062.
120Safari/537.
36Mozilla/5.
0(WindowsNT6.
1;WOW64)AppleWebKit/537.
36(KHTML,likeGecko)Chrome/32.
0.
1700.
107Safari/537.
36Table13-User-AgentHeadersSeveralautonomousagentssawsub-half-secondresourceaccesstimes.
Tables14and15displaytheresourcesaccessedandtheirheaders.
Resourcesaccessedinclude"robots.
txt",whichisgenerallyaresourceaccessedbyrobots.
TheheadersforthesamerecordsspecifythattheUser-Agentisarobot.
IPAddressUrlDate52.
26.
82.
4http://www.
trytologin.
com/10/24/20157:05:17.
67752.
26.
82.
4http://www.
trytologin.
com/10/24/20157:05:17.
77052.
26.
82.
4http://www.
trytologin.
com/Home/About10/24/20157:05:18.
11366.
249.
79.
217http://www.
trytologin.
com/robots.
txt10/31/20151:42:35.
25766.
249.
79.
217http://www.
trytologin.
com/10/31/20151:42:35.
33366.
249.
79.
217http://www.
trytologin.
com/robots.
txt11/7/201517:17:34.
66066.
249.
79.
217http://www.
trytologin.
com/11/7/201517:17:34.
767158.
69.
225.
37http://www.
trytologin.
com/robots.
txt12/21/20159:31:31.
087158.
69.
225.
37http://www.
trytologin.
com/12/21/20159:31:31.
230216.
145.
14.
142http://www.
trytologin.
com/12/25/20151:49:41.
457216.
145.
14.
142http://www.
trytologin.
com/robots.
txt12/25/20151:49:38.
847Table14-ResourcesAccessedbyRobots44IPAddressDateUser-Agent52.
26.
82.
410/24/20157:05:17.
677BusinessBot:Nathan@lead-caddy.
com52.
26.
82.
410/24/20157:05:17.
770BusinessBot:Nathan@lead-caddy.
com52.
26.
82.
410/24/20157:05:18.
113BusinessBot:Nathan@lead-caddy.
com66.
249.
79.
21710/31/20151:42:35.
257Mozilla/5.
0(compatible;Googlebot/2.
1;+http://www.
google.
com/bot.
html)66.
249.
79.
21710/31/20151:42:35.
333Mozilla/5.
0(compatible;Googlebot/2.
1;+http://www.
google.
com/bot.
html)66.
249.
79.
21711/7/201517:17:34.
660Mozilla/5.
0(compatible;Googlebot/2.
1;+http://www.
google.
com/bot.
html)66.
249.
79.
21711/7/201517:17:34.
767Mozilla/5.
0(compatible;Googlebot/2.
1;+http://www.
google.
com/bot.
html)158.
69.
225.
3712/21/20159:31:31.
087Mozilla/5.
0(compatible;Lipperhey-Kaus-Australis/5.
0;+https://www.
lipperhey.
com/en/about/)158.
69.
225.
3712/21/20159:31:31.
230Mozilla/5.
0(compatible;Lipperhey-Kaus-Australis/5.
0;+https://www.
lipperhey.
com/en/about/)216.
145.
14.
14212/25/20151:49:41.
457Mozilla/5.
0(Windows;U;WindowsNT5.
1;en;rv:1.
9.
0.
13)Gecko/2009073022Firefox/3.
5.
2(.
NETCLR3.
5.
30729)SurveyBot/2.
3(DomainTools)216.
145.
14.
14212/25/20151:49:38.
847Mozilla/5.
0(Windows;U;WindowsNT5.
1;en;rv:1.
9.
0.
13)Gecko/2009073022Firefox/3.
5.
2(.
NETCLR3.
5.
30729)SurveyBot/2.
3(DomainTools)Table15-FastResourceAccessHeadersTheseresultsagreewithotherresearchers'resultsandreportnonewresults.
5.
1.
2SubmittedformInformationWhenevaluatingsubmittedforminformationfromseveraldifferentsources,itwasapparentthatwildrobotsdidnotsubmittheWebform.
Thisislikelyduetotherobotsbeingindexers.
Asaresult,theseagentswerenotassessedforbiometricinformation.
However,scriptedagentsandhumanagentssubmittedforminformation.
OnereasontheformsubmissionfunctionalityisexcludedisinconsiderationofthelosttimeandresourceswhenaWebSiteperformsinputvalidation.
455.
1.
3HumanAgentsHumanagentsareoftenidentifiedprimarilybyslowerrequesttimesandbysubmittingparticulartimingvectors.
Table16displaystimingvectorsforoneofthehumanagentsthatprovidedinformationontheWebform.
NegativevaluesintheTimePressedcolumnindicatethatthekeywaspressedbeforethepreviouskeywasreleased.
ShiftPressedTimeReleasedTime-Pressed02140010160274-790374-15417041761544-20805614410664-960737-870175800263-710176320232-104022510406420239400278-1190321153Table16-UsersubmittedtimingvectorforEmailFieldThisagentiscategorizedasahumanduetothevarianceinthetimingvector.
Theseresultsareprovidedforcomparisonofroboticbehaviorinthefollowingsection.
5.
1.
4VisualStudioWebTestsVisualStudioWebTestsdonotallowfordirectinteractionwiththeWebpage;instead,itoptstosendpredefinedHTTPPOSTrequests.
VisualStudiotestsareidentifiedasroboticbehaviorby46evaluatingtheinter-requesttimebetweenpageaccess,thesubsequentpostingofdata,andthepostingoftheidenticalinformation.
Successfullycapturingandanalyzingdatahasprovidedstrikingresults.
Requestspeedisalreadyknowntobeacommonrobotdetectionmarker.
Nevertheless,repetitivedatapostingmightindicateanewrobotdetectionmarker.
However,tovalidatethisnewmarkermoreresearchisneeded.
5.
1.
5SeleniumSeleniumusesaWebbrowsertosendcommands,resultingintheidentificationoftypicalbrowserheaders.
However,itisdeterminedthatarobotisinteractingwiththeformwhentheframeworkisusednaivelyduetotherapidinsertionofkeypresses.
Theseinter-keypresstimesareexecutedwithsub-twomillisecondtimings,aspeedthatequatestosuperhumantypingspeeds.
Thoughthereexistlongerdelays,itdepictsatypicalhumantypingspeeds.
Table17showsthekeyboarddynamicsprofileascapturedfromtheSeleniumagent.
47ShiftPressedTimeReleased(ms)Time-Pressed(ms)02300810410510500510400520761040-3503934062041051052061Table17-SeleniumsubmittedtimingvectorforEmailFieldHowever,sinceaprogramminglanguagedrivesSelenium,itispossibletocreatedelaysbetweenkeypressestomodifythisbehavior.
Assuspected,thisagentiseasilyidentifiablethroughitsspeedoftyping.
Thespeedofdatainputisnotawell-researchedtopic,andthisprovidessomevalidityintokeystrokedynamicsasadetectionmarker.
5.
1.
6iMacrosiMacrospostsstandardheaderinformation,asitusestheactualbrowsertoperformwork.
However,wheninputtingdataintotheloginform,nokeystrokesaresenttotheform.
Sincenokeystrokesaresenttotheform,itbecomesapparentthatthiswasnotausertypinglogininformation.
Inthisscenario,notimingvectorswerecaptured.
Thesubmissionofaformwithoutanycommoninputmarkersissubmittedonbehalfofanautomationtoolandisanadditionalinputmarkerfordetectionofrobots.
485.
1.
7SahiSahiusesaWebbrowsertoperformautomatedsteps.
Requestheaderscannotdetermineifarobotisanagentperformingactions,duetotheagentusingaWebbrowser.
Determinationofarobotagentwascompletedbyevaluatingthesubmittedkeypressintervalandholdtimes;theseweretypicallylessthantwomillisecondsinduration.
Table18showsacapturedtimingvectorfromtheSahiagent.
Theinputtimingfromthisagentshowsrapidkeypressandkeyreleasetimings,fasterthanhumanscanconsistentlydoso.
TheshiftpressedcolumnshowsaBooleanvalues(1=true,0=false)iftheshiftkeywaspressedwithanotherkey.
TheTimeReleasedandTimePressedcolumnsshowtiminginmilliseconds.
ShiftPressedTimeReleasedTime-Pressed010011011021000011010010010011011010010010000011Table18-SahisubmittedtimingvectorforEmailFieldSimilarlytotheSeleniumexperiment,resultscapturedherealsoreflectthatrapidinputtimesareduetoanautomatedagententry.
Thisfurtheraddstothevaliditythatrapidinputtimingisamarkerforrobotagents.
495.
2TimingVectorAnalysisThebehaviorpatternstheautomatedagentsandhumansexhibitarevastlydifferent.
Often,HTTPheadersdifferanddisplayinformationindicativeofrobotpresence.
Othertimes,interactionofthewebpagedifferssignificantly.
Evaluationoftheforminteractionpatternspresentsaclearindicationofthedifferencesininputpatterns.
Humans,forexample,exhibitvaryinginputpatterns.
Robotstendtomaintainconstantinteractionpatterns-iftheyareprovidedatall.
Inaddition,robotpagerequesttimeisoftenmuchfasterthanthatofhumanagents;aspreviouslyshowninTable14-ResourcesAccessedbyRobots.
5.
2.
1HumanvHumananalysisInthelatersections,variousalgorithmsincludingbackpropagation,supportvectormachines,decisiontreeclassification,anddissimilaritycomparisonareevaluated.
Foreachofthesealgorithms,thedatathatwasusedisdescribedinthemethodology.
5.
2.
1.
1BackpropagationWhilepromisinginindividualresearchapplications,thelengthoftimerequiredbybackpropagationtrainingprohibitsitsuseinmodernWebapplications.
Manyresearchers,whoexperimentedwiththismethodtoprovideabiometricsignatureforlogin,utilizedseveraldifferenthumanagentstoprovidenegativetrainingmodelsintheirresearch.
Sincethisamountoftrainingisnotpracticalinthenormaliterationoftraining,additionalmeasuresneedtobeevaluatedthateitherextendordifferinmethodologyprovidedhere.
Itispossiblethatwhenregisteringanewuseraccounttocaptureacommonphraseanduselazylearningviabackpropagationtotrainthemodelusingasinglephraseorword.
However,thismethodrequiresretrainingforeveryindividualduringaccountregistration.
Thisdoesnotseemlikeapracticalarchitecturetouseinproduction50readysystemssincethetrainingandcomparisonneedstocomparethousands(ormore)userinputstoprovideinvalidlogindata.
Initialtestingofbackpropagationyieldedunexpectedresults.
Whilecreatingvalidtrainingusingsixinputsamples,fiveyieldedsimilardatainputprofilesasshowninFigure6-KeyHoldTimeswithFakeLoginandFigure7-KeyPressTimeswithFakeLogin.
Evenaftertrainingthenetworkwith2000randomlygeneratedinvalidloginattempts,thebackpropagationalgorithmcalculatedtheseattemptswiththesamecertaintyrateasvalidloginattempts.
Becauseofthesmallsampleofvalidloginsandtrainingwiththeinvaliddataset,thereisahighprobabilityofacceptinginvaliduserlogins.
Whentrainingtheneuralnetworkwithouttheserandomlygeneratedloginattempts,thereareonlyminordifferencesinclassificationcertainty.
Theseresultsclearlyshowthatamorerobusttrainingsystemisrequiredtoidentifythebiometricmarkersoftypingpatternscorrectly.
Theexperimentsperformedusingbackpropagationwereultimatelyunsuccessful.
Mostresearchonthistopicsupportsbackpropagationasahighlysuccessfulmeansofprovidingauthentication.
However,theresultsgatheredwiththisresearchsuggestotherwise.
Sincethismethodrequiresasubstantialamountoftrainingperpotentialuserofthesystem,thismethodisnotagoodchoicefordetermininginvalidattemptsusingWebforms.
5.
2.
1.
2SupportVectorMachinesAsanotherpromisingmethodofclassifyingdata,utilizationofsupportvectormachinesprovidedsuccessfulresults.
Withadequaterandomrecordstrainedasinvalidlogins,supportvectormachinesaccuratelydetermineifauserisvalidbasedonkeyboarddynamics.
ResultsofthistrainingareshowninTable19.
Eachtestperformed10,000attemptedlogins.
51Table19showstheresultsoftrainingandclassification,aswellasperformancemetrics.
#RandomTrainingRecordsFRRFARMemoryUsage*ProcessingTime*500011.
1%0.
0007%684MB1Second10001.
3%0.
0013%43MB288ms1000.
1%0.
0018%10MB82ms100%0.
0069%9.
7MB54ms0100%100%9.
7MB51msTable19-ResultsofSupportVectorMachineTrainingItisapparentthattheinclusionoftheinvalidtrainingrecordssignificantlyincreasestheaccuracyoftheFAR.
Whenutilizing1000ormorerandomtrainingrecords,memoryandprocessorconsumptionexplodes.
Trainingwithmanygeneratedloginscausesperformanceissuesifthesystemisinconstanthighdemand.
Supportvectormachinesaretoutedashighlyaccuratemethodsofclusteringdata,andtheresultsabovesuggestthatthisholdstruefortheseexperiments.
SVMsdisplayaccuracy,whichallowsforsecondaryauthenticationusingauser'skeystrokedynamics.
5.
2.
1.
3DecisionTreeDecisiontreeclassificationshowsmorepromisethandoesbackpropagationclassificationforthisdata.
Sinceonlyafewinputsthatclassifyasvalidloginsexist,usingrandomlygeneratedinvalidvaluescontributetoaveryhighdegreeofaccuracytowardsclassifyinginvalidlogins.
Whenusingthe6knownloginattemptsfortheuserinputthatwascapturedandoneinvalidattempttotrain,thefalseacceptancerate(FAR)isconsistentlybetween55and120attemptsper10,000attempts(.
55%to1.
12%falseacceptance,withanaverageof.
7977%)whenusing10,000generatedloginattemptstotestthemodelagainst.
Thefalserejectionrate(FRR)forthismodelyieldednoerrorsinthenumberoffalselyrejectedattemptswhenusingzerogeneratedloginattempts;theseattemptsincludetheinitialvalidmodelandanadditionalattemptedlogin.
52Asstatedinthemethodology,attemptstodecreasethefalseacceptancerateusedadditionalinvalidlogintraining.
Whilethedecreaseinfalseacceptancewhentrainedwithmoreinvalidlogins,theincreaseinfalserejectionratesmaydesignatetheneedfortruetrainingratherthangeneratedtraining.
FARandFRRvaluesforalltestsareshowninTable20.
Thesevaluesareper10,000generatedlogins.
Maxandminvaluesarecalculatedfromrepeatingtestingiterations1000times.
GeneratedInvalidLoginCountFalseAcceptanceMinFalseAcceptanceMaxFalseAcceptanceAverageFalseRejections05511079.
77010012160.
6814181001913479.
708535100008227.
73682650000326.
079775Table20-DecisionTreeResults(valuesareper10,000)Whileitisapparentthatthefalseacceptancerateisreducedoncesignificanttrainingisundertaken,thenumberoffalserejectionsincreasesformostiterations.
Thisexperimentwasunsuccessfulatclassifyingvalidauthenticationsusingkeyboarddynamics.
Thisisduetothehighfalserejectionrateseenwhenusingmoreinvalidtrainingdata.
5.
2.
1.
4DissimilarityComparisonUsingthetestdatawiththismethodproducedpromisingresults.
Utilizingonlysixsamplestotrainthemodel,thismethodhasgivenanFARofNETFramework,"[Online].
Available:http://accord-framework.
net/.
[Accessed4102015].
[29]J.
Jin,J.
Offutt,N.
Zheng,F.
Mao,A.
KoehlandH.
Wang,"EvasiveBotsMasqueradingasHumanBeingsontheWeb,"IEEE,2013.
[30]A.
Rappaport,"Robots&Spiders&Crawlers:HowWebandintranetsearchenginesfollowlinkstobuildindexes,"infoseeksoftware.
[31]K.
Rivett,M.
Gorunescu,F.
Gorunescu,M.
Ene,S.
TenreirodeMagalahaesandH.
Santos,AUTHENTICATINGCOMPUTERACCESSBASEDONKEYSTROKEDYNAMICSUSINGAPROBABILISTICNEURALNETWORK,London:UniversityofWestminster.
61APPENDICESADissimilarityAlgorithmCode1-ValidatebyDissimilarity6263BDataCaptureLayoutTemplateMarkup64CLoginPageMarkup656667DNon-IntrusiveDataCapturingJavaScript6869EInteractionFormController707172FObjectModelEntities737475GBackpropagationClassifier76HSupportVectorMachineClassifier77IDecisionTreeClassifier78JDissimilarityComparisonClassifier7980KCommonExperimentVariablesandFunctions818283LBackpropagationTest84MSupportVectorMachineTests8586NDecisionTreeTests878889ODissimilarityComparisonTests90919293PDatabaseScript9495