relevancenofollow

AnatomyofCommentSpam2014,Imperva,Inc.
1.
ExecutiveSummarySpamisdefinedasirrelevantorunsolicitedmessagessentovertheInternet,typicallytolargenumbersofusers,forthepurposesofadvertising,phishing,spreadingmalware,etc.
Byspammingmultipletargetsoveralongperiodoftime,spammersareabletogainprofit,anddoharm.
Liketheflyersinourmailboxes,digitalspamstarteditspathtogloryviaemail.
However,withtheevolutionofwebtechnologiesandwebsiteinteraction,spammershavemovedtoreachingusersviatheweb,injectingspamcommentsintoforums,commentfields,guestbooks,andevenwebsiteslikeWikipedia,whichallowusergeneratedcontenttobepublished.
Andthus,commentspamwasborn.
Commentspammersaremostoftenmotivatedbysearchengineoptimization,sothattheycanuseapromotedsiteforadvertisementandmalwaredistribution.
Attackersarealsoknowntousecommentspamforthepurposeofclickfraud.
Thecommentspamissuehasbecomesoprevalentthatorganizationsarefightingback,byimplementingmitigationservices.
Interestingly,therehavebeenincidentsofspammersfightinganti-spammersinanattempttoshutdownthosemitigationservices,andmanyofthosecounterattackshavebeensuccessful.
Wedecidedtostudythecommentspamspacefrombothends.
Inourresearch,weexaminedtheattacker'spointofview,includingthecommentspamtechniquesandtools.
Inaddition,weexaminedthevictim'spointofviewtounderstandhoworganizationsdealwithcommentspamtoday.
1.
1KeyFindingsOverthecourseoftwoweeks,fromSeptember1toSeptember14in2013wemonitoredcomment-spammeractivityagainstmorethan60differentapplications.
Herearesomeofourkeyfindings:58percentofallcommentspammersareactiveforlongperiodsoftime.
17percentofallcommentspammersgeneratedthemajorityofcommentspam.
Inordertounderstandhowacommentspammerattacksawebapplication,welookedcloselyatasinglevictim.
Thisreportincludesthiscasestudyandwhatwehavelearned:80percentofcommentspamtrafficisgeneratedby28percentofattackers.
Overtime,commentspammersincreasedtheirvelocityagainsttheattackedwebsite.
1.
2MainConclusionsOurconclusionswerestraightforward:Identifyingtheattackerasacommentspammerearlyon,andblockingtherequests,preventsmostofthemaliciousactivityIPreputationwillhelpinsolvingthecommentspamproblem,byblockingcommentspammersearlyintheirattackcampaigns2TableofContents1.
ExecutiveSummary.
12.
Introduction.
33.
TheAttacker'sPointofView.
44.
CommentSpaminPractice.
45.
TheVictim'sPointofView106.
MitigationTechniques.
126.
1ContentInspection.
126.
2SourceReputation126.
3Anti-automation136.
4Demotivation.
136.
5ManualInspection.
137.
CaseStudies.
147.
1AnalyzingaSingleVictim.
147.
2AnalyzingaSingleAttacker.
157.
3AttackersAbuseGoogleAppEngineforCommentSpam.
198.
SummaryandConclusions.
2032.
IntroductionWikipedia'sdefinitionforcommentspam1:"Commentspamisatermusedtorefertoabroadcategoryofspambotpostingswhichabuseweb-basedformstopostunsolicitedadvertisementsascommentsonforums,blogs,wikisandonlineguestbooks.
"Anexampleforacommentspammedsite:Figure1–ASpammedSiteExampleAttackersusecommentspamforvariousreasons.
Themostsignificantoneis'SearchEngineOptimization'(SEO)–improvingasite'srankingwithinasearchengineresultset(withrespecttogivensearchterms).
Asiterankingwithinasearchengineresultsetisbasedonthenumberandqualityofwebsitesthatholdlinkstoit(AKA"backlinks").
Thus,postingmanycommentscontaininglinkstoatargetsiteincreasesitsrankingwithinsearchengineresultsets(especiallywithrespecttokeywordssurroundingthelink).
Attackersthenusethepromotedsiteforadvertisement(usuallyofdubiousmerchandise)andmalwaredistribution.
AttackersarealsoknowntousecommentspamforthepurposeofClickFraud.
1http://en.
wikipedia.
org/wiki/Comment_spam43.
TheAttacker'sPointofViewThereareafewbasicstagesanattackerfollowswhenaspiringtoproducecommentspamtraffic.
Eachofthesestagescanbeperformedseparatelyandneedstobefine-tuned:TargetAcquisition(AKAURLharvesting):Thetaskoffindingqualityvulnerablewebsitestopostcommentsonisnamed"URLharvesting".
TheURL'squalityismeasuredbytherelevancetothepromotedsite;theURL'sownsearchengineranking;thedifficultyofpostingcomments(forexampleun-protectedpublicpostsorCaptchaprotectedposts)andthesite'spolicyregardingsearchengines(forexamplethefollow/nofollowvalueofthe"rel"attributeofhyperlinks).
Posting:PostthecommentsonthechosenURLs.
Verification:Verifythatthecommentswereindeedpublished.
4.
CommentSpaminPracticeTheattacker'ssuccessreliesonpublishingcommentspaminlargescales.
Largescalecommentspamisachievedbyautomatingtheaforementionedcommentingprocess.
Forthispurpose,automatictoolsweredevelopedwhichsupportthisprocessandoffercomplementaryservices.
Thetools'inputisasetofkeywordsrelevantforthepromotedsite.
Theautomatedtoolsmayencompassallofthefollowingsteps,oronlypartofthem:URLharvesting:Automatictoolsusepopularsearchenginestolocaterelevantwebsitesbasedoninputkeywords.
Uponsuccess,thetoolexploresthefoundwebsites,inordertolocatesuitableURLsforcommenting.
Blogsarethemostpopularwebsitesforcommentspamposting.
Infact,sometoolsarelimitedtoharvestingonlyblogs,andspecificallyWordPressblogs.
Figure2-AutomatedTool(G-LockBlogFinder)forHarvestingshowsanexampleofanautomatedtool(G-LockBlogFinder)whichspecificallyofferstoharvestblogs.
Theuserspecifiedaninputkeyword:"music",andthetoollocatedarelevantsetoftargets.
Figure2–AutomatedTool(G-LockBlogFinder)forHarvestingBlogs5Someattackersskiptheharvestingstage,bypurchasinglistsofhighqualityURLs–URLswithhighsearchenginerankingandwhichautomaticallyapprovecomments.
Thusmany'qualityURLs'listsareavailableforpurchaseonblackhatSEOforumsandspecificsites(Figure3showsanexample).
AtypicalpriceforaURLlistis$40forapproximately13,000URLs.
Figure3–URLListsforSaleCommentgeneration:Relevantverbalcommentsareattachedtothepromotedsitelinks.
ThisservestheSEOtechniqueandprovidesamoreauthenticcomment.
Theverbalcommentsareproducedaccordingtotheinputkeywords.
Figure4showsanexampleofacommentthatwasautomaticallygeneratedbythe'CommentBlaster'toolfortheinputkeyword'music'.
Figure4–AutomaticCommentExampleThecommentinFigure4iswrittenin"Spintax"format.
Onewaytomitigatecommentspamistoblockduplicatecomments.
Spintaxisanautomaticmethodthatwasdevelopedbyspammers,inordertoavoidthispitfall.
Theideaisforthespammertocreateagenericcommentusingaspeciallyformattedsyntax.
Thisgenericcommentcanbespunintomanydifferentcommentswithasimilarmeaning.
Figure5showsanexampleforthe"Spintaxphrase"andtheresultingcomments2.
2http://umstrategies.
com/what-is-spintax/6TheSpintaxphraseinFigure5hasafewpossiblevariations.
Foreachuniquecomment,thetoolselectsaspecificcombination.
Theresultisafull(hopefullysensible)comment.
Figure6showsanexampleofaSpintaxcreatedbyScrapeBox.
Thistoolenablestheusertoinputakeyword,orinput/edittheSpintaxitself.
Figure6–ScrapeBoxSpintaxExample7Posting:ToolsoffertoautomaticallypostcommentsonmanyURLsatonce.
Sometargetsrequiredifferentformstobefilledinordertosubmitcomments,suchas:userauthentication,Captchaformsoruserdetails.
Sophisticatedtoolsincorporateservicestohandlethesechallenges.
Figure7showshowtoconfiguretheScrapeBoxtooltohandleCaptchachallenges.
Figure7–ScrapeBoxCaptchaSolving8Verification:Toolsprovidefeedbacktotheuserspecifyingwhetherornotacommentwasposted.
Figure8showsanexampleoftheScrapeBoxtoolstatusreport.
Figure8–ScrapeBoxPostingStatusReport9Therearemanyautomatedtoolsthatholdallorpartofthefunctionalitiesdiscussedinthissection.
ThepopularimplementationsaretheScrapeBoxtool,whichoffersallthementionedfeaturesandisshowninFigure9.
TheGscrapertoolisanewalternativethatofferssimilarfeatureswithsimilarpricing.
Figure9–AScrapeBoxScreenshotWeexploredonesideofthecommentspamattack–theattackerpointofview.
Inthenextsection,weexaminetheotherside–thevictim'spointofview.
Thisprovidesusabetterunderstandingoftheessenceoftheattack,andtheoptionalmitigations.
105.
TheVictim'sPointofViewWeobservedalargeamountofdatainordertothoroughlyunderstandthequantitativeaspectsofcommentspamtraffic.
Thedatawascollectedthroughthereal-timemonitoringofattackdataagainstmorethan60webapplications.
Wefocusedonaperiodoftwoweeks,fromSeptember1toSeptember14in2013anduseddifferentfilterstoleaveonlytrafficthatisclearlycommentspam.
Wethenanalyzedthebehavioroftheseattacksovertime,andacrosstargets.
Wealsoperformedcalculationsofstatisticalpropertiesofthemalicioustraffic.
Wediscoveredthatmostofthecommentspamtrafficoriginatedfromattackerswhohavebeenactiveforlongperiods,andattackedmultipletargets.
Toillustratetheexactrelationshipbetweenthenumberofattackedtargetsperattacksource,andthedurationoftheattacker'sactivity,wedesignedan"Attack-SourceReputationQuadrant"graph(SeeFigure10.
ThisgraphwasfirstintroducedinaourpreviousHII3).
Figure10–AttackSourceReputationQuadrantforCommentSpamInan"Attack-SourceReputationQuadrant"graph,theY-axisrepresentsthenumberoftargetsthatwereattacked,andtheX-axisrepresentsthedurationofanattack.
Accordingly,eachdotinthegraphrepresentsanattacksourceandcorrespondstothesource'slongevityandthenumberoftargetsithasattackedduringthecourseofouranalysis.
ToexpresstheAttack-SourceReputationQuadrantasagraph,weaddedtwomoredivisions.
ThefirstisaverticallinealongtheY-axiswhichseparatesattacksourcesofthoseactiveonlyduringasingleday,fromthoseactiveformorethanasingleday.
Thesecondisahorizontallinewhichsimilarlyisolatesattacksourcesthatattackedonlyasingletargetfromthosethatattackedmultipletargets.
Therearefourdifferentquadrants:Theupperleftquadrant(inpurple)includesallattacksourcesthatwereactiveforonlyonedayandattackedmorethanonetarget.
Theupperrightquadrant(inblue)includesallattacksourcesthatwereactiveformorethanonedayandattackedmorethanonetarget.
Thelowerleftcorner(inred)includesallattacksourcesthatwereactiveforonlyonedayandattackedonlyasingletarget.
Thelowerrightquadrant(ingreen)includesallattacksourcesthatwereactiveformorethanonedayandattackedonlyasingletarget.
3http://www.
imperva.
com/resources/hacker_intelligence.
asp11Toquantifythedata,we'veenhancedtheAttack-SourceReputationQuadrantwithtwopie-charts(color-codedtothequadrants,respectively):Thetoppiechartrepresentsthepercentageofattacksources,withineachquadrant.
Thebottompiechartrepresentsthepercentageoftraffic,withineachquadrant.
Figure10showsthatmostoftheattackers(72percent)areintheredzone,whichmeanstheywereactiveonlyforasingleday,andattackedonlyasingletarget.
Nonetheless,mostofthecommentspamtraffic(58percent)isinthebluezone,whichmeanstheywereactivemorethanoneday,andattackedmorethanonetarget.
Wefocusedontheupperrightquadrant(blue)andexploredthetraffic.
Wediscoveredthatarelativelysmallnumberofattackersareresponsibleforalargeamountofthecommentspamtraffic.
Figure11showsthecumulativepercentageofcommentspamtrafficfromtheattackersinthebluequadrant.
Theattackersaresortedbydominance:attacker#1producedthehighestnumberofattacksinthegivenperiod,etc.
Figure11–TheCumulativePercentageofCommentSpamTrafficThegraphshowsthat80percentofthecommentspamtrafficwasgeneratedby28percentoftheattackers.
126.
MitigationTechniquesWebsitescandefendthemselvesagainstcomment-spamattacksusinganumberofmitigationtechniques.
Following,aresomeofthepopularonesatusetoday.
6.
1ContentInspectionThecontentinspectiontechniqueisbasedoninspectingthecontentofthepostedcomments,accordingtoapredefinedsetofrules.
Rules,forexample,mightbe:toomanylinksinonecomment;logicalsentencesthatarerelatedtothesubjectathand;andnoduplicatecomments.
Insuchsystemsatradeoffexistsbetweenfalse-positiveandtrue-negativerates,dependingontherulesdefinitions.
Akismet4isacommentspamdetectionservicethatusesacombinationofmitigationmethods,amongthemthecontentbasedtechnique.
Whenusingit,eachcommentissenttotheAkismetservers.
Theserverscheckthereceiveddata,andreturnatrue/falseanswer.
Figure12–AkismetMotoContentbasedmitigationcanrelyonthereputationofthehyperlinkspostedwithinthecomments5.
Oncealinktoaspecificwebsiteappearsintoomanycommentsontheweb,orinrequeststhataresuspiciousenoughtobecreatedusingcommentspamtools,thepromotedwebsitemaygainabadreputation.
Thisreputationcanbeusedtoblockcommentscontainingthesehyperlinks.
"Penguin"6isarecentupdatetotheGooglesearchenginethatusesthiskindofinformation,andpenalizeswebsitesthatareknowntousecommentspamtools.
6.
2SourceReputationThismitigationtechniqueisbasedonidentifyingwhetheracommentisspamaccordingtothereputationoftheposter.
Sourcereputationisbasedonwhetherpreviouslyseentrafficfromthatsourcewasconsideredcommentspam.
Onlinerepositories,basedoncrowdsourcing,wereset-upforthesepurposes.
Therepositoriesareusedtobothreportspamandtocheckacommentsourcereputation.
Thetwomostpopularrepositoriesarewww.
projecthoneypots.
organdwww.
stopforumspam.
com.
Ourresearchfoundthemratherreliable.
4http://www.
akismet.
com5http://www.
securelist.
com/en/analysis/204792295/Redirects_in_Spam6http://en.
wikipedia.
org/wiki/Google_Penguin136.
3Anti-automationAnti-automationtechniquescanbeusefulforcommentspammitigation,asautomatictoolsarefrequentlyusedtoproducecommentspamtraffic.
Onesimpleoptionisaddingacheckboxtoindicatewhetherauserwishestopostacomment.
RegularlychangingtheHTTPfieldnameforthischeckboxisusefulagainstthemoresophisticatedtools.
AmorecomplexoptionisusingtheCaptcha7mechanism.
Whenusingit,eachcommentpostrequiresenteringanobfuscatedtextdisplayedonthepage.
Figure13–CaptchaChallengeforPostingaComment6.
4DemotivationThedemotivationtechniquestrivestomakecommentspamuseless.
Thiscanbeachievedbythefollow/nofollowvaluethatcanbeassignedtothe"rel"attributeofanHTMLanchor()elementwhichdefinesahyperlink8.
Itspecifieswhetheralinkshouldbefollowedbythesearchengine'sindexingalgorithm.
Settingthe"nofollow"valueforpostedcommentsdecreasesthecommentspammotivation.
ThisisdemonstratedinFigure14.
Frameworkscanusethisvaluetodemotivatecommentspammers.
Forexample,WordPress1.
5andaboveautomaticallyassignsthenofollowvaluetoalluser-submittedlinks9.
AnotherexamplefordemotivationisthePenguin10updatetoGooglesearchenginealgorithmthatfocusesondecreasingthesearchenginerankingofwebsitesthatareconsideredtousecommentspamtechniques.
6.
5ManualInspectionManualinspectionisveryeffectiveforidentifyingcommentsasspam.
Itsprimarydrawbackisitslossofscalability–asspamincreases,manualinspectionofitbecomesimpractical.
Thistechniqueiseffectiveagainstmanualcommentspam,duetotherelativelysmallamountofspamthatcanbemanuallyposted(andinspected).
7ttp://en.
wikipedia.
org/wiki/CAPTCHA8http://en.
wikipedia.
org/wiki/Nofollow9http://codex.
wordpress.
org/Nofollow10http://en.
wikipedia.
org/wiki/Google_Penguin147.
Casestudies7.
1AnalyzingaSingleVictimInordertobetterunderstandthecommentspamattackpattern,wetookacloserlookatthespamtrafficdirectedatasinglevictim.
Wechoseonewebsitethatwasreceivingagreatamountofcommentspamtraffic.
Itconsistsofasinglehost,withmanyURLs.
Thevictimisanon-profitorganizationthatsuppliesinformationandsupportsacommunityofusers.
Wegathereddataoveraperiodofonemonth,thatproduced384eventsfromSeptember1sttoSeptember30th2013.
Wediscoveredahighdiversityinthevolumeofcommentspamtrafficfordifferentpages.
OurtheoryassociatespopularphraseswithintheURLaddressandpagecontent,totheattackrate.
Wedocumentedattackson119URLs.
Figure15showsthenumberofeventsforeachURLindescendingorder,i.
e.
URLonereceivedthehighestnumberofevents,andsoon.
Figure15–URLPopularityGraphThegraphshowsthattargetonehadsignificantlysufferedmorecommentspamcomparedtotheothertargetsonthatsamehost.
Ithadapproximately10timesmoreeventscomparedtothenextURLinorder.
Apotentialexplanationcanbethattargetonecontainsthepopularphrase'weightgain'initsURLaddresswhichdrawscommentspamattackers.
Thisphraseappearsfrequentlywithinthepagesuchas"causesofweightgain"and"Howcanthisweightgainbeprevented".
15Wediscoveredthatasmallnumberofsourcesproducedmostofthetraffic.
Figure16showsthecumulativepercentageofcommentspamtrafficgeneratedbysourceIPs,tothetarget,athand.
Figure16–TheCumulativePercentageofCommentSpamTraffictoaSingleTargetWecanseeinFigure16,52percentofsourceIPsproduceapproximately80percentofthetraffic.
7.
2AnalyzingaSingleAttackerInordertothoroughlyunderstandthecommentspamtrafficwefocusedonahighlyactiveattacker,andexaminedbothitstrafficquantitativeandqualitativeaspects.
Wediscoveredthattheattackerwasactiveforalongperiod.
Weidentified61HTTPrequestsascommentspamduringaperiodoftwoweeks.
Figure17showsthenumberofrequeststheattackersenteachday,duringthosetwoweeks.
Figure17–RequestsperDayforaSingleAttackerTheattackerwasactivefortendaysandthenumberofrequestsperdayhadincreasedduringtheperiod.
Identifyingthisattackerasacommentspammerearly,andblockingitsrequests,wouldhavepreventedmostofitstraffic.
16Theattackerhadafewtargets,andmostofthemsufferedarelativehighamountofcommentspamattacks.
Theinvestigateddataincluded61eventstargetingfivedifferentwebsites.
Figure18showsthepercentageofthetrafficreceivedbyeachtarget.
Figure18–PercentageofTrafficperTargetAsshowninFigure18,mostofthetargetsreceivedroughly,anevenportionofthetraffic(exceptfromtargetfive).
Inaddition,oncetheattackerattackedacertaintarget,itwaslikelytoattackitagain,inthefollowingdays.
Forexample,theattacksontargettwohavespannedovereightdays,ofthetwoweekperiod.
Theautomatedtoolusesinputparameterswithnoreassurancetheyarebeingpubliclyavailableonthesite.
Figure19showsascreenshotofatargetedpage.
Theattacker'stargetwasthe"comments"field,howeverthecommentsareforanorderbeingmade,andwillnotbepublishedonthesite.
Webelievethishappensduetolackofverificationbytheautomatedtools–theyharvesttargetswitha"comment"parameter,anddonotverifythatthevalueisactuallybeingpublishedonthesite.
17Figure19–ACommentSpamTargetWeanalyzedthecontentofcommentsandlearnedthehyperlinksinasinglerequestarefordifferentsitesandconsecutiverequestshavesimilarhyperlinks.
Therequestsproducedbythetoolwillholdcommentscontainingdifferentwebsitestopromote.
Thosecommentswillberestructuredinconsecutiverequests,inordertoavoiddefensemechanisms.
Inthecaseathand,theattackersent48requestscontainingsevendifferentURLs.
Thecommentshaveabasicreoccurringstructure:18Error!
Referencesourcenotfound.
showstwocomments,forexample,thatholdeightURLs,withfiveuniquevalues.
Figure20–ExamplesofCommentsWhenfeedingthemtoabrowser,wesawthatsixofthemleadtothesamewebsiteofapharmaceuticalcompany(markedinyellow).
TheothertwoURLs(markedinblue)belongtoanotherwebsite.
UsingtheseURLsasjumpingboards,preventsthepromotedwebsitesfromgainingabadreputationfromusingcommentspamtools,andavoidshavingidenticalcomments(seeSection6:MitigationTechniques).
197.
3AttackersAbuseGoogleAppEngineforCommentSpamThe"GoogleAppEngine"11isaserviceprovidedbyGoogle,whichallowsuserstorunwebapplicationsonGoogle'sinfrastructure.
Oneespeciallyeasy-to-createapplicationisawebproxy,whichcanbeusedtogeneratecommentspamtraffic.
Figure2112showsaseriesofsimplestepsforcreatingaproxyusingtheGoogleAppEngine.
Figure21–StepstoTurnGoogleAppEngineintoaProxy11https://developers.
google.
com/appengine/docs/whatisgoogleappengine12http://www.
labnol.
org/internet/setup-proxy-server/12890/HackerIntelligenceInitiativeOverviewTheImpervaHackerIntelligenceInitiativegoesinsidethecyber-undergroundandprovidesanalysisofthetrendinghackingtechniquesandinterestingattackcampaignsfromthepastmonth.
ApartofImperva'sApplicationDefenseCenterresearcharm,theHackerIntelligenceInitiative(HII),isfocusedontrackingthelatesttrendsinattacks,Webapplicationsecurityandcyber-crimebusinessmodelswiththegoalofimprovingsecuritycontrolsandriskmanagementprocesses.
Inourresearch,wemonitoredalistofIPaddressesknowntogeneratecommentspam.
ThemostdominantIPinthegroupwasanaddressregisteredtoGoogleAppEngine.
ThistechniqueisusedbyspammerstobypassreputationcontrolsbasedonIPaddresses,sincemostoftenaddressesofGoogleAppEngine(andthoseofothercloudservices)areexplicitlywhitelisted.
Infact,inourowndataset,weareabletoidentifylegitimatetrafficfromthesameIPaddresswhichbelongstoadifferentapplication(forexamplethe"feedly"application13).
AmorecarefulinspectionoftherequeststructurerevealedthatanapplicationID(appID)isspecifiedintheHTTPuser-agentfield–probablyinsertedbyAppEngineinfrastructure.
8.
SummaryandconclusionsStudyingthecommentspamspacefrombothends,andtakingintoaccountallexistingmitigationtechniques,wehavecometothefollowingconclusions:Identifyingtheattackerasacommentspammerearlyonandblockingitsrequestspreventsmostofthemaliciousactivity.
IPreputationwillhelpinsolvingthecommentspamproblem,byblockingcommentspammersearlyonintheirattackcampaignsAsofApril2014,ImpervaoffersaCommentSpamIPreputationfeedthroughitsThreatRadarservices,tohelpcustomersmitigatethecommentspamproblem.
13http://cloud.
feedly.
com/#welcome