Scenarioswinrar5

www.
SELabs.
ukinfo@SELabs.
uk@SELabsUKwww.
facebook.
com/selabsukblog.
selabs.
ukJAN-MAR2020EMAILSECURITYSERVICESPROTECTION2SELabstestedarangeofemailhostedprotectionservicesfromarangeofwell-knownvendorsinanefforttojudgewhichwerethemosteffective.
Eachservicewasexposedtothesamethreats,whichwereamixtureoftargetedattacksusingwell-establishedtechniquesandpublicattacksthatwerefoundtobeliveontheinternetatthetimeofthetest.
Theresultsindicatehoweffectivelytheserviceswereatdetectingand/orprotectingagainstthosethreatsinrealtime.
EmailSecurityServicesProtectionJan-Mar20203CONTENTSIntroduction04EmailSecurityServicesProtectionAwards05ExecutiveSummary06HowweTested071.
ThreatDetectionResults102.
TotalAccuracyRatings113.
ProtectionandLegitimateHandlingAccuracy124.
Conclusion15AppendixA:AttackDetails16AppendixB:DetailedResults18AppendixC:TermsUsed25AppendixD:FAQs26AppendixE:ServicesTested26Documentversion1.
0Written:16thMarch20201.
01editedon3rdMarch,correctedKasperskyLabsproductnameEmailSecurityServicesProtectionJan-Mar2020MANAGEMENTChiefExecutiveOfficerSimonEdwardsChiefOperationsOfficerMarcBriggsChiefHumanResourcesOfficerMagdalenaJurenkoChiefTechnicalOfficerStefanDumitrascuTESTINGTEAMThomasBeanSolandraBrewsterDimitarDobrevLiamFisherGiaGorboldJonThompsonDaveTogneriJakeWarrenStephenWitheyITSUPPORTDannyKing-SmithChrisShortPUBLICATIONSteveHainesColinMackleworthWebsitewww.
SELabs.
ukTwitter@SELabsUKEmailinfo@SELabs.
ukFacebookwww.
facebook.
com/selabsukBlogblog.
selabs.
ukPhone02038755000PostSELabsLtd,55AHighStreet,Wimbledon,SW195BA,UKSELabsisISO/IEC27001:2013certifiedandBSENISO9001:2015certifiedforTheProvisionofITSecurityProductTesting.
SELabsisamemberoftheMicrosoftVirusInformationAlliance(VIA);theAnti-MalwareTestingStandardsOrganization(AMTSO);andtheMessaging,MalwareandMobileAnti-AbuseWorkingGroup(M3AAWG).
AMTSOStandardreference:https://tinyurl.
com/essp20202020SELabsLtdEmailSecurityServicesProtectionJan-Mar20204INTRODUCTIONThisemailsecuritytestreportistheproductoftwoyearsofadvancedthreatresearch.
Wehaveworkedwiththesecuritycompaniesthemselvesandwiththeircustomers.
Wehavemonitoredwhatthebadguyshavebeendoingandidentifiedandreplicatedreal-worldemailthreatsthataffecteveryonegenerally,andalsospecifictypesofbusinesses.
Thereisnoreportlikethisanywhereinthepublicdomain.
Weareextremelyproudtopresenttheresultshere.
Asyouscantheheadlines,awardsanddatatablesyoumaywonderwhysomanyofthemajorplayersintheemailsecurityindustryareabsent.
Overthelast24monthswe'veworkedwithmostofthemprivately,butthisisanewtestand,frankly,theyareworriedabouttheirresults.
ItistothemassivecreditofcompanieslikeFortinet,MimecastandPerceptionPointthattheyhaveenoughconfidenceintheirproductstoentersuchachallengingtest.
Andtobethefirst.
Wewillalwayswelcometheparticipationofanyvendorintheemailsecurityspacebut,aswemoveonwithtestingsecurityproducts,pleasecheckintoseewhichcompaniesareinvolved.
Askyourselfwhycertaincompaniescontinuetorefusetobetested.
Dotheyhavesomethingtohide,oristhetestjustnogoodTobefair,emailsecurityisinitsinfancywhencomparedtoothercomputersecurityservices.
Weexpectservicestoimproveovertimeastheyfacegoodindependenttesting.
Buttheseservicesareforsalenowandyoudeservetoknowwhicharethestrongest.
Webelievethatthistestisthebestthere'severbeeninthisspace,butwedon'texpectyoutojusttakeusatourword.
ToaddfurthercredibilitytoourclaimsinthisreportwehavesubmittedittotheAnti-MalwareTestingStandardsOrganization,whichassessessecuritytestsfortransparency.
Wewon'tknowuntilafterthetestispublishedifitcomplieswiththeAMTSOtestingStandard,butwehaveenoughconfidenceintheintegrityofourselvesandthetestingmethodsthatwe'reopeningourselvesuptojudgment.
ToverifyitscompliancepleasechecktheAMTSOreferencelinkatthebottomofpagethreeofthisreportorhere.
Aswithallofourreports,ifyouhaveanyquestionspleasecontactusviaourwebsite,TwitterorFacebook.
Emailsecurity:IsitanygoodagainsthackersWorld'sfirstin-depth,publictestofsecurityservicesvs.
targetedattacks5ThefollowingproductswinSELabsawards:EmailSecurityServicesProtectionAwardsPerception-PointFortinetFortiMailMimecastSecureEmailGatewayKasperskySecurityforOffice365GoogleGSuiteBusinessJAN-MAR2020EMAILSECURITYSERVICESPROTECTIONEmailSecurityServicesProtectionJan-Mar2020MicrosoftOffice365MicrosoftOffice365AdvancedThreatProtectionJAN-MAR2020EMAILSECURITYSERVICESPROTECTIONJAN-MAR2020EMAILSECURITYSERVICESPROTECTIONJAN-MAR2020EMAILSECURITYSERVICESPROTECTIONGoogleGSuiteEnterpriseJAN-MAR2020EMAILSECURITYSERVICESPROTECTIONEmailSecurityServicesProtectionJan-Mar20206ExecutiveSummaryThistestpittedanumberofemailsecurityservicesagainstlivetargetedattacksthatusedthesameorsimilartacticstowell-knowngroupsoperatingoverthelastfewyears.
Advancedmalwareandsocialengineeringtacticswereusedtoreplicatenation-state-levelattackers,aswellascybercriminalstargetingindividualsandthegeneralpublic.
Theservicestestedwerestandaloneemailsecuritygatewaysandplatforms,whichareintegratedemailservicesthatincludesecurityfeatures.
Common'commodity'threatsweremostlydetected.
Noproductwasabletodetectandpreventalltargetedthreats.
Thehighestoveralldetectionratewas96%.
Thelowestoveralldetectionratewas73%.
Falsepositivesweresurprisinglycommon,particularlywiththeemailplatforms.
Legitimatemessagehandlingwasgenerallysuccessful,rangingfrom72%to100%accuracy.
TheTotalAccuracyRatings(seeleft)showhowwelleachservicehandledthreatsandlegitimatemessagesinacombined,weightedrating.
Productshighlightedingreenwerethemostaccurate,scoring40percentormoreforTotalAccuracy.
Thoseinorangescoredbetween20to40percent.
Anyproductsshowninredscoredlessthan20percent.
EXECUTIVESUMMARYProductProtectionAccuracyRatingLegitimateAccuracyRatingTotalAccuracyRatingTotalAccuracyRating(%)Perception-Point2,6037003,30394%FortinetFortiMail2,5256403,16590%MimecastSecureEmailGateway2,4127003,11289%KasperskySecurityforOffice3651,6815502,23164%GoogleGSuiteEnterprise9565051,46142%GoogleGSuiteBusiness8255351,36039%MicrosoftOffice3654635501,01329%MicrosoftOffice365AdvancedThreatProtection42655097628%ServicesSomeservicestestedmaybelistedinthisreportusingjustthevendors'namesforclarityandbrevity.
ForalistoffullservicenamespleaseseeAppendixE:ServicesTestedonpage26EmailSecurityServicesProtectionJan-Mar202077CommodityBasicSophisticatedSocialPhishingBusinessEmailCompromiseMalwareLegitimateExampleScenariosExampleTestCasesFreeMoneytoTransferFBIBlackmailEmergencyPayPalRequestLotteryWinFundBeneficiaryMoneyMuleBasicexamplesmightincludeplaintext,poorspellingandgrammaralongsideobviouslyunsuitableemailaddresses(e.
g.
anFBIscamsentfromaGmailaccount).
Moreadvancedoptionscanincludemessagere-coding,morebelievableemailaddressesandmalwareequippedwithanti-virusevasionabilities.
Targeted110CategoriesHowWeTestedThecommoncommoditythreatsweregatheredfromthewildandreplayedthroughtheemailsecurityservices.
Wherepossible,dataabouttheoriginalattackers'IPaddresseswereprovidedtoallowservicesthathavereliableIPaddressreputationsystemstousetheirthreatintelligenceduringtesting.
Legitimatemessageswereconstructedin-house.
Targetedattackscomprisefourdistinctcategories:SocialEngineering;Phishing;MalwareandBusinessEmailCompromise.
ForeachoftheseTestCaseStructurecategorieswecreatedanumberofmainTestCaseStructurevariations.
Intheexamplebelowyoucanseethatthesocialengineeringmessagesareformedintosixgroups(scenarios),includingfreemoneytransfer,lotterywinandlawenforcementblackmailscams.
Foreachscenariowecreatevariantsthatrangeinsophisticationfromextremelybasictoveryadvanced.
Thegoalistotesthoweffectiveeachemailsecurityserviceiswhenfacingarangeofdifferenttypesofattacker,oratleastarangeofdifferentattackapproaches.
EmailSecurityServicesProtectionJan-Mar20208ResultsandScoring8ServiceUnderTestLegitimateLegitimateSendingServerStoppedQuarantined(Admin)Quarantined(User)InboxNotifiedEdited(Allow)Edited(Deny)JunkJunk(Allow)Junk(Deny)BlockedRejectedCommoditySocialPhishingMalwareBusinessEmailCompromiseEmailmessagestravelovertheinternettotheirrecipients.
Beforetheyreachtheinboxtheynegotiatetheirwaythroughvarioussecurityservicesbeforereachingthetarget'sowninfrastructure.
Thereareopportunitiesfordetectionandprotectionatdifferentstagesinthisjourney.
Badmessagesmightbepreventedfromenteringtheserviceundertest,beingblockedorotherwiserejected.
Oncewithintheservice,themessagemightbedetectedandpreventedfromprogressingfurther,oritmightbeplacedintoaquarantinefromwhicheitherauseroradministratormayreleaseit.
MessagesthathavesuccessfullyrunthegauntletfacepossibledetectionbyOffice365orwhicheveremailserviceisinuse.
Messagesmayendupintheinboxorquarantine,withorwithoutchangessuchasremovedorrewrittenURLs,attachmentsandotherelements.
TargetEmailSecurityServicesProtectionJan-Mar20209Attacker/APTGroupMethodTargetSandwormFIN7APT19APT28DridexAPT33(2019)APT33(2017)FIN4Whentestingservicesagainsttargetedattacksitisimportanttoensurethattheattacksusedarerelevant.
Anyonecanrunanattackrandomlyagainstsomeoneelse.
Itisthesecurityvendor'schallengetoidentifycommonattacktypesandtoprotectagainstthem.
Astesters,weneedtogeneratethreatsthatinsomewayrelatetotherealworld.
Alloftheattacksusedinthistestarevalidwaystocompromiseanorganisation.
Withoutanysecurityinplace,allwouldsucceedinattackingthetarget.
Outcomeswouldincludesystemsinfectedwithransomware,remoteaccesstonetworksanddatatheft.
Butwedidn'tjustsitdownandbrainstormhowwewouldattackdifferentcompanies.
Insteadweusedcurrentthreatintelligencetolookatwhatthebadguyshavebeendoingoverthelastfewyearsandcopiedthemquiteclosely.
Thiswaywecantesttheservices'abilitiestohandlesimilarthreatstothosefacedbyglobalgovernments,financialinstitutionsandnationalinfrastructure.
Thegraphiconthispageshowsasummaryoftheattackgroupsthatinspiredthetargetedattacksusedinthistest.
Ifaservicewasabletodetectandprotectagainstthesethenthere'sagoodchancetheyareontracktoblockingsimilarattacksintherealworld.
Iftheyfail,thenyoumighttaketheirboldmarketingclaimsaboutdefeatinghackerswithapinchofsalt.
FormoredetailsabouteachAPTgroupseeAppendixA:AttackDetailsonpage16.
DetailsDocumentscontaininghiddenlinkstoscriptsDocumentscontaininghiddenlinkstoscriptsMicrosoftOfficemacrosWindowsvulnerabilitiesviaOfficedocumentsWinRARexploitHTMLapplicationfilesMan-in-the-middlespearphishingWindowsvulnerabilitiesviaOfficedocumentsEnergyBankingGovernmentespionageFinancialmarketAviationDemocraticNationalComitteeUSretail,restaurantandhospitalityKeyAttackersvs.
TargetsEmailSecurityServicesProtectionJan-Mar2020101.
ThreatDetectionResultsWhiletestingandscoringemailsecurityservicesiscomplex,itispossibletoreportstraight-forwarddetectionrates.
Thefiguresbelowsummarisehoweachservicehandlesthreatsinthemostgeneral,leastdetailedway.
ThreatsthatMicrosoftmovedtotheJunkfolderarecountedashitsforMicrosoft,whileanymessagesthatpassthroughanon-MicrosoftserviceandendupintheJunkfolderaremissesforthatservice.
THREATDETECTIONRESULTSPRODUCTDetectionRateMissesDetectionRate(%)Perception-Point2701096%MimecastSecureEmailGateway2661495%FortinetFortiMail2641694%MicrosoftOffice365AdvancedThreatProtection2443687%GoogleGSuiteEnterprise2384285%KasperskySecurityforOffice3652305082%GoogleGSuiteBusiness2305082%MicrosoftOffice3652057573%DetectionratesareausefulbutunsubtlewaytocompareservicesFortinetFortiMail94%DetectionMicrosoftOffice365ATP87%DetectionGoogleGSuiteBusiness85%DetectionKasperskySecurityforOffice36582%DetectionGoogleGSuiteBusiness82%DetectionMicrosoftOffice36573%DetectionPerception-Point96%DetectionMimecastSecureEmailGateway95%DetectionEmailSecurityServicesProtectionJan-Mar2020112.
TotalAccuracyRatingsJudgingtheeffectivenessofanemailhostedprotectionserviceisasubtleartandmanyfactorsneedtobeconsideredwhenassessinghowwellitperforms.
Tomakethingseasierwe'vecombinedallofthedifferentresultsintooneeasy-to-understandtable.
Thegraphicbelowtakesintoaccountnotonlyeachservice'sabilitytodetectandprotectagainstthreats,butalsoitshandlingofnon-maliciousmessagesandcomponentsofthosemessages,suchasattachmentsandlinkstowebsites.
Notallprotectionmeasures,ordetectionsforthatmatter,areequal.
Aservicemightcompletelydeleteanincomingmaliciousemailandneverallowtheintendedrecipienttosee(andsubsequentlyinteractwith)it.
Servicesmaycondemnsuspiciousmessagestoa'quarantine'areaifitlackstheutterconvictionthatthemessageisunwanted.
Thiskeepsthreatsawayfromrecipientsunlesstherecipientjudgesthatthemessageisreallysafe.
Attheweakerendofthescale,theservicemightsimplyaddawarningtotheemail'sSubjectline.
Wetakethesedifferentpossibleoutcomesintoaccountwhenattributingpointsthatformfinalratings.
Forexample,aservicethatcompletelyblocksamaliciousmessagefromfallingintothehandsofitsintendedrecipientisratedmorehighlythanonethatprefixestheSubjectlinewith"Malware:"orTOTALACCURACYRATINGSPRODUCTTotalAccuracyRatingTotalAccuracyRating(%)Perception-Point3,30394%FortinetFortiMail3,16590%MimecastSecureEmailGateway3,11289%KasperskySecurityforOffice3652,23164%GoogleGSuiteEnterprise1,46142%GoogleGSuiteBusiness1,36039%MicrosoftOffice3651,01329%MicrosoftOffice365AdvancedThreatProtection97628%"Phishingattempt:",orsendsthemessagetoa'Junk'folder.
Categorisinghowaservicehandleslegitimatemessagesissimilar,butinreverse.
MakingasmallchangetotheSubjectlineismuchlessseriousafailingthandeletingthemessageandfailingtonotifytherecipient.
TotalAccuracyRatingscombineprotectionandfalsepositives.
Perception-Point94%TotalAccuracyFortinetFortiMail90%TotalAccuracyMimecastSecureEmailGateway89%TotalAccuracyKasperskySecurityforOffice36564%TotalAccuracyGoogleGSuiteEnterprise42%TotalAccuracyGoogleGSuiteBusiness39%TotalAccuracyMicrosoftOffice36529%TotalAccuracyMicrosoftOffice365ATP28%TotalAccuracyEmailSecurityServicesProtectionJan-Mar2020123.
ProtectionandLegitimateHandlingAccuracyTheresultsbelowindicatehoweffectivelytheservicesdealtwiththreatsandlegitimateemail.
Pointsareearnedfordetectingthreatsandforblockingorotherwiseneutralisingthem.
Pointsarealsoearnedforallowinglegitimateemailentryintotherecipient'sinboxwithoutsignificantdamage.
Stopped;Rejected;Notified;Editedeffectively(+10forthreats;-10forlegitimate)Iftheservicedetectsthethreatandpreventsanysignificantelementofthatthreatfromreachingtheintendedrecipientweawardit10points.
Ifitmiscategorisesandblocksorotherwisesignificantlydamageslegitimateemailthenweimposeaminus10pointpenalty.
Quarantined(Between+8forthreats;-8forlegitimate)Servicesthatinterveneandmovemaliciousmessagesintoaquarantinesystemareawardedeithersixoreightpointsdependingonwhetherornottheuseroradministratorcanrecoverthemessage.
However,thereisasixtoeightpointdeductionforeachlegitimatemessagethatisincorrectlysenttoquarantine.
Junk(+5forthreats;-5forlegitimate)Themessagewasdeliveredtotheuser'sJunkfolder.
Inbox(-10forthreats;+10forlegitimate)Maliciousmessagesthatarriveintheuser'sinboxhaveevadedthesecurityservice.
Eachsuchcaselosestheservice10points.
Alllegitimatemessagesshouldappearintheinbox.
Foreachonecorrectlyroutedthereisanawardof10points.
RatingcalculationsForthreatresultswecalculatetheprotectionratingsusingthefollowingformula:Protectionrating=(10xnumberofStoppedetc.
)+(6-8xnumberofQuarantined)+(5xnumberofJunk)+(-10xnumberofInbox)etc.
SCORINGDIFFERENTOUTCOMESActionThreatLegitimateInbox-1010JunkFolder5-5Quarantined(admin)8-8Quarantined(user)6-6Notified10-10Stopped10-10Rejected10-10Blocked10-10Edited(Allow)-1010Edited(Deny)10-10Junk(Deny)10-10Junk(Allow)-77Forlegitimateresultstheformulais:(10xnumberofInbox)+(-5xnumberofJunk)+(-6-8xnumberofQuarantined)+(-10xnumberofStoppedetc.
)etc.
Theseratingsarebasedonouropinionofhowimportantthesedifferentoutcomesare.
Youmayhaveadifferentviewonhowseriousitisforalegitimateemailtoendupinquarantine,orforamalwarethreattoendupintheinbox.
Youcanusetherawdatafromthisreport(SeeAppendixB:DetailedResultsonpage18)torollyourownsetofpersonalisedratings.
EmailSecurityServicesProtectionJan-Mar202013PROTECTIONACCURACYRATINGSPRODUCTProtectionAccuracyRatingProtectionAccuracyRating(%)Perception-Point2,60393%FortinetFortiMail2,52590%MimecastSecureEmailGateway2,41286%KasperskySecurityforOffice3651,68160%GoogleGSuiteEnterprise95634%GoogleGSuiteBusiness82529%MicrosoftOffice36546317%MicrosoftOffice365AdvancedThreatProtection42615%Perception-Point93%AccuracyFortinetFortiMail90%AccuracyMimecastSecureEmailGateway86%AccuracyKasperskySecurityforOffice36560%AccuracyGoogleGSuiteEnterprise34%AccuracyGoogleGSuiteBusiness29%AccuracyMicrosoftOffice36517%AccuracyMicrosoftOffice365ATP15%AccuracyEmailSecurityServicesProtectionJan-Mar202014Thistableshowshowaccuratelytheserviceshandledlegitimateemail.
Theratingsystemisdescribedindetailin3.
ProtectionandLegitimateHandlingAccuracyonpage12.
LEGITIMACYACCURACYRATINGPRODUCTLegitimateAccuracyRatingLegitimateAccuracyRating(%)MimecastSecureEmailGateway700100%Perception-Point700100%FortinetFortiMail64091%KasperskySecurityforOffice36555079%MicrosoftOffice36555079%MicrosoftOffice365AdvancedThreatProtection55079%GoogleGSuiteBusiness53576%GoogleGSuiteEnterprise50572%LegitimateAccuracyRatingsgiveaweightedvaluetoservicesbasedonhowaccuratelytheyhandlelegitimatemessages.
MimecastSecureEmailGateway100%AccuracyPerception-Point100%AccuracyFortinetFortiMail91%AccuracyKasperskySecurityforOffice36579%AccuracyMicrosoftOffice36579%AccuracyMicrosoftOffice365ATP79%AccuracyGoogleGSuiteBusiness76%AccuracyGoogleGSuiteEnterprise72%AccuracyEmailSecurityServicesProtectionJan-Mar2020154.
ConclusionThistestpittedanumberofemailsecurityservicesagainstlivetargetedattacksthatusedthesameorsimilartacticstowell-knowngroupsoperatingoverthelastfewyears.
Whilemalwarewasofteninvolved,therewasfarmoretotheattacksusedthanjustsendingaransomwarefileasanattachment.
Advancedmalwareandsocialengineeringtacticswereusedtoreplicatenation-state-levelattackers,aswellascybercriminalstargetingindividualsandthegeneralpublic.
Inotherwords,wedidn'tjustcreatealistofbrand-newwaystoattacktargetsoveremail.
Wewereinspiredbyattackgroupswhosebehaviourhasbeenmonitored,analysedandpublished.
Theservicesthatwetestedcanberoughlyorganisedintotwogroups:emailsecuritygateways,suchasMimecastSecureEmailGatewayandFortinetFortiMailCloud–GatewayPremium;andemailplatformsthatincludeemailsecurityfeatures,suchasMicrosoftOffice365andGoogleGSuite.
Allservicesclaimtoprotecttheirusersfromthreatsandourgoalwastotestthatclaim.
Beforewegettothejuicystuffit'sworthrememberingthatemailsecurityproductsaresupposedtoletrealemailthrough,whilefilteringoutthedangerousmessages.
Toensurethattheproductsweren'tconfiguredtoblockeveryincomingemail,wealsotestedwithlegitimatemessages.
Weexpectedeveryservicetoallowalloftheseintotheinbox.
Additionally,wetestedwithsomeverywell-knownthreatsthataffectthegeneralpubliconanongoingandnon-discriminatorybasis.
Inotherwords,allofthecompaniesbehindtheseservicesshouldbeawareofthemanddetectthem.
Thisreportcontainsresultsforallofthesetestcases:targetedattacks;commoditythreats;andlegitimatemessages.
Wehaveaweightedscoringsystemthatgeneratesoneeasy-to-understandTotalAccuracyRating,whichtakesalloftheresultsintoaccount.
Aservicethatblockseverymessagewillscorewellintermsofprotectionbutfacestrongpenaltiesforblockingtheusefulemails.
Similarly,aservicethatletseverymessagethroughwillbepenalisedforallowingthreatsthrough.
ThestrongestservicesoverallwerefromPerceptionPoint,FortinetandMimecast.
AllthreeachievedhighenoughratingstowinAAAawards.
Theymanagedthisbycorrectlydetectingandhandlingthreats,whileallowingthevastmajorityofthelegitimatemessagesintotheinboxes.
Ifyouwantmoreprecisedetailsabouthowtheyhandledtargetedsocialengineering,phishingandmalwareattackspleaseseeAppendixB:DetailedResultsonpage18.
EmailSecurityServicesProtectionJan-Mar202016AttackGroup:DridexmalwarecampaignMethodofAttack:WindowsvulnerabilitiesviaOfficedocumentsTargets:BankingThisattackcampaigninvolvedsendinginvoicerequeststofinancedepartments.
Themessagescontainedmaliciousdocumentsthatpromptedtherecipienttoupdatethedocumentwithdatafromotherlinkedfiles.
However,userinteractionwasnotrequired,andtheattackwouldinitiateregardless.
References:https://attack.
mitre.
org/software/S0384/https://www.
proofpoint.
com/us/threat-insight/post/dridex-campaigns-millions-recipients-unpatched-microsoft-zero-dayAPPENDICESAppendixA:AttackDetailsAttackGroup:SandwormMethodofAttack:WindowsvulnerabilitiesviaOfficedocumentsTargets:EnergyindustriesInlate2015agroupknownastheSandwormTeammadeuseofazero-dayvulnerabilitytocauseawidespreadpoweroutageinUkraine.
ThisthreatactorisalsoknownasVoodooBearandBlackEnergyAPTGroup.
References:https://attack.
mitre.
org/groups/G0034/AttackGroup:APT19MethodofAttack:DocumentscontaininghiddenlinkstoscriptsTargets:Defence;financialmarkets;education;andlegalservicesUsingsimilartechniquestothoseoutlinedinthedescriptionforFIN7(above),theAPT19attackgroupsentspearphishingemailswithhiddenlinkstomaliciouscode.
Whiletechnicallysimilar,thegroupfocussedondifferenttypesoftarget.
References:https://attack.
mitre.
org/groups/G0073/AttackGroup:APT28MethodofAttack:MicrosoftOfficemacrosTargets:GovernmentMacro-basedattacksareapopularchoiceasastartingpointofatargetattack.
Thereisalowbarriertoentryandawidedistributionofvulnerabletargets.
InfamouscampaignsconductedbyAPT28,andassociatedgroupsFancyBearandSednit,usuallystartwithspear-phishingemailmessagesdesignedtoconvinceuserstoopenspeciallycrafted,attachedMicrosoftOfficedocumentsthatleadtofurthercompromiseoftheirsystems.
References:https://attack.
mitre.
org/groups/G0007/AttackGroup:FIN7MethodofAttack:DocumentscontaininghiddenlinkstoscriptsTargets:RetailandhospitalityindustriesFIN7usedspearphishingattackstargetedatretail,restaurantandhospitalitybusinesses.
Whatappearedtobecustomercomplaints,CVs(resumes)andfoodorderssentinWordandRTFformatteddocuments,wereactuallyattacksthathidmalicious(VBS)codebehindhiddenlinks.
References:https://attack.
mitre.
org/groups/G0046/AttackGroup:FIN4MethodofAttack:Man-in-the-middlespearphishingTargets:FinancialmarketsThisgroupstolecleanOfficedocumentsfromthetargetandeditedthem,embeddingmaliciousmacros.
Byusingcorrectlyformatteddocumentscontainingrealinformation,stolenfromcompromisedaccounts,theattackersincreasedthelikelihoodthatrecipientswouldbetrickedintoopeningthedocumentsandallowingtheirownsystemstobecompromised.
References:https://attack.
mitre.
org/groups/G0085/TargetedAttackTypesEmailSecurityServicesProtectionJan-Mar202017CommodityAttackTypesCategorySub-categoryTotalsMalwareAttachment15SocialAdvancedFee43FakeLove2Sextortion3MoneyMule2PhishingLinks4Attachment1AttackGroup:APT33(2017)MethodofAttack:HTMLapplicationfilesTargets:AviationIn2017thisgroupsentspearphishingemailstoemployeesintheaviationindustry.
TheemailmessagesweresupposedlyrelatedtorecruitmentbutcontainedlinkstomaliciousHTMLapplication(.
hta)files.
These.
htafilescontainedjobdescriptionsandlinkstorealrecruitmentadvertisements,aswellaslinkstomalware.
References:https://attack.
mitre.
org/groups/G0064/AttackGroup:APT33(2019)MethodofAttack:WinRARexploitTargets:GovernmentAttacksinFebruary2019involvedsendingspearphishingemailswithmaliciousWinRARfileattachments.
ThegroupfocusedonSaudiArabiaandtheUnitedStates,aimingtoattacksupplychainsinvolvedingovernmentandrelatedindustriesincludingresearch,chemical,engineeringandmanufacturing.
References:https://attack.
mitre.
org/groups/G0064/Themaincategoriesofthecommodityattacksusedrepresentverycommontypesofapproachtoengagingwithatargetoveremail.
Thesearebysendingmalware;tryingtosociallyengineeravictimthroughpersuasiontodosomething(likesendmoney);andphishing,whichisanattempttotricktheuserintosendingimportantinformationlikeaccountdetailsorpasswords.
Inthistestweattachedallofthemalwaresamplestotheemails.
Forsocialengineeringtestcaseswetriedtotrickthetargetintosendingmoneyforservicesthatwillneverbedelivered,suchasfakelotterywins(AdvancedFee),aswellasblackmailattempts(Sextortion),promisesofsexualrelationships(FakeLove)andenticementtocybercriminalenterprises(MoneyMule).
Phishingattacksincludedlinkstofakewebsitespurportingtobewell-knownbanks,socialmediasitesetc.
(Links),andsimilarlog-informsembeddedintheemails(Attachment).
EmailSecurityServicesProtectionJan-Mar202018FortinetFortiMailProductStoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social4310001015Phishing435390000Malware650500000BusinessEmailCompromise200000000TOTAL17168901015AppendixB:DetailedResultsTargetedAttackDetailsThefollowingtablesshowhoweachservicehandleddifferenttypesoftargetedattack.
Thetableattheendoftheseriesalsosummariseshowtheyhandleddifferentcategoriesofcommoditythreats.
Therearefourmaincategoriesoftargetedattackusedinthistest:SocialEngineeringPhishingMalwareBusinessEmailCompromiseEachservicehasanumberofoptionswhenhandlingsuchthreats.
Thetablesshowhoweachservicehandledeachcategory.
Forexample,youcanseehowmanysocialengineeringsamplesmadeitthroughtotheinbox;howmanyweresenttotheJunkfolder;andhowmanywerepreventedfromcominganywhereneartheuser-Stopped,rejectedorEdited(deny)arecommonoptions.
Noteverypossibleoptionneedstobetakenbyaserviceundertest,sothetablesshowonlythoseoutcomesthatoccurred.
SocialProtection73%SocialProtection88%PhishingProtection100%SocialProtection88%BusinessEmailCompromiseProtection100%TotalProtection92%SocialProtection88%MalwareProtection100%EmailSecurityServicesProtectionJan-Mar202019SocialProtection88%PhishingProtection55%SocialProtection88%PhishingProtection62%19GoogleGSuiteEnterpriseStoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social10000401000Phishing9002800023Malware04500213010BusinessEmailCompromise100001900TOTAL20450284242033GoogleGSuiteBusinessStoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social10000401000Phishing9002406021Malware04500015010BusinessEmailCompromise100001900TOTAL20450244050031TotalProtection64%BusinessEmailCompromiseProtection5%BusinessEmailCompromiseProtection5%TotalProtection61%SocialProtection83%SocialProtection83%MalwareProtection64%MalwareProtection67%EmailSecurityServicesProtectionJan-Mar20202020MicrosoftOffice365StoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social13000291800Phishing9000113901Malware5400011500BusinessEmailCompromise000019100TOTAL76000607301KasperskySecurityforOffice365StoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social12013201401Phishing9001203504Malware700000000BusinessEmailCompromise000190001TOTAL91016304906SocialProtection88%CommodityProtection100%MalwareProtection100%BusinessEmailCompromiseProtection95%SocialProtection75%PhishingProtection35%TotalProtection74%SocialProtection70%MalwareProtection79%BusinessEmailCompromiseProtection95%TotalProtection65%PhishingProtection33%EmailSecurityServicesProtectionJan-Mar20202121MimecastSecureEmailGatewayStoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social1139000802Phishing936630060Malware358900000BusinessEmailCompromise016000202TOTAL2314915301064MicrosoftOffice365AdvancedThreatProtectionStoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social11000301900Phishing10031013411Malware4717011400BusinessEmailCompromise000020000TOTAL68110151343411SocialProtection88%BusinessEmailCompromiseProtection100%SocialProtection68%PhishingProtection23%MalwareProtection80%TotalProtection62%SocialProtection83%PhishingProtection90%BusinessEmailCompromiseProtection80%MalwareProtection100%TotalProtection90%EmailSecurityServicesProtectionJan-Mar202022Perception-PointStoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)Social510000900Phishing600000000Malware690000001BusinessEmailCompromise200000000TOTAL2000000901SocialProtection85%MalwareProtection99%TotalProtection95%PhishingProtection100%BusinessEmailCompromiseProtection100%EmailSecurityServicesProtectionJan-Mar202023CommodityAttackDetailsCommodityAttackDetailsPRODUCTStoppedRejectedEdited(deny)Junk(deny)JunkFolderInboxEdited(allow)Junk(allow)FortinetFortiMail4030000000GoogleGSuiteBusiness1060000000GoogleGSuiteEnterprise1060000000MimecastSecureEmailGateway367000000Perception-Point700000000KasperskySecurityforOffice3653500330101MicrosoftOffice3653600032200MicrosoftOffice365AdvancedThreatProtection3500032201FortinetFortiMail100%ProtectionGoogleGSuiteBusiness100%ProtectionGoogleGSuiteEnterprise100%ProtectionPerception-Point100%ProtectionKasperskySecurityforOffice36599%ProtectionMicrosoftOffice36597%ProtectionMicrosoftOffice365ATP97%ProtectionMimecastSecureEmailGateway100%ProtectionEmailSecurityServicesProtectionJan-Mar202024Theseresultsshowhoweffectivelyeachservicemanagedmessagesthatposednothreat.
Inanidealworldalllegitimatemessageswouldarriveintheinbox.
Whentheyarecategorisedasbeingathreatthena'falsepositive'resultisrecorded.
Itisimportanttotestforfalsepositivesbecausetoomanyindicateaproductthatistooaggressiveandwillblockusefulemailaswellasthreats.
Itwouldbeeasytocreateaproductthatblockedallthreatsifitwasalsoallowedtoblockalllegitimateemail.
Findingthebalancebetweenallowinggoodandblockingbadisthekeytoalmosteverytypeofsecuritysystem.
LEGITIMATEMESSAGEDETAILSProductInboxJunkFolderStoppedMimecastSecureEmailGateway7000Perception-Point7000KasperskySecurityforOffice36560100MicrosoftOffice36560100MicrosoftOffice365AdvancedThreatProtection60100GoogleGSuiteBusiness59110GoogleGSuiteEnterprise57130FortinetFortiMail6703LegitimateMessageDetailsFortinetFortiMail96%EffectiveMimecastSecureEmailGateway100%EffectivePerception-Point100%EffectiveKasperskySecurityforOffice36599%EffectiveGoogleGSuiteEnterprise100%EffectiveMicrosoftOffice365ATP100%EffectiveGoogleGSuiteBusiness100%EffectiveMicrosoftOffice365100%EffectiveEmailSecurityServicesProtectionJan-Mar202025Theresultsbelowusethefollowingterms:NotifiedTheservicepreventedthethreatfrombeingdeliveredandnotifiedtheuser.
Therewasnooptionfortheusertorecoverthethreat.
StoppedTheservicesilentlypreventedthethreatfrombeingdelivered.
RejectedTheservicepreventedthethreatfrombeingdeliveredandsentanotificationtothesender.
Edited(deny)Theservicedeliveredthemessagebutalteredittoremovemaliciouscontent.
Junk(deny)Theservicemodifiedthemessage,whichwassenttothetargetJunkfolder.
Themaliciouscontentwasremoved.
BlockedForsomereason,otherthantheinvolvementofthetestedservice,themessagewaspreventedfromarriving.
Quarantined(admin)Theservicepreventedthethreatfrombeingdeliveredandkeptacopyofit,whichcouldberecoveredbytheadministratoronly.
Quarantine(user)Theservicepreventedthethreatfrombeingdeliveredandkeptacopyofit,whichcouldberecoveredbytheuser.
JunkFolderThemessagewasdeliveredtotheuser'sJunkfolderbytheemailserviceprovider(e.
g.
MicrosoftOffice365;GoogleGSuiteBusiness)orbyanotherintegratedservice.
Junk(allow)Theservicemodifiedthemessage,whichwassenttothetargetJunkfolder,butdidn'tremovethemaliciouscontent.
InboxTheservicefailedtodetectorprotectagainstthethreat.
Edited(allow)Theservicemodifiedthemessage,whichwassenttothetargetinbox,butdidn'tremovethemaliciouscontent.
AppendixC:TermsUsedEmailSecurityServicesProtectionJan-Mar202026AppendixD:FAQsAfullmethodologyforthistestisavailablefromourwebsite.
TheproductschosenforthistestwereselectedbySELabs.
Thetestwasunsponsored.
Thetestwasconductedbetween3rdand17thofFebruary2020.
Allproductswereconfiguredaccordingtoeachvendor'srecommendations,whensuchrecommendationswereprovided.
Maliciousemails,URLs,attachmentsandlegitimatemessageswereindependentlylocatedandverifiedbySELabs.
TargetedattackswereselectedandverifiedbySELabs.
Maliciousandlegitimatedatawasprovidedtopartnerorganisationsoncethetestwascomplete.
SELabsconductedthisemailsecurityservicesprotectiontestusingrealemailaccountsrunningonpopularcommercialservices.
QWhatisapartnerorganisationCanIbecomeonetogainaccesstothethreatdatausedinyourtestsAPartnerorganisationsbenefitfromourconsultancyservicesafteratesthasbeenrun.
Partnersmaygainaccesstolow-leveldatathatcanbeusefulinproductimprovementinitiativesandhavepermissiontouseawardlogos,whereappropriate,formarketingpurposes.
Wedonotsharedataononepartnerwithotherpartners.
Wedonotpartnerwithorganisationsthatdonotengageinourtesting.
QIamasecurityvendorandyoutestedmyproductwithoutpermission.
MayIaccessthethreatdatatoverifythatyourresultsareaccurateAWearewillingtoshareacertainleveloftestdatawithnon-partnerparticipantsforfree.
Theintentionistoprovidesufficientdatatodemonstratethattheresultsareaccurate.
Formorein-depthdatasuitableforproductimprovementpurposeswerecommendbecomingapartner.
AppendixE:ServicesTestedThetablebelowshowstheservice'snameasitwasbeingmarketedatthetimeofthetest.
SERVICESTESTEDVendorServiceFortinetFortiMailCloud-GatewayPremiumGoogleGSuiteBusinessGoogleGSuiteEnterpriseKasperskySecurityforOffice365MicrosoftOffice365MicrosoftOffice365withAdvancedThreatProtectionMimecastSecureEmailGatewayPerception-PointPerception-Point27SELabsReportDisclaimer1.
TheinformationcontainedinthisreportissubjecttochangeandrevisionbySELabswithoutnotice.
2.
SELabsisundernoobligationtoupdatethisreportatanytime.
3.
SELabsbelievesthattheinformationcontainedwithinthisreportisaccurateandreliableatthetimeofitspublication,whichcanbefoundatthebottomofthecontentspage,butSELabsdoesnotguaranteethisinanyway.
4.
Alluseofandanyrelianceonthisreport,oranyinformationcontainedwithinthisreport,issolelyatyourownrisk.
SELabsshallnotbeliableorresponsibleforanylossofprofit(whetherincurreddirectlyorindirectly),anylossofgoodwillorbusinessreputation,anylossofdatasuffered,pureeconomicloss,costofprocurementofsubstitutegoodsorservices,orotherintangibleloss,oranyindirect,incidental,specialorconsequentialloss,costs,damages,chargesorexpensesorexemplarydamagesarisinghisreportinanywaywhatsoever.
5.
Thecontentsofthisreportdoesnotconstitutearecommendation,guarantee,endorsementorotherwiseofanyoftheproductslisted,mentionedortested.
6.
Thetestingandsubsequentresultsdonotguaranteethattherearenoerrorsintheproducts,orthatyouwillachievethesameorsimilarresults.
SELabsdoesnotguaranteeinanywaythattheproductswillmeetyourexpectations,requirements,specificationsorneeds.
7.
Anytrademarks,tradenames,logosorimagesusedinthisreportarethetrademarks,tradenames,logosorimagesoftheirrespectiveowners.
8.
Thecontentsofthisreportareprovidedonan"ASIS"basisandaccordinglySELabsdoesnotmakeanyexpressorimpliedwarrantyorrepresentationconcerningitsaccuracyorcompleteness.