temsopendns

End-UserMapping:NextGenerationRequestRoutingforContentDeliveryFangfeiChenAkamaiTechnologies150BroadwayCambridge,MAfachen@akamai.
comRameshK.
SitaramanUniversityofMassachusetts,Amherst&AkamaiTechnologiesramesh@cs.
umass.
eduMarceloTorresAkamaiTechnologies150BroadwayCambridge,MAmtorres@akamai.
comABSTRACTContentDeliveryNetworks(CDNs)delivermuchoftheworld'sweb,video,andapplicationcontentontheInternettoday.
AkeycomponentofaCDNisthemappingsystemthatusestheDNSprotocoltorouteeachclient'srequesttoa"proximal"serverthatservestherequestedcontent.
WhiletraditionalmappingsystemsidentifyaclientusingtheIPofitsnameserver,wedescribeourexperienceinbuildingandrolling-outanovelsystemcalledend-usermappingthatidentiestheclientdirectlybyusingaprexoftheclient'sIPad-dress.
UsingmeasurementsfromAkamai'sproductionnet-workduringtheroll-out,weshowthatend-usermappingprovidessignicantperformancebenetsforclientswhousepublicresolvers,includinganeight-folddecreaseinmap-pingdistance,atwo-folddecreaseinRTTandcontentdown-loadtime,anda30%improvementinthetime-to-rst-byte.
Wealsoquantifythescalingchallengesinimplementingend-usermappingsuchasthe8-foldincreaseinDNSqueries.
Fi-nally,weshowthataCDNwithalargernumberofdeploy-mentlocationsislikelytobenetmorefromend-usermap-pingthanaCDNwithasmallernumberofdeployments.
1.
INTRODUCTIONContentDeliveryNetworks(CDNs)delivermuchoftheworld'swebsites,videoportals,e-commerceapplications,socialnetworks,andledownloads.
Asanexample,Aka-mai'sCDNcurrentlyserves15-30%ofallwebtrafcfromalargedistributedplatformofover170,000serversdeployedinover102countriesand1300ISPsaroundtheworld[2].
TheCDNhostsanddeliverscontentonbehalfofthousandsofenterprisesandorganizationsthatrepresentamicrocosmoftheInternetasawhole,includingbusinessservices,nan-Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.
Copyrightsforcomponentsofthisworkownedbyothersthantheauthor(s)mustbehonored.
Abstractingwithcreditispermitted.
Tocopyotherwise,orrepublish,topostonserversortoredistributetolists,requirespriorspecicpermissionand/orafee.
Requestpermissionsfrompermissions@acm.
org.
SIGCOMM'15,August17-21,2015,London,UnitedKingdomc2015Copyrightheldbytheowner/author(s).
PublicationrightslicensedtoACM.
ISBN978-1-4503-3542-3/15/08.
.
.
$15.
00DOI:http://dx.
doi.
org/10.
1145/2785956.
2787500Content&Server&Client&Origin&Mapping&System&Overlay&Transport&Figure1:AContentDeliveryNetworkcialservices,travel,manufacturing,automotive,media,en-tertainment,e-commerce,software,gaming,andthepublicsector.
Theclients1whoaccesscontentonAkamai'sCDNarearepresentativesampleofInternetusersfromnearlyev-erycountry,everymajorISP,anduseeverymajordevice.
ThegoalofaCDNistohostanddelivercontentandap-plicationstoclientsaroundtheworldwithhighavailabil-ity,performance,andscalability[13,21].
Akamai'sCDNachievesitsgoalbydeployingalargenumberofserversinhundredsofdatacentersaroundtheworld,soastobe"proximal"inanetworksensetoclients.
TounderstandtheoverallarchitectureoftheCDN,weenumeratethestepsin-volvedwhenaclientaccessescontenthostedontheCDN.
AsshowninFigure1,whentheclientaccessesaWebpage,thedomainnameoftheWebpageistranslatedtotheIPad-dress(shortenedto"IP"inthispaper)ofaserverthatislive,hassufcientcapacity,andisproximaltotheclient.
ThedomainnametranslationisprovidedbytheCDN'smappingsystemthatwestudyinthispaper.
Theclientrequestscon-tentfromtheserverassignedtoitbythemappingsystem.
Iftheserverhasthecontentincache,itservesthecontenttotheclient.
Otherwise,theserverrequeststhecontentfromtheoriginserversthatareoperatedbythecontentprovider1Inthispaper,weusetheterm"client"todenotetheend-userorhis/herdevicesuchasacellphone,desktoporlaptopthatisconnectedtotheInternetandisrunningsoftwaresuchasabrowsercapableofaccessingWebpages.
andandservesittotheclient.
ForamoredetaileddiscussionofCDNevolutionandarchitecture,wereferto[21].
AcentralcomponentofAkamai'sCDNisitsmappingsystem.
Thegoalofthemappingsystemistomaximizetheperformanceexperiencedbytheclientbyensuringquickerdownloadsoftheaccessedcontent.
Tospeedupthedown-loads,themappingsystemrouteseachclientrequesttoa"proximal"serverthatcanbereachedbytheclientwithlowlatencyandloss.
Further,themappingsystemensuresthatthechosenserverislive,notoverloaded,andislikelytocon-taintherequestedcontent.
Thelastconsiderationreducestheprobabilityofacachemissattheserverthatwouldresultinthecontentbeingfetchedfromoriginwithlongerresponsetimes.
Conceptually,themappingsystemcanbeviewedascomputingthefollowingcomplextime-varyingfunction:MAPt:InternetAkamDomainLDNS!
IPs.
(1)Ateachtimet,MAPttakesasinputthecurrentstateoftheglobalInternetInternet,includingdetailedreal-timeknowl-edgeofconnectivity,liveness,latency,loss,andthroughputinformation;thecurrentstateofAkamai'sCDNAkam,in-cludingreal-timeknowledgeofliveness,load,andotherin-formationaboutserversandroutersoftheCDN;thedomainnameofthecontentthatisbeingaccessedbytheclient;and,thelocalrecursivedomainnameserver(LDNS)thatmakestherequestfordomainnameresolutiononbehalfoftheclient.
ThemappingsystemreturnstwoormoreIPs2oftheCDN'sserversthatcanservetherequestedcontent.
Akamai'smappingsystemroutestrillionsofclientrequestsperday,controllingtensofterabitspersecondofcontenttrafcservedtoclientsworld-wide.
Onatypicalday,thereare6.
4millionLDNSserverslocatedin240countriesmak-ing1.
6millionDNSqueriespersecond(cf.
Figure2),rep-resenting30millionclientrequestspersecondaroundtheglobe.
EversincetherstCDNswerebuiltatAkamainearlysixteenyearsago[13],themappingsystemhasbeenthesub-jectofmuchresearchtoimproveitsresponsivenessandac-curacy.
However,asrepresentedinEquation1,traditionalmappingsystemsatCDNsmakerequestroutingdecisionsbasedontheidentityoftheclient'sLDNSratherthanthatoftheclientitself.
WecallthisNS-basedmappingandcanbeinaccurateincaseswhentheLDNSisnotina"similar"lo-cationastheclient,i.
e.
,whenthenetworkcharacteristicsoftheLDNSisnotagoodproxyforthatoftheclient.
Thisfun-damentallimitationarisesfromtheuseofthedomainnamesystem(DNS)protocolthatallowsthemappingsystemtolearntheidentityoftheLDNSbutnottheclientonwhosebehalfthedomainnametranslationrequestismade.
TorectifythelimitationsoftheDNSprotocol,Google,AkamaiandothersindustryplayershaverecentlyproposedanextensiontotheDNSprotocolthatallowsarecursivenameservertospecifyaprexoftheclient'sIP(usuallya/24prex)whenrequestingdomainnametranslationsonbehalfofaclient[11].
Forthersttime,thismechanismal-2Whilethemappingsystemcheckslivenessbeforereturn-ingtheIPofaserver,morethanoneserverisreturnedasaadditionalprecautionagainsttransientfailures.
0102030405060Requestspersecond(million)Jan07Jan10Jan13Jan16Jan190123456Queriespersecond(million)ClientrequestsDNSqueriesFigure2:Clientrequestsserved(leftaxis)andDNSqueriesresolved(rightaxis)bythemappingsystem.
Whenthemap-pingsystemresolvesaDNSqueryfromaLDNS,multiplecontentrequestsfromclientsthatusethatLDNSmayfollow.
lowsthenovelpossibilityofbuildingamappingsystemthathasdirectknowledgeabouttheclientandusesittoprovidemoreaccuratemappings.
Wecallsuchasystemend-usermapping.
Theinsightsgainedfrombuildingandrolling-outAkamai'send-user(EU)mappingsystemisourfocus.
Ourcontributions.
Welistkeycontributionsbelow.
1.
ThelimitationsofNS-basedmappingcausedbydis-crepanciesinthelocationsofclientsandLDNSeshavebeenknownforoveradecade[24].
However,weprovidetherstpublicanalysisofclientsandtheirLDNSesataglobalscaleacrosstheentireInternetusingdatafrom584thousandnameserversand3.
76million/24clientIPblocksacross37294AS'esand238countriesacrosstheworld.
2.
Ourworkpresentsthearchitectureandreal-worldroll-outofAkamai'send-usermapping,amajorconceptualad-vanceinmappingtechnology.
Wecapturetheperformanceimpactoftheroll-outonactualclientsaroundtheworld.
WebperformanceisacomplexphenomenathatisinuencedbytheglobalstateoftheInternet,theconnectivityoftheclient,propertiesofWebsitesandtheirhostinginfrastruc-ture,andamultitudeofotherfactors.
Ourworkcapturestheimpactofthenewmappingparadigminareal-worldset-tingprovidinginsightsthatarehardtoobtaininacontrolledexperimentalsetting.
3.
End-usermappingrequiresbothmeasurementsandanalysistobeperformedatamuchlargerscale,asmappingdecisionsaremadeatamuchnergranularity.
Usingex-tensivedatafromclientsandtheirLDNSarchitecturesintheglobalInternetandmeasurementstakenduringtheend-usermappingroll-out,weprovideinsightsintothescalingcon-siderationsinusingtheEDNS0client-subnetextensionoftheDNSprotocol.
4.
Usinglatencymeasurementsfromover2500+serverdeploymentlocationsaroundtheworldto8KrepresentativeclientIPblocksthatgeneratethemosttrafcontheInternet,westudytheimportantquestionofhowdeploymentsimpacttheperformanceoftraditionalNS-basedandend-usermap-ping.
Weshowthatend-usermappingprovidesmoreincre-mentalbenetsforaCDNwithserversinalargenumberofdeployedlocationsthanaCDNdeployedinfewerloca-tions.
Further,weexposeaninherentlimitationofNS-basedmappinginreducinglatenciesfortheworst1%ofclients.
Roadmap.
InSection2,wedescribethearchitectureofatraditionalNS-basedmappingsystemandhowend-usermappingcanbeincorporatedintothisarchitectureusingtheEDNS0client-subnetextension.
InSection3,weanalyzetherelativelocationsofclientsandtheirLDNSesintheglobalInternetwiththeviewofunderstandingthebenetsthatwearelikelytoseefromrolling-outend-usermapping.
InSec-tions4and5,weanalyzetheperformanceandscalabilityimpactofourroll-outofend-usermappingsystemtoclientswhousepublicresolvers.
InSection6,westudytheroleofserverdeploymentsinend-usermapping.
InSection7wepresentrelatedworkandconcludeinSection8.
2.
THEMAPPINGSYSTEMAWebsitehostedonAkamaitypicallydelegatesauthor-ityforitsdomainnamestoauthoritativenameserversthatarepartofthemappingsystem.
Further,eachclientusesa"local"domainnameserver3(LDNS)thatworksinare-cursivefashiontoprovidedomainnametranslationsfortheclient.
TheLDNSthatprovidesdomainnameservicefortheclientistypicallyhostedbytheInternetServiceProvider(ISP)whoprovidesInternetconnectivitytotheclient.
Alter-nately,theLDNScouldbeanpublicresolverthatisanameserverdeployedbyathird-partyproviderthatcanbeusedbytheclient.
ThelocationoftheLDNSwithrespecttotheclientdependsontheDNSarchitectureofthenameserviceprovider,whetheritbeanISPorapublicresolverprovidersuchasGoogleDNS[5]orOpenDNS[7].
TobetterillustrateAkamai'smappingsystem,wetracethroughthestepsofhowaclientinteractswiththesystemtoobtainadomainnameresolution(seeFigure3).
(1)SupposethattheclientwantstoaccesscontentatsomeWebsitethatishostedonAkamai.
TheclientrequestsitsLDNStoresolvethedomainnameoftheWebsite.
(2)LDNSworksinarecursivemodeasfollows.
IftheLDNShasavalidnameresolutionfortherequesteddomaininitscache,itrespondstotheclientwiththerelevantIPs.
Otherwise,theLDNSforwardstherequesttoanauthorita-tivenameserverfortherequesteddomain.
(3)TheauthoritativenameserverrespondswithavalidresolutiontotheLDNS.
LDNScachestheresponseandinturnforwardstheresponsetotheclient.
NotethataDNSresponsefromanauthoritativenameserverisassociatedwithaTTL(time-to-live)thatdictateshowlongtheresponseisvalid.
TTL'saretrackedandenforcedastheresponseisforwardedandcacheddownstreambynameserversandresolvers,includingtheLDNSandtheclient'sresolver.
WhentheTTLexpires,thecachedentryismadeinvalid,requiringanewDNSresolution.
NS-basedversusend-usermapping.
InatraditionalNS-basedmappingsystem,theLDNSdoesnotforwardanyin-formationabouttheclientwhenitcontactstheauthorita-tivenameserversinstep(2)above.
Hence,themappingsystemdoesnotknowtheIPoftheclientthatrequestedthenameresolutionandassignsedgeserversentirelybased3Despiteitsname,aLDNSmaynotbevery"local"totheclient,thekeyrationaleforend-usermapping.
TOPOLOGYDISCOVERYSCORING(authoritative)LOADBALANCINGLOCALGLOBALTOPLEVELLOWLEVELPERIODICREAL-TIMENETWORKMEASUREMENTINTERNETServerAssignmentDNSLDNS(recursive)CLIENTCONTENTDATAMappingSystemNAMESERVERSFigure3:ThearchitectureofthemappingsystemLDNS(recursive)CLIENT(A.
B.
C.
D)ContentDownload(authoritative)NAMESERVERSSERVER(E.
F.
G.
H)E.
F.
G.
Hfoo.
netfoo.
netforA.
B.
C.
D/24E.
F.
G.
HforA.
B.
C.
D/20Figure4:Exampleofinteractionbetweentheclient,LDNS,andAkamai'snameserverswiththeEDNS0extension.
ontheIPofitsLDNS.
However,inend-usermapping,theLDNSforwardsaprexoftheclient'sIPtotheauthorita-tivenameserversinstep(2)aboveusingthenewly-proposedEDNS0client-subnetextension.
Thisenablestheend-usermappingtouseadditionalclientinformationinprovidingdomainnametranslationsaswedescribenext.
2.
1End-UserMappingEnd-usermappingdeployedrecentlyatAkamaiusestheidentityoftheclientratherthanitsLDNS.
Conceptually,end-usermapping(EUMAP)computesthefollowingtimevaryingfunction.
EUMAPt:InternetAkamDomainClient!
IPs.
(2)ComparedtoNS-basedmapping(seeEquation1),end-usermappingusestheclientinformationtomakemoreaccuratemappingdecisions,evenincaseswheretheLDNSandtheclientarenotproximaltoeachother.
Akeyenablerforourend-usermappingdesignisarecentInternetdrafttoextendtheDNSprotocolcalledtheEDNS0client-subnetextensionthatallowsrecursivenameserverstoforwardinformationabouttheclientasapartoftheirDNSrequest[11].
Figure4showstheinteractionbetweentheclient,recursive,andau-thoritativenameserversforanexampledomainfoo.
netwhenthenameserverssupporttheEDNS0protocolexten-sion.
TheclientwithIPA.
B.
C.
DcontactsitsLDNStoresolvefoo.
net.
Withextensionsupport,whentheLDNSforwardstherequestforfoo.
nettoanauthoritativenameserveritcanappenda/xprexoftheIPoftheclientwhoinitiatedtherequest,wheretheprex4usedisgenerally/24.
(By/xprexwemeantherstxbitsoftheIP.
)Theauthori-tativenameservers,whichinthecaseofadomainhostedonAkamaiispartofthemappingsystem,respondswithserverIPsappropriatefora/yprexoftheclient'sIPwhereyx,i.
e.
,thenameservercanreturnaresolutionthatisvalidforasupersetoftheclient's/xIPblock.
(Byclient's/xIPblock,wemeanthesetofIPsthathavesamerstxbitsastheclient'sIP.
)TheDNSresolutionprovidedbytheauthoritativenameservercanbecachedforthedurationoftheTTLbydownstreamrecursivenameserverssuchastheLDNS.
However,thecachedresolutionisonlyvalidfortheIPblockforwhichitwasprovidedandnotforanyclientIPsthatdonotbelongtotheblock.
2.
2MappingSystemArchitectureThemappingsystemconsistsofthreemajorfunctionalcomponentsasshowninFigure3thatwedescribeinturn.
Wealsousedatacollectedfromthenetworkmeasurementcomponentbelowforouranalysis.
1)NetworkMeasurement.
BoththeglobalInternetandAkamai'sCDNaremonitoredandmeasured.
Thedatathatneedstobecollectedonbothcountsisenormousandvaried.
TheInternetisalarge"patchwork"of71Kautonomoussys-tems(AS's)thatinterconnectwitheachotherincomplexandever-changingways.
TheserverandnetworkcomponentsofAkamai'sCDNaredeployedinclustersinmorethanathou-sandnetworksaroundtheglobe.
Afewmajorsourcesofdatacollectedinclude:(i)AS-levelinformationiscollectedbyAkamai'sBGPcollectorsinstalledaroundtheInternetthatinitiateBGPses-sionswithISP'sandperiodicallyrecordstheBGPsessionstate.
ThisinformationisusedtounderstandwhichIPsbe-longtowhichAS,howAS'esconnectwitheachother,etc.
(ii)Geographicinformationsuchasthecity,state,coun-try,andcontinentisdeducedforIPsaroundtheworldusingvariousdatasourcesandgeolocationmethods[1].
(iii)NameserverinformationiscollectedusingtheDNSrequestlogsforAkamai-hosteddomainsfromnameservers(i.
e.
,LDNSes)aroundtheworld.
(iv)Network-levelmeasurementsincludepathinforma-tion,latency,loss,andthroughputbetweendifferentpointsontheInternet.
(v)Livenessandload.
LivenessandloadinformationofallcomponentsofAkamai'sCDNiscollectedinreal-time,includingserversandrouters.
2)ServerAssignment.
Theserverassignmentcomponentusesnetworkmeasurementdatatocreateareal-timetopo-logicalmapoftheInternetthatcaptureshowwellthediffer-entpartsoftheInternetconnectwitheachother,aprocess4Aprexlongerthan/24isdiscouragedtoretainclient'sprivacy.
calledtopologydiscovery.
ThetopologicalmapisthenusedtoevaluatewhatperformanceclientsofeachLDNSislikelytoseeiftheyareassignedtoeachAkamaiservercluster,aprocesscalledscoring.
Differentscoringfunctionsthatin-corporatebandwidth,latency,packetloss,etccanbeusedfordifferenttrafcclasses(web,video,applications,etc).
Theloadbalancingmoduleassignsserverstoeachclientrequestintwohierarchicalsteps:rstitassignsaserverclusterforeachclient,aprocesscalledgloballoadbalancing.
Next,itassignsserver(s)withinthechosencluster,aprocesscalledlocalloadbalancing.
Toperformthesetasks,theloadbal-ancerusestheoutputofscoringtoevaluatecandidateserverchoicesthatyieldthehighestperformanceforeachclientre-questandcombinesthatinformationwithliveness,capacity,andotherreal-timeinformationabouttheCDN.
Theloadbalancingalgorithmsaredescribedingreaterdetailin[19].
3)NameServers.
AkamaihasalargedistributedsystemofnameserversaroundtheworldthatactasauthoritiesforAkamai-hosteddomainnames.
Forexample,acontentprovi-derhostedonAkamaicanCNAMEtheirdomaintoanAka-maidomain,forexample,www.
whitehouse.
govcouldbeCNAME'dtotheAkamaidomainofe2561.
b.
akamaiedge.
net.
TheauthorityforthelatterdomainisinturndelegatedtoanAkamainameserverthatistypicallylocatedinanAka-maiclusterthatisclosetotheclient'sLDNS.
Thisdele-gationstepimplementsthegloballoadbalancerchoiceofclusterfortheclient'sLDNS,sodifferentclientscouldre-ceivedifferentnameserverdelegations.
Finally,thedele-gatednameserverreturns"A"recordsfortwoormoreserverIPstobeusedbytheclientforthedownload,implementingthechoicesmadebythelocalloadbalancer.
3.
UNDERSTANDINGCLIENTSANDTHEIRNAMESERVERSTomotivatetheneedforend-usermapping,westartbyanalyzingthelocationsofclientsrelativetotheirrecursivenameservers(i.
e.
,LDNS)intheglobalInternet.
ToobtainanaccuratepictureweneedtomatchalargecharacteristicsetofclientsaroundtheworldwiththeirrespectiveLDNSes.
Thematchedclient-LDNSpairscanthenbelocatedusingourgeo-locationdatabase[1]toprovidethegeographiclo-cationandnetworkinformationneededfortheanalysis.
3.
1CollectingClient-LDNSpairsAssociatingaclientwithitsLDNShassomeintrinsicdif-culties.
BoththeLDNS'srequestforadomainnamereso-lutionandtheclient'ssubsequentrequestforanURLonthatdomainareloggedatAkamai'sauthoritativenameserversandcontentserversrespectively.
Onepotentialapproachistomatchtheserequeststoobtainclient-LDNSpairings.
However,matchingtherequestsistrickyandinexactsincethetworequestscanbespacedwithinatimewindowequaltotheTTLofthedomainname.
Further,whentheclientre-ceivesacachedresponsefromitsLDNS,theLDNSmakesnocorrespondingdownstreamrequesttoAkamai'sauthor-itativenameservers.
Whilethereareheuristicwaysofob-tainingasmallersampleofclient-LDNSpairs[24],ourchal-lengeistoobtainalargecharacteristicanddenitivesetofpairsthathavegoodcoverageoftheclientswhogeneratetrafcontheglobalInternet.
Toobtainalargesetofpairs,weuseAkamai'sdownloadmanagercalledNetSession[3].
NetSessionisinstalledonclientdevicesandisusedtoperformdownloadsinafasterandmorereliablefashion.
Softwareandmediapublishersopt-intouseNetsessionfeaturestoimprovehttpdeliveryperformancefortheircontent.
Oncetheyopt-in,clientsuseNetsessiontodownloadthatcontent.
Thus,Netsessionhasalarge,representativeinstalledbaseofclientsaroundtheworld,makingitanidealmeasurementplatformforouranal-ysis.
Morethan30millionuniqueNetSessionclientsper-formtransactionseverymonth.
NetSessionwasinstrumentedtocollectLDNSinforma-tionasfollows.
EachNetSessionclientmaintainsapersis-tentconnectionwithaNetSessioncontrolplane.
EveniftheclientisbehindaNAT,itcanreliablylearnitsexternalclientIPfromthispersistentconnection.
NetSessionclientsalsofoundtheirLDNSserverperforminga"dig"commandonaspecialAkamainamewhoami.
akamai.
net.
Theclient-LDNSassociationwasthensenttoAkamai'scloudstorageforprocessing.
TheLDNSinformationforclientsaroundtheworldwerethenaggregatedinthecloudtothegranular-ityof/24clientIPblocks.
Specically,foreach/24clientIPblock,theprocessgeneratesthesetofIPscorrespondingtotheLDNSesusedbytheclientsinthataddressblock.
ForeachLDNSintheset,therelativefrequencywithwhichthatLDNSappearedwascomputed.
Usingtheaboveprocess,wecollectedLDNSdatafromMarch24toApril7,2014.
Onaverage,about14.
8mil-lionrecordswereprocessedperdayduringthecourseofourdatacollection.
Client-LDNSassociationdataforatotalof3.
76million/24clientIPblockswascomputedinaggre-gate.
WhiletheclientsthatuseNetSessionaregenerallyafractionofthetotalactiveclientsinanygiven/24clientIPblock,ourcoverageof/24clientIPblocksisrepresentativeandsignicantoftheoverallInternet.
Inparticular,the/24clientIPblocksinourdatasetaccountforabout84.
6%ofthetotalglobalclientdemand5servedbyAkamai.
Thenum-berofdistinctLDNSesinourdatasetwasjustover584,000.
Thus,ourdatasetisalargerepresentativecross-sectionofclientsandLDNSesintheglobalInternet.
Toderiveclient-LDNSdistanceestimates,weuseAka-mai'sEdgescape[1]geo-locationdatabasethatusesregistrydataandnetworkdatadistilledfromtransactionshandledbyover170,000Akamaiserversin102countriesandoverathousandISPdeploymentsaroundtheworldtoestablishge-ographicallocationandnetworkinformationforIPsaroundtheworld.
Edgescapecanprovidethelatitude,longitude,countryandautonomoussystem(AS)foranIP.
ForIPsinmobilenetworks,themobilegatewaylocationisusedasthereferencelocation.
Toderivethedistancebetweenaclient-LDNSpairweusethelatitudeandlongitudeinformationtocomputethegreatcircledistancebetweenthetwolocations.
5Clientdemandisameasureoftheamountofcontenttrafcdownloadedbyaclient(orbyclientsinanIPblock).
0%10%20%30%40%10100100010000ClientLDNSdistance(miles)PercentofclientdemandFigure5:Histogramofclient-LDNSdistanceforclientsacrosstheglobalInternet.
0250050007500INTRVNMXBRIDAURUITJPUSMYCADEFRGBNLARTHCHESHKKRSGTWCountryClientLDNSdistance(miles)Figure6:Client-LDNSdistancesbycountry.
3.
2HowfarareclientsfromtheirLDNSesFigure5showstheoverallglobaldistributionofclientLDNSdistances.
Nearlyhalfoftheclientpopulationislo-catedveryclosetoitsLDNS.
Themosttypicaldistanceliesinarangethatisnogreaterthanthediameterofametropoli-tanarea.
Ataround200-300miles,thereisanoteworthyincreaseinthemarginaldistribution.
Ataround5000miles,thereisanotherincreasethatcanbeattributedtothesmallnumberofclientsthatuseLDNSthatareeitheracrosstheAtlanticorPacicoceans.
Breakdownbycountry.
Breakingthesedistancesdownbycountry,Figure6isabox-plot6representingthe5th,25th,median,75th,and95thquantilesoftheper-countrydistribu-tions.
Welistdataforthetop25countriesasmeasuredbyaggregateclientdemand.
Overall,mostcountrieshaveame-diandistancethatisfairlysmall,thoughIndia,Turkey,Viet-namandMexicohavemediandistancesover1000miles.
In-dia,Brazil,Australia,andArgentinahavesignicantpopu-lationswhoseLDNSesareveryfarawayasoveraquarterofthepopulationisservedbyLDNSeswhosedistanceisover6Allboxplotsinthispapershow5th,25th,50th,75thand95thpercentiles.
4500miles.
WesternEuropeseeslowdistancesappearinginasmallband.
However,KoreaandTaiwanaresignicantinhavingthesmallestdistances.
Thisisnotsurprisingconsid-eringthewell-developedInternetinfrastructureandthecon-centrationofpopulationswithinasmallgeographicalareainthemajorcitiesinthesecountries.
JapanhasasmallmediandistancebutasignicantfractionofclientshaveLDNSesthatarefaraway.
Onereasonisclientsatmulti-nationalcor-porationswithcentralizedLDNSesdeployedoutsideJapan.
Publicresolvers.
Wenowevaluatetheclient-LDNSdis-tanceforpublicresolverswhereaclientusesanLDNSpro-videdbyapublicthird-partyprovidersuchasGooglePub-licDNSorOpenDNS.
SuchprovidershaveadistributednameserverinfrastructureanduseIPanycast[14]torouteclientstothe"closest"LDNSdeployment.
However,thepublicresolversusetheirunicastaddresseswhencommu-nicatingwithAkamai'sauthoritativenameserversallowingustogeo-locatethepublicLDNSes.
Figure7showsclient-LDNSdistanceforclientsthatusepublicresolvers.
Weseethattheclient-LDNSdistancesaresignicantlyhigherwithmediandistanceat1028miles,comparedtoamediandis-tanceof162milesintheoverallclientpopulation.
Thisre-ectsthefactthattheLDNSdeploymentsofapublicDNSprovidermayoftennotbelocaltotheclient.
0.
0%2.
5%5.
0%7.
5%10.
0%12.
5%10100100010000ClientLDNSdistance(miles)PercentofclientdemandFigure7:Histogramoftheclient-LDNSdistanceforclientswhousepublicresolvers.
ThecountrybreakdowninFigure8showsdisproportion-atelylargedistancesforclientsusingpublicresolversinsomecountriesinSouthAmerica,SouthEastAsiaandOceania.
ThetwoSouthAmericancountriesofArgentinaandBrazilhadthelargestdistances.
Inthisregard,itisnotablethatthelargestpublicresolverprovider,GooglePublicDNS,doesnotcurrentlyhaveapresenceinmanySouthAmericancountries.
SingaporeandMalaysiaarewellservedbythepublicresolvershostedinSingapore.
However,presumablyduetopeeringarrangements,manyclientsinthesecoun-triesareroutedtomoredistantpublicresolvers.
ClientswhousepublicresolversinWesternEurope,HongKongandTai-wanarerelativelyclosetotheirLDNSincomparisonwithothercountries,thoughtheyaremuchmoredistantwhencomparedtoclientsinthosesamecountrieswhodonotusepublicresolvers.
0100020003000400050006000ARBRAUINIDSGMYTHTRMXJPVNKRCAESUSRUHKITCHDEGBFRNLTWCountryClientLDNSDistance(miles)Figure8:Client-LDNSdistanceforclientswhousepublicresolvers.
0%20%40%VNTRITIDMYBRARINRUMXTHESTWUSGBHKCACHFRNLDESGAUJPKRCountryPercentofclientdemandfrompublicresolversFigure9:Percentofclientdemandoriginatingfrompublicresolvers,bycountry.
Theadoptionofpublicresolversalsovarycountrybycoun-try.
Figure9showsthepercentageofclientdemandorig-inatingfrompublicresolversbrokendownbycountryasseeninourNetSessiondataset.
ClientsinVietnamandTurkeyareveryheavyusersofpublicresolvers.
Remark-ably,despitethesignicantclient-LDNSdistances,asig-nicantfractionofclientsinIndia,Brazil,andArgentinausepublicresolvers.
Overall,percentofclientdemandfrompublicresolversapproaches8percentworldwide.
BreakdownbyAS.
Figure10showsthedistributionoftheclient-LDNSdistanceasafunctionoftheASsize,whereASsizeistheclientdemandoriginatingfromthatASasapercentageofthetotalclientdemandservedbyAkamai,i.
e.
,anASwithsize21hasclientsthataccountfor0.
5%ofthetotalclientdemandservedbyAkamai.
Atotalof37,294ASeswiththemostdemandwereanalyzed.
Ascanbeseeninthegure,whentheASsizeissmall,theclient-LDNSdistancesarelarge,especiallythehigherpercentilesofthedistance.
Thismayseemcounter-intuitive.
010002000300040005000210292827262524232221PercentofclientdemandfromASMedianclientLDNSdistance(miles)Figure10:Client-LDNSdistanceasafunctionofASsize.
But,thereasonisthatsmallerAS'esincludesmalllocalISPswhoaremorelikelyto"outsource"theirnameserverinfras-tructuretootherproviders.
ThereasonfortheoutsourcingiseconomicinnatureastheISPmaynotwanttoownandop-erateanameserverinfrastructure.
So,theISPmaychoosetheinexpensiveoptionofusingapublicresolveroperatedbyaprovidersuchasGooglePublicDNS,OpenDNS,Level3,UltraDNS,etc.
The"outsourcing"ofDNSservicesoftencausestheLDNSestobenon-local,leadingtolargerclient-LDNSdistances.
AdifferentcategoryofsmallAS'eswithlargeclient-LDNSdistancesareenterpriseswithgeographi-callydiversebranchofceswhoforoperationalconvenienceuseacentralizednameserverinfrastructuredeployedinonlyoneofthoseofces.
Giventhelargeclient-LDNSdistances,weexpectend-usermappingtobenetalargefractionofclientsofsmallAS'es.
LargeISPstypicallyoperatetheirownnameserverinfras-tructuresfortheirclients.
SuchinfrastructureoftenconsistsofLDNSesthataredeployedinmultiplegeographicallydis-tributedlocations.
Todirectclientstothe"nearest"LDNS,theIPanycast[16,15]mechanismisoftenused.
Thisex-plainsthesmallervaluesofclient-LDNSdistancedespitethelargegeographicalareacoveredbytheseglobalISPs.
How-ever,IPanycasthasmanyknownlimitations[23]thatcanresultinafractionoftheclientsbeingroutedtofarawayLDNSlocations.
Thus,end-usermappingmaybebenecialforclientsoflargeISPsalso.
3.
3HowfarareclientsthatusethesameLDNSfromeachotherAclientclusterisasetofclientsthatusethesameLDNS.
TheclientsontheInternetcanbepartitionedintoclientclus-ters,oneclusterforeachLDNS.
Wedenetheradiusofaclientclustertobethemeandistanceoftheclientsintheclustertothecentroidofthecluster7.
IntraditionalNS-basedmapping,aclientclusteristheunitformakingserverassign-mentdecisions,i.
e.
,allclientsinaclientclusterareassigned7Distancesarecomputedusingthelatitudeandlongitudeoftheclientsfromourgeo-locationdatabase.
Theradiusandcentroiduseclientdemandsastheweights.
thesamesetofserverIPs,sincetheyusethesameLDNS(cf.
Equation1).
IfaclientclusterofaLDNShasasmallra-dius,i.
e.
,theclientsareclosetogether,amoresophisticatedformofNS-basedmappingcouldstillbeeffective,eveniftheclient-LDNSdistancesarelarge.
Thereasonisthatthemappingsystemcoulddiscovertheclientclusterandassignserversthatprovidegoodperformancefortheentireclus-ter.
However,iftheclientclusterhasalargeradius,i.
e.
,theclientsarefarawayfromeachother,theremaybenosingleserverassignmentfortheentireclusterthatisoptimalforallclientsinit.
Thus,itisinherentlydifcultforNS-basedmappingtoperformwellwhentheclientclusterhasalargeradius,evenknowingclient-LDNSpairings.
Figure11reafrmsthatonanoverallbasisalargefractionofclientsareclosetotheirLDNSesandtheclusterradiiaresmall.
However,focusingonthesubsetofLDNSesthatarepublicresolvers,weseethatnotonlyareclient-LDNSdis-tanceslarge,butclusterradiiarelargeaswell.
Infact,99%ofthepublicresolverdemandoriginatesfromclientclus-terswithradiibetween470to3800miles.
Thegurealsoshowsthatforpublicresolversthemeancluster-LDNSdis-tancetendstobelargerthantheclusterradius.
ThisimpliesthattheLDNSisoftennotdeployedata"central"locationwithintheclientclusterthatitserves,i.
e.
,nearthecentroidthatminimizethemeanclient-LDNSdistance.
Thisisinpartduetothefactthatapublicresolverproviderdoesnothavene-grainedcontroloverwhichclientsinwhichloca-tionsusetheirservice.
e.
g.
,clientsfromcountrieswheretheproviderhasnodeploymentsoftenusetheservice.
0%25%50%75%100%1025005000Distance(miles)CumulativepercentofclientdemandClusterradius(allLDNS)ClientLDNSmeandistance(allLDNS)Clusterradius(publicresolvers)ClientLDNSmeandistance(publicresolvers)Figure11:CDFsofmeanclient-LDNSdistanceandclus-terradiusforallLDNSesandforthesubsetthatarepublicresolvers.
4.
PERFORMANCEIMPACTWepresentourexperienceandinsightsobtainedinde-ployingend-usermappingforclientsaroundtheworldinthersthalfof2014.
Duringthisperiod,Akamaibegantheroll-outofend-usermappingforclientswhousepublicresolverssuchasGooglePublicDNSandOpenDNS.
Thereasonsforinitiallytargetingclientswhousepublicresolversweretwo-fold.
Basedonouranalysisofclient-LDNSdistancesinSec-tion3,weconcludedthatclientswhousepublicresolversaremorelikelytobenetfromend-usermapping,sincetheytendtobefartherawayfromtheirLDNSes(cf.
Figure7)andalsohadlargeclientclusterradii(cf.
Figure11).
Fur-ther,publicresolverproviderssuchasGooglePublicDNSandOpenDNSsupporttheEDNS0client-subnetextensionthatisrequiredforend-usermapping.
Theend-usermap-pingroll-out8startedonMarch28th2014andcompletedonApril15th2014.
Wepresentinsightsbasedonperformancemeasurementsmadebefore,during,andaftertheroll-out.
4.
1PerformancemetricsTheperformanceexperiencedbyclientswhodownloadwebcontentcanbecharacterizedinmanydifferentbutcom-plimentaryways.
Weusethefollowingfourmetricsmea-suredfromreal-worldclientsdownloadingcontentfromAka-maitoevaluatetheperformance.
Eachmetricshedslightonadifferentfacetofmappingandclient-perceivedperfor-mance.
Notethatweexpectallthesemetricstodecrease(smallerisbetter)whenend-usermappingisrolledout.
1)Mappingdistanceisthegreatcircledistancebetweenaclientandtheservertowhichitwasassignedbythemap-pingsystem.
Thisisapurelygeographicalmetricwithnonetwork-relatedcomponent.
2)Roundtriptime(RTT)betweentheclientandtheservertowhichitwasassigned.
ThisissimplytheTCPRTTmeasuredfromtheserver'sTCPstack.
Thisispurelyanetwork-relatedmetric.
3)Timetorstbyte(TTFB)isthedurationfromwhentheclientmakesaHTTPrequestforthebasewebpagetowhentherstbyteoftherequestedwebpagewasreceivedbytheclient.
Thisquantityismeasuredfromtheclient'sbrowserandincludesthreecomponents:(i)thetimefortheclient'srequesttoreachtheserver,(ii)timefortheservertoconstructthewebpage,and(iii)timefortherstchunkofthewebpagetoreachtheclient.
Notethatend-usermappingisexpectedtodecreaseboththerstandthirdcomponentofTTFBabovebyreducingtheserver-clientRTT.
However,sincemanybasewebpagesare"dynamic"andneedtobepersonalizedfortheclient,thesecondcomponentofcon-structingthewebpagemayinvolvefetchingpersonalizedel-ementsfromtheorigin.
Overlaytransportisusedtospeeduporigin-servercommunication[26],thoughsuchtransportisnotimpactedbytheend-usermappingroll-out.
Thus,weexpectTTFBtoshowmoremodestreductionsasend-usermappingimpactsonlysomeofitstimecomponents.
4)Contentdownloadtimeisthedurationfromthere-ceivingoftherstbyteofthepagetocompletingthedown-loadoftherestofthewebpage,includingthecontentem-beddedinthepage.
Thismetricisalsomeasuredfromtheclient'sbrowser.
TheembeddedcontentofwebpagesaretypicallymorestaticandcacheableandincludesCSS,im-ages,andJavaScriptthatarenotpersonalizedtotheclient.
Thus,unlikeTTFB,weexpectthismetrictobesignicantly8WeareunawareofanyotherAkamaisoftwarereleasesorInterneteventshappeningduringtheroll-outperiodthatcouldconfoundourmeasurementsandconclusions.
impactedbytheend-usermappingroll-outasthismetricisdominatedbyclient-serverlatencies.
4.
1.
1HighandlowexpectationcountriesTobetterunderstandtheperformanceimpact,weclassifythecountriesintotwogroups:a"highexpectation"groupwhereweexpectend-usermappingtohaveagreaterimpactanda"lowexpectation"groupwhereweexpecttheimpacttobelower.
Ourclient-LDNSanalysisinSection3.
2givesusanideaofwhatbenetstoexpectinwhichcountries.
Specically,Figure8showstheproximityofclientstotheirLDNSformajorcountries.
Usingthisanalysis,wesplitthemajorcountriesintotwohalves.
Wedenethehighexpecta-tiongrouptobethoseclientswhoresideincountrieswherethemediandistancetoapublicresolverismorethan1000milesandthelowexpectationgrouptobethosewhoseme-diandistanceisunder1000miles.
Weaggregateandpresenttheperformancemetricsseparatelyforthesetwogroups,asweexpectthemtoshowdifferentbehaviors.
4.
2CollectingperformanceinformationWecollectedperformancemetricsfromalargeandchar-acteristicsetofclientsaroundtheworldbefore,during,andaftertheend-usermappingroll-out.
WeusedAkamai'sRealUserMeasurement(RUM)system[4]forourclient-sideper-formancemeasurements.
RUMinsertsJavaScriptintose-lectwebpagesdeliveredbyAkamai.
ThatJavaScriptrunsinsidetheclient'sbrowserwhenthepageisdownloadedbytheclient.
Theperformancemeasurementismadeus-ingtheindustry-standardnavigationtiming[6]andresourcetimingAPIs[8].
UsingtheseAPIs,theJavaScriptrunninginsidetheclient'sbrowsercollectsprecisetiminginforma-tionwhenthepagedownloadisinprogress,includingwhentheDNSlookupstartedandcompleted,whentheTCPcon-nectionwasinitiated,whenthefetchrequestwassentout,whentherstbyteoftheresponsewasreceived,andwhenallthepagecontentwasfullydownloaded.
Usingthesetim-ingmilestones,metricssuchasTTFBandcontentdownloadtimecanbecomputed.
Thetimingmeasurementsperformedinclientbrowsersaroundtheworldwassenttoabackendcloudstoragesystemandwassubsequentlyanalyzedtopro-ducetheaggregatestatisticsweprovideinthissection.
WecollectedRUMmeasurementsfromawideselectionofWebsitesandclientsaroundtheworldfromJan1,2014toJune30th,2014,aperiodthatincludestheend-usermappingrolloutfromMarch28thtoApril15th.
Sincetheroll-outonlyimpactsclientswhousepublicresolvers,weidentiedsuchclientsusingourclient-LDNSpairingdatadescribedinSection3.
1andextractedRUMdatafromonlythosequal-iedclients.
Figure12showsthetotalnumberofqualiedRUMmeasurementscollectedandusedinouranalysisfrombothhighandlowexpectationcountries.
Ourdatasethas33millionto58millionmeasurementspermonth,eachmonthfromJantoJune2014,foratotalof273millionmeasure-ments.
ThemeasurementvolumeshowsanincreasingtrendonaccountofmoredownloadsfromqualiedclientsofthepagesmeasuredbyRUM.
Ourgoalistomeasureperformanceforalargeandchar-acteristiccrosssectionofclients,Websites,devices,andconnectivitiesacrosstheglobalInternet.
Toachievethatwemeasured6,388domainnamesand2.
5millionuniqueURLsaccessedby149,826uniqueclients.
OurdatasetincludesallmajorclientplatformssuchasWindows,FreeBSD,Linux,Android,iOS,andgameconsoles,andallmajorbrowsersin-cludingFirefox,Opera,Chrome,andIE.
Further,ourclientsuseavarietyofwaystoaccesstheInternetincludingcellu-lar,WiFi,3G,4G,DSL,cablemodem,andber.
0102030DecJanFebMarAprMayJunMeasurementspermonth(millionmeasurements)expectationhighlowFigure12:NumberofRUMmeasurementspermonth.
4.
3PerformanceAnalysisWeanalyzethemappingdistance,RTT,TTFB,andcon-tentdownloadtimeforclientswhousepublicresolversbe-fore,during,andaftertheroll-out.
1)Mappingdistance.
Mappingdistanceshowsasignif-icantimprovementduringtheroll-outperiodofMarch28thtoApril15th.
Figure13showsforthehighexpectationgroup,themeanmappingdistancedroppedfromover2000milesonaveragetoaround250miles.
Eventhelowexpec-tationcountriesexperiencedshortermappingdistance:theaveragemappingdistancewentfrom400milesto200miles.
Figure14showstheCDFofthemappingdistancesforbothhighandlowexpectationcountriesbothbeforeandaf-tertheroll-outiscompleted.
Theperiodaftertheroll-outisApril15thorlaterandtheperiodbeforetheroll-outisMarch28thorearlier.
Notethatallpercentilesseeimprove-ment.
But,thereisadrasticdecreaseinthemappingdis-tancearoundthe90thpercentileforhighexpectationcoun-triesfrom4573milesto936miles.
ThedecreaseisduetoimprovedmappingdistanceforclientsinlargecountrieslikeIndiaandBrazilwhousepublicresolverslocatedinSouth-eastAsiaandNorthAmericarespectively(cf.
Figure8).
2)RTT.
RecallthatRTTmeasuresthelatencybetweentheclientandtheserverassignedtothatclient.
Unlikemappingdistance,RTTreectsthestateofthenetworkpathsuchaspropagationdelay,andcongestion.
AsshowninFigure15,theaverageRTTforthehighexpectationgroupdroppedfrom200msto100ms,asignicant50%decrease.
But,theimprovementforthelowexpectationgroupwasmod-est.
Figure16showstheCDFoftheRTTforbothhighandlowexpectationcountriesbeforeandaftertheroll-out.
All010002000JanFebMarAprMayJunJulMappingdistance(miles)expectationhighlowFigure13:Dailymeanofmappingdistance.
0%25%50%75%100%02000400060008000Mappingdistance(miles)CumulativepercentofRUMmeasurementslowexpectationafterrolloutlowexpectationbeforerollouthighexpectationafterrollouthighexpectationbeforerolloutFigure14:CDFsofmappingdistance.
percentilesshowimprovement.
Forinstance,the75thper-centileoftheRTTdecreasessignicantlyfrom220msto137msforthehighexpectationcountries.
3)Time-to-First-Byte.
Asnotedearlier,TTFBincludesaspectsthatarenotimpactedbybettermappingdecisions,suchasthecomputationtimetogenerateandtransmitady-namicwebpage.
Consequently,thegainsexpressedasapercentagearelowerbutstillsignicant.
Figure17showsthatthemeanTTFBofthehighexpectationcountriesde-creasedfromaround1000msto700ms,a30%improve-ment.
Figure18showstheCDFoftheTTFBforbothhighandlowexpectationcountriesbeforeandaftertheroll-out.
Allpercentilesshowimprovement.
Forinstance,the75thpercentileoftheTTFBdecreasesfrom1399msto1072msforthehighexpectationcountriesandfrom830msto667msforthelowexpectationones.
4)ContentDownloadTime.
Figure19showsareductionfrom300msto150msforthehighexpectationcountries,a50%reduction.
Recallthatcontentdownloadtimeisdomi-natedbyserver-clientlatenciesandthedecreaseismorecor-relatedwithcorrespondingdecreaseinRTT.
Theimprove-mentforthelowexpectationgroupissmallasthedownloadtimeisalreadysmall.
Figure20showstheCDFofthecon-tentdownloadtimeforhighandlowexpectationcountriesbeforeandaftertheroll-out.
Allpercentilesshowimprove-ment,e.
g.
,the75thpercentileofthedownloadtimereduces0100200300JanFebMarAprMayJunJulRTT(ms)expectationhighlowFigure15:DailyMeanofRoundTripTime(RTT).
0%25%50%75%100%0200400600RTT(ms)CumulativepercentofRUMmeasurementslowexpectationafterrolloutlowexpectationbeforerollouthighexpectationafterrollouthighexpectationbeforerolloutFigure16:CDFsofRoundTripTime(RTT).
from272msto157msforthehighexpectationgroupandfrom192msto102msforthelowexpectationone.
4.
4WhyDownloadPerformanceMattersFromourresultsabove,wecanconcludethatend-usermappingprovidessignicantperformancebenetstoclientswhousepublicresolvers,especiallyinthosecountrieswhereclient-LDNSdistancesarehigh.
Fasterdownloadtimessuchasthoseprovidedbyend-usermappingarekeytoabetterInternetexperience,resultinginwebpagesthatloadmorequicklyandvideosthatstartplayingsooner.
Betterdown-loadperformanceenhancestheclient'sexperienceofacon-tentprovider'sWebsite,moresatisedclientsinturnfavor-ablyimpactthebusinessofthecontentprovider,allowingthecontentprovidertoinvestinevengreaterperformanceenhancements,forminga"virtuouscycle"[25].
Asanex-ample,anoft-citedrecentstudybyWalmartlabs[12]con-cludedthatthedownloadtimeofWebpagesinWalmart'se-commercesiteimpactsthebuyingbehaviorofitsusers.
BycorrelatingRUMperformancemeasurementscollectedforWalmart.
comwithback-endbusinessmetrics,thestudyconcludedthata100msdecreaseinwebpagedownloadtimecanresultina1%increaseinrevenueanda1sec-onddecreasecanresultinuptoa2%increaseinconversion50075010001250JanFebMarAprMayJunJulTimetofirstbyte(ms)expectationhighlowFigure17:DailyMeanofTimetoFirstByte(TTFB).
0%25%50%75%100%0100020003000Timetofirstbyte(ms)CumulativepercentofRUMmeasurementslowexpectationafterrolloutlowexpectationbeforerollouthighexpectationafterrollouthighexpectationbeforerolloutFigure18:CDFsofTimetoFirstByte(TTFB).
rates9.
Numerousotherstudiesshowhowevenafew100msincreaseinpagedownloadtimesofaWebsitecande-creaserevenues,pageviews,searchesperuser,etc[9].
Infact,itiswidelyheldinindustrythataWebsitethatisfasterthanitscompetingsitesbyaslittleas250mshasasignif-icantbusinessadvantagetobereckonedwith[18].
Inad-dition,searchenginesrankfasterWebsitesaheadofsloweronesandclientsoftenassociategreaterbrandreputationwithfasterWebsites.
Thus,the"needforspeed"isasingularfo-cusforcontentprovidersandtheCDNsalikeand"shavingoff"eventensofmillisecondsofWebdownloadtimesforacross-sectionofclientsisdeemedworthyandimportant.
Besidesfasterdownloadtimes,thedecreaseinmappingdis-tanceandRTTduetoend-usermappingoftenmeansthattheclient-serverpathcrossesfewerASboundaries,peeringpointsandtransnationalcablelinks,hencereducingthelike-lihoodofcongestionandfailure.
Thus,end-usermappingmayresultinmorestableandreliableclient-serverpaths.
4.
5TheBenetsofEDNS0AdoptionTodeployend-usermappingbeyondthecurrentsetofclients,theclient'sISPneedstoadopttheEDNS0exten-9Conversionrateisakeymetricfore-commercesitesandisthepercentageofvisitorstothesitewhobuyaproduct.
0100200300400JanFebMarAprMayJunJulContentdownloadtime(ms)expectationhighlowFigure19:Dailymeanofcontentdownloadtime.
0%25%50%75%100%02505007501000Contentdownloadtime(ms)CumulativepercentofRUMmeasurementslowexpectationafterrolloutlowexpectationbeforerollouthighexpectationafterrollouthighexpectationbeforerolloutFigure20:CDFsofcontentdownloadtime.
sionfortheirDNSservices.
Ourresultsshedlightontheperformancebenetssuchadoptionwouldyieldandpro-videsastrongimpetusforitsadoption.
Forinstance,ex-cludingthepublicresolvers,weknowthat6.
2%ofthere-mainingclientdemandoriginatesfromclientswhoseLDNSareatleast1000milesaway.
Extrapolatingfromourresultsforsimilarclientsusingpublicresolvers,wecouldexpectasimilar50%reductioninRTTandcontentdownloadtimefortheseclients.
Likewise,excludingthepublicresolvers,clientswithLDNSesbetween500to1000milesaccountfor5.
3%ofremainingclientdemand.
Extrapolatingfromsim-ilarclientswhousepublicresolvers,wecanspeculatethattheseclientswillseea24%decreaseinRTTsandcontentdownloadtimes.
Ofcourse,54%oftheremainingclientdemandwillseenobenetatallfromend-usermapping,sincetheyhavelocalLDNSes.
However,thefactthatatleast11.
5%oftheremainingclientdemandwillseeasignicantenoughperformanceimprovementissufcientimpetustoEDNS0adoption.
5.
SCALINGCHALLENGESEnd-usermappingischallengingsinceitmakesmappingdecisionsatpotentiallyamuchnergranularitythantradi-tionalNS-basedmapping.
ThereareordersofmagnitudemoreclientsthantherearenameserversontheglobalIn-ternet.
Anend-usermappingsystemmustperformmorene-grainnetworkmeasurementsandprovideresolutionsatanerscaleacrosstheglobalInternetthanaNS-basedmap-ping,leadingtoscalingconsiderationsdiscussedbelow.
5.
1TradeoffsinchoosingthemappingunitsAmappingunitisthenest-grainsetofclientIPsforwhichserverassignmentdecisionsaremadebythemap-pingsystem.
AtraditionalNS-basedmappingsystemusesaLDNSasthemappingunit,i.
e.
,allclientsintheclientclus-terthatuseaLDNSaremappedtogetherasaunit.
Anend-usermappingsystemcoulduse/xclientIPblocksthatparti-tiontheclientIPspace,wherex24.
Anaturalrstchoiceis/24clientIPblockssinceLDNSesthatsupporttheEDNS0extensioncurrentlyuse/24IPblocksintheirqueries.
TounderstandthescalingissuesinswitchingfromNS-basedtoend-usermapping,letusrstexaminethenumberofrelevant/24clientIPblocksontheInternetincompari-sontothenumberofrelevantLDNSes.
WeuseourNetSes-siondatatorstcomputethedemandgeneratedbyclientsineach/24clientIPblock.
WealsocomputedthedemandgeneratedbyeachLDNS,wheretheLDNSdemandissim-plythedemandgeneratedbyclientswhousethatLDNS.
Wethensortedallthe/24clientIPblocks(resp.
,LDNSes)indecreasingorderofdemandandplottedaCDFofthede-mandinFigure21.
Inthedataset,thetotalnumberof/24clientIPblockswithnon-zerodemandis3.
76million,while584thousandLDNSeshavenon-zerodemand.
Supposethemappingsystemisrequiredtomeasureandprovidemappingdecisionsfor95%ofthetotalclientdemandontheInternet.
Asthegureshows,anNS-basedmappingsystemneedonlymeasureandanalyzethetop25,000LDNSeswiththemostdemand,whereasanend-usermappingsystemmustmeasureandanalyzethetop2.
2million/24clientIPblocks,whichisseveralmagnitudeshigher.
Likewise,tocover50%ofthetotalclientdemand,thetop1800LDNSeswiththemostde-mandsufce,whereasnearly430,000ofthe/24clientIPblockswiththemostdemandareneeded.
0%25%50%75%100%1e+011e+021e+031e+041e+051e+06NumberofclientIPblocksorLDNSCumulativepercentoftotaldemandClientIPblocksLDNSFigure21:Numberof/24clientIPblocksorLDNSesthatproduceagivenpercentoftotalglobaldemand.
Oneheuristicapproachtoreducingthenumberofmap-0%25%50%75%100%10100100010000Clusterradius(miles)Cumulativepercentoftotalclientdemandprefix/24/22/20/18/16/14/12/10/8(a)Histogramoftheclusterradiusfor/xclientIPblocks.
0100000200000300000400000/8/10/12/14/16/18/20/22/24PrefixlengthNumberofclusters(b)Numberof/xclientIPblockswithnon-zerodemand.
Figure22:Asmallervalueofxyieldsfewermappingunitsbutlargerclusterradiuswithlessmappingaccuracy.
pingunitsforend-usermappingistousetheIPblocks(i.
e.
,CIDRs)inBGPfeedsthataretheunitsforroutingintheIn-ternet.
Inparticular,ifasetof/24IPblocksbelongwithinthesameBGPCIDR,theseblockscanbecombinedsincetheyarelikelyproximalinthenetworksense.
Weextracted517KuniqueCIDRswithnon-zerotrafcfromBGPfeedsacrosstheInternetfromthenetworkmeasurementcompo-nentofthemappingsystem.
Bycombining/24IPblockswhenevertheybelongtothesameBGPCIDR,wereducethenumberofmappingunitsfrom3.
76millionto444K.
Notethatthesametechniquemaybeappliedtoreducethenumberofmappingunitsfor/xIPblocks,foranyvalueofx.
AfterapplyingtheBGPCIDRstoreducethenumberofmappingunits,thereisstillthetradeoffofwhat/xclientIPblockstochooseasthemappingunit.
Onecouldreducethenumberofmappingunitsbyusingcoarser/xclientIPblocks,i.
e.
,bychoosingasmallervalueofx.
However,whencoarserIPblocksareused,thesetofclientsinagivenblockislargerandspanalargergeographicalarea.
Thisre-ducesmappingaccuracyastheclientclustersthataretheunitsofmappinghavealargerradius.
Figure22providestheexacttradeoffbetweentheclusterradiuswhichisaproxyformappingaccuracyandthenumberofclustersthatneedtobemeasuredandanalyzed.
Itcanbeseenthat/20clientIPblocksareaworthyoptionastheyreducethenumberofmappingunitsbyafactorof3incomparisonto/24blocks.
However,theclustersarestillrelativelysmallwith87.
3%oftheclustershavingaradiusofnomorethan100miles.
5.
2DealingwithgreaterDNSqueryratesInNS-basedmapping,eachLDNSstoresoneresolutionperdomainname.
However,withend-usermapping,differ-entclientIPblockswithinthesameclientclustermaygetdifferentresolutionsforthesamedomainname.
Thus,anLDNSthatservesmultipleclientIPblocksmaystoremulti-pleentriesforthesamedomainname.
Therefore,anLDNSmaymakemultiplerequeststoanauthoritativenameserverforthedomainname,oneforeachclientIPblock.
ThiscanleadtoasharpincreaseintheLDNSqueriesseenbytheauthoritativenameserversofthemappingsystem.
Fig-ure23showsthetotalDNSqueriespersecondservedbythemappingsystembefore,during,andafterenablingend-usermappingforclientswhousepublicresolvers.
Priortotheroll-out,thetotalqueriespersecondservedbyAkamai'snameserverswas870Kqueriespersecondofwhichpub-licresolverstargetedbytheroll-outaccountedforroughly33.
5Kqueriespersecond.
Butaftertherollout,thetotalqueriespersecondontheAkamainetworkwas1.
17millionqueriespersecondofwhichpublicresolversaccountedfor270Kqueriespersecond.
Thus,thequeriesfrompublicre-solversincreasedbyafactorof270K/33.
5K=8,anincreaselargelyattributabletotheroll-out10.
Thegradualincreaseinqueryrateseenoutsideoftheroll-outwindowissimplyduetothenormalincreaseinInternettrafcovertime.
6000008000001000000120000014000001600000JanFebMarAprMayJunJulQueriespersecondFigure23:DNSqueriesreceivedbyAkamai'snameserversfromLDNSesshowedasignicantincreaseduringtheend-usermappingrollout.
10DNSqueriesincreasewhenpublicresolversturnontheEDNS0extension.
But,theperformanceimprovementsinSection4.
3occurwhenAkamaigraduallyturnedonend-usermappingforthesepublicresolvers.
110100100000.
10.
20.
30.
40.
50.
60.
70.
80.
91PopularityofDomainnameandLDNSpairs(inqueriesperTTL)FactorincreaseinqueryrateFigure24:MorepopulardomainnameandLDNSpairsshowagreaterincreaseinqueryrateaftertheroll-out.
ThepopularityofadomainnameamongtheclientsofanLDNSinuencesthefactorincreaseinDNSqueriesforthatdomainnamewhenEDNS0andend-usermappingareturnedon.
Priortotheend-usermappingroll-out,thequeryrateforadomainnamefromaparticularLDNSisatmostonequeryperTTL,sincetheLDNScancachethetransla-tionforthetimeoftheTTL.
WebucketeachdomainnameandLDNSpairaccordingtothenumberofqueriesreceivedperTTLpriortotheroll-out.
Figure24showsthefactorin-creaseinqueryratefordomainnameandLDNSpairsthatfallintoeachbucket.
NotethatthemorepopulardomainnameandLDNSpairsthathavepre-roll-outqueryratescloseto1queryperTTLsawthelargestincreaseinqueryratewhenend-usermappingwasrolledout,whilelesspopu-lardomainssawlittleornoincrease.
ThereasonisthatamorepopulardomainnameismorelikelytobeaccessedbyclientsinmultipleclientIPblocksoftheLDNS'sclientcluster,eachIPblockrequiringaseparatedomainnameres-olutionwhenEDNS0isused.
Fortunately,thedomainnameandLDNSpairsinthehighestpopularitybucketinFigure24accountedforonly11%oftotalpre-roll-outqueries.
6.
ROLEOFSERVERDEPLOYMENTSServerdeploymentsplayanimportantroleindeterminingclientperformance.
Moreserverdeploymentlocationsmeanbetterperformanceforclients,sincethemappingsystemhasmoreoptionstochooseaproximalserverforeachclient.
But,whatroledodeploymentsplayindeterminingthead-ditionalperformancebenetsprovidedbyend-usermappingoverNS-basedmappingShouldaCDNwithasmallnum-berofdeploymentlocationsadoptend-usermappingForaCDNwithagivensetofdeploymentlocations,whatismorebenecial:addingmoredeploymentlocationsorincorporat-ingend-usermappingHowmuchcanNS-basedmappingbeimprovedbymakingitclient-awareToprovideintuitiononthesekeywhat-ifquestions,con-siderasimpliedmodel.
LetaCDNhaveNdeploymentlocations.
ThedeploymentspartitiontheIPaddressspaceoftheglobalInternetintosetsEi,1iN,suchthatEiisthesetofIPsforwhomtheithdeploymentlocationisthemostproximalamongalldeployments.
Observethatforanyclientc,ifcanditsLDNSarebothinsomesetEi,bothend-usermappingandtraditionalNS-basedmappingwillpickaserverintheithdeploymentlocationforclientc,i.
e.
,thereisnoadditionalbenetforclientcfromusingend-usermapping.
Thus,ifaCDNhasfewerdeployments,eachsetEiislikelylargerandishencemorelikelytocon-tainboththeclientanditsLDNS.
Thus,wewouldexpectaaCDNwithfewerdeploymentstobenetlessfromend-usermappingthanaCDNwithmoredeployments.
Wequantifyanswerstothisandotherkeyquestionsusingsimulations.
SimulationMethodology.
WecreateauniverseUofpossibledeploymentlocationsbyusing2642differentlo-cationsaroundtheglobewithAkamaiservers.
Thesede-ploymentsarespreadover100countriesandwerechosentoprovidegoodcoverageoftheglobalInternet.
Next,wechoosearound20K/24IPblocksthataccountformostoftheloadontheInternetandfurtherclustertheminto8K"pingtargets",soastocoverallmajorgeographicalareasandnet-worksaroundtheworld.
Wethenperformlatencymeasure-mentsusingpingsfromeachdeploymentUtoeachofthe8Kpingtargets.
ForanyclientorLDNS,wendtheclos-estofthe8Kpingtargetsandusethatasaproxyforla-tencymeasurements,i.
e.
,thelatencymeasurementstothepingtargetareassumedtobethelatencymeasurementstotheclientorLDNS.
Usingthepinglatencymeasurementsdescribedabove,wesimulatethreemappingschemes,eachwithavaryingnumberofdeploymentlocations.
(1)NS-basedmapping(NS):MapclienttothedeploymentlocationthathastheleastlatencytotheLDNSofthatclient.
(2)End-usermapping(EU):Mapclienttothedeploymentlocationthathastheleastlatencytotheclient's/24IPblock.
(3)Client-AwareNS-basedMapping(CANS):Foreachclient,ndtheclusterofclientsthatsharesitsLDNS.
Mapclienttothedeploymentlocationthatminimizesthetrafc-weightedaverageofthelatenciesfromthedeploymenttoitsclusterofclients.
NotethatCANSmappingisanenhancementofpureNSmappingbyusingthelatencymeasurementstotheclientsoftheLDNS,ratherthanjustthelatencymeasurementtotheLDNS.
InsituationswhereLDNSisfarawayfromitsclients,butitsclientsarethemselvesrelativelyclosetogether,CANSmappingcouldprovidelowlatencymappings.
CANsrequirestrackingclient-LDNSassociationsonanongoingbasisontheglobalInternet,anadditionalcomplexityincom-parisonwithNSmapping.
However,CANScanbeviewedasahybridbetweenNSandEUthatusesclientmeasure-mentsbutrequiresnospecicknowledgeabouttheclient'sIP,i.
e.
,itdoesnotrequiretheEDNS0protocolextension.
Wesimulatedthethreemappingschemesaboveforavary-ingnumberofdeploymentlocationsNchosenfromtheuni-verseU.
Weperformed100randomrunsofoursimulation,wherewedothefollowingineachrun.
Werandomlyor-derthedeploymentsinU.
Then,foreachN,wesimulateallthreemappingschemesassumingtherstNdeploymentsintherandomordering.
Thesimulationcomputesthetrafc-weightedmean,95th,and99thpercentilelatenciesachievedbythethreeschemes.
Finally,foreachvalueofN,weav-eragedthemetricsobtainedacrossthe100simulationrunsandthosevaluesarereportedinFigure25.
050100150200250408016032064012802560NumberofdeploymentlocationsPinglatency(ms)CANSmeanCANS95pctCANS99pctEUmeanEU95pctEU99pctNSmeanNS95pctNS99pctFigure25:LatenciesachievedbyEU,CANS,andNSmap-pingasafunctionofCDNdeploymentlocations.
AnimportantcaveatininterpretingFigure25isthatthepinglatenciesshownareanunderestimateoftheactualla-tencyorRTTfromtheservertotheclient,sinceonlyapingtarget(typicallyarouter)enroutetotheclientis"pinged".
So,whiletheabsolutevaluesofthepinglatenciesarelessmeaningfulexceptasalowerboundontheactuallatencies,therelativevaluesarestillmeaningful.
Asshownintheg-ure,allmappingschemesprovidesmallerpinglatencieswithalargerdeployment.
Further,meanpinglatencyisnearlyidenticalforallthreemappingschemes,reectingthefactthatinmanycasesaclientanditsLDNSareproximaltoeachotherandLDNSisagoodproxyfortheclient.
Evenso,EUperformedthebestofthethreewithmeanpingla-tencydroppingfrom35msforasmalldeploymenttounder10msasthedeploymentsincrease.
However,meanlatencyacrossallclientsontheglobeislessinterestingthanlatencyoftheworst-performingclients.
Infact,bothCDNsandcontentprovidersarefocusedonimprovingtheperformanceoftheworst-performingclient.
Thus,wecomputedthe95thand99thpercentilesofthela-tencies,i.
e,latenciesfor1-5%oftheworstclients.
ItisclearthatEUprovidesalargebenetovertheotherschemesforhigherpercentilesofpinglatency.
Inparticular,NS-basedmappingprovidesdiminishingbenetsbeyond160deploymentlocationsforthe99thpercentilelatency,andisinparticularunabletoreduceitbelow186msevenwith1280deploymentlocations.
ThereasonisthatNS-basedmappingdoesnotworkwellforclientswhoseLDNSesarenotproximalwhoarelikelyamongtheworst-performingclients.
However,EUcontinuestoreducethelatencieswithincreasingdeployments,evenbeyond1280deployments.
ItcanalsobeseeninthegurethataCDNwithlargerde-ploymentlocationsseesaproportionallylargerreductioninhigherpercentilesofpinglatencybyswitchingtoEUfromNSthanaCDNwithsmallerdeployments.
CANmappingprovidesanintermediatepointbetweentheextremesofNSandEUM.
Inparticular,theknowledgeoflatenciestoclientsbehindagivenLDNSprovidessufcientknowledgetoim-proveNS-basedmappingforhigherpercentilesforlatency.
7.
RELATEDWORKWhiletheEDSN0extensionprovidesasystematicmech-anismforend-usermappingimplementation,othermecha-nismshavebeenexploredinlimitedwaysinindustry.
AvideoCDNatAkamaiincirca2000usedmetaleredirec-tiontoimplementend-usermapping.
Whenaclientstartsavideo,themediaplayerfetchesametalethatcontainstheserver'sIPfromwhichtodownloadthevideo.
TheserverIPembeddedinthemetaleisdynamicallygeneratedbythemappingsystemusingtheclient'sIPderivedfromthemetaledownload.
However,suchamechanismishardtoextendtotheWebandothertrafcthatdonotusemetales.
Analogoustometaleredirection,systemsthatusehttpredirectionhavealsobeenbuiltwheretheclientisrstas-signedaserverusingNS-basedmapping.
Therstserverusesitsknowledgeoftheclient'sIPtoredirecttheclienttoa"better"secondserverifappropriate.
Thesecondserverthenservesthecontenttotheclient.
However,thisprocessincursaredirectionpenaltythatisacceptableonlyforlargerdownloadssuchasmedialesandsoftwaredownloads.
Toolsfordiscoveringclient-LDNSpairingshavealsoex-istedinindustryforthepast15years.
Inprinciple,suchpairingscanbeusedtocreateaclient-awareNS-basedmap-pingsystem(cf.
Section6),thoughitwillnotbeeffectiveforLDNSeswithlargeclientclusters(cf.
Section3.
3).
WethinkthattheEDNS0extensioniskeytobuildinglarge-scaleend-usermappingthatovercomestheshortcom-ingsofpriorimplementations.
TheEDNS0extensionre-movestheoverheadofexplicitclient-LDNSdiscovery,avoidsaredirectionperformancepenalty,andiseffectiveevenforLDNSeswithlargegeo-distributedclientclusters.
Fromaresearchperspective,client-LDNSdistancesandtheirpotentialimpactonserverselectionhasbeenstudiedin[24],andsubsequentlyin[20,17].
Thepriorliteratureob-servedlargerclient-LDNSdistancesandpoorerperformanceforclientsusingpublicresolversthatareincreasinglyinuse[22].
Ourmeasurementstudyofclient-LDNSdistancesinSection3isbasedonamuchwiderglobalcross-sectionofclientsandLDNSesthanpriorworkandlargelyconrmpriorconclusionsonpublicresolvers.
However,wegoastepfurtherbydescribinganend-usermappingsystemtorem-edytheissue.
TheEDNS0extensionhasalsobeenstudiedastoolforguringoutdeploymentsofCDNproviderswhosupporttheextensionsuchasGoogle[10,27].
ExtensionsotherthanEDSN0forovercomingclient-LDNSmismatchhavealsobeenproposed[16].
8.
CONCLUSIONInthispaper,wedescribedourexperienceinrolling-outanewmappingsystemcalledend-usermapping.
Byanalyz-ingclientsandLDNSesfromaroundtheworld,weshowedthatasignicantfractionofclientshaveLDNSesthatarenotintheirproximityandcouldbenetfromend-usermapping.
Weconrmedtheperformancebenetsbymeasuringmap-pingdistance,RTT,Time-To-First-Byte(TTFB),andcon-tentdownloadtimeduringtheroll-out.
Weshowedthatfor"high-expectation"countries,clientsusingpublicresolverssawaneight-folddecreaseinmeanmappingdistance,atwo-folddecreaseinRTTandcontentdownloadtime,anda30%improvementintheTTFB.
Wealsoquantiedthescalingchallengesinimplementingend-usermappingsuchasthe8-foldincreaseinDNSqueriesandthegreaternumberofmap-pingunitsthatneedtobemeasuredandanalyzed.
Finally,weshedlightontheroleofdeploymentsandshowedthataCDNwithalargernumberofdeploymentlocationsislikelytobenetmorefromend-usermappingthanaCDNwithasmallernumber.
Whileweonlydescribetheroll-outofend-usermappingtoclientswhoareusingpublicresolvers,ouranalysisshowsthatabroadroll-outofthistechnologyacrosstheentireInternetpopulationwillbequitebenecial.
Forsucharoll-outtooccur,moreISPswouldneedtosupporttheEDNS0extension.
Weexpectourworkthatquantiesthereal-worldbenetsofend-usermappingtoprovideim-petustoabroaderadoptionoftheEDNS0extension.
9.
ACKNOWLEDGEMENTSFirstandforemost,wethankthemanyengineersatAka-maiwhodesigned,implementedandrolled-outend-usermap-ping,makingitpossibleforustoevaluateitsimpact.
SpecialthankstoMikeConlenwhohelpedcollectDNSquerydata,toPabloAlvarezwhomadekeycontributionstoend-usermappingscoring,andtoJasonMoreauwhomademajorcon-tributionstonameserverdesign.
Wethankouranonymousrefereesforcopiousreviewsthathelpedimprovethepaper.
AspecialthankstoourshepherdEthanKatz-Bassettwhoprovidedlotsofgreatfeedbackthatstrengthenedthepaper.
10.
REFERENCES[1]AkamaiEdgescape.
http://goo.
gl/P68U6q.
[2]AkamaiFacts&Figures.
http://goo.
gl/Megx1b.
[3]AkamaiNetSessionInterface.
http://goo.
gl/FOtjlz.
[4]AkamaiRealUserMonitoring.
http://goo.
gl/8oiQyC.
[5]GooglePublicDNS.
https://goo.
gl/p8cfJm.
[6]NavigationTiming.
http://goo.
gl/ePcQrG.
[7]OpenDNS.
https://www.
opendns.
com/.
[8]ResourceTiming.
http://goo.
gl/5eYQtL.
[9]Velocityandthebottomline.
http://goo.
gl/KTlcYR.
[10]M.
Calder,X.
Fan,Z.
Hu,E.
Katz-Bassett,J.
Heidemann,andR.
Govindan.
MappingtheexpansionofGoogle'sservinginfrastructure.
InProceedingsoftheACMInternetMeasurementConference,pages313–326,2013.
[11]C.
Contavalli,W.
vanderGaast,D.
Lawrence,andW.
Kumari.
ClientsubnetinDNSrequests.
IETFInternetDraft,Nov.
2014.
[12]C.
Crocker,A.
Kulick,andB.
Ram.
Realusermonitoringatwalmart.
com:Astoryinthreeparts.
InSanFranciscoandSiliconValleyWebPerformanceGroup,Feb2012.
http://minus.
com/msM8y8nyh.
[13]J.
Dilley,B.
M.
Maggs,J.
Parikh,H.
Prokop,R.
K.
Sitaraman,andW.
E.
Weihl.
Globallydistributedcontentdelivery.
IEEEInternetComputing,6(5):50–58,2002.
[14]X.
Fan,J.
Heidemann,andR.
Govindan.
Evaluatinganycastinthedomainnamesystem.
InProceedingsoftheIEEEINFOCOM,pages1681–1689,2013.
[15]T.
Hardie.
Distributingauthoritativenameserversviasharedunicastaddresses.
RFC3258,Apr.
2002.
[16]C.
Huang,I.
Batanov,andJ.
Li.
Apracticalsolutiontotheclient-LDNSmismatchproblem.
SIGCOMMComput.
Commun.
Rev.
,42(2):35–41,Mar.
2012.
[17]C.
Huang,D.
A.
Maltz,J.
Li,andA.
Greenberg.
PublicDNSsystemandglobaltrafcmanagement.
InProceedingsoftheIEEEINFOCOM,pages2615–2623,2011.
[18]S.
Lohr.
Forimpatientwebusers,aneyeblinkisjusttoolongtowait.
NewYorkTimes,Feb2012.
http://goo.
gl/y70JgH.
[19]B.
M.
MaggsandR.
K.
Sitaraman.
Algorithmicnuggetsincontentdelivery.
SIGCOMMComput.
Commun.
Rev.
,July2015.
[20]Z.
M.
Mao,C.
D.
Cranor,F.
Douglis,M.
Rabinovich,O.
Spatscheck,andJ.
Wang.
ApreciseandefcientevaluationoftheproximitybetweenWebclientsandtheirlocalDNSservers.
InUSENIXAnnualTechnicalConference,GeneralTrack,pages229–242,2002.
[21]E.
Nygren,R.
Sitaraman,andJ.
Sun.
TheAkamaiNetwork:Aplatformforhigh-performanceInternetapplications.
ACMSIGOPSOperatingSystemsReview,44(3):2–19,2010.
[22]J.
S.
Otto,M.
A.
Sánchez,J.
P.
Rula,andF.
E.
Bustamante.
ContentdeliveryandthenaturalevolutionofDNS:remoteDNStrends,performanceissuesandalternativesolutions.
InProceedingsoftheACMInternetMeasurementConference,pages523–536,2012.
[23]S.
Sarat,V.
Pappas,andA.
Terzis.
OntheuseofanycastinDNS.
InProceedingsoftheIEEEICCCN,pages71–78,2006.
[24]A.
Shaikh,R.
Tewari,andM.
Agrawal.
OntheeffectivenessofDNS-basedserverselection.
InProceedingsoftheIEEEINFOCOM,volume3,pages1801–1810,2001.
[25]R.
K.
Sitaraman.
Networkperformance:DoesitreallymattertousersandbyhowmuchInFifthInternationalConferenceonCommunicationSystemsandNetworks(COMSNETS),pages1–10.
IEEE,2013.
[26]R.
K.
Sitaraman,M.
Kasbekar,W.
Lichtenstein,andM.
Jain.
Overlaynetworks:AnAkamaiperspective.
InAdvancedContentDelivery,Streaming,andCloudServices.
JohnWiley&Sons,2014.
[27]F.
Streibelt,J.
Bttger,N.
Chatzis,G.
Smaragdakis,andA.
Feldmann.
ExploringEDNS-client-subnetadoptersinyourfreetime.
InProceedingsoftheACMInternetMeasurementConference,2013.