equal114dns

TrendsinWideAreaIPTrafficPatternsAViewfromAmesInternetExchangeSeanMcCreary(mccreary@caida.
org)kcclaffy(kc@caida.
org)UniversityofCalifornia,SanDiegoCooperativeAssociationforInternetDataAnalysis(CAIDA)U.
S.
A.
Contents1.
Introduction2.
Background1.
MonitorSite2.
MonitoringMethodologyandTools3.
AnalysisMethodologyandTools3.
Results1.
PacketLengths2.
ProtocolMix1.
RawData2.
LongTermTrends3.
ShorterTermTrends4.
ConclusionsandFutureDirections5.
Acknowledgments6.
ReferencesAbstractWereportresultsfromalongitudinalanalysisoftheIPtrafficworkloadseenatasinglemeasurementsiteinsideamajorInternettrafficexchangepoint.
UsingdatacollectedbytheNLANR/MOATNetworkAnalysisInfrastructure(NAI)project[NAI]andanalysissoftwarefromCAIDA'sCoralReefproject[CoralReef],wepresenttrendsinapplicationusageseenattheNASAAmesInternetExchangeover10months,fromMay1999throughMarch2000.
Weshowchangesinthefractionoftrafficfromstreamingmediaandonlinegaming,aswellasanincreaseintrafficfromnewapplicationssuchasNapsterandIPSECtunneling.
WealsoshowthatourdatadoesnotindicateanyoverallchangeintheTCP/UDPtrafficratioattheAmesInternetExchangeduringthisperiod,orsignificantdifferencesfromtheanalysesbyMCIWorldcomandCAIDAin1998.
IntroductionOvertheyearstherehasbeenagreatdealof'commonwisdom'developedaboutthenatureofwideareaInternettraffic.
Unfortunately,theserulesofthumbhavedevelopedintheabsenceofobservationsaboutthecompositionofactualtrafficcarriedintheInternetbackbone.
ManyanalysesofInternettrafficbehaviorrequireaccurateknowledgeofthetrafficcharacteristicsinthebackbonetoday,forpurposesrangingfromtheoptimizationoffuturenetworkingequipmenttomodelingtheeffectsofnewprotocolsontheexistingtrafficmix.
CAIDAaimstofillthisgap,supplyingaccurateinformationaboutbackbonetrafficcharacteristicstobothindustryandtheacademiccommunity.
Otherstudies[Feldmann98]haveusedInternettraffictracedataasinputforevaluationofspecificprotocolperformanceissues,buthavenotconcentratedoncharacterizingtheoverallworkloadcontainedinthetraces.
Therearealsoseveralongoingprojects[Feldman98,AIXstats]thatmonitortheutilizationofnetworklinksatourmeasurementsiteintermsofthenumberofpacketsandbytestransferredovertime.
However,thesemonitoringprojectsdonotattempttoprovideafinergrainedpictureofthelinkutilizationintermsoftheprotocolsinuseonthelinks.
EarlierworkloadanalysisstudiesintheNSFnetbackbone[Heimlich89,Claffy93,Merit95]includedcourse-grainestimatesofthefractionoftrafficduetothemostpopularapplicationsbeforetheadventofwebbrowsers.
ThisincludedbasicstatisticsabouttheprevalenceofFTP,e-mail,netnews,telnet,DNS,andasinglecategoryfor'other'TCPandUDPapplications.
Severalstudiesdetailingtheworkloadpresentedattheborderoflargeorganizations[Caceres89,Caceres91,Paxson94,WISCstats]haveincludedtrafficbreakdownsbyprotocolaswell.
Thesesite-specificstudiesformacomplementtobackbonestudies,astheycanrevealdifferencesinthewaydifferentkindsoforganizationsusetheInternet.
Previousstudiesofbackbonetraffic[Apisdorf97,Thompson97,claffy98]haveanalyzedtheworkloadpresentedattwositesinsidetheInternetMCIbackbone(AS3561),nowownedbyCableandWireless.
Thesestudiesspannedarelativelyshortperiodoftime,rangingfromasingledaytoafullweek.
Theseshort'snapshots'ofbackbonetrafficarenotlongenoughtoaccuratelydeterminetrendsinthetrafficmix,asweseealargeamountofvariabilityonthesetimescales.
InthispaperwepresentananalysisoftheworkloadseenatAIXoveramuchlongerperiod,spanningmorethan10months.
NLANR/MOAT'sDatacubeproject[MOAT99]isaframeworkforstoringandanalyzingtracescollectedfromasetofCoralmonitorsatHPCsites.
Thethreedimensionsofthe'cube'are:dataorigin(measurementlocationwherethedataoriginated);projectname(e.
g.
,analysismethod);anddatacollectiondate.
MOATmakesthepublicdataavailableonitswebserverviathisDataCubestructure,toallowdataexplorationalongeachdimensionofthecube.
Inthenextsection,wepresentbackgroundinformationaboutourdatacollectionsiteandthetoolsweusedtocollectandanalyzethedata.
Wethenpresentsomesampleresultsfromourstudy,ananalysisofthedistributionofpacketlengths,andsometrendsintheprotocolmixseenatoursamplingsite.
WeconcludewithasummaryofthechallengeswefaceincontinuingandextendingourmeasurementprogramatCAIDA.
BackgroundTheMonitorSiteThetracesusedforthisstudywerecollectedfromtheNASAAmesInternetexchange(AIX)inMountainView,CA[AIX]aspartofanNSF/NASAcollaborativeeffortwithNLANR/MOAT.
Theywerecollectedfromoneoffour(nowfive)OC-3ATMlinksthatinterconnectAIXandMAE-WestinSanJose,CA.
Fig1:Adiagramshowingthelocationoftheopticalsplitterusedtocollectthedata.
NotethattherearecurrentlyfivelinksbetweenNASA-AmesandMAE-West.
ThankstoHans-WernerBraunandNLANR/MOATforuseofthisfigure.
ThisgroupoflinksformastripedconnectionbetweentwoDECGigaswitches,withanaggregatebandwidthofapproximatelyasingleOC-12link.
TheGigaswitchesuseaproprietaryschedulingalgorithmforsendingpacketsacrossthislink,buteachpacketissentacrossanindividuallinkinsideanAAL5PDU.
ThismeanstheschedulinginsidetheGigaswitcheshappensatthepacketlevel,sinceallcellsfromaPDUaresentoverthesamelink.
Consequently,thedatawecollectfromthissiteisessentiallysub-sampledfromtheactualdatatraversingthislinkusingtheproprietaryschedulingalgorithminsidetheGigaswitches.
Thisalgorithmisapproximatelyround-robin,butalsodependsoninternalloadcharacteristicsintheGigaswitchswitchingfabric.
However,weknowthatthedistributionofpacketsamongtheOC-3linksisnotentirelyuniform,sincemeasurementsofthelinkutilizationsshowtwoofthemcarryapproximatelytwicethetrafficoftheothertwo(measuredbybytevolume,notpacketvolume)[Feldman98].
However,weassumethattheGigaswitchschedulingalgorithmisindependentoftheencapsulatedprotocol,e.
g.
notdependentuponpacketlength.
Becauseofthiscomplexschedulingalgorithmwearenotabletoaccuratelyestimatethenumberofconversationstraversingthemonitoredlinks,orthelengthoftheseconversationsinpacketsorbytes.
Consequently,weonlycharacterizetheworkloadobservedatAIXintermsofrelativefractionsofpacketsandbytes.
MonitoringMethodologyandToolsThedatacollectionsystemisessentiallysimilartotheoneusedin[Thompson97].
ACoral/OC3monplatformwasconnectedtoonelinkineachdirectionusingopticalsplitters.
ThetraceswestudiedwerecollectedaspartofNLANR/MOAT'sNetworkAnalysisInfrastructure(NAI)project[Braun98].
Foreachpacketthatpassesthemonitor,onlythefirstATMcellfromtheAAL5PDUiscapturedandwrittentodisk.
Thefirstcellcontainsthefirst40bytesofeachpacket,whichisusuallyenoughtoextracttheTCPorUDPportnumbersfromthetransportlayerheaders.
However,themonitordoesnotverifythattheentireAAL5PDUiscarriedbythelink,andsoestimatesofthedataratecarriedbythelinkmaybeinflatedinthepresenceofcellloss.
Sixtoeighttraceswerecollectedeachday,usuallywithadurationof90secondseach.
Thestartingtimeforeachtracewassetatequalintervalsduringthe24hourperiod,andrandomizedoverarangeofanhouratthebeginningofeachinterval.
Aftercollection,thetracesareprocessedtoremoveanyinformationthatmightcompromisetheprivacyoftheindividualsgeneratingthetraffic.
ThisprocessingmasksthesourceanddestinationIPaddresses,anddeletesalldatafromtheIPpayloadexceptforaTCPorUDPheader(ifpresent),ortheICMPorIGMPtypeandcodefields.
IfthepacketcarriesenoughbytesofIPheaderoptions,thentheTCPorUDPportnumbersmaynotbepresentinthefirstcellofthePDU.
Inthiscase,weignorethatpacketinsubsequentapplicationworkloadanalysis.
SincethefractionofpacketswithIPheaderoptionsistypicallylessthan0.
003%,thisdoesn'tseriouslyimpactourmeasurementsofthetrafficfractiongeneratedbythemostpopularTCPandUDPapplications.
AnalysisMethodologyandToolsWeusedCoralReef[CoralReef]toreduceeachrawtracetoasetofsummarytablesthatwearchivedforlateranalysis.
ThetablesincludeaggregatenumberssuchasthenumberofpacketsandbytesinthetraceaswellasdistributionsofpacketlengthsandthenumberofpacketsandbytesseenforeachIP-layerprotocol.
ForTCPandUDP,weanalyzeapplicationusageusingportaddresspairs.
ThepackettracesavailablefromtheNAIarchive[Braun98]onlyincludeIPandtransportlayerheaders,soourmethodologydoesnotuseencapsulateddatatoidentifytheapplicationthatgeneratedthepackets.
TracesintheNAIarchivehavehadallpayloaddataremovedtoprotecttheprivacyofInternetusers.
Inmostcases,wehaveassumedthatpacketssentbetweenanyportnumberhigherthan1023andawell-knownportnumberbelow1023aregeneratedbythesameprotocol(e.
g.
,HTTPonport80).
Thismatchestypicalendhostbehavior,inwhichclientsallocateephemeralportsfromtherange1024to32767[Stevens94].
Forsomeoftheprotocols,wehavecondensedrangesofportnumbersinboththesourceanddestinationfields.
Forexample,theRealAudiocategoryintheUDPtableincludesalltrafficwithdestinationportsbetween6970and7170inclusive[RealNetworks].
Unfortunately,thisrangealsoincludestheportsusedbyAFS,andsowearepotentiallyconfusinganunknownamountofAFStrafficwithRealAudio.
However,themajorityofRealAudiotrafficappearsonUDPports6970,6971,and6972,noneofwhichareusedbyAFS.
ByonlyconsideringtrafficonUDPportsfromthisrangethatarenotusedbyAFS,theamountofRealAudiotrafficcanbeestimatedindependentlyfromtheamountofAFStrafficthatmaybepresentaswell.
WearecurrentlyinvestigatingbettertechniquesfordifferentiatingRealAudioandAFStrafficusingpacketsizedistributionandpacketinterarrivalpatterns,andwehopetobeabletoconclusivelydifferentiatebetweenthetwointhefuture.
ArecentanalysisofthetrafficpatternsexhibitedbyRealAudiotraffic[Mena00]hasshownseveralparametersthatmaybeusedtodifferentiatebetweenRealAudioandotherprotocols.
AfurtherstudycharacterizingAFStrafficpatternsneedstobeundertakentoidentifythebestmetricstousetoseparatethetwo.
ForbothTCPandUDPtraffic,thereisasignificantfractionoftrafficthatcannotbemappedtoapplicationsusingwellknownportnumbers.
Manyprotocolsdonotdependonwell-knownportnumbers,buteitheruseawell-knownservicefornegotiatingtheportnumbersusedbysecondaryconnections,orusearbitrarybutfixedportnumbersthatarenotregisteredwithIANA.
Themostpopularapplicationwithnegotiatedportnumbersispassive-modeFTP,inwhichtheclientsendstheportnumbertouseforadataconnectionoverthecommandchannel.
Therearemanyotherprotocolsthatusesimilarbehavior,suchasNapsterandInternettelephonyapplications.
Mostonlinegamesdonotregisterwell-knownportswithIANA,butusearbitraryportnumbersabove5000.
Wehavecollectedtheportnumbersusedbyseveralofthepopulargamesandusethisinformationtoestimatethefractionoftrafficgeneratedbythem.
OuranalysisofonlinegametrafficincludesgametrafficonthefollowingUDPports:HalfLifeanytoorfrom27005anytoorfrom27015Quake3:Arenaanytoorfrom27960Starcraft6112to6112QuakeIIanytoorfrom27901anytoorfrom27910QuakeWorldanytoorfrom27500anytoorfrom27001Unrealanytoorfrom7777Table1:UDPportsusedbyOnlineGamesAsisthecasewithRealAudioandAFS,therearemanypossibilitiesforconfusionbetweengametrafficandotherapplicationswhenonlyportnumbersareusedtomaketheclassification.
Weassumethattherearenootherprotocolsthatpreferentiallyusethesesameports,andthatapplicationsthatephemerallyusetheseportscontributeequalamountsoftrafficacrossalltrafficcategories.
Thisassumptioncarriessignificantrisks,andneedsfurtheranalysistofullyevaluateitsimpactonourdata.
ResultsPacketlengthsThefollowinggraphsshowthedistributionofIPpacketsizesseenatAIX.
Thesedistributionsarebuiltfromtwoapproximatelyoneweekperiodsnearthebeginningandendofourstudy.
Theycontaincontributionsfromthedifferentworkloadscarriedbythenetworkatdifferenttimesofday,andsotheyshouldrepresentmoreofan'average'pictureofthepacketsizedistributionthananyindividualtrace.
However,noattempthasbeenmadetonormalizethecontributionsofindividualtraces.
Thedistributionspresentedherearesimplythoseoftheconcatenatedtraces.
Additionally,thesedistributionshaveonlybeenplottedforpacketsizeslessthan1600bytes.
Thisallowsthestructureofthedistributionstobepresentedingreaterdetail,buthidestheverysmallfractionoflargerpacketsthatappearinthesetraces(typicallylessthan0.
005%ofpackets).
Fig2:IPpacketlengthdistributionfrom39tracefilescapturedbetweenThursday,May13th1999at19:14:36PDTandWednesday,May19th1999at13:02:20PDT,overanintervalofalmostsixdays.
Statisticsfortheunderlyingpacketlengthdistribution:Mean:413bytes,StandardDeviation:509bytesMedian:93bytes,Percentiles:5th40bytes,25th40bytes,75th576bytes,95th1500bytesNumberofObservations:127millionpackets[127710031packets]Thesenumberscorrespondtotheuppercurveinthefigureabove.
Fig3:IPpacketlengthdistributionfrom43tracefilescapturedbetweenSunday,February20th2000at16:01:51PSTandSunday,February27th2000at13:59:18PST,overanintervalofalmost7days.
Statisticsfortheunderlyingpacketlengthdistribution:Mean:420bytes,StandardDeviation:521bytesMedian:78bytesPercentiles:5th40bytes,25th40bytes,75th576bytes,95th1500bytesNumberofObservations:84millionpackets[84415871]Thesenumberscorrespondtotheuppercurveinthefigureabove.
TheprimaryfeaturesofthisdistributionoriginateinthewaycommonTCPimplementationsdivideadatastreamintopackets.
Approximately85%ofthetrafficinthesetracesisTCP,andalargeproportionofthisTCPtrafficisgeneratedbybulktransferapplicationssuchasHTTPandFTP.
Consequently,themajorityofthepacketsseenareoneofthreesizes:40bytepackets(theminimumpacketsizeforTCP)whichcarryTCPacknowledgmentsbutnopayload;1500bytepackets(themaximumEthernetpayloadsize)fromTCPimplementationsthatusepathMTUdiscovery;and552byteand576bytepacketsfromTCPimplementationsthatdonotusepathMTUdiscovery.
Thesetwodistributionsarestrikinglysimilardespitethefactthatthesecondisbasedondatacollectedmorethan9monthsafterthefirst.
Theseconddistributionhasaslightlylargercontributionfrompacketssmallerthan100bytes,butthedifferenceisquitesmall.
Thefollowinggraphshowshowthemeanandmedianvaluesofthepacketsizedistributionvaryovertheentiredurationofourstudy.
Fig4:MeanandmedianpacketlengthvaluesforeachpackettracecollectedbetweenThursday,May13th1999at19:14:36PDTandSatMar1113:57:38PST2000.
Valueshavebeenbinnedbyweek,andthemedianforeachbinisplottedwithfirstandthirdquartileerrorbars.
Itisnotsurprisingthattheseparametersdonotshowanysignificantlongtermtrend.
AslongasTCPisthedominantprotocolinuseintheInternet,thepacketlengthdistributionisunlikelytochangeverymuchunlesstheTCPprotocolitselfchanges.
ProtocolMixRawDataInthetablesbelow,weprovidesomesampledatafromthemonthofFebruary,2000.
Eachofthe216tracesfromthisperiodwereconcatenated,andthesummaryinformationpresentedbelowisforallthetracescombined.
TotalIPBytes193692014407TotalIPPackets451971619TotalDurationofTraces24345secIPpacketswithDFset329241464FragmentsofIPDatagrams1023206FragmentsofUDPDatagrams597228FragmentsofTCPDatagrams9702IPPacketswithoptions3213Non-IPpackets462085Table2:AggregatetotalsforalltracescollectedinFebruary,2000Table3presentsthetop10IPlayerprotocolsseeninthesetraces.
TCPandUDPtypicallyaccountforalmostallofthetrafficseeninanytrace,withGREandICMPasthenexttwomostpopularprotocolsProtocolNumberPacketsBytesAverageSizeTCP6374801201176706563104471UDP17624567319842511709157GRE4775664155272240819696ICMP159380441350011401227ESP50517353197216792381IPinIP4265103179257606676AH517442343454671583IPIP9414350241707350290SKIP5711705041633952355IGMP2684047729038112Table3:Top10protocolsseenduringFebruary,2000Inthenexttwotables,wehaveconsolidatedtheTCPandUDPportaddresspairsintocategoriesbyapplication.
Asdescribedpreviously,wehaveassumedthattrafficbetweenanyportnumberhigherthan1023andawell-knownportnumberbelow1023isexclusivelygeneratedbythesameprotocol(e.
g.
HTTPonport80).
Wedenotethecondensedsetofportnumbersinthetablewiththelabel'0'.
Forsomeoftheprotocols,wehavecondensedrangesofportnumbersinboththesourceanddestinationfields.
Forexample,theRealAudiocategoryintheUDPtableincludesalltrafficwithdestinationportsbetween6970and7170inclusive,representedwiththelabel'7070'.
NotethatasignificantportionofbothTCPandUDPtrafficfallsintoacategorylabeledwith'0'inboththesourceanddestinationfields.
Thisincludesallthetrafficthatwasnotmappedtoaspecificprotocolusingourmethodology.
Thistrafficiseithergeneratedbyprotocolsthatusenegotiatedportnumbersatbothends(e.
g.
,passive-modeFTP),orbynewapplicationsthatuseunregisteredfixedports.
Wecontinuetoaddportutilizationprofilesfornewprotocolsasweobtainnewandupdatedinformation.
ProtocolSourceDestinationPacketsBytesAverageSizeHTTP800140780543100044030753710004531984217319763013382NNTP01191789548115992967942893HTTP08094578965784416385082FTPData20067280976689611587994SMTP02588789256071084052683NNTP119088572175399672480609HTTP/WebProxy8080023316692327032104998Napster0669933311091838804438552HTTPS443030358091535037132505Napster6699033778281528188686452FTPData02054980971262294037229Napster668801230335935883810760175501182358908626624768POP311001820887798255125438Hotline550106855367871220081148RTSP5540754508616087123816Napster066881149382348845629303SMTP250643867233978802052RealAudio70700445551298598442670Shoutcast80000353545296161291837WebCache31280325447280739543862HTTPS04432635048280418901106NetBIOSSSN139031265826496521284721890294654174140223590Table4:Top25TCPapplicationcategoriesseenduringFebruary,2000ProtocolSourceDestinationPacketsBytesAverageSize00151088222568130721169RealAudio0707046100702029483625440DNS535392908721064980650114DNS5303444558638796849185HalfLife27015270052199098452384485205DNS05352865543325982496206770619334230280312371Starcraft61126112416762521778375552EverQuest90019000908432171755388189HalfLife2700527015275441616080617658RealAudio69700532356154663054290Unreal777701109005141327485127EverQuest90059000777485138637166178Unreal07777189261310761327956Quake3:Arena279602796078425880146922102HalfLife27015042461477756002183QuakeII27901279109466745884258362HalfLife02700531773558062692182022503374599504691328001025508144789873175NetBIOSNS1371375098544399607586QuakeII27910279012328654394243818803714653140574996871CU-SeeMe7648764813684339681727289QuakeII2790106014273732778562HalfLife2700504983002919106258Table5:Top25UDPapplicationcategoriesseenduringFebruary,2000LongTermTrendsIneachofthetimeseriesgraphsbelow,wehavecollectedthedatavaluesintoweek-longbins,andweplotonlythemedianandfirstandthirdquartilesforeachbin.
Forgraphswithmorethanonedataseries,wehaveintroducedaslightoffsettothebinsforeachseriestopreventoverlap.
Wepresentonlythefractionoftrafficcontributedbyeachapplicationorapplicationgroupratherthanabsolutemeasurementsoftrafficvolumeinanycategory.
Shiftsintherelativefractionsamongmultipleprotocolsorapplicationsovertimerepresentchangesinhowthenetworkisused,ratherthanhowbusyourmeasuredlinkisatanygiventime.
However,duringtheperiodofthisstudy,themonitoredlinkhadamedianutilizationofapproximately85Mb/s.
Medianandquartilevalueshavebeencalculatedusingthemethodologydescribedin[RFC2330].
Specifically,thequartilevaluesareactualdatapointsandnotinterpolatedvalues,andthemedianiseitheranactualdatapointortheaverageofthetwomiddlevalues.
Eachweeklybintypicallycontains40-50observations.
ThefirstgraphshowsthefractionofpacketsduetoTCPandUDPseenatAIX.
Thegraphdoesnotshowanyclearlongtermshiftsinthebalancebetweenthesetwodominanttransportlayerprotocols.
DespitetheincreaseinthenumberofUDPapplicationsinrecentyears,growthinthetotalamountofUDPtrafficisoffsetbygrowthinTCPtrafficaswell.
Figure5:FractionofTCPandUDPpacketsseenatAIX.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
ThisnextgraphshowsthefractionofIPSECtrafficseenatAIX,includingbothauthenticationheader(AH)andencapsulatingsecuritypayload(ESP)traffic.
Thegraphshowsanincreaseofalmostanorderofmagnitudeduringthefirst5monthsofourstudy,butthenlevelsofforevendeclinesslightly.
Thissuggeststhatafteraninitialperiodofseriousinterest,IPSECtraffichasstoppedgrowingfasterthannon-IPSECtraffic.
Figure6:Fractionofauthenticationheader(AH)andencapsulatingsecuritypayload(ESP)trafficseenatAIX.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
Theamountoffragmentedtrafficinwideareanetworkshasbeenatopicofgreatinterestinthepastfewmonths,especiallyasitrelatestoIPtracebacktechniques[Savage00].
Ourdataindicatesthatthefractionoffragmentedtrafficisontherise,andthatthemajorityofthisgrowthisintheformofUDPpackets.
WeneedtoperformfurtheranalysistodeterminewhichUDPprotocolsaregeneratingthesefragments.
Notsurprisingly,TCPtrafficisvirtuallyneverfragmented.
MostlikelythisisduetowidespreaddeploymentofpathMTUdiscoverycombinedwithTCP'srelativelysmalldefaultpacketsize.
Figure7:Fractionoffragmentedtraffic.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
FTPisthetraditionalbulktransferprotocolwidelyusedbeforetheadventofHTTPandtheweb.
However,weseeacleardeclineinthecontributionofFTPtotheoveralltrafficmixsinceOctober1999.
ThisdeclinemayactuallybeduetoashiftfromactivetopassivemodeFTPassociatedwithanincreaseinpacketfilteringfirewallsintheInternet,oritmaybeduetoashiftawayfromFTPtoalternativeprotocolsforfiletransfer.
However,thegraphofHTTPtrafficfractionoverthissametimeperioddoesnotshowacorrespondingincrease.
Figure8:ThefractionofactivemodeFTPtrafficisdeclining.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
ThefollowinggraphshowsarelativelysurprisingdecreaseinthefractionofRealAudiotrafficseenatAIX.
BothTCPandUDPRealAudiotraffichavedecreasedinpacketvolumecomparedtonon-RealAudiotraffic,althoughthistrendseemstohaveflattenedoutinthelastfewmonthsofourstudy.
AlthoughRealNetworkshasreleasednewerversionsoftheirRealPlayersoftwaresincethisstudybegan,thenewversionsusethesamesetofTCPandUDPportsastheoldversions,withtheadditionofTCPport554forRTSPtraffic[RealNetworks].
Consequently,thetrendweobserveisnotduetoashiftfromtheoldersoftwaretothenewerversions,butrepresentseitheraslowinginthegrowthofRealAudiotrafficoradeclinerelativetothegrowthinnon-RealAudiotraffic.
Figure9:ThefractionofRealAudiotraffichasdeclinedoverthepast10months.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
Thisgraphshowsadeclinetheinthefractionoftrafficgeneratedbyseveralpopularonlinegamesoverthefirst8monthsofourstudy.
TheonlinegamesincludedinthisgraphareStarcraft,QuakeII,andQuakeWorld(avariantofQuakeII).
Thesegameswerepopularwhenwestartedcollectingourdata,butasthegraphshows,theirpopularityhasdeclinedfairlysteadilysinceJuly,1999.
Figure10:Fractionofonlinegametraffic,includingQuakeII,QuakeWorld,andStarcraft.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
Thissecondgraphofonlinegamingtrafficshowsquiteadifferenttrendfromthepreviousgraph.
Unlikethepreviousgraph,thisoneincludestrafficfromseveralnewergamesinadditiontotheolderones.
ThenewgraphincludestrafficgeneratedbyHalfLife,Quake3:Arena,andUnrealinadditiontoStarcraft,QuakeII,andQuakeWorld.
Althoughthereisnotenoughdatatodetermineatrend,themediantrafficfractionsaremuchhigheronthisgraphthanthepreviousone.
Hence,theoverallfractionofonlinegametrafficseemstobeontherise,butitisamovingtarget.
Theincreaseisprimarilygeneratedbynewgamesastheygainpopularity,whileoldergamesseemtodecreaseinpopularityovertime.
Figure11:Fractionofonlinegametraffic,includingHalfLife,QuakeII,QuakeWorld,Quake3:Arena,Starcraft,andUnreal.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
AlthoughNapsterdoesnotuseafixedsetofportsforfiletransfers,weidentifiedthethreemostcommonlyusedTCPportsinuseinlateJanuary:TCPports6688,6697,and6699.
Bulktransfertrafficmaybeeithersenttoorreceivedfromtheseports,sinceNapstersupportsbothactiveandpassivemodetransfers[Napster].
AsUniversitiesandothersitesmovetoblocktrafficontheseports,Napstertrafficwillundoubtedlymigratetoothers.
However,theshorttermtrendclearlyshowsdramaticgrowth,increasingbyover50%inthelasttwomonthsofourstudy.
Figure12:FractionofNapsterbulk-transfertrafficseenatAIX.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
ShorterTermTrendsSomeshortertermtrendscanbeeasilyassociatedwithcommonuserbehavior.
Forexample,thefractionofemailtraffic(SMTPpacketstoorfromTCPport25)increasedsignificantlyinNovemberandearlyDecember,onlytodropoffagainduringtheholidaysattheendofDecember.
Perhapsthiswasrelatedtoonlinecommerce,asmanypeopleusedtheInternettopurchasetheirChristmasgiftsFigure13:FractionofmailtrafficseenatAIX,showingapeakinusagejustbeforetheChristmasholidays.
Tracesarecollectedintoweeklybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
Onlinegamingisclearlymorepopularonweekends,asthefollowinggraphshows.
Thefractionofgametrafficcannearlydoubleonweekendscomparedtothetypicalweekday.
Thischangeinworkloadoverthecourseofaweekhasbeensmoothedoutinourpreviousgraphsbyourchoiceofbinsize.
Figure14:WeeklyvariationsinthefractionofonlinegamingtrafficseenatAIX.
Tracesarecollectedintodailybins,andthemedianandfirstandthirdquartilesforeachbinareplotted.
ConclusionsandFutureDirectionsInourlongtermstudyofthetrafficworkloadattheAmesInternetExchange,wefoundnosignificanttrendsovertimeineithertheoverallpacketsizedistributionorchangesintheratioofTCPtoUDPtraffic.
WedidfindanorderofmagnitudeincreaseinthevolumeofIPSECtraffic,andastrongincreasingtrendinthevolumeoffragmentedUDPtraffic.
Someotherinterestingtrendsincludedecreasesinthevolumeofactive-modeFTPandRealAudiotraffic,andincreasesinthevolumeofonlinegamingandNapstertraffic.
Wealsofoundshorttermtrends,includingatransientincreaseinthevolumeofe-mailtrafficduringtheholidayshoppingseason,andastrongweekday/weekendvariationinthevolumeofonlinegamingtraffic.
Thecollectionofbackbonetrafficstatisticshasbeenseverelyimpactedbythebreakupofthenetworkcoreintoanumberofmutuallycompetitiveorganizations.
DatacollectioneffortsmustnowprotectboththeprivacyofindividualInternetusersandtheproprietaryinterestsofthehostorganizations.
CAIDAisinauniquepositiontomonitorandanalyzetrafficcollectedfrombackbonelinks,asithasongoingrelationshipswithseveralcoreISPsandexchangepointproviders.
Ourworkloadcharacterizationeffortsalsofaceadditionaltechnicalchallenges,bothinextendingourmethodologytoincreasetheaccuracyofourmeasurements,andincopingwiththeintroductionofnewprotocols.
Ourportbasedtrafficclassificationschemehassomeseverelimitations,asillustratedbythevolumeoftrafficitfailstomaptoanyspecificprotocol.
Additionaltechniquesarenecessarytofurtherdifferentiatethistraffic,aswellasreduceconfusionamongprotocolsthatusethesamesetofsourceordestinationports.
Furthermore,inordertoaccuratelyclassifyprotocolsthatusenegotiatedportsonbothendsitwillbenecessarytocorrelatemultipletrafficstreamsfromthesamehosttothesameordifferentdestinations[Plonka00].
However,thesetechniquesmayhavelimitedusabilityintheInternetcorewherethemeasurementsiteisfarawayfromboththesourceanddestinationofthetraffic.
Consequently,wemaybeunabletoeffectivelysortthistrafficbyprotocolwithoutusingtechniquesthatmonitorthealltrafficexchangedontheassociatedcontrolchannels,asthesechannelstypicallyuseawell-knownportatoneendevenifthedatachanneldoesnot.
IPSECrepresentsanotherpotentialproblemforworkloadanalysis,asESPtrafficencryptsthesourceanddestinationportsweusefortrafficclassification.
IfuseofESPbecomeswidespread,wemaylosetheabilitytoestimatehowthenetworkisbeingutilizedaltogether.
[Bellovin99]proposesamodificationtoESPthatsendscleartextportnumbersforencapsulatedtraffic.
Ifthismodificationwasadopted,ESPwouldnotposesuchaproblemforourmeasurements.
TheinterconnectionlinkbetweenAIXandMAE-WesthasrecentlybeenupgradedtoasingleOC-12PacketoverSONET(POS)link.
Thiswillimprovethedatacollectionsite,andallowustoperformpacketflowanalysissinceeverypacketsentbetweenAIXandFIXWestwillbevisibletoourmonitor.
However,itwillalsorequirePOSsupportforCoral/OCXmonhardware.
DevelopmentofadditionalhardwareandsoftwaretosupportPOSlinksatOC-3andOC-12speedsiscurrentlyunderwayatCAIDA,andwehopetocontinuethisanalysisafterthelinkupgradewithminimalinterruptionindatacollection.
CAIDAisalsointerestedinotherbackbonesamplingpointsforcomparisonagainsttheworkloadseenatAIX.
PreviousstudiesdonewithdatacollectedbyMCI[claffy98,Apisdorf97,Thompson97]bearsomesimilaritiestoourresults,butmoredatacollectionmustbeperformedbeforeanyconclusionscanbedrawnabouthowrepresentativeourresultsareofotherwide-areaInternetinfrastructure.
AcknowledgmentsThisanalysiswouldnothavebeenpossiblewithouttheworkofHans-WernerBraunandtheNationalScienceFoundation'sNLANR/MOATprojectforcollectingandarchivingInternettraffictraces.
TheparticularmonitorthatweusedattheAMESInternetExchange(AIX)existsasaresultofacollaborationamongNASA,NLANR,andCAIDA.
WearegratefultoBillJones,LanceTatman,BobbyCates,andJeffOsbourne(allfromNASA-Ames),andtoSteveFeldman(previouslyatMCI/Worldcom)fortheircontinuedsupportandmaintenanceofthisandothertrafficmonitorsattheexchangepoints.
WewouldalsoliketothankKenKeys,DavidMoore,andtherestoftheCoralReefteamfortheanalysislibraryusedthroughoutthisproject.
References1.
[AIX]AmesInterneteXchange,http://aix.
arc.
nasa.
gov/.
2.
[AIXstats]AmesInterneteXchangeLinkUtilizationStatistics,http://aix.
arc.
nasa.
gov/graphlink.
html.
3.
[Apisdorf97]J.
Apisdorf,kclaffy,K.
Thompson,andR.
Wilder.
OC3mon:Flexible,Affordable,High-PerformanceStatisticsCollection,http://www.
isoc.
org/isoc/whatis/conferences/inet/97/proceedings/F1/F1_2.
HTM.
4.
[Bellovin99]S.
Bellovin.
Transport-FriendlyESP,http://www.
research.
att.
com/~smb/talks/tfesp-ndss/index.
htm.
5.
[Braun98]H.
-W.
Braun.
TowardsasystemicunderstandingoftheInternetorganism:aframeworkforthecreationofaNetworkAnalysisInfrastructure,http://moat.
nlanr.
net/NAI.
6.
[Caceres89]R.
Caceres.
MeasurementsofWide-AreaInternetTraffic.
UCB/CSD89/550,UniversityofCalifornia,Berkeley,CA,December1989,http://sunsite.
berkeley.
edu/Dienst/UI/2.
0/Describe/ncstrl.
ucb/CSD-89-550.
7.
[Caceres91]R.
Caceres,P.
Danzig,S.
Jamin,andD.
MitzelCharacteristicsofwide-areaTCP/IPconversationsProceedingsofACMSIGCOMM'91,September1991,http://www.
acm.
org/pubs/citations/proceedings/comm/115992/p101-caceres.
8.
[CAIDA]CooperativeAssociationforInternetDataAnalysis,http://www.
caida.
org.
9.
[claffy93]K.
ClaffyandH.
-W.
BraunandG.
Polyzos.
Long-termtrafficaspectsoftheNSFNET,ProceedingsofINET'93.
http://wwwdev.
caida.
org/outreach/papers/lta.
html.
10.
[claffy95]K.
C.
Claffy,Hans-WernerBraun,GeorgeC.
Polyzos.
AparameterizablemethodologyforInternettrafficflowprofiling,IEEEJSACApril1996,http://wwwdev.
caida.
org/outreach/papers/pmi.
html.
11.
[claffy97]kclaffyandT.
Monk.
What'snextforInternetdataanalysisIEEESpecialIssueonCommunicationsinthe21stCentury85,1563-1571(1997).
.
12.
[claffy98]kclaffy,G.
Miller,andK.
Thompson.
thenatureofthebeast:recenttrafficmeasurementsfromanInternetbackbone,http://www.
isoc.
org/inet98/proceedings/6g/6g_3.
htm.
13.
[claffy99]kcclaffy.
Internetmeasurementanddataanalysis:topology,workload,performanceandroutingstatistics,,NAE99,http://wwwdev.
caida.
org/outreach/papers/Nae/.
14.
[CoralReef]CoralReefhomepage,http://www.
caida.
org/Tools/CoralReef.
15.
[Feldman98]S.
Feldman.
MAE-WestLinkUtilizationStatistics,http://www.
mae.
net/~feldman/gigaswitch/ames.
16.
[Feldmann98]A.
Feldmann,J.
Rexford,andR.
Caceres.
EfficientpoliciesforcarryingWebtrafficoverflow-switchednetworks,,IEEE/ACMTransactionsonNetworking,December1998,pp.
673-685,http://www.
research.
att.
com/~jrex/papers/ton98.
ps.
17.
[Heimlich89]H.
Heimlich.
TrafficCharacterizationoftheNSFNETBackbone,,USENIXConferenceProceedings,Winter1989,http://www.
research.
att.
com/~jrex/papers/ton98.
ps.
18.
[Merit95]NSFNETStatistics,MeritNetwork,Inc.
,1995,http://www.
merit.
edu/merit/archive/nsfnet/statistics.
19.
[Mena00]A.
MenaandJ.
Heidemann.
AnEmpiricalStudyofRealAudioTraffic,,IEEEINFOCOM2000,http://www.
ieee-infocom.
org/2000/papers/84.
ps.
20.
[MOAT]NationalLaboratoryforAppliedNetworkResearch(NLANR)/MeasurementandOperationsAnalysisTeam(MOAT),http://moat.
nlanr.
net/.
21.
[MOAT99]NationalLaboratoryforAppliedNetworkResearch(NLANR)/MeasurementandOperationsAnalysisTeam(MOAT),DataanalysisbasedonCoralpackettraceshttp://moat.
nlanr.
net/Datacube.
22.
[Napster]NapsterProtocolSpecification,http://opennap.
sourceforge.
net/napster.
txt.
23.
[OC3mon]Coral/OC3MonitoringHardware,http://moat.
nlanr.
net/OC3mon-monitors.
24.
[OC3mon-sw]OC3monDataCollectionSoftware,http://moat.
nlanr.
net/Software/OC3mon.
25.
[RFC2330]V.
Paxson.
GrowthTrendsinWide-AreaTCPConnections,IEEENetwork,8(4),pp.
8-17,July/August1994.
ftp://ftp.
ee.
lbl.
gov/papers/WAN-TCP-growth-trends.
ps.
Z.
26.
[Plonka00]D.
Plonka.
UW-MadisonNapsterTrafficMeasurement,http://net.
doit.
wisc.
edu/data/Napster/.
27.
[RealNetworks]RealNetworksRealSystemFirewallSupport,http://service.
real.
com/firewall/adminfw.
html.
28.
[RFC2330]V.
Paxson,G.
Almes,J.
Mahdavi,andM.
Mathis.
FrameworkforIPPerformanceMetricsRFC2330,ftp://ftp.
isi.
edu/in-notes/rfc2330.
txt.
29.
[Savage00]S.
Savage,D.
Wetherall,A.
KarlinandT.
Anderson.
PracticalNetworkSupportforIPTraceback,,insubmission.
CurrentlyavailableasUW-CSE-00-02-01.
.
30.
[Stevens94]W.
RichardStevens.
TCP/IPIllustrated,Volume1:TheProtocolsAddison-Wesley,1994.
31.
[Thompson97]K.
Thompson,G.
Miller,andR.
Wilder.
WideAreaInternetTrafficPatternsandCharacteristicsIEEENetwork,Vol.
11No.
6,pp.
10-23,Nov/Dec1997.
http://www.
vbns.
net/presentations/papers/MCItraffic.
ps.
32.
[WISCstats]D.
Plonka.
UniversityofWisconsinNetworkUsageStatistics,http://wwwstats.
net.
wisc.
edu/.
Formoreinformation:info@caida.
orglastupdate:thispage: