goodcorrectly

APracticalGuideto(Correctly)APracticalGuideto(Correctly)TroubleshootingwithTracerouteRichardASteenbergennLayerCommunications,Inc.
IntroductionIntroductionTroubleshootingproblemsontheInternetgpThenumberonego-totoolis"traceroute"EveryOScomeswithatraceroutetoolofsomekind.
Therearethousandsofwebsiteswhichcanrunatraceroute.
Therearedozensof"visualtraceroute"toolsavailable,bothcommerciallyandfree.
AnditseemslikesuchasimpletooltouseItypeinthetargetIPaddressanditshowsmesomerouters.
AndwherethetraceroutestopsorwherethelatencygoesupAndwherethetraceroutestops,orwherethelatencygoesupalot,that'swheretheproblemis,rightHowcouldthispossiblygowrongUfllild'bfhUnfortunately,realitycouldn'tbeanyfurtheraway.
ByRichardSteenbergen,nLayerCommunications,Inc.
2IntroductionIntroductionSowhat'swrongwithtracerouteMostmodernnetworksareactuallywellrunSosimpleissueslikecongestionorroutingloopsarebecomingasmallerpercentageofthetotalnetworkissuesencountered.
asmallerpercentageofthetotalnetworkissuesencountered.
Andmorecommonly,theencounteredissuesarecomplexenoughthatanavetracerouteinterpretationisutterlyuseless.
FewpeopleareskilledatinterpretingtracerouteFewpeopleareskilledatinterpretingtracerouteMostISPNOCsandevenmostmid-levelengineeringstaffarenotabletocorrectlyinterpretcomplextraceroutes.
Thisleadstoasignificantnumberofmisdiagnosedissuesand"falsereports",whichfloodtheNOCsofnetworksworldwide.
Inmanycasestheproblemoffalsereportsissobad,itisallfbutimpossibleforaknowledgeableoutsidepartytosubmitatracerouterelatedticketaboutarealissue.
ByRichardSteenbergen,nLayerCommunications,Inc.
3TracerouteTopicsTracerouteTopicsTopicstodiscusspHowtracerouteworksInterpretingDNSintracerouteUnderstandingnetworklatencyAsymmetricpathsMultiplepathsMPLSandtracerouteRdTtFtidRandomTracerouteFactoidThedefaultstartingportinUNIXtracerouteis33434.
Thiscomesfrom32768(2^15orthemaxvalueofaThiscomesfrom32768(215,orthemaxvalueofasigned16-bitinteger)+666(themarkofSatan).
ByRichardSteenbergen,nLayerCommunications,Inc.
4Traceroute–The10,000FtOverviewTracerouteThe10,000FtOverview1.
LaunchaprobepackettowardsDST,withaTTLof12.
EachrouterhopdecrementstheTTLofthepacketby13.
WhenTTLhits0,routerreturnsICMPTTLExceeded4.
SRChostreceivesthisICMP,displaysatraceroute"hop"5.
Repeatfromstep1,withTTLincrementedby1,until…6DSThostreceivesprobereturnsICMPDestUnreach6.
DSThostreceivesprobe,returnsICMPDestUnreach.
7.
Tracerouteiscompleted.
ICMPDestUnreachICMPTTLExceedICMPTTLExceedICMPTTLExceedICMPTTLExceedByRichardSteenbergen,nLayerCommunications,Inc.
5SRCRouter1Router2Router3Router4DSTTTL=1TTL=2TTL=3TTL=4TTL=5Traceroute–ALittleMoreDetailTracerouteALittleMoreDetailMultipleProbesMultipleProbesMosttracerouteimplementationssendmultipleprobes.
Thedefaultis3probesperTTLincrement("hop").
pp(p)Hencethenormal3latencyresults,or3*'sifnoresponse.
EachprobeusesadifferentDSTPorttodistinguishitselfSoanylayer4hashingcansendeachprobeondifferentpaths.
ThismaybevisibletotracerouteinthecaseofECMPhashing.
Orinvisible,inthecaseof802.
3adstyleLayer2aggregation.
Buttheresultisthesame,someprobesmaybehavedifferently.
NotalltracerouteimplementationsuseUDPWindowsusesICMP,othertoolsmayevenuseTCP.
ByRichardSteenbergen,nLayerCommunications,Inc.
6Traceroute–LatencyCalculationTracerouteLatencyCalculationHowistraceroutelatencycalculatedHowistraceroutelatencycalculatedTimestampwhentheprobepacketislaunched.
TimestampwhenthereplyICMPisreceived.
ppySubtractthedifferencetodetermineround-tripvalue.
Routersalongthepathdonotdoanytime"processing"Theysimplyreflecttheoriginalpacket'sdatabacktotheSRC.
Manyimplementationsencodetheoriginallaunchtimestampintotheprobepacket,toincreaseaccuracyandreducestate.
ppyButremember,onlytheROUNDTRIPismeasured.
Tracerouteisshowingyouthehopsontheforwardpath.
BthiltbdthfdPLUSButshowingyoulatencybasedontheforwardPLUSreversepaths.
Anydelaysonthereversepathwillaffectyourresults!
ByRichardSteenbergen,nLayerCommunications,Inc.
7Traceroute–WhatHopsAreYouSeeingTracerouteWhatHopsAreYouSeeingICMPTTLExceedICMPReturnInterface192.
168.
2.
1/30ICMPReturnInterface192.
168.
3.
1/30ICMPTTLExceedTTL=1ICMPTTLExceedIngressInterfaceIngressInterfaceTTL=2EgressInterfaceSRCRouter1gessteace172.
16.
2.
1/30PacketwithTTL1entersrouterviaingressinterfaceRouter210.
3.
2.
2/3010.
3.
2.
1/30ICMPTTLExceedisgeneratedastheTTLhits0ICMPsourceaddressisthatoftheingressrouterinterface.
Thisishowtracerouteseestheaddressofa"hop"theingressIPThisishowtracerouteseestheaddressofahop,theingressIP.
Theabovetraceroutewillread:172.
16.
2.
110.
3.
2.
2Randomfactoid:Thisbehaviorisactuallynon-standardByRichardSteenbergen,nLayerCommunications,Inc.
8RFC1812saystheICMPsourceMUSTbefromtheegressiface.
Ifobeyed,thiswouldpreventtraceroutefromworkingproperly.
HowtoInterpretDNSinaTracerouteByRichardSteenbergen,nLayerCommunications,Inc.
9InterpretingDNSinaTracerouteInterpretingDNSinaTracerouteInterpretingDNSisoneofthemostusefulimportantaspectsofcorrectlyusingtraceroute.
Informationyoucandiscoverincludes:LocationIdentifiersInterfaceTypesandCapacitiesRouterTypeandRolesNetorkBondariesandRelationshipsNetworkBoundariesandRelationshipsByRichardSteenbergen,nLayerCommunications,Inc.
10InterpretingTraceroute-LocationInterpretingTracerouteLocationKnowingthegeographicallocationoftheroutersisanimportantfirststeptounderstandinganissue.
Toidentifyincorrect/suboptimalrouting.
Tohelpyouunderstandnetworkinterconnections.
Andeventoknowwhenthereisn'taproblematall,i.
e.
knowingwhenhighlatencyisjustifiedandwhenitisn'tknowingwhenhighlatencyisjustifiedandwhenitisnt.
Themostcommonlyusedlocationidentifiersare:IATAAirportCodesIATAAirportCodesCLLICodesAttemptstoabbreviatebasedonthecitynameAttemptstoabbreviatebasedonthecityname.
Butsometimesyoujusthavetotakeaguess.
ByRichardSteenbergen,nLayerCommunications,Inc.
11LocationIdentifiers–IATAAirportCodesLocationIdentifiersIATAAirportCodesIATAAirportCodesGoodInternationalcoverageofmostlargecities.
MostcommoninnetworkswithafewbigPOPs.
Examples:SantoDomingo=SDQSanJoseCalifornia=SJCSometimesrepresentedbypseudo-airportcodesEspeciallywheremultipleairportsservearegionOhthitdiititiOrwheretheairportcodeisnon-intuitiveNewYork,NYisservedbyJFK,LGA,andEWRairports.
ButisfrequentlywrittenasNYC.
NthVAidbIADWhitDCbDCANorthernVAisservedbyIAD,WashingtonDCbyDCA.
ButbothmaybewrittenasWDC.
ByRichardSteenbergen,nLayerCommunications,Inc.
12LocationIdentifiers–CLLICodesLocationIdentifiersCLLICodesCommonLanguageLocationIdentifierCommonLanguageLocationIdentifierFullcodesmaintained(andsold)byTelecordia.
MostcommonlyusedbyTelephoneCompaniesyyppExample:HSTNTXMOCG0Inanon-Telcorole,mayonlyusethecity/stateidentifiersExamples:HSTNTX=HoustonTexasASBNVA=AshburnVirginiaWelldefinedstandardcoveringalmostallUS/CAcitiesCommonlyseeninnetworkswithalargernumberofPOPs.
NotanactualstandardoutsideofNorthAmericaNotanactualstandardoutsideofNorthAmericaSomeprovidersfudgethese,e.
g.
AMSTNL=AmsterdamNLByRichardSteenbergen,nLayerCommunications,Inc.
13LocationIdentifiers–ArbitraryValuesLocationIdentifiersArbitraryValuesAndthensometimespeoplejustmakestuffupAndthensometimespeoplejustmakestuffupChicagoILAirportCode:ORD(O'Hare)orMDW(Midway)CLLICode:CHCGILExampleArbitraryCode:CHITorontoONTorontoONAirportCode:YYZorYTCCLLICode:TOROONExampleArbitraryCode:TORFrequentlybasedonthegoodintentionsofmakingthingreadableinplainEnglisheventhoughthesethingreadableinplainEnglish,eventhoughthesemaynotfollowanystandards.
ByRichardSteenbergen,nLayerCommunications,Inc.
14CommonLocations–USMajorCitiesCommonLocationsUSMajorCitiesLocationNameAirportCodesCLLICodeOtherCodesAshburnVAIADASBNVAWDC,DCAAtlantaGAATLATLNGAChicagoILORD,MDWCHCGILCHIgDallasTXDFWDLLSTXDALHoustonTXIAHHSTNTXHOULosAngelesCALAXLSANCALALosAngelesCALAXLSANCALAMiamiFLMIAMIAMFLNewarkNJEWRNWRKNJNEW,NWKNewYorkNYJFK,LGANYCMNYNYC,NYMSanJoseCASJCSNJSCASJO,SV,SFPaloAltoCAPAOPLALCAPAIX,PA,SeattleCASEASTTLWAByRichardSteenbergen,nLayerCommunications,Inc.
15CommonLocations–Non-USMajorCitiesCommonLocationsNonUSMajorCitiesLocationNameAirportCodesCLLICode(*)OtherCodesAmsterdamNLAMSAMSTNLFrankfurtGEFRAFRNKGEHongKongHKHKGNEWTHKggLondonUKLHRLONDENLONMadridSPMADMDRDSPMontrealCAYULMTRLPQMTLMontrealCAYULMTRLPQMTLParisFRCDGPARSFRPARSingaporeSGSINSNGPSISeoulKRGMP,ICNSEOLKOSELSydneyAUSYDSYDNAUTokyoJPNRTTOKYJPTYOyTorontoCAYYZ,YTCTOROONTORByRichardSteenbergen,nLayerCommunications,Inc.
16InterpretingDNS–InterfaceTypesInterpretingDNSInterfaceTypesMostnetworkswilltrytoputinterfaceinfoinDNSOftentohelpthemtroubleshoottheirownnetworks.
ThhhilbdThoughthismanynotalwaysbeuptodate.
ManylargenetworksuseautomaticallygeneratedDNS.
CanpotentiallyhelpyouidentifythetypeofinterfaceCanpotentiallyhelpyouidentifythetypeofinterfaceAswellascapacity,andmaybeeventhemake/modelofrouter.
Examples:pxe-11-1-0.
edge1.
NewYork1.
Level3.
netXE-#/#/#isJuniper10GEport.
Thedevicehasatleast12slots.
G/GCIt'satleasta40G/slotroutersinceithasa10GEPICinslot1.
ItmustbeJuniperMX960,nootherdevicecouldfitthisprofile.
ByRichardSteenbergen,nLayerCommunications,Inc.
17CommonInterfaceNamingConventionsCommonInterfaceNamingConventionsInterfaceTypeCiscoIOSCiscoIOSXRJuniperFastEthernetFa#/#fe-#/#/#GigabitEthernetGi#/#Gi#/#/#/#ge-#/#/#10GigabitEthernetTe#/#Te#/#/#/#xe-#/#/#gSONETPos#/#POS#/#/#/#so-#/#/#T1Se#/#t1-#/#/#T3t3#/#/#T3t3-#/#/#EthernetBundlePo#/Port-channel#BE####ae#SONETBundlePosCh#BS####as#TunnelTu#TT#orTI#ip-#/#/#orgr-#/#/#ATMATM#/#AT#/#/#/#at-#/#/#VlanVl###Gi#ge-gByRichardSteenbergen,nLayerCommunications,Inc.
18InterpretingDNS–RouterTypes/RolesInterpretingDNSRouterTypes/RolesKnowingtheroleofaroutercanbeusefulKnowingtheroleofaroutercanbeusefulButeverynetworkisdifferent,andusesdifferentnamingconventions.
Andjusttobeextraconfusion,theydon'talwaysfollowtheirownnamingrules.
GllkithttGenerallyspeaking,youcanguessthecontextandgetabasicunderstandingoftheroles.
Corerouters–CR,Core,GBR,BB,CCR,EBRPeeringrouters–BR,Border,Edge,IR,IGR,PeerCustomerroutersARAggrCustCARHSAGWCustomerrouters–AR,Aggr,Cust,CAR,HSA,GWByRichardSteenbergen,nLayerCommunications,Inc.
19NetworkBoundariesandRelationshipsNetworkBoundariesandRelationshipsIdentifyingNetworkBoundariesisImportantIdentifyingNetworkBoundariesisImportantThesetendtobewhereroutingpolicychangesoccur.
Forexample,differentreturnpathsbasedonLocalPreference.
Thesealsotendtobeareaswherecapacityandroutingarethemostdifficult,thuslikelytobeproblems.
IdtifithltihibhlfltIdentifyingtherelationshipcanbehelpfultooTypically:a)TransitProvider,b)Peer,orc)Customer.
MtkillttiditdithiDNSManynetworkswilltrytoindicatedemarcsintheirDNSExamples:Clearnameslikenetwork.
customer.
alter.
netOralwayslandingcustomersonroutersnamed"gw"ByRichardSteenbergen,nLayerCommunications,Inc.
20NetworkBoundariesandRelationshipsNetworkBoundariesandRelationshipsIt'seasytospotwheretheDNSchanges4te1-2-10g.
ar3.
DCA3.
gblx.
net(67.
17.
108.
146)5sl-st21-ash-8-0-0.
sprintlink.
net(144.
232.
18.
65)Or,lookfor"remoteparty"nameintheDNS4po2-20G.
ar5.
DCA3.
gblx.
net(67.
16.
133.
90)5cogent-1.
ar5.
DCA3.
gblx.
net(64.
212.
107.
90)Commonwhereonesidecontrolsthe/30DNS,andtheothersidedoesn'tprovideinterfaceinformationtheothersidedoesntprovideinterfaceinformation.
Formoreinfo,lookattheothersideofthe/30>nslookup6421210789>nslookup64.
212.
107.
89Result:te2-3-10GE.
ar5.
DCA3.
gblx.
netByRichardSteenbergen,nLayerCommunications,Inc.
21UnderstandingNetworkLatencyByRichardSteenbergen,nLayerCommunications,Inc.
22UnderstandingNetworkLatencyUnderstandingNetworkLatencyThreeprimarytypesofnetworkinducedlatencyThreeprimarytypesofnetworkinducedlatencySerializationDelayThedelaycausedbyhavingtotransmitdatathroughThedelaycausedbyhavingtotransmitdatathroughrouters/switchesinpacketsizedchunks.
QueuingDelayThetimespentinarouter'squeueswaitingfortransmissionThisThetimespentinarouter'squeueswaitingfortransmission.
Thisismostlyrelatedtolinecontention(fullinterfaces),sincewithoutcongestionthereisverylittleneedforameasurablequeue.
PtiDlPropagationDelayThetimespent"inflight",inwhichthesignalistravelingoverthetransmissionmedium.
Thisisprimarilyalimitationbasedonthespeedoflight,orotherelectromagneticpropagation.
ByRichardSteenbergen,nLayerCommunications,Inc.
23Latency–SerializationDelayLatencySerializationDelayDelaycausedbypacketbasedforwardingDelaycausedbypacket-basedforwardingPacketsmovethroughthenetworkasasingleunit.
Can'ttransmitthenextpacketuntillastoneisfinishedCanttransmitthenextpacketuntillastoneisfinished.
NotmuchasanissueinmodernnetworksSpeedshaveincreasedbyordersofmagnitudeovertheSpeedshaveincreasedbyordersofmagnitudeovertheyears,whilepacketsizeshavestayedthesame(small).
1500bytesovera56klink(56Kbps)=214.
2msdelay1500bytesoveraT1(1.
536Mbps)=7.
8msdelay1500bytesoveraFastE(100Mbps)=0.
12msdelay1500bytesoveraGigE(1Gbps)=0.
012msdelayByRichardSteenbergen,nLayerCommunications,Inc.
24Latency–QueuingDelayLatencyQueuingDelayFirstyoumustunderstand"Utilization"A1GEdoing500Mbpsissaidtobe"50%utilized"Butinreality,aninterfaceiseithertransmitting(100%utilized)ornottransmitting(0%utilized)atanyinstantutilized)ornottransmitting(0%utilized)atanyinstantTheaboveisreally"used50%ofthetime,over1second"QueueingisanaturalfunctionofroutersQueueingisanaturalfunctionofroutersWhenapacketisreadytosendbuttheinterfaceisinuse,itmustbequeueduntiltheinterfaceisfree.
qAsaninterfacereachessaturation,theprobabilityofapacketbeingqueuedrisesexponentially.
Whenaninterfaceisextremelyfull,apacketmaybequeuedformanyhundredsorthousandsofmiliseconds.
ByRichardSteenbergen,nLayerCommunications,Inc.
25Latency–PropagationDelayLatencyPropagationDelayDldbiltiditDelaycausedbysignalpropagationoverdistance.
Lighttravelsthroughavacuumat~300,000km/secFibercoreshavearefractiveindexof148Fibercoreshavearefractiveindexof~1.
481/1.
48=~0.
67c,lightthroughfiber=~200,000km/sec200000km/sec=200km(or125miles)permillisecond200,000km/sec200km(or125miles)permillisecond.
Divideby2forround-triptime(RTT)measurements.
Example:Example:Around-triparoundtheworldattheequator,viaaperfectlystraightfiberroute,wouldtake~400msduesolelytospeed-of-lightpropagationdelays.
ByRichardSteenbergen,nLayerCommunications,Inc.
26IdentifyingtheLatencyAffectingYouIdentifyingtheLatencyAffectingYouSo,howdoyoudetermineiflatencyisnormalUselocationidentifierstodeterminegeographicaldata.
Seeifthelatencyfitswithpropagationdelay.
FlForexample:3xe-3-0-0.
cr1.
nyc3.
us.
nlayer.
net(69.
22.
142.
74)6.
570ms4xe-0-0-0.
cr1.
lhr1.
uk.
nlayer.
net(69.
22.
142.
10)74.
144msy()NewYorkNYtoLondonUKin67.
6ms4200milesYup!
Anotherexample:5cr2wswdcipattnet(12122338)[MPLS:Label17221Exp0]8msec8msec8msec5cr2.
wswdc.
ip.
att.
net(12.
122.
3.
38)[MPLS:Label17221Exp0]8msec8msec8msec6tbr2.
wswdc.
ip.
att.
net(12.
122.
16.
102)[MPLS:Label32760Exp0]8msec8msec8msec7ggr3.
wswdc.
ip.
att.
net(12.
122.
80.
69)8msec8msec8msec8192.
205.
34.
106[AS7018]228msec228msec228msec9t14d01id01tlt(154543222)[AS174]2282282289te1-4.
mpd01.
iad01.
atlas.
cogentco.
com(154.
54.
3.
222)[AS174]228msec228msec228msecWashingtonDCtoWashingtonDCin220msNope!
ByRichardSteenbergen,nLayerCommunications,Inc.
27PrioritizationandRateLimitingByRichardSteenbergen,nLayerCommunications,Inc.
28"ToIt"vs.
"ThroughIt"ToItvs.
ThroughItArchitectureofamodernrouter:Packetsforwardedthroughtherouter(dataplane)FastPath:hardwarebasedforwardingofordinarypacketsExample:AlmosteverypacketinnormalInternettraffic.
SlowPath:softwarebasedhandlingof"exception"packetsExample:IPOptions,ICMPGeneration(includingTTLExceeded)PacketsbeingforwardedTOtherouter(controlplane)Example:BGP,IGP,SNMP,CLIaccess(telnet/ssh),ping,oranypacketssentdirectlytoalocalIPaddressontherouter.
anypacketssentdirectlytoalocalIPaddressontherouter.
TheseCPUstendtoberelativelyunderpoweredA320-640+Gbpsroutermayonlyhavea600MHzCPUICMPGenerationis*NOT*apriorityfortherouter.
ByRichardSteenbergen,nLayerCommunications,Inc.
29TheInfamousBGPScannerTheInfamousBGPScannerOnmanyplatformstheslow-pathdataplaneandypppthecontrol-planesharethesameresources.
Andoftendon'thavethebestschedulersfortheCPUAsaresult,control-planeactivitysuchasBGPchurn,CLIuse,andperiodicsoftwareprocessescanconsumeCPUandslowthegenerationofICMPTTLExceedsCPUandslowthegenerationofICMPTTLExceeds.
Thisresultsinrandom"spikes"intraceroutelatency,whichisoftenmisinterpretedasanetworkissue.
pThemostinfamousprocesswhichcausesthesespikesiscalled"BGPScanner",andrunsevery60p,ysecondsonallCiscoIOSdevices.
ByRichardSteenbergen,nLayerCommunications,Inc.
30RateLimitedICMPGenerationRateLimitedICMPGenerationMostroutersalsoratelimittheirICMPgenerationOftenwitharbitraryhard-codedlimits.
Whichmaybeinsufficientunderheavytracerouteload.
JuniperHardlimitof50ppsperinterface,250ppsonFPC3sHardlimitof500ppsperPFEasofJUNOS8.
3+FoundryHardlimitof400ppsperinterfaceForce10Hardlimitof200ppsor600ppsperinterfaceByRichardSteenbergen,nLayerCommunications,Inc.
31SpottingTheFakeLatencySpottingTheFakeLatencyThemostimportantruleofallpIfthereisanactualissue,thelatencywillcontinueorincreaseforallfuturehops:Example(Notarealissueinhop2):1ae3.
cr2.
iad1.
us.
nlayer.
net0.
275ms0.
264ms0.
137ms2xe-1-2-0.
cr1.
ord1.
us.
nlayer.
net18.
271ms18.
257ms68.
001ms3tge2-1.
ar1.
slc1.
us.
nlayer.
net53.
373ms53.
213ms53.
227Latencyspikesinthemiddleofatraceroutemeanabsolutelynothingiftheydonotcontinueforwardabsolutelynothingiftheydonotcontinueforward.
Atworstitcouldbetheresultofanasymmetricpath.
Butitisprobablyanartificialrate-limitorprioritizationissue.
BdfiitiifllfddktbifftdBydefinition,ifregularlyforwardedpacketsarebeingaffectedyoushouldseetheissuepersistonallfuturehops.
ByRichardSteenbergen,nLayerCommunications,Inc.
32AsymmetricPathsByRichardSteenbergen,nLayerCommunications,Inc.
33AsymmetricPathsAsymmetricPathsThenumberoneplagueoftraceroutepgTracerouteshowsyoutheforwardpathonlyButthelatencyshownforeachhopisbasedonButthelatencyshownforeachhopisbasedonThetimeittookfortheprobepackettoreachthehop,PLUSThetimeittookfortheTTLExceedreplytocomeback.
ThereversepathitselfiscompletelyinvisibleNotonlydoestraceroutenotrevealanythingaboutit,but…ItcanbecompletelydifferentateveryhopintheforwardpathItcanbecompletelydifferentateveryhopintheforwardpath.
TheonlysolutionistolookatbothforwardandreversetraceroutesAdthit'tthttiltithiAndeventhen,itcan'tcatchpotentialasymmetricpathsinthemiddle.
ByRichardSteenbergen,nLayerCommunications,Inc.
34AsymmetricPathsandNetworkBoundariesAsymmetricPathsandNetworkBoundariesAsymmetricpathsoftenstartatnetworkboundariesWhyBecausethatiswhereadminpolicieschange.
te1-1.
ar2.
DCA3.
gblx.
net(69.
31.
31.
209)0.
719ms0.
560ms0.
428mste1-2-10g.
ar3.
DCA3.
gblx.
net(67.
17.
108.
146)0.
574ms0.
557ms0.
576mste1210g.
ar3.
DCA3.
gblx.
net(67.
17.
108.
146)0.
574ms0.
557ms0.
576mssl-st21-ash-8-0-0.
sprintlink.
net(144.
232.
18.
65)100.
280ms100.
265ms100.
282ms144.
232.
20.
149(144.
232.
20.
149)102.
037ms101.
876ms101.
892mssl-bb20-dc-15-0-0.
sprintlink.
net(144.
232.
15.
0)101.
888ms101.
876ms101.
890msWhat'swronginthepathaboveItCOULDbecongestionbetweenGBLXandSprint.
ButitcouldalsobeanasymmetricreversepathButitcouldalsobeanasymmetricreversepath.
AtthisGBLX/Sprintboundary,thereversepathpolicychanges.
Thisisoftenseeninmulti-homednetworkwithmultiplepaths.
Intheexampleabove,Sprint'sreverseroutegoesviaacircuitthatiscongested,butthatcircuitisNOTshowninthetraceroute.
ByRichardSteenbergen,nLayerCommunications,Inc.
35UsingSourceAddressinyourTracerouteUsingSourceAddressinyourTracerouteHowcanyouworkaroundasymmetricpathsyypThemostpowerfuloptionistocontrolyourSRCaddress.
Inthepreviousexample,assumethat:Youaremulti-homedtoGlobalCrossingandLevel3GlobalCrossingreachesyouviaGlobalCrossingSprintreachesyouviaLevel3SprintreachesyouviaLevel3ThereisaproblembetweenSprintandLevel3.
Howcanyouprovetheissueisn'tbetweenGXandSprintRunatracerouteusingyoursideoftheGBLX/30asyoursource.
This/30comesfromyourprovider(GBLX)'slargeraggregate.
ThereversepathwillbeguaranteedtogoSprint->GBLXpggpIfthelatencydoesn'tpersist,youknowtheissueisonthereverse.
ByRichardSteenbergen,nLayerCommunications,Inc.
36AsymmetricPathsAsymmetricPathsButremember,asymmetricpathscanhappenanywhereButremember,asymmetricpathscanhappenanywhereEspeciallywherenetworksconnectinmultiplelocationsAnduseclosest-exit(hotpotato)routing,asistypicallydone.
dusecosestet(otpotato)outg,asstypcaydoeHop1(red)returnsviaaChicagointerconnectionHop2(green)returnsviaaSanJoseinterconnectionChicagoILByRichardSteenbergen,nLayerCommunications,Inc.
37WashingtonDCSanJoseCAUsingSourceAddressinyourTracerouteUsingSourceAddressinyourTracerouteButwhatifthe/30isnumberedoutofmyspaceButwhatifthe/30isnumberedoutofmyspaceAsinthecaseofacustomerorpotentiallyapeer.
YoucanstillseesomebenefitsfromsettingSRCsYoucanstillseesomebenefitsfromsettingSRCsConsidertryingtoexaminethereversepathofapeerwhoyouhavemultipleinterconnectionpointswith.
AtraceroutesourcedfromyourIPspace(suchasaloopback)maycomebackviaanyofmultipleinterconnectionpoints.
Butiftheremotenetworkcarriesthe/30sofyourinterconnectionyintheirIGP(i.
e.
theyredistributeconnectedintotheirIGP)…Thenthetrafficwillcomebackovertheirbackbone,andreturntoyouviathe/30youaretestingfrom.
yygTryingbothoptionscangiveyoudifferentviewpoints.
ByRichardSteenbergen,nLayerCommunications,Inc.
38DefaultSourceAddressesDefaultSourceAddressesWhentraceroutingfromarouterWhentraceroutingfromarouter…Mostroutersdefaulttousingthesourceaddressoftheegressinterfacethattheprobeleavesfrom.
gpThismayormaynotbewhatyouwanttosee.
Someplatformscanbeconfiguredtodefaulttoaloopbackaddressratherthantheegressinterface.
Forexample,Juniper.
ByRichardSteenbergen,nLayerCommunications,Inc.
39MultiplePathsandLoadBalancingByRichardSteenbergen,nLayerCommunications,Inc.
40MultiplePathsMultiplePathsBecauseeach(UDP/TCP)tracerouteprobeusesadifferentlayer4port,equal-costmulti-pathmaymakemultiplepathsshowupwithinasingle"hop"Thisisrelativelyharmless,butmaybeconfusing.
Example:6ldbb2liktlit(809125114)74139741266ldn-bb2-link.
telia.
net(80.
91.
251.
14)74.
139ms74.
126msldn-bb1-link.
telia.
net(80.
91.
249.
77)74.
144ms7hbg-bb1-link.
telia.
net(80.
91.
249.
11)89.
773mshbg-bb2-link.
telia.
net(80.
91.
250.
150)88.
459ms88.
456ms8s-bb2-link.
telia.
net(80.
91.
249.
13)105.
002mss-bb2-linktelianet(80239147169)102647ms102501mssbb2link.
telia.
net(80.
239.
147.
169)102.
647ms102.
501msOfthe3probes,2gooveronepath,1goesoveranother.
ByRichardSteenbergen,nLayerCommunications,Inc.
41MultiplePaths-ExamplesMultiplePathsExamplesAslightlymorecomplexexample4p16-1-0-0.
r21.
asbnva01.
us.
bb.
verio.
net(129.
250.
5.
21)0.
571ms0.
604ms0.
594msp()5p16-1-2-2.
r21.
nycmny01.
us.
bb.
verio.
net(129.
250.
4.
26)7.
279ms7.
260msp16-4-0-0.
r00.
chcgil06.
us.
bb.
verio.
net(129.
250.
5.
102)25.
981ms6p16-2-0-0.
r21.
sttlwa01.
us.
bb.
verio.
net(129.
250.
2.
180)71.
027msp16-1-1-3.
r20.
sttlwa01.
us.
bb.
verio.
net(129.
250.
2.
6)66.
730ms66.
535msECMPbetweentwoparallelbutdifferentpathsECMPbetweentwoparallelbutdifferentpathsAshburnVA–NewYorkNY–SeattleWAAshburnVA–ChicagoIL–SeattleWAAlsoharmless,butpotentiallyconfusing.
ByRichardSteenbergen,nLayerCommunications,Inc.
42MultipleUnequalLengthPathsMultipleUnequalLengthPathsAmuchworsescenarioisECMPwheretheload-balancedpathsareofunequalhoplength.
Thiscanmakethetracerouteappeartogobackandforth,andisextremelyconfusinganddifficulttoread.
TraceroutehopsenduplookinglikethisTraceroutehopsenduplookinglikethis1AAA2BXB3CBC4DCD5EDEByRichardSteenbergen,nLayerCommunications,Inc.
43HandlingMultiplePathsHandlingMultiplePathsWhenindoubt,onlylookatasinglepathSetyourtracerouteclienttoonlysendasingleprobe.
BbhhibhhhihButbeawarethatthismaynotbethepathwhichyouractualtrafficforwardsover.
OnewaytotryoutdifferentpathswhichmaybeavailableisyypytoincrementthedestIPby1ortrydifferentsourceIPs.
ByRichardSteenbergen,nLayerCommunications,Inc.
44MPLSandTracerouteMPLSandTracerouteByRichardSteenbergen,nLayerCommunications,Inc.
45MPLSICMPTunnelingMPLSICMPTunnelingManylargenetworksoperateanMPLSbasedcoreManylargenetworksoperateanMPLSbasedcoreSomedevicesdon'tevencarryanIProutingtableThisisfineforswitchingMPLSlabeledpacketsThisisfineforswitchingMPLSlabeledpacketsButpresentsaproblemwhenICMPsaregeneratedHowdoestheMPLS-onlyrouterdeliveranICMPHowdoestheMPLSonlyrouterdeliveranICMPOnesolutioniscalledICMPTunnelingIfgeneratinganICMPaboutapacketinsideanLSPIfgeneratinganICMPaboutapacketinsideanLSPThenputthegeneratedICMPbackintothesameLSPWorksfordeliveringthemessage,but…ItcanmaketracerouteslookreallyWEIRD!
ByRichardSteenbergen,nLayerCommunications,Inc.
46MPLSICMPTunnelingDiagramMPLSICMPTunnelingDiagramICMPDestUnreachICMPTTLExceedTTL=1TTL=2TTL=3TTL=4TTL=5ICMPTTLExceedICMPTTLExceedICMPTTLExceedICMPTTLExceedSRCRouter1Router2Router3Router4DSTTTL=1TTL=2TTL=3TTL=4TTL=5AllreturnedICMPpacketsmusttraveltotheendoftheLSPbeforegoingbacktothesenderICMPDestUnreachICMPTTLExceedICMPTTLExceedICMPTTLExceedICMPTTLExceedAllreturnedICMPpacketsmusttraveltotheendoftheLSPbeforegoingbacktothesender.
ThismakeseveryhopintheLSPappeartohavethesameRTTasthefinalhop.
TTL=1TTL=2TTL=3TTL=4TTL=5CCByRichardSteenbergen,nLayerCommunications,Inc.
47SRCRouter1Router2Router3Router4DST35MPLSICMPTunnelingExampleMPLSICMPTunnelingExample1.
te2-4.
ar5.
PAO2.
gblx.
net(69.
22.
153.
209)1.
160ms1.
060ms1.
029ms2.
192.
205.
34.
245(192.
205.
34.
245)3.
984ms3.
810ms3.
786ms3.
tbr1.
sffca.
ip.
att.
net(12.
123.
12.
25)74.
848ms74.
859ms74.
936ms4cr1sffcaipattnet(12122191)74344ms74612ms74072ms4.
cr1.
sffca.
ip.
att.
net(12.
122.
19.
1)74.
344ms74.
612ms74.
072ms5.
cr1.
cgcil.
ip.
att.
net(12.
122.
4.
122)74.
827ms75.
061ms74.
640ms6.
cr2.
cgcil.
ip.
att.
net(12.
122.
2.
54)75.
279ms74.
839ms75.
238ms7.
cr1.
n54ny.
ip.
att.
net(12.
122.
1.
1)74.
667ms74.
501ms77.
266ms8.
gbr7.
n54ny.
ip.
att.
net(12.
122.
4.
133)74.
443ms74.
357ms75.
397ms9.
ar3.
n54ny.
ip.
att.
net(12.
123.
0.
77)74.
648ms74.
369ms74.
415ms9.
ar3.
n54ny.
ip.
att.
net(12.
123.
0.
77)74.
648ms74.
369ms74.
415ms10.
12.
126.
0.
29(12.
126.
0.
29)76.
104ms76.
283ms76.
174ms11.
route-server.
cbbtier3.
att.
net(12.
0.
1.
28)74.
360ms74.
303ms74.
272msByRichardSteenbergen,nLayerCommunications,Inc.
48Sendquestions,complaints,to:RichardASteenbergen