pressesservicetemporarilyunavailable

CiscoSystems,Inc.
www.
cisco.
comCiscohasmorethan200officesworldwide.
Addresses,phonenumbers,andfaxnumbersarelistedontheCiscowebsiteatwww.
cisco.
com/go/offices.
CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideSoftwareRelease11.
0LastRevised:June10,2016Senddocumentationfeedbacktoscc-docfeedback@cisco.
comTHESPECIFICATIONSANDINFORMATIONREGARDINGTHEPRODUCTSINTHISMANUALARESUBJECTTOCHANGEWITHOUTNOTICE.
ALLSTATEMENTS,INFORMATION,ANDRECOMMENDATIONSINTHISMANUALAREBELIEVEDTOBEACCURATEBUTAREPRESENTEDWITHOUTWARRANTYOFANYKIND,EXPRESSORIMPLIED.
USERSMUSTTAKEFULLRESPONSIBILITYFORTHEIRAPPLICATIONOFANYPRODUCTS.
THESOFTWARELICENSEANDLIMITEDWARRANTYFORTHEACCOMPANYINGPRODUCTARESETFORTHINTHEINFORMATIONPACKETTHATSHIPPEDWITHTHEPRODUCTANDAREINCORPORATEDHEREINBYTHISREFERENCE.
IFYOUAREUNABLETOLOCATETHESOFTWARELICENSEORLIMITEDWARRANTY,CONTACTYOURCISCOREPRESENTATIVEFORACOPY.
TheCiscoimplementationofTCPheadercompressionisanadaptationofaprogramdevelopedbytheUniversityofCalifornia,Berkeley(UCB)aspartofUCB'spublicdomainversionoftheUNIXoperatingsystem.
Allrightsreserved.
Copyright1981,RegentsoftheUniversityofCalifornia.
NOTWITHSTANDINGANYOTHERWARRANTYHEREIN,ALLDOCUMENTFILESANDSOFTWAREOFTHESESUPPLIERSAREPROVIDED"ASIS"WITHALLFAULTS.
CISCOANDTHEABOVE-NAMEDSUPPLIERSDISCLAIMALLWARRANTIES,EXPRESSEDORIMPLIED,INCLUDING,WITHOUTLIMITATION,THOSEOFMERCHANTABILITY,FITNESSFORAPARTICULARPURPOSEANDNONINFRINGEMENTORARISINGFROMACOURSEOFDEALING,USAGE,ORTRADEPRACTICE.
INNOEVENTSHALLCISCOORITSSUPPLIERSBELIABLEFORANYINDIRECT,SPECIAL,CONSEQUENTIAL,ORINCIDENTALDAMAGES,INCLUDING,WITHOUTLIMITATION,LOSTPROFITSORLOSSORDAMAGETODATAARISINGOUTOFTHEUSEORINABILITYTOUSETHISMANUAL,EVENIFCISCOORITSSUPPLIERSHAVEBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES.
CiscoandtheCiscologoaretrademarksorregisteredtrademarksofCiscoand/oritsaffiliatesintheU.
S.
andothercountries.
ToviewalistofCiscotrademarks,gotothisURL:www.
cisco.
com/go/trademarks.
Third-partytrademarksmentionedarethepropertyoftheirrespectiveowners.
TheuseofthewordpartnerdoesnotimplyapartnershiprelationshipbetweenCiscoandanyothercompany.
(1110R)AnyInternetProtocol(IP)addressesusedinthisdocumentarenotintendedtobeactualaddresses.
Anyexamples,commanddisplayoutput,andfiguresincludedinthedocumentareshownforillustrativepurposesonly.
AnyuseofactualIPaddressesinillustrativecontentisunintentionalandcoincidental.
2016CiscoSystems,Inc.
Allrightsreserved.
1CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideCONTENTS1Preface5Overview5Audience5Purpose5Organization6RelatedDocumentation6CHAPTER1REMTroubleshooting1-1ChapterOverview1-1Introduction1-2Prerequisites1-2ComponentsUsed1-2Conventions1-2TroubleshootingMethodology1-2PossibleErrors&TroubleshootingTips1-4UnabletoBringUpKioskFlexApplicationontheCustomerPod'sHomePage1-5UnabletoEstablishCallfromCustomerPod('SystemError'Message)1-7CustomerPodDisplays'Error:ManagementServerisnotreachable'Message1-8CustomerPodDisplays'StartupURLisnotconfigured'Message1-8IECDisplays'Cannotregister'Message1-9CustomerPodTouchScreenFunctionisNotWorkingCorrectly1-9CustomerPodDisplaysVirtualKeyboardWhenCallConnects1-9CustomerPodDisplays'Serverisdown'Message1-10CustomerPodDisplays'ServiceTemporarilyUnavailable'Message1-10CustomerPodDisplays'Kioskisnotregistered'Message1-11ConnectedPeripheralsareNotDetectedbytheIEC1-11IECisNotReflectingtheAppliedPolicy1-11IEC'sProfileConfigurationisNotActive1-11CustomerPodDisplays'Systemisnotavailable,Pleasetryaftersometime'Message1-12CustomerPodDisplays'Expertnotavailable'Message1-13CustomerPodDisplaysRequesttoProvideFeedbackAfterPressingtheConnectButton1-13Contents2CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideCustomerPodDoesNotDisplaytheHomePage1-14IECRebootsTwice1-14UnabletoSeeWaitVideoWhileCallisintheQueue1-15UnabletoSeeOn-HoldVideoWhileCallisOnHold1-15UnabletoBringUpStaticGraphicPageinTP(DuringNon-TPCalls)1-15UnabletoUploadandUpdateImagestoREM1-16READisNotShowingUp1-16eREADDocumentPanelHasChangedItsSize1-17VideoCallisNotEstablishedBetweenDifferentTypesofTelePresenceVideoEndpoints1-17ExpertsarenotGettingRegisteredinREAC1-17REIC(CobraBrowser)Hangs1-18SessionHangs1-18REMErrorMessage1-18EX90FirmwareErrorMessage1-18TroubleshootingGuidelinesforIECasVideoEndpoint1-18SIPVideoCallisNotEstablishedBetweenIECVideoEndpointandAgent1-18TroubleshootingGuidelinesforRemoteExpertApplicationsRunningonVMware1-19vCenterSettings1-19VMwarePerformanceIndicators1-19CPUTroubleshooting1-20MemoryTroubleshooting1-20DiskTroubleshooting1-21NetworkTroubleshooting1-21CPUOversubscriptionImplicationsonPerformance1-22OtherConsiderations1-22InformationtoHelpwithTroubleshooting1-22REEventandRESCLogs1-22NormalIECBootupSequence1-23NormalCallFlowanditsAssociatedresc.
log1-25TraceFilesforTAC1-28CHAPTER2ServiceabilityAdministration2-1ChapterOverview2-1ServiceabilityOverview2-2Monitoring2-2ErrorsandAlertsfromREAC2-2ErrorsandAlertsfromREAD2-4ErrorsandAlertsfromREIC2-6REMServer(RESC)Errors2-8Contents3CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideLogMonitoring2-8EventsTabintheIEM2-9CallRelatedErrors2-9JTAPIIssues2-10AgentRegistrationWhenDoneFromREACShowsaRedStatus2-10REICDisplaystheErrorMessage"Systemerror,pleasetryaftersometime"2-10CustomerPodDisplays"ExpertnotAvailable"Message2-10CustomerPodDisplays"Kioskphoneisoutofservice"Message2-11ExpertRegistrationIssue2-11RsyncCommandFailure2-11REMNodeFailure2-12CommunicationFailure2-12CallConnect2-13CallDisconnect2-13CallHoldandReleasefromHold2-13CallTransfer2-14CallConference2-14WebService2-15DualContentIssue2-16FailureofRESCWeb(API)Services2-16VerifyingVMSettings2-17MasterConfiguration2-17NTP2-17NetworkConfiguration2-18EffectonDatabase2-19Document2-20CallSession2-20Video2-20REMEvents2-20LogFiles2-21TUITroubleshootingTools2-21Ping2-21System,RE,andApplicationServerLogs2-21TroubleshootingToolsinREAC2-22Contents4CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuide5CiscoRemoteExpertManagerTroubleshooting&ServiceabilityGuideOL-xxxxx-xxPrefaceNoteAlladvertisingmaterialsmentioningfeaturesoruseofthissoftwaremustdisplaythefollowingacknowledgment:"ThisproductincludessoftwaredevelopedbytheUniversityofCalifornia,Berkeleyanditscontributors.
"OverviewThisprefacedescribestheaudience,organization,andconventionsofthisguide.
Italsoprovidesinformationonrelateddocumentation.
Thisprefaceincludesthefollowingsections:"Audience""Purpose""Organization""RelatedDocumentation"AudienceThisguideisintendedforcustomers,partners,andthefieldteamwhowilltroubleshootthehardwareandsoftwareatthedatacenter,contactcenter,andbranches.
Itisalsointendedforadministratorswhowillkeeptheapplicationupandrunning.
PurposeThisguideprovidestheinformationthatyouneedtotroubleshootandmaintainserviceabilityfortheCiscoRemoteExpertManager(REM).
6CiscoRemoteExpertManagerTroubleshooting&ServiceabilityGuideOL-xxxxx-xxPrefaceOrganizationOrganizationThisguideisorganizedintothefollowingchapters:RelatedDocumentationThesedocumentsprovideadditionalinformationabouttheCiscoRemoteExpertSmartsolution:CiscoRemoteExpertManagerInstallationGuidehttp://www.
cisco.
com/c/en/us/support/customer-collaboration/remote-expert-manager/products-installation-guides-list.
htmlCiscoRemoteExpertManagerPortUsageGuidehttp://www.
cisco.
com/c/en/us/support/customer-collaboration/remote-expert-manager/products-installation-guides-list.
htmlCiscoRemoteExpertManagerAdministrationGuidehttp://www.
cisco.
com/c/en/us/support/customer-collaboration/remote-expert-manager/products-maintenance-guides-list.
htmlCiscoRemoteExpertManagereREADUserGuidehttp://www.
cisco.
com/c/en/us/support/customer-collaboration/remote-expert-manager/products-user-guide-list.
htmlCiscoRemoteExpertManagerReleaseNoteshttp://www.
cisco.
com/c/en/us/support/customer-collaboration/remote-expert-manager/products-release-notes-list.
htmlChapterTitleDescription1REMTroubleshootingExplainshowtotroubleshootcommonproblems2ServiceabilityAdministrationAssistsadministratorswithmaintainingtheREMapplicationCHAPTER1-1CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuide1REMTroubleshootingJune10,2016ChapterOverviewThischapterexplainshowtotroubleshootcommonproblems.
Topicsinthischapterinclude:"Introduction""TroubleshootingMethodology""PossibleErrors&TroubleshootingTips""TroubleshootingGuidelinesforIECasVideoEndpoint""TroubleshootingGuidelinesforRemoteExpertApplicationsRunningonVMware"–"vCenterSettings"–"VMwarePerformanceIndicators"–"CPUTroubleshooting"–"MemoryTroubleshooting"–"DiskTroubleshooting"–"NetworkTroubleshooting"–"CPUOversubscriptionImplicationsonPerformance"–"OtherConsiderations""InformationtoHelpwithTroubleshooting"–"REEventandRESCLogs"–"NormalIECBootupSequence"–"NormalCallFlowanditsAssociatedresc.
log"–"TraceFilesforTAC"1-2CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingIntroductionIntroductionThisdocumentdiscussesandsolvesthemostcommonproblemsthathavebeenreportedduringtestingandalsofromthefield.
Thisdocumentalsoexplainsnecessarycallflowtohelpyouunderstandthelogictomaketroubleshootingeasier.
PrerequisitesThisdocumentassumesthatyouknowtheworkingofCiscoUnifiedCommunicationManager(UCM),CiscoUnifiedContactCenterExpress(UCCX),CiscoUnifiedContactCenterEnterprise(UCCE),andCiscoTelePresence(TP)videoendpoints.
ComponentsUsedThisdocumentisnotrestrictedtospecificsoftwareandhardwareversions.
ConventionsRefertoCiscoTechnicalTipsConventionsformoreinformationondocumentconventions:http://www.
cisco.
com/en/US/tech/tk801/tk36/technologies_tech_note09186a0080121ac5.
shtml.
TroubleshootingMethodologyInordertoisolatetheissueandresolveit,itisimportanttofollowamethodology.
Ahighleveltroubleshootingprocessflowisagoodwaytoisolateproblemcomponentsfromtheoverallsolution.
TherearemainlythreecomponentsinvolvedintheCiscoRemoteExpertSolution:1.
UnifiedCommunicationManageranditsendpoints2.
UnifiedContactCenterExpressandexperts3.
RemoteExpertManageranditscomponentsA"DivideandConquer"troubleshootingapproachissuggestedinordertoisolatetheproblemcomponentwithinthesolution.
Forexample,withoutusingthecustomerpod,determineiftheothercomponentsofthesolutionarefunctioningcorrectly.
Isitpossible,forexample,tomanuallyinitiateacallfromthebranchTPvideoendpointandsuccessfullyroutethecalltoanexpertviaContactCenterIftheansweris"no",thenatleastpartoftheproblemlieswithintheCUCM/UCCX/UCCEinfrastructure.
CorrectthatproblembeforedeterminingifREMtroubleshootingisnecessary.
ReviewtheflowchartbelowtotroubleshootUCMandUCCX/UCCEissues.
1-3CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingTroubleshootingMethodologyFigure1-1TroubleshootingFlowchartforUCMandUCCX/UCCEIssuesIfcustomer-side/branchsitemanuallyinitiatedcallcanbesuccessfullyroutedtoanexpertandansweredbyanexpert,butdoesnotworkfromthecustomerpod,thenitcouldbeaREM-relatedissue.
Inthatinstance,thefollowingflowchartmaybehelpful.
1-4CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsFigure1-2TroubleshootingFlowchartforREM-RelatedIssuesPossibleErrors&TroubleshootingTipsThissectiondocumentserrorsthatcustomersmayencounterandhowtofixthem.
Table1-1PossibleErrorsandReasonsPossibleErrorsComponentsInvolvedPossibleReasonsCustomerpodhasnodisplayMonitorIEC1.
Checkwhethermonitorisworkingandthepowerisconnected.
2.
IfmonitorwasconnectedaftertheIECwasbootedup,reboottheIEC.
3.
TryconnectingthemonitortoIEC'salternatevideoport(VGAorHDMI).
IECdisplays'Managementfailure:ProductVEPisnotfound'IECTheIEChasanolderversionoffirmware.
UpgradetheIEC'sfirmware.
1-5CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsAdditionalissuesandhowtosolvethemaredescribedbelow.
UnabletoBringUpKioskFlexApplicationontheCustomerPod'sHomePageThecallflowthatisrelatedtothecustomerpod'sdisplaypageisthefollowing:1.
WhiletheIECbootsup,theIECcontactstheIEMforitshomepageURL(startupURLintheIEM)whichishostedonREM.
2.
TheIECgoestotheURLdestinationlistedinthestartupURLoftheIEM'spolicy.
3.
TheREM(thedestinationoftheURL)requeststheIECtoloadtheKioskFlexapplication.
4.
WhilerequestingtoloadtheKioskFlexapplication,theKioskFlexapplicationqueriestheIECforitsserialnumber.
5.
TheIECrespondswithitsserialnumber.
6.
IftheIEC'sserialnumbermatcheswhatisconfiguredintheREM,theKioskFlexapplicationisloadedintotheIECandtheIECdisplaysthehomepagewithaconnectbuttononthecustomerpod.
7.
TheIECalsodisplaysastaticimageontheTPvideoendpoint(e.
g.
officehours).
IECdisplays'NetworkError'IECLANCable/SwitchDHCPServerIfLANcablewasconnectedaftertheIECwasbooted,reboottheIEC.
Checkthenetworkconnection.
IfitisDHCP-based,checkwhethertheDHCPserveriscorrectlyleasinganIPaddresstotheIEC.
IECdisplays'KioskPhoneisoutofService'Customerpod-sidevideoendpointCustomerpod-sidevideoendpointisunregisteredfromtheUCM.
IECrebootsincertaincasesREMCustomerpod-sidevideoendpointThefollowingaresomeofthescenariosinwhichrebootisinitiatedfromREM:Wheneverthereisapowerfailure,REMrebootstheIECtoidentifyallperipheralsthatareconnected.
Ifcustomerpod-sidevideoendpointgoesdownandcomesup,REMrebootstheIEC.
IftheCobraapplicationisnotworkingnormally,REMrebootstheIEC.
IECisnotstreamingvideosuchastheOnHoldVideooravideothatwaspushedfromanexpertRTMPCompliantStreamingServer1.
CheckRTMPCompliantStreamingServerisup.
2.
Iftheagentisnotregistered,thecallwillnotbeconnectedtotheagentwhothenwillnotbeabletoloadREAD.
Also,theKioskwillnotmovetotheconnectedscreen.
AgentDesktopdisplays'CheckingforActiveSession'continually,evenafteracceptingthecallREMContactCenterCADAdminWorkflow1.
ValidateExpertconfigurationinREAC.
2.
VerifyCADWorkflowsettings.
PossibleErrorsComponentsInvolvedPossibleReasons1-6CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsFigure1-3KioskFlexApplicationCallFlowIfyouareunabletobringuptheKioskFlexapplication,thefollowingarepossiblereasons:TheIECwasconfiguredwiththewrongIEMhostinformation.
VerifythattheIEM'sURLiscorrectintheIECandtheIEMshowsthattheIECisactive.
TheURLconfiguredintheIEMisincorrect.
Firstputthedefaultkioskserialnumberinrem.
propertiesandruntheconfigurationtool.
ThenregisterthekioskwiththedefaultIECinREAC.
VerifythattheURLiscorrectbytyping"https://:8443/reic"inabrowser.
Youshouldseethesamehomepagewiththeconnectbuttononthebrowserwindowandbeabletoclicktheconnectbuttonwithamouse.
Asaresult,acallwilltrytoconnectwiththevideoendpointusingtheUCCX/UCCEPilotnumber.
TheIECdoesnothavethecorrectpolicyapplied.
VerifythatthepolicywiththeREM'sURLasthestartupURLwasappliedtotheIECintheIEM.
TheIECisunabletoreachtheIEM.
Thiscouldbeanetworkproblem(e.
g.
LocalLANissue,WANissue,orcableissue).
IftheREM'sFQDNisused,itcouldbethatDNSserveriseitherincorrectlyconfiguredornotreachable.
PingtheREMfromIEC'sconsoletoensurethatitispingable.
TheLANconnectionmaynothavebeenupwhentheIECwasbootingup.
ReboottheIEC.
Thewebservicemaynotbeup.
Verifythatallninewebservices(listedbelow)areupbyenteringtheURL"https://:8443/resc/services/listServices".
1.
PrintandVideoShareService2.
FileUploadService3.
SpringInit4.
CustomerService5.
AdminService6.
NotificationService7.
Version1-7CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTips8.
VirtualAgentServices9.
ReportingServiceTheIEC'sserialnumberislistedincorrectlyintheREM.
VerifythattheREM'sKioskconfigurationwindowisshowingthecorrectserialnumberoftheIEC.
ChecktheIEC'seventlogfromIEMformoreinformation.
Refertothe"EventsTabintheIEM"sectionintheServiceabilityAdministrationchapterfordetailedinformationonhowtocheckanIEC'slogs.
UnabletoEstablishCallfromCustomerPod('SystemError'Message)WhenacustomerpressestheConnectbutton,thecallflowisthefollowing:1.
TheKioskapplicationinvokestheREM'scallconnectwebservice.
2.
TheREMrequeststheCUCMtosetupacallviaJTAPIbetweenthecustomerpod'svideoendpointDNandUCCX/UCCEPilotDN.
3.
TheCUCMinitiatesacallfromthecustomerpod'sTPvideoendpointtoUCCX/UCCEPilotDNandthecallisestablished.
Figure1-4ConnectButtonCallFlow1-8CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsWhenacallisunabletobeestablished,thefollowingarepossiblereasons:ConfigurationdataisincorrectintheREM:–TheTelePresenceDNofthecustomerpod'svideoendpointislistedincorrectlyinREM.
ChecktheKioskconfigurationintheREAC.
–TheIVRPhonenumberislistedincorrectlyintheREM.
ChecktheExpertTypeconfigurationintheREAC.
–Thecustomerpod'svideoendpointisnotlistedintheCUCM'sapplicationuser(ragent)controllist.
Includethecustomerpod'svideoendpointintotheREM'scontrollist.
–TheCUCM'sapplicationuser(ragent)eitherdoesnotexistortheuser'spasswordisnotmatchingthepasswordintheREMconfiguration.
–TheREMhasincorrectCUCMcredentials(username,password,IPaddresses).
–TheREMmayhaveduplicateentriesforthecustomerpod'svideo-endpoint(DN).
ProblemwiththeJTAPIlinkstoeithertheREMortoUCCX/UCCE:–CheckthattheJTAPIlinkbetweenREMandCUCMisupandrunning.
–CheckthattheJTAPIlinkbetweenCUCMandUCCX/UCCEisupandrunningbycheckingtheCTIRoutePoint'sstatusintheCUCMadministrationpage.
Problemwithendpoints:–Thecustomerpod'svideoendpointisnotintheactivestate(e.
g.
upgradingfirmwareorrebootingstate).
–TheUCCX/UCCEportsaremaxedoutandhenceunabletoestablishacallwithUCCX/UCCE(e.
g.
UCCX/UCCEgettingmorecallsthanitcansupport).
CustomerPodDisplays'Error:ManagementServerisnotreachable'MessageThemessage"ManagementServerisnotreachable"indicatesthattheIEMIPAddressisnotcorrectorinaccessible.
Thismessagemayalsoappearduetoanetworkproblem,aproxyserverconfigurationerror,oranincorrectIEMURL.
Checkifthefirewallpolicyisblockingaccess.
IftheIEMisdownbuttheIEChasaccessedthestartupURLpreviously,itwillloadthestartupURLfromitscache.
Inotherwords,thefailureoftheIEMdoesnotpreventtheIECfromfunctioning.
Ifanyconfigurationchangesareneeded,thenIEMhastobeactiveforpushingthenewpolicyconfigurationstotheIEC.
CustomerPodDisplays'StartupURLisnotconfigured'MessageThefollowingarethepossiblereasonsandresolutionsforthemessage"StartupURLisnotconfigured":IECdoesnothaveapolicy(initialconfigurationthatincludesstartupURL)enforcedinIEM.
VerifytheIEChasthecorrectpolicyappliedandtheIEChasbeenrebooted.
AnotherpossibilityisthattheIECisnotregisteredintheIEM,insteaditisinstandalonemode.
ChecktheIEMtoensurethattheproperserialnumberisaddedfortheIEC.
IftheIECisnotrebootedafterconfigurationchanges,reboottheIEC.
1-9CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsIECDisplays'Cannotregister'MessageClicktheShowDetailsbuttontorevealinformationaboutservicethatisdisabled.
ThisisduetothefacttheIEMisnotenabledforregistration.
Step1LogintotheIEMasroot/administratoruser.
Otherwise,userscannotseetheMaintenancelink.
TheMaintenancelinkisnotshowntoregularusers.
Step2ClicktheMaintenancelink.
Step3ClickServerSettings.
Step4ChecktheDevicegatewayenabledcheckbox.
Step5ClickApply.
CustomerPodTouchScreenFunctionisNotWorkingCorrectlyIfthetouchscreen(customerpod)isnotworkingcorrectly:1.
EnsurethattheUSBinterfacecableispluggedintotheIECandthetouchscreen.
2.
Usethecalibrationutilitytorecalibratethescreen.
a.
PressCtrl+Alt+S.
b.
EntertheDMC(DeviceMaintenanceCode).
c.
ClickCalibrator.
3.
RebootthesystemifthetouchscreenUSBcablewasnotconnectedbeforeboottime.
CustomerPodDisplaysVirtualKeyboardWhenCallConnectsWhenconnectingtoaremoteexpert,thetouchscreenconnectedtotheIECdisplaysavirtualkeyboardonthecustomerpod.
ThisisduetoanincorrectpolicyappliedtotheIEC.
DisablethefollowingconfigurationsettingsinthepolicythatisappliedtotheIEC.
IfthereisnopolicyappliedtotheIEC,thenthesechangesshouldbesetintheIECprofileconfiguration.
Step1LogintotheIEM.
Step2GotothepolicythatisappliedtotheIECortheIEC'sprofile.
Step3Gotokeyboard>virtual>enabledproperty.
Step4Setenabledvaluetofalse.
Step5Gotobrowser>input>popup>keyboard>enabledproperty.
Step6Setenabledvaluetofalse.
Step7SavethepolicybyclickingtheApplybutton.
Step8ReboottheIECtoactivatethepolicyonit.
1-10CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsCustomerPodDisplays'Serverisdown'MessageTherearemultiplepossiblereasonsforthiserrormessage:1.
ChecktheIEC'sPolicyorProfilesettingsintheIEMwithrespecttostartupURLconfiguration.
1.
CheckREM-UCMJTAPIconfiguration(applicationusername/passwordinUCMandREM).
2.
TheREM'stomcatservicemaybedown.
3.
ThemostlikelyreasonforthiserrormessageisthattheREM'sserviceisnotupandrunning.
AsuccessfulREMinstallationshouldshownineservices(PrintandVideoShareService,FileUploadService,SpringInit,CustomerService,AdminService,NotificationService,Version,VirtualAgentServices,andReportingService).
Tolistthoseservices,gotohttps://:8443/resc/services/listServiceswhere""istheIPaddressoftheREM.
Ifyouseeonlyoneservice,youneedtocheckresc.
log.
CheckthattheJTAPIlinkbetweentheREMandUCMisupandrunning.
Ifthelinkisnotupandrunning,theUCMusernameandpasswordthatislistedintherem.
propertiesfilearenotwhatisconfiguredinUCMapplicationuserpage.
Trythefollowingtoresolvetheissue:1.
RestarttheTomcatservice.
SeetheCiscoRemoteExpertManagerInstallationGuideforinstructions.
2.
Lookforthemessage'badlogin'intheresc.
log.
[root@localhost~]#grep--color'badlogin'/var/rem/resc/logs/resc.
log2012-06-0822:33:44,442ERROR[pool-2-thread-1]org.
apache.
axis2.
deployment.
ServiceDeployer-TheVirtualAgentServices.
aarservice,whichisnotvalid,causedErrorcreatingbeanwithname'virtualAgentService'definedinclasspathresource[applicationContext.
xml]:Cannotresolvereferencetobean'callService'whilesettingbeanproperty'callService';nestedexceptionisorg.
springframework.
beans.
factory.
BeanCreationException:Errorcreatingbeanwithname'callService'definedinclasspathresource[applicationContext.
xml]:Instantiationofbeanfailed;nestedexceptionisorg.
springframework.
beans.
BeanInstantiationException:Couldnotinstantiatebeanclass[com.
cisco.
big.
call.
CallService]:Constructorthrewexception;nestedexceptioniscom.
cisco.
jtapi.
PlatformExceptionImpl:Unabletocreateprovider--badloginorpassword.
Ifitisabadloginorpassworderror,correcttheJTAPIusernameandpasswordeitherintherem.
propertiesfileorintheUCMapplicationconfigurationpage.
Iftherem.
propertiesfileismodifiedtocorrectthisusernameandpasswordissue,executetheIAS.
SeetheCiscoRemoteExpertManagerInstallationGuideforinstructions.
CustomerPodDisplays'ServiceTemporarilyUnavailable'MessageThiserroroccurswhentheIECcannotpullthehomepagefromtheREMbecausethestartupURLconfiguredintheIEMisnotreachableortheREMserverorservicesaredown.
Toresolvethiserror:VerifytheREMisupandfunctioningbycheckingthisURLonabrowser:https://:8443/reicwhere""istheIPaddressoftheREM.
ChecktheIEC'seventlogintheIEM.
VerifytheIPaddressoftheREMasconfiguredintheIEMpolicythatisappliedtotheIECortheIEC'sprofile.
1-11CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsCustomerPodDisplays'Kioskisnotregistered'MessageIfthecustomerpoddoesnotdisplaythe'Connecthomepage'messagebutinsteaddisplaysthe'Kioskisnotregistered'message,thereasonforthiserrorisduetothefactthatREACdoesnothaveanentryforthisIECinitsKioskmenupage.
AddtheIECtotheREAC.
Ifitisalreadyadded,verifythattheIEC'sserialnumberiscorrect.
Theresc.
logfilewilldisplaythisissueas'uniqueserialNumber':[root@localhost~]#grep--color'uniqueserialNumber:'/var/rem/resc/logs/resc.
logWhentheIECisaddedorupdatedwiththecorrectserialnumber,theresc.
logshoulddisplay'kiosk:isalive'.
ConnectedPeripheralsareNotDetectedbytheIECPeripheralssuchasaprinter,scanner,keyboard,ormousemustbeconnectedtotheIECbeforeitisbootedupinorderfortheIECtodetectthem.
IfyouconnectaperipheralaftertheIEChasbootedup,reboottheIECtodetectthatperipheral.
IECisNotReflectingtheAppliedPolicyAfterapplyingthepolicy,IECneedstoberebootedtohavethepolicyenforced.
AlsochecktheIEMtoensurethattheIEChastheproperpolicyappliedtoit.
IEC'sProfileConfigurationisNotActiveIfthereisapolicyassignedtotheIEC,itsconfigurationtakeprecedenceovertheIEC'sprofileconfiguration.
IftheIEC'sprofileconfigurationisrequiredinstead,removetheappliedpolicy.
1-12CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsFigure1-5ProfileandPolicyHierarchyforIECsCustomerPodDisplays'Systemisnotavailable,Pleasetryaftersometime'MessageIfthecustomerpoddisplaysthe'Systemisnotavailable,Pleasetryaftersometime'message,itcouldbeduetothevideoendpointatthecustomerpodisnotintheUCM'sREMapplicationuser'scontrollist.
[root@localhost~]#grep--color'notinprovider'/var/rem/resc/logs/resc.
logcom.
cisco.
jtapi.
InvalidArgumentExceptionImpl:Address2512isnotinprovider'sdomain.
Toresolvethisissue,gototheUCMapplicationuserconfigurationthatisusedbytheREMinitsrem.
propertiesfile.
Addthevideoendpointintoitscontrollist.
UsetheUnix"less"commandtoviewandsearchtheresc.
logfileforREcallissues.
Withinthe"less"command,searchforthefollowingstringstofindcallissues:'MakingaCallconnectviaJtapi'istheresc.
logentryindicatinganewREsessionattempt.
'ObserverThread'areresc.
logentriescontainingObserverThreadthatcorrespondtotheclientpodvideoendpointinitiatingandparticipatingintheREsession.
'AgentObserver'areresc.
logentriescontainingAgentObserverthatcorrespondtotheexpertpod'svideoendpointparticipatingintheREsession.
ThefollowingcommandshowswhichtheDNsthatsuccessfullygotconnectedinthepast:1-13CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTips[root@localhost~]#grep--color'CallConnected'/var/rem/resc/logs/resc.
logTheresultofthecommandisthefollowingoutput:2012-06-0722:15:25,632INFO[ObserverThread(com.
cisco.
big.
call.
BIGObserver@6f221448)]com.
cisco.
big.
call.
BIGObserver-CallConnected25122012-06-0722:15:25,634INFO[ObserverThread(com.
cisco.
big.
call.
BIGObserver@6f221448)]com.
cisco.
big.
call.
BIGObserver-CallConnected11342012-06-0722:15:43,017INFO[ObserverThread(com.
cisco.
big.
call.
BIGObserver@6f221448)]com.
cisco.
big.
call.
BIGObserver-CallConnected25042012-06-072:22:24,026INFO[ObserverThread(com.
cisco.
big.
call.
BIGObserver@1539d980)]com.
cisco.
big.
call.
BIGObserver-CallConnected2512CustomerPodDisplays'Expertnotavailable'MessageWhenevertheIECisrebootedorpoweredup,theREMwillcheckreachabilitytotheIEC'slocalvideoendpointthatisconfiguredintheREAC'sKioskmenupage.
IftheIEC'svideoendpointisnotpingablefromtheREM,theREMinstructstheIECtodisplay'Expertnotavailable'.
ThismayalsohappenwhentheREMorREAC'sKioskpagehasmultipleinstancesofthesameIEC.
Ifyouchecktheresc.
log,youwillseefollowingerrormessages:[root@localhost~]#grep--color'isnotreachable'/var/rem/resc/logs/resc.
logWARN[http-bio-80-exec-60]com.
cisco.
big.
admin.
util.
VepManagementUtilCTSforkiosk:656015330015isnotreachable4****WARN[http-bio-80-exec-60]com.
cisco.
big.
admin.
util.
VepManagementUtilCTSforkiosk:656015330015isnotreachable5****WARN[http-bio-80-exec-60]com.
cisco.
big.
admin.
util.
VepManagementUtilCTSforkiosk:656015330015isdead.
after5triesAsimilarmessagewillalsobeloggedintheIEC'sEventscreenintheIEM.
CustomerPodDisplaysRequesttoProvideFeedbackAfterPressingtheConnectButtonIfthecustomerpoddisplaysascreenrequestingthecustomertoprovidefeedbackaboutthesessionafterthecustomerpressestheConnectbutton,thewrongCCPilotDNwasconfiguredortheDNisnotworking.
ContactCentercouldalsobepossiblydownorbusyortheUCM-ContactCenterintegrationhasissues.
Checktheresc.
logforCalledAddress:Unknown:[ObserverThread(com.
cisco.
big.
call.
BIGObserver@5b7127)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCalledAddress:UnknownINFO[ObserverThread(com.
cisco.
big.
call.
BIGObserver@5b7127)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCallingAddress:2512INFO[http-bio-80-exec-73]com.
cisco.
big.
call.
CallService-TimetakenforCallConnect:4INFO[ObserverThread(com.
cisco.
big.
call.
BIGObserver@5b7127)]com.
cisco.
big.
call.
BIGObserver-Settingthesessiontostatus2INFO[ObserverThread(com.
cisco.
big.
call.
BIGObserver@5b7127)]com.
cisco.
big.
call.
BIGObserver-Calldisconnected1-14CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsCustomerPodDoesNotDisplaytheHomePageThereareseveralpossiblereasonswhythecustomerpoddoesnotdisplaytheHomepage:TheIECdoesnothavethepolicywiththestartupURLpointingtotheREMappliedtoit.
ApplythatpolicytotheIECintheIEM.
IfIECdoesnothaveapolicybutusingitsprofileintheIEMforthestartupURL,verifythatthestartupURLintheprofileispointingtotheREM.
NoteUseofpoliciesisthepreferredmethodforconfiguringIECproperties.
VerifythattheIEChasbeenaddedtotheREAC'sKioskmenu.
IftheIECwasupdatedrecentlybutnotrebooted,theIECneedstoberebooted.
CheckifREMServerisreachableandwebservicesareaccessible.
Onceyouhavefixedtheissueabove,verifythatthecontentintheHomepageiscorrectbyfollowingthesesteps:Step1CheckwhichimageswillloadintheIECbygoingtothefollowinglink:https://:8443/resc/services/VirtualAgentServices/getKioskDetailsBySerialNumserialNumber=.
ThePNGfilesaretheonesthatactuallydisplayonthecustomerpod.
Step2ThosePNGfilesdisplayedareinfactmappedtospecificContactCenterPilotDNs,ExpertTypesaregroupedtogetherandtheIVR(DN)numbercanbemodifiedaltogetherforanExpertTypegroup.
Step3ChecktheIEC'seventlogintheIEMtoseeiftherequestfromtheIECwassuccessful.
NoteTheIPaddressusedintherequestisalsoinitsresponse.
ThiswouldhelptoidentifytheREMIPaddressusedbytheIECinitsrequest.
IECRebootsTwiceIncertaincircumstances,suchasduringapowerfailureatthebranch,theIECrebootstwice.
ThisisexpectedbecausetheIECwasdesignedtoonlydetectthoseperipheralsthatareconnectedtoitwhenitisrebooted.
Inapowerfailurescenario,theTPvideoendpointtakeslongertimetorebootcomparedtotheIEC.
Asaresult,theIECdoesnotdetecttheTPvideoendpointconnectedtoitsHDMIport.
Inordertocircumventthisissue,IECrebootsagainafteritfindstheTPvideoendpointisreachable.
Thefollowingresc.
logshowsthattheREMrebootedtheIEC:2012-06-1116:45:40,326INFO[http-bio-80-exec-144]com.
cisco.
big.
common.
util.
CommandExec-reboot[root:localhost~]$echo"RE_CHANNEL_TERM";RE_CHANNEL_TERM[root:localhost~]$Broadcastmessagefromroot@localhost(pts/2)(MonJun1109:45:282012):ThesystemisgoingdownforrebootNOW!
1-15CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsUnabletoSeeWaitVideoWhileCallisintheQueueTheWaitVideothatisstreamedwhenthecallisinaqueueishandledbyREM.
Thecallflowwhenacustomercallisbeingqueuedisasfollows:1.
Whenthereisnoexpertavailable,theUCCX/UCCEputsthecallinaqueue.
2.
BasedontheJTAPIevent,theRECSrequeststhemediaserver(Adobe)tostreamapre-configuredvideofiletotheIEC.
PossiblereasonsfortheWaitvideonotplayingare:TheWaitvideohasnotbeenaddedintheREM.
Goto/opt/apache-tomcat-7.
0.
23/webapps/reic/assets/videoandensurethattheWaitvideohasbeenadded.
IfaparticularvideohasbeenprovidedastheWaitvideoforanexperttypeinREACandstilldoesnotplay,checkthefollowing:–ThevideoformatprovidedinREACintheVideotabiscorrect(httpsorrtmpformat).
–ThevideofilenameinREACisthesameasthatgivenintheAdobeserver(FolderPath:Adobe>webroot>vod>filename).
UnabletoSeeOn-HoldVideoWhileCallisOnHoldTheOn-HoldvideoishandledbytheexternalmediaserversuchasAdobeMediaServer(AMS).
Therearetwodeploymentmodels:customerscaninstallAMSonadifferentserverorcustomerscaninstallAMSandREMonthesameserver.
PossiblereasonsfortheOn-Holdvideonotplayingare:TheAdobeserverdoesnothavetherequestedmediafile.
ChecktheREACVideotabtoensurethatthefile'sURLisprovidedinthecorrectformat.
ThefileisnotpresentintheAdobemediaserverlocation(FolderPath:Adobe>webroot>vod>filename).
CheckifthefilenameprovidedintheREACisthesameasthatstatedinthislocation.
Onlyfilenamesoftypes.
flv,.
f4v,.
mp4and.
movaresupported.
Videoclipsarestreamedviathertmpprotocol.
OnlyfilenamesandformatsofthesetypeswouldplayifprovidedastheOn-Holdvideo.
Convertthevideoformatorchangethefilenameifnecessary.
UnabletoBringUpStaticGraphicPageinTP(DuringNon-TPCalls)TheIECcandisplayastaticimageorstreamavideofiletotheTPvideoendpointwhenitisnotbeingused.
Ifthisfeatureisnotworking,aconfigurationmismatchisthelikelycause.
Step1VerifytheTPvideoendpointdeviceconfigurationinUCMiscorrectandmakesurethatthe'DaysDisplayNotActive','DisplayOnTime',and'DisplayonDuration'settingsarecorrectlyconfigured.
Step2VerifytheREM'sContentconfigurationpagehascorrectimagefilenamesandhours.
Step3GotothefollowingURL:https://:8443/reic/dualcontent.
htmlThisURLresponseshoulddisplaythegraphicsthatsupposedtoshowupatthecurrenttime.
1-16CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipsStep4VerifyintheIEMthattheIECshowsdualimages(oneimageforthecustomerpodandtheotherimageisforTPvideoendpointwhenitisnotonacall).
ClickDevicesintheleftpaneandthenclicktheShowScreenshoticonlocatedatthetopofthecenterpane.
IfthedualimagesshowupintheIEM,thentheissueisprobablywiththeTPvideoendpoint.
UnabletoUploadandUpdateImagestoREMThestaticgraphicimagesandvideofilescanbeuploadedtotheREMfromREAC.
However,suchuploadingfailsfromcertainbrowsers.
OnlyusesupportedbrowserswhileaccessingtheREAC.
READisNotShowingUpWhentheexpertanswersthecall,theREADwillbedisplayedintheCiscoAgentDesktop'sintegratedbrowser.
Ifitdoesnot,thefollowingarepossiblereasons:UCCX/UCCEdoesnothaveaPremiumPackagelicense.
TheCADneedsthePremiumPackagelicense.
CiscoAgentWorkflowisnotcorrectlyconfigured.
VerifythattheCiscoAgentWorkflowiscorrectlyconfiguredtobringuptheREADfromtheREMserver.
NoteTheconfigurationparametersintheHTTPActionSetuparecasesensitivesoensurethatthevaluesenteredintofieldsarecorrect.
Thewebservicesarenotrunning.
Thislinkshouldshownineactiveservices:https://:8443/resc/services/listServices.
Verifyeachofthecallevents.
dationofREAD.
Fordifferentcalleventstates,theexpert'sdesktopwilldisplaydifferentwebpages.
NoteReplacewiththeactualIPaddressandreplacewiththeactualuserID.
–NotReady:https://:8443/read/Common.
jspagentDn=&state=0&request=welcome–Ready:https://:8443/read/Common.
jspagentDn=&state=1&request=welcome–Ringing:https://:8443/read/desktoppageagentDn=&calling=–Dropped:https://:8443/read/Common.
jspagentDn=&request=disconnect–Logout:https://:8443/read/Common.
jsprequest=logout1-17CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingPossibleErrors&TroubleshootingTipseREADDocumentPanelHasChangedItsSizeIfthedocumentpanelsizedoesnotappearasexpectedontheagent'sscreen,theREAD_GADGET_WIDTHandREAD_GADGET_HEIGHTpropertieshavebeenchangedintheREMPropertiesfile.
ThedefaultvaluesareREAD_GADGET_WIDTH='100%'andREAD_GADGET_HEIGHT='790px'.
TheminimumvalueforREAD_GADGET_HEIGHTis500px.
Onceyouhavechangedtheirvalues,restarttheFinesseTomcatserviceforthechangestotakeeffect.
VideoCallisNotEstablishedBetweenDifferentTypesofTelePresenceVideoEndpointsWhenanexpertisavailable,thecustomercallthathasbeenqueuedinUCCX/UCCEwillberedirectedtotheavailableexpert'svideoendpoint.
Ifthevideoendpointsusedbetweenthecustomerandtheexpertareofdifferenttypes(i.
e.
oneisanEX90andtheotherisaC40),therecouldbeinteroperabilityissuesassociatedwithMTP.
UCM-basedMTPdoesnotsupportvideoandiftheend-to-callusesMTPresources,theresultingcallwillbeanaudiocall.
Therearetwoworkarounds:1.
RemoveMTPresourcesbeingusedinsuchcallsbetweencustomer-sidevideoendpointandexpert'svideoendpoint.
2.
IfMTPisneededforsomeotherreasons,useIOS-basedMTPwithpass-throughconfiguration.
IOS-basedpass-throughfeaturesupportsvideocallsgettingestablished.
ExpertsarenotGettingRegisteredinREACWhileregisteringexpertsinREAC,iftheregistrationfailsandshowsbothnodes(inadualnodesetup)inred,dothefollowing:Step1GototheconsoleoftheVMinVMWareVSphere.
Step2Typethecommandsystem-config-network.
Step3GototheoptionEditDNSConfiguration.
Step4CheckiftheHostnameisdifferentinboththeseVMs.
TheHostnamemustbeuniqueforeachVM.
Ifitisthesamename,changetheHostnameinoneoftheVMs.
Step5ValidatetheUCMconfiguration.
Step6Restartthenetworkbytypingthecommand/etc/init.
d/networkrestart.
NoteIfthehostnamehaschanged,restartTomcatforREMtofunctionproperly.
1-18CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingTroubleshootingGuidelinesforIECasVideoEndpointREIC(CobraBrowser)HangsWhentheREIChangsorgoesintoalimbostateduetosomereason,usethefollowingcommandtoclearcache:https://:8443/resc/services/AdminService/cleanCallCacheSessionHangsTheadministratorcancleanthecallcacheofkiosksifasessionhangs.
Whenasessionhangs,anewcallcannotbeinitiatedtoorfromthecustomerpod.
Bycleaningthecallcache,thesession'sstatusintheSessiontabischangedto"Completed",whichthenallowsthecustomerpodtostartorreceiveanewcall.
Step1ClicktheCleanCallCachetab.
AStatusboxappearsinthelowerrightcornerofthescreenindicatingthatthecallcachewascleaned.
REMErrorMessageIfyouseetheerrormessage"Addressisnotinprovider'sdomain",thedevicewhoseDirectoryNumber(DN)isisnotinthecontrol-listofapplicationuser(ragent)inCUCM'sconfiguration.
AddthedeviceintheCUCMundertheappropriateApplicationUser.
EX90FirmwareErrorMessageIfyouseetheerrormessage"Theagentorworkflow-initiatedactionrequestfailed",updatetheEX90firmwaretoTC6.
1.
TroubleshootingGuidelinesforIECasVideoEndpointIftheIECisbeingusedasthevideoendpointasopposedtoaCiscoTelePresenceunitsuchastheEX-60,youmayencounterthefollowingissuesrelatedtotheSIPconfiguration.
SIPVideoCallisNotEstablishedBetweenIECVideoEndpointandAgentIfthecallcannotbemadeorreceived,verifythefollowingwasconfiguredcorrectlyintheIEM:TheformatoftheCUCMphoneis:sip.
targetorsip.
target1ThevalueoftheCUCMphoneisthedirectorynumberinCUCM.
TheformatoftheCUCMusernameis:sip.
usernameThevalueoftheCUCMusernameistheusernameinCUCM.
TheformatoftheCUCMpasswordis:sip.
passwordThevalueoftheCUCMpasswordistheCUCMpassword.
TheformatoftheCUCMdomainis:sip.
domain1-19CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingTroubleshootingGuidelinesforRemoteExpertApplicationsRunningonVMwareThevalueoftheCUCMdomainistheIPaddressoftheCUCM.
TheformatoftheCUCMprotocolis:sip.
transportThevalueoftheCUCMprotocolis:udpNoteEnsurethatalloftheformatsandvaluesareinlowercase.
TroubleshootingGuidelinesforRemoteExpertApplicationsRunningonVMwarePerformanceindicatorsfromwithinthevirtualmachinesarestillvalid.
FortheUCapplicationsthatsupportit,useRTMTortheperfmondataforanalysisoftheperformanceoftheUCapplication.
Datafromthesetoolsprovidesviewoftheguestperformancesuchasdisk,CPU,memory,etc.
MovetotheVMwareinfrastructurewhenthereisaneedtogettheperspectivefromtheESXihost.
UsethevSphereClienttoviewdata.
IfvCenterisavailable,historicaldataisavailablethroughtheclient.
IfvCenterisnotavailable,livedatafromthehostisavailablethroughtheclient.
vCenterSettingsJustlikesomeoftheUCapplications,vCentercanbeconfiguredtosavemoreperformancedata.
Themorehistoricaldatasaved,thebiggerdiskspaceneededbythedatabaseusedbyvCenter.
NoteThisisoneofthemainareaswhereyouneedvCenterratherthangoingdirectlytotheESXihostforperformancedata.
vCentercansavehistoricaldatathattheESXihostdoesnotkeep.
TheconfigurationstochangetheamounthistoricaldatasavedbyvCenterislocatedinthevSphereclientunderAdministration>ServerSettings.
Foreachintervalduration,thedurationthatthestaticswillbesavedfor(days,weeks,months,oryears)andthestatisticslevelcanbeset.
Thestatisticlevelsrangefrom1to4withlevel4containingthemostdata.
Lookatthedatasizeestimatestomakesurethereisenoughspacetokeepallofthestatistics.
VMwarePerformanceIndicatorsThefollowingtableliststheperformanceindicatorstomonitorandviewfromaVMwareperspectivewhenavirtualmachine'sperformanceisnotoptimal.
MostcountersarefromtheESXihost,whichcangiveaperspectiveofVMinteractionsandoverallhostanddatastoreutilization.
Table1-2VMwarePerformanceIndicatorsPerformanceAreaObjectCounterAcceptableRangeCPUHostUsagelessthan80%CPUVirtualMachineReadylessthan3%1-20CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingTroubleshootingGuidelinesforRemoteExpertApplicationsRunningonVMwareCPUTroubleshootingHighCPUusagecouldbeduetoasmallnumberofVMsusingalloftheresourcesortoomanyVMsrunningonthehost.
IFtherearetoomanyVMsrunningonthehost,seeifCPUreservationsareinuse(seeoversubscriptionsection).
ToisolateaCPUissueforaparticularVM,considermovingittoanotherESXihost.
ToviewtheCPUperformanceindicators,gototheESXihost'sperformancetabandselecttheadvancedbutton.
Underchartoptions,selectCPU,timeframe,andthenonlythehost(notindividualcores)toviewoverallCPUusageonthehost.
EachvirtualmachineCPUusagecanalsobeseenfromtheVirtualMachinestabonthehost.
TogetaviewofthereservationssetbyalloftheVMs,usetheResourceAllocationtabofthecluster.
NoteThisisonlyavailableviavCenter.
MemoryTroubleshootingTheguidelinesdonotsupportmemorysharingbetweenVMs.
Toverify,theswappingandballooningcountersshouldbesettozero.
IfagivenVMdoesnothaveenoughmemoryandtherearenotmemoryissuesonthespecifichost,considerincreasingtheVM'smemory.
Toviewthememoryperformanceindicators,gototheESXihost'sperformancetabandselecttheadvancedbutton.
Underchartoptions,selectmemoryandtimeframe,andthenselectthefollowingcounters:UsedmemorySwapusedBalloonUsedmemorycanbeusedtolookatgeneraltrends.
Swapandballoonshouldalwaysbe"0",otherwisememorysharingisbeingused,whichshouldnotbethecase.
MemoryHostConsumedgeneraltrendisstableMemoryHostBalloon/Swapused0kbDiskSpecificdatastoreKernelcommandlatencylessthan3msDiskSpecificdatastorePhysicaldevicecommandlatencylessthan20msDiskSpecificdatastoreAveragecommandsissuedpersecondlessthanLUNcapacityNetworkHostReceivepacketsdropped/Transmitpacketsdropped0packetsPerformanceAreaObjectCounterAcceptableRange1-21CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingTroubleshootingGuidelinesforRemoteExpertApplicationsRunningonVMwareDiskTroubleshootingBaddiskperformanceoftenshowsupashighCPUusage.
IOPSdatacanprovideinformationonhowhardtheapplicationandVMisworkingthedisks.
SpecificactivitiescancausespikesinIOPSsuchasupgradesanddatabasemaintenance.
IfVMsrunningonthesamedatastorearealldoingtheseactivitiesatthesametime,thedisksmightnotbeabletokeepup.
IOPSdatacanbeseenfromvCenterortheSAN.
Disklatency(responsetime)isagoodindicatorofdiskperformance.
Step1Toviewthediskperformanceindicators,gototheESXihost'sperformancetab.
Step2Choosetheadvancedbutton.
Step3Choosetheappropriatedatastore,whichcanbefoundonthedatastorepage.
Step4Underchartoptions,selectdiskandtimeframe.
Step5Choosethefollowingcounters:PhysicaldevicecommandlatencyKernelcommandlatencyAveragecommandsissuedpersecondThekernelcountershouldnotbegreaterthan2-3ms.
Thephysicaldevicecountershouldnotbegreaterthan15-20ms.
The'Averagecommandsissuedpersecond'countercanbeusedifIOPSarenotavailablefromtheSAN.
IOPSshouldbeconsideredifitlookslikedatastoreisoverload.
ThisIOPSdataisviewablefromthehostandeachVM.
NoteForNFSdatastores,thephysicalandkernellatencydataisnotavailable.
StartinginVMware4.
0update2andbeyondtheesxtopcommandcanbeusedtoviewNFScountersandinparticulartheguestlatency.
Theguestlatencyisasummationofthephysicaldeviceandkernellatencies.
NetworkTroubleshootingGenerally,networkperformanceissuescanbeseenbydroppedpackets.
IfdroppedpacketsareseenfromanESXihost,thenetworkinfrastructureshouldbeinvestigatedfortheissue,whichmightincludeavirtualizedswitch(Nexus1000V).
InESXi4.
1,issueshavebeenseenwithlargefiletransfers(e.
g.
SFTP/FTPtransfers).
Forthisissue,theLargeReceiveOffloadoptionsneedtobedisabledontheESXihost.
Thissettingisfoundonthehost'sConfigurationtab>AdvancedSettings>Net.
*.
NoteThereareseveralLROsettingsonthispageandallofthemneedtobedisabled.
IfaVMhasbeenclonedandusesstaticMACaddresses,verifytherearenotduplicateMACaddressesinthenetwork.
Toviewthenetworkperformanceindicators,followthesesteps:Step1GototheESXihost'sperformancetab.
Step2Choosetheadvancedbutton.
Step3Underchartoptions,selectNetworkandtimeframe.
1-22CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingInformationtoHelpwithTroubleshootingStep4Choosethefollowingcounters:ReceivepacketsdroppedTransmitpacketsdroppedThemainthingtocheckisthatnopacketsaregettingdroppedinthenetwork.
NoteAdvancednetworkdebuggingandconfigurationcanbedoneonNexus1000vifused,whichrequiresvCenter.
CPUOversubscriptionImplicationsonPerformanceSomedeploymentsallowCPUoversubscription,whichisthesharingofCPUcoresbetweenVMs.
Inthiscase,afewmoredatapointsneedtobeconsidered.
First,looktoseeifCPUreservationsaresetfortheVM.
CiscohasrecommendedCPUreservationswhileusingCPUoversubscriptionforsomeapplicationswhicharepartoftheofficialOVA.
Second,lookattheCPUreadytime(i.
e.
howlongaVMiswaitingtorunonacore).
Thiscountercanbeconvertedintoapercentage.
Generalguidelinesaretokeepthepercentagebelow6%andthenanythingabove3%shouldbemonitoredfordesiredresponsetimes.
OtherConsiderationsOntheC-seriesUCSserverstherehavebeenissueswiththewritecachebatterybackup.
Ifthisbatteryisnotoperatingcorrectly,performancewillsuffer.
UseatoollikeWBEMCommandLineInterface(wbemcli)toverifythebatteryisokay.
Thefollowingoutputisanexampleofusingthewbemcli:wbemcliei-noverify'https://root:@:5989/root/cimv2:VMware_HHRCBattery'Gotothefollowinglinktolearnmoreabouttroubleshootingvirtualizedenvironments:http://docwiki.
cisco.
com/wiki/Troubleshooting_and_Performance_Monitoring_Virtualized_EnvironmentsInformationtoHelpwithTroubleshootingREEventandRESCLogsREEventlogscanhelpwithtroubleshootingissues.
Followthesesteps:Step1DownloadtheREEventlogfromREAC.
Step2Reviewtheexceptionmessagecolumntodetermineifthedesiredoperationthrewanexception.
Step3IftherootcausecannotbeidentifiedintheREEventlog,thendownloadtheresclogfromREAC.
1-23CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingInformationtoHelpwithTroubleshootingStep4Scantheresclogfilefordetailedinformation.
NormalIECBootupSequenceFigure1-6IECBootupSequenceThefollowinghappensduringtheIECbootupsequence:1.
REMreceivesinitialnotificationasindicatedby"ReceivedgetKioskNotificationInfo:656015330015":[http-bio-80-exec-44]com.
cisco.
big.
scheduler.
services.
NotificationService-ReceivedgetKioskNotificationInfo:656015330015[http-bio-80-exec-44]com.
cisco.
big.
scheduler.
services.
NotificationService-ReturngetKioskNotificationInfo[org.
springframework.
scheduling.
quartz.
SchedulerFactoryBean#0_Worker-3]com.
cisco.
big.
scheduler.
job.
SchedulingJob-SchedulingJobinvoked[org.
springframework.
scheduling.
quartz.
SchedulerFactoryBean#0_Worker-3]com.
cisco.
big.
scheduler.
job.
SchedulingJob-SchedulingJobexecutedsuccessfully[org.
springframework.
scheduling.
quartz.
SchedulerFactoryBean#0_Worker-1]com.
cisco.
big.
scheduler.
job.
SchedulingJob-SchedulingJobinvoked[org.
springframework.
scheduling.
quartz.
SchedulerFactoryBean#0_Worker-1]com.
cisco.
big.
scheduler.
job.
SchedulingJob-SchedulingJobexecutedsuccessfully[http-bio-80-exec-37]com.
cisco.
big.
va.
services.
VirtualAgentServices-ReceivedgetKioskDetailsuniqueserialNumber:656015330015[http-bio-80-exec-37]com.
cisco.
big.
va.
dao.
KioskDAO-[com.
cisco.
big.
va.
pojo.
domain.
Locale@57d04f0a][http-bio-80-exec-41]com.
cisco.
big.
va.
services.
VirtualAgentServices-ReceivedgetKioskDetailsuniqueserialNumber:656015330015[http-bio-80-exec-41]com.
cisco.
big.
va.
dao.
KioskDAO-[com.
cisco.
big.
va.
pojo.
domain.
Locale@6555121a]1-24CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingInformationtoHelpwithTroubleshooting[http-bio-80-exec-44]com.
cisco.
big.
admin.
service.
AdminService-getContentByTimeInterval:Time:09-06-201205:18:23[http-bio-80-exec-44]com.
cisco.
big.
admin.
service.
AdminService-ParsedDate:SatJun0905:18:23GMT+00:0020122.
REMinstructstoloaddualcontentimage:[http-bio-80-exec-44]com.
cisco.
big.
va.
dao.
ContentDAO-ReturnDCValue:http://172.
21.
57.
111:80/images//bg_img_1920X1200.
jpg[http-bio-80-exec-37]com.
cisco.
big.
admin.
service.
AdminService-reportPostIECRebootinvokedforKioskId:6560153300153.
REMroutinelychecksTPvideoendpointreachability:WARN[http-bio-80-exec-60]com.
cisco.
big.
admin.
util.
VepManagementUtilCTSforKiosk:656015330015isnotreachable4****WARN[http-bio-80-exec-60]com.
cisco.
big.
admin.
util.
VepManagementUtilCTSforKiosk:656015330015isnotreachable5****WARN[http-bio-80-exec-60]com.
cisco.
big.
admin.
util.
VepManagementUtilCTSforKiosk:656015330015isdead.
after5triesTheselogswillonlybecreatedwhentheIEC'slocalTPvideoendpointisnotreachable.
IftheTPvideoendpointisreachable,thennosuchlogsarecreated.
HowevertheroutinecheckingofTPreachabilitylogswillbeavailableinIEC'seventloginIEM.
4.
REMregularlymonitorsforalivenessofIEC,inparticulartheCobraApplication:[http-bio-80-exec-46]com.
cisco.
big.
admin.
service.
AdminService-Checkingforaliveness:656015330015time:SatJun0912:19:11GMT+00:002012[http-bio-80-exec-46]com.
cisco.
big.
admin.
service.
AdminService-Kiosk:656015330015isalive1-25CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingInformationtoHelpwithTroubleshootingNormalCallFlowanditsAssociatedresc.
logFigure1-7NormalCallFlowInthescenariobelow,thecustomerpressestheConnectbuttononthecustomerpodtohaveanimmersivevirtualcollaborationwithanexpertandthefollowingvaluesareused:CustomerpodIEC'sserialnumber:656015330015Customer-sideEX90videoendpointDN:2512ContactCenterPilotDN:1134Expert'sDN:2504Logfile:resc.
log1.
REMreceivesCall-ConnectrequestafterthecustomerpressedtheConnectbuttononthecustomerpodindicatedby"getKioskNotificationInfo:656015330015"inthefirstlineofthefollowingoutput:[http-bio-80-exec-87]com.
cisco.
big.
scheduler.
services.
NotificationService-ReceivedgetKioskNotificationInfo:656015330015[http-bio-80-exec-87]com.
cisco.
big.
scheduler.
services.
NotificationService-ReturngetKioskNotificationInfo[pool-17-thread-2]com.
cisco.
big.
call.
CallService-CallConnectTimeoutCallnotestablishedyet.
.
.
.
[pool-17-thread-2]com.
cisco.
big.
call.
CallService-ExpertTypenamedecoratorEWT:10[pool-17-thread-4]com.
cisco.
big.
call.
CallService-CallConnectTimeoutCallnotestablishedyet.
.
.
.
[pool-17-thread-4]com.
cisco.
big.
call.
CallService-ExpertTypenamedecoratorEWT:10[http-bio-80-exec-87]com.
cisco.
big.
va.
services.
VirtualAgentServices-CallConnectRequest:KioskSerial:656015330015ExpertType:1-26CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingInformationtoHelpwithTroubleshooting2.
REMgetstheDNofthecustomerpod-sidevideoendpointREAC/REMindicatedby"Addressinserviceeventfor:2512"inthefirstlineofthefollowingoutput:[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-Addressinserviceeventfor:2512[http-bio-80-exec-87]com.
cisco.
big.
call.
CallService-TimetakentocheckifAddressisinservice:03.
REMabletostartJTAPIMonitoringon2512:[http-bio-80-exec-87]com.
cisco.
big.
call.
CallService-IsAddressinservice:true[http-bio-80-exec-87]com.
cisco.
big.
call.
CallService-Timetakentocheckifcallispresent:1002[http-bio-80-exec-87]com.
cisco.
big.
call.
CallService-IscallPresent:false[http-bio-80-exec-87]com.
cisco.
big.
call.
BIGObserver-StartinganSSHconnectiontoCTStoexecutecallStartcommand[http-bio-80-exec-86]com.
cisco.
big.
scheduler.
services.
NotificationService-ReceivedgetKioskNotificationInfo:656015330015[http-bio-80-exec-86]com.
cisco.
big.
scheduler.
services.
NotificationService-ReturngetKioskNotificationInfo[pool-17-thread-2]com.
cisco.
big.
call.
CallService-CallConnectTimeoutCallnotestablishedyet.
.
.
.
[pool-17-thread-2]com.
cisco.
big.
call.
CallService-ExpertTypenamedecoratorEWT:104.
REMconnectstocustomerpod-sideendpointviaSSHindicatedby"StartinganSSHconnectionto10.
17.
161.
51"inthefirstlineofthefollowingoutput:[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-StartinganSSHconnectionto10.
17.
161.
51[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-StartingPreconditionCheck[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-InitializingPreCommandExecutionMatchString[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-available:0[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-totalReadTimeout:25000[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-sleepReadtime:2000[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-available:95[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-readBytes:95[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-Welcometo.
0.
290141Alpha35.
REMsuccessfullyconnectedtovideoendpointviaSSHindicatedby"CompletedpreCommandExecutionMatchString"inthesecondlineofthefollowingoutput:R[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-ProcessBytesAndMatchStringexecutedsuccessfullyforCommand:callConnect[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-CompletedpreCommandExecutionMatchString6.
REMdialsContactCenterPilotDN(1134)indicatedby"Executingxcommanddialnumber:1134"inthefirstlineofthefollowingoutput:[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-Executingxcommanddialnumber:1134[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-StartingPostconditionCheck[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-InitializingPreCommandExecutionMatchString[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-available:0[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-totalReadTimeout:23000[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-sleepReadtime:20001-27CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingInformationtoHelpwithTroubleshooting[pool-17-thread-4]com.
cisco.
big.
call.
CallService-CallConnectTimeoutCallnotestablishedyet.
.
.
.
[pool-17-thread-4]com.
cisco.
big.
call.
CallService-ExpertTypenamedecoratorEWT:10[http-bio-80-exec-86]com.
cisco.
big.
admin.
service.
AdminService-Checkingforaliveness:656015330015time:FriJun0814:39:25GMT+00:002012[http-bio-80-exec-86]com.
cisco.
big.
admin.
service.
AdminService-Kiosk:656015330015isalive[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-available:108[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-readBytes:108[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-xcommanddialnumber:1134[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-ProcessBytesAndMatchStringexecutedsuccessfullyforCommand:callConnect[http-bio-80-exec-87]com.
cisco.
big.
call.
tp.
SSHCommandExecutor-CompletedPostCommandExecutionMatchString[http-bio-80-exec-87]com.
cisco.
big.
call.
BIGObserver-TimetakentoexecuteSSHcommandonCTS:47.
REMshowscallconnectedbetweenDN2512and1134indicatedby"1134"inthefirstline,"2512"inthesecondline,and"CallConnected1134"inthethirdlineofthefollowingoutput:[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCalledAddress:1134[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCallingAddress:2512[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-CallConnected1134[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-classcom.
cisco.
jtapi.
ConnCreatedEvImpl[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCalledAddress:2504[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCallingAddress:2512[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-classcom.
cisco.
jtapi.
ConnConnectedEvImpl8.
REMfindscallisnotconnectedtoexpert(queuedcall):[pool-17-thread-2]com.
cisco.
big.
call.
CallService-CallConnectTimeoutCallnotestablishedyet.
.
.
.
[pool-17-thread-2]com.
cisco.
big.
call.
CallService-ExpertTypenamedecoratorEWT:10[pool-17-thread-4]com.
cisco.
big.
call.
CallService-CallConnectTimeoutCallnotestablishedyet.
.
.
.
[pool-17-thread-4]com.
cisco.
big.
call.
CallService-ExpertTypenamedecoratorEWT:10pool-17-thread-6]com.
cisco.
big.
call.
CallService-CallConnectTimeoutCallnotestablishedyet.
.
.
9.
REMshowscallconnectedtotheexpert(DN:2504)indicatedby"2504"inthefirstline,"2512"inthesecondline,and"CallConnected2504"inthethirdlineofthefollowingoutput:[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCalledAddress:2504[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCallingAddress:2512[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-CallConnected250410.
REMdetectscallisonholdbyexpertindicatedby"2504InmanageOnHold"inthelastlineofthefollowingoutput:[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)].
AgentObserver-AgentObserver-DN:2504[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-Event:1-28CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter1REMTroubleshootingInformationtoHelpwithTroubleshooting[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-2504Call:2512-->2504[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-2504CallId:9245[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-2504InmanageOnHold11.
REMdetectscallisresumedbyexpertindicatedby"2504InmanageTalking"inthethirdlineofthefollowingoutput:[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-2504Call:2512-->2504[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-2504CallId:9245[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-2504InmanageTalking[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
BaseObserver-Sessionreferencenot[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@5e6276e5)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-ListofSessiononAgent12.
REMdetectscallisdisconnectedindicatedby"Calldisconnected"inthelastlineofthefollowingoutput:[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-classcom.
cisco.
jtapi.
ConnDisconnectedEvImpl[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-classcom.
cisco.
jtapi.
ConnDisconnectedEvImpl[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCalledAddress:2504[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCallingAddress:2512[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-classcom.
cisco.
jtapi.
TermConnDroppedEvImpl[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCalledAddress:2504[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCallingAddress:2512[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-classcom.
cisco.
jtapi.
ConnDisconnectedEvImpl[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCalledAddress:2504[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-BIGObserver-CurentCallingAddress:2512[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-Settingthesessiontostatus2[ObserverThread(com.
cisco.
big.
call.
BIGObserver@403bfa49)]com.
cisco.
big.
call.
BIGObserver-CalldisconnectedTraceFilesforTACTherearenormallytwologfilesthatareneededbytheCiscoTACteamtotroubleshootREMrelatedissues:1.
resc.
logfile2.
IEC'seventlogfromtheIEMGototheIEC'seventscreen,selectallthecheckboxesunderfacility(bydefaultitisallselected),andthenpresstheSaveasbutton.
Whenthelocalworkstation'sWindowsExplorerwindowopens,savethelogfiletothelocaldisk.
CHAPTER2-1CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuide2ServiceabilityAdministrationJune10,2016ChapterOverviewThischapterisintendedforserviceengineersandserveradministratorswhoareaccountableforkeepingtheREMupandrunningtoitscompleteusabilityandperformance.
ThescopeofthischapterisrestrictedtotopicsthatassistadministratorswithmaintainingtheREMapplication.
ThischapteralsoexplainshowtobackupandrestoretheREM.
Topicsinthischapterinclude:"ServiceabilityOverview""Monitoring"–"ErrorsandAlertsfromREAC"–"ErrorsandAlertsfromREAD"–"ErrorsandAlertsfromREIC"–"REMServer(RESC)Errors"–"LogMonitoring"–"EventsTabintheIEM"–"CallRelatedErrors"–"JTAPIIssues"–"WebService"–"DualContentIssue"–"FailureofRESCWeb(API)Services""VerifyingVMSettings""TUITroubleshootingTools""TroubleshootingToolsinREAC"ServiceabilityOverviewServiceabilityadministrationprovidesdescriptionsandproceduresforthefollowingfunctionality:2-2CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringSupportsdailymanagementofREMapplicationsSchedulesmonitoringandmaintenanceoftheeachcomponentCaptureseventsfortroubleshootingandprovidesalert/errormessagedefinitionsSavestraceinformationtovariouslogfilesfortroubleshootingMonitorsthediskusageofthelogpartitiononaserverMonitoringMonitoringinvolveskeepingtrackofvariouserrorsandwarningbeingthrownbyvariouscomponentsoftheREM.
ItinvolvescheckingthestateorhealthofallREMcomponentsforavailabilityandappropriatefunctioningofallsolutionfeatures.
TherearevariouserrorsandalertsthatarethrownbyREMwhichneedstoberesolvedtokeeptheREMupandrunning.
ErrorsandAlertsfromREACThetablesbelowdocumenttheerrorsandalertsthatcouldbethrownbytheRemoteExpertAdminConsole(REAC)applicationandshownontheuserinterface(UI).
Table2-1GenericREACErrorsError/AlertDescriptionSolutionUnabletocontactthewebserver.
Pleasecheckwebserverstatus.
OccurswhentheREMserverisdownBringtheREMserverupbyusingtheTUI:MainMenu>ServicesControl>ApplicationServer>b)StartServiceAninternalerroroccurredOccursduringKioskrestartifRESChasencounteredaninternalerrorChecktheRESClogfileforthespecificerrorconditionbyusingtheTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>a)RESCLogs:resc.
logUnabletoconnecttoREMServerOccursduringKioskrestartwhenREACisunabletoconnecttoRESC(ifRESCisdeployedonaseparateserver)1.
BringtheREMserverupbyusingtheTUI:MainMenu>ServicesControl>ApplicationServer>b)StartService2.
ChecktheRESCIPbyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties>REMCorePropertiesUnabletorestartkioskOccursduringKioskrestartifREMhasfailedtorestartduetoissuesontheserver1.
ChecktheKiosksettings>IECSerialNumber2.
Performthesameoperationagain3.
RestartIECfromIEM4.
RestartIECmanually2-3CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringTable2-2RemoteServiceCallErrorsIllegalinputprovidedOccursduringKioskrestartifKioskhasbeendeletedfromREMChecktheREAClogfileforthespecificerrorconditionbyusingtheTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>d)REACLogs:reac.
logUnabletoconnecttoKioskOccursduringKioskrestartwhenREMserverisunabletomakeconnectiontotheIEMVerifyiftheIEMIPaddressiscorrectintheREMPropertiesfilebyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMPropertiesHTTP(500):TheserverreportedanErrorTheserverhasencounteredaninternalerror,whichpreventeditfromfulfillingtherequest1.
CheckREAClogfileforthespecificerrorconditionbyusingtheTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>d)REACLogs:reac.
log2.
CheckRESClogfileforthespecificerrorconditionbyusingtheTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>a)RESCLogs:resc.
logAnerroroccurredduringFileTransferTheserverhasencounteredaninternalerror,whichpreventeditfromfulfillingtherequest1.
CheckRESCIPbyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties>REMCoreProperties2.
CheckthefollowingparametersinREACPROPERTIESbyusingtheTUI:MainMenu>REMServerAdministration>EditREMTemplates>a)REACProperties:remote_server_ip=$*[RESC_IP]*remote_server_uname=ENC($*[RESC_SERVER_USER]*)remote_server_pwd=ENC($*[RESC_SERVER_PASSW]*)UnabletosynchronizeresourcesacrossclusternodesOccursduringfiletransferwhenhandshakingbetweenthetwonodesfails1.
RefertotheJTAPIIssuessectionofthischaptertofixtheissue2.
UploadthefilefromREACagainError/AlertDescriptionSolution2-4CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringTable2-3ClusterManagementErrorsErrorsandAlertsfromREADTheerrorsgeneratedbytheREADapplicationcanbegroupedintwoscenarios:1.
DirectConnect(DC)errorsintheREADapplication2.
ErrorsoralertsfromtheREADapplicationError/AlertDescriptionSolutionRemotecalltoxxxxfailedOccurswhenREACisunabletocommunicatetoREMserver1.
CheckREAClogfileforthespecificerrorconditionbyusingtheTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>d)REACLogs:reac.
log2.
VerifyifRESCservicesareavailablebyinvoking"https://:8443/resc/services/listServices"RemotecalltoxxxxreturnedanemptyresponseOccurswhenREMserversendsinvalidresponseCheckRESClogfileforthespecificerrorconditionbyusingtheTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>a)RESCLogs:resc.
logError/AlertDescriptionSolutionTherequestedoperationcannotbeexecutedduetoincompleteconfigurationOccursiftheREMserverdetailsandDBdetailsaremissinginREMconfigurationfile1.
Verifythevaluesof$*[NODE_IP_1]*and$*[DATABASE_1]*intheREMPropertiesfilebyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties2.
UsetheTUI:MainMenu>REMServerAdministration>c)RunConfigurationToolTheselectedoperationfailedOccursifthereisanerrorwhileactivatingtheDBChecktheREAClogfileforthespecificerrorconditionbyusingtheTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>d)REACLogs:reac.
logTheselectednode''{0}''isnotfoundOccursiftheREMserverdetailsandDBdetailsaremissinginREMconfigurationfile1.
Verifythevaluesof$*[NODE_IP_1]*and$*[DATABASE_1]*inREMPropertieslocationbyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties2.
UsetheTUI:MainMenu>REMServerAdministration>c)RunConfigurationTool2-5CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringTable2-4DirectConnectErrorsTable2-5READErrorsError/AlertDescriptionSolutionDCpopsupinREADandafterclickingontheStartsharingbuttonbutitremainsintheWaitingstateThe'cv_service_url'inDirectConnect.
exe.
configisincorrect.
FiledirectionC:\CSI\DirectConnect\bin\DirectConnectisincorrect1)Fix'cv_service_url'topointtothecorrectREMIPaddress2)RestarttheDCServerfromthedesktopTheExpertisnotregisteredinREMVerifyintheExperttabinREACthattheexpertisaddedDClicensehasexpired.
Theerrordisplayedinerror.
logavailableatC:\CSI\DirectConnect\Server\tomcat\logs\error.
logisthefollowing:2012-04-1311:46:47,584ERROR[/]Exceptionsendingcontextinitializedeventtolistenerinstanceofclasspureweb.
servlet.
ContextListenerjava.
lang.
RuntimeException:com.
reprisesoftware.
rlm.
RlmException:Licensehasexpired(-3)atpureweb.
servlet.
ContextListener.
contextInitialized(ContextListener.
java:88)~[ContextListener.
class:na]RenewDClicenseError/AlertDescriptionSolutionREADnotinitializedOccurswhenREADisunabletocontactRESCVerifyifservicesofRESCareavailablebyinvoking"https://:8443/resc/services/listServices"PreviouscollaborationrequestisstillinprogressOccursiftryingtostartaREADmodulewhenthecurrentoneisstillinprogressInREAD,checkjobstatusifDocumentCamera,Scan,SignatureCapture,orDirectConnectarerunningbeforetryingtostartanothermoduleCheckingforanactivesession…OccursifCADAdminhasincorrectvaluesRefertotheCADAdministrationGuideCommunicationFailure:PleasecontactAdministratorOccursiftheREMservergoesdownduringthesessionBringtheREMserverupbyusingtheTUI:MainMenu>ServicesControl>ApplicationServer>b)StartService2-6CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringErrorsandAlertsfromREICTherearedifferenttypesoferrorpagesusedinREIC.
Alltheerrorcodeswithdetaileddescriptionaretabulatedbelow.
AmessagespecifictoparticularerrorcodeisdisplayedontheREICscreen.
Ifthereisnomessageassociatedwiththeerrorcode,REICdisplaystheerrorcodeitselfonthetouchscreencollaborationpanel.
Table2-6REICErrors404errorindocumentsharingpanelforlocalfileuploadUnabletofetchtheurloftheuploadedfileCheckifthe'docstore.
url'or'docstore.
path'parameterisconfiguredproperlyinREADpropertiesbyusingtheTUI:MainMenu>REMServerAdministration>EditREMTemplates>f)READpropertiesNoroutetohostOccurswhenprintingalocalfileCheckifthe'webservice.
url'parameterismissinginREADpropertiesbyusingtheTUI:MainMenu>REMServerAdministration>EditREMTemplates>f)READpropertiesError/AlertDescriptionSolutionError/AlertDescriptionSolutionSystemisnotavailable,PleasetryaftersometimeOccursduetothefollowingreasons:1.
IfTPdetails(TPDirectoryNumberorTPType)isincorrect2.
TPMACaddressisnotaddedinCUCM>Applicationuser3.
IVRPhoneNumberisincorrect1.
VerifytheTPdetailsinREAC>Kiosk2.
AddtheTPMACaddressinCUCM>Applicationuser3.
CheckIVRPhoneNumberinREAC>ExpertTypeKioskphoneisoutofserviceOccursifconnectionwithintheKioskphoneisnotavailable1.
VerifythevaluesofCUCMinREMPropertiesbyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties2.
Kioskphoneisnotreachable.
Pleasechecktheconnectionstatus.
Sorry,thereisalreadyanactivecalltoanExpert.
PleasetryagainafterthecurrentcallendsOccurswhenanexpertisengagedinanothercallMakesurethatthecurrentcallendsontheTPandtryagainaftersometimeLocaleFiledoesnotexistOccursifthefileismissinginthesystemMakesureMessageBundleisaddedinREAC>Locale2-7CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringKioskserialnumberisnotconfigured,cannotstartacallOccurswhenKioskserialnumberisnotconfiguredproperlyVerifytheIECSerialNumberinREAC>KioskSelectedLocaleisnotassociatedtoanyexperttype.
PleasecontactadminOccursifthelocaleassociatedtothekioskisnotassociatedwithanyexperttypeAssociatetheExpertTypewithlocaleinREAC>ExpertTypeAgentsarenotavailablecurrently.
PleasetryaftersometimeOccurswhenallagentsarebusyTryagainafteraperiodoftimeExpertnotAvailableOccurswhen1.
TPisdown2.
TPIPaddressinREAC>Kioskisincorrect1.
PowerontheTPandrestartIECfromtheIEM2.
EditTPdetails(TPIPaddressinREAC>Kiosk)ServerisdownREMserverisdownBringtheREMserverupbyusingtheTUI:MainMenu>ServicesControl>ApplicationServer>b)StartServiceKioskisnotregisteredOccursiftheparticulardevicenumberisnotaddedinREACMakesurethiskioskisaddedinREAC>KioskSYSTEMERRORPleasecontactadmin.
Probablereasonsforthiserror:RESCisnotrunningOccursduringtheloadingofKioskhomepage1.
VerifyifservicesofRESCareavailablebyinvoking"https://:8443/resc/services/listServices"2.
VerifythevaluesofCUCMinREMPropertiesbyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties3.
RestarttheREMserverbyusingtheTUI:MainMenu>ServicesControl>ApplicationServer>c)StopService>b)StartServiceImproperloadingofREIC1.
Occursifthevalueof$*[RESC_IP]*inrem.
propertiesisincorrect2.
ExperttypeisnotconfiguredproperlyinREAC3.
IfStartupURLinIEMisincorrect1.
CheckRESCIPbyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties>REMCoreProperties2.
CheckExperttypeinREAC>ExpertType3.
CheckStartupURLintheIEMREIChangsOccursifthereareun-terminatedsessionsTerminatesessionsinREAC>CleanCallCacheError/AlertDescriptionSolution2-8CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringREMServer(RESC)ErrorsInordertocheckifRESCisrunningproperly,runthisURLinthebrowser:https://:8443/resc/services/AdminService/getVersionInfoIfRESCisdeployedorrunningproperly,thenyouwillseearesultsimilartothefollowing:REM-1.
9.
6IfRESCisnotdeployedorrunningproperly,thenyouwillseeadifferentresultsuchas:Theservicecannotbefoundfortheendpointreference(EPR)/resc/services/AdminService/getVersionInfoRefertothe"CallRelatedErrors"sectionofthischaptertotroubleshoottheseissues.
LogMonitoringThelogfilesforalltheREMcomponentsaretabulatedbelow.
Table2-7LogFilesLookintotherespectivelogfilesmentionedaboveifthereareanyerrorsinanyoftheREMcomponents.
REICisshowingCobrabrowserpagewithmessage"StartupURLisnotconfigured.
"OccursifthepolicyismissinginIEMCheckiftheIEMhascorrectpolicyassignedtothisKioskREICisshowing"ServiceTemporarilyUnavailable"OccursiftheStartupURLinIEMisincorrectorREMserverisunabletomakeconnectiontotheIEM1.
VerifyiftheIEMIPaddressiscorrectinREMPropertiesbyusingtheTUI:MainMenu>REMServerAdministration>b)EditREMProperties2.
VerifycorrectStartupURLintheIEMpolicyError/AlertDescriptionSolutionComponentNameLogFileLocationRESCTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>a)RESCLogs:resc.
logREACTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>a)REACLogs:reac.
logREICRefertothe"EventsTabintheIEM"forREICloglocationREADTUI:MainMenu>Troubleshooting>Logs>RemoteExpertLogs>a)READLogs:read.
log2-9CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringEventsTabintheIEMTheIEMallowsyoutodothefollowing:ViewalltheIECsregisteredwiththeIEMChecktheREIClogsToperformthetasksabove,followthesesteps:Step1Enterhttp://inabrowser.
Step2EnterthecredentialsandclickEnter.
Step3ClickDevicesontheleftnavigationpane.
AlltheIECsregisteredwiththeIEMinyouraccountarevisibleinboththeleftandcenterpanes.
Step4ClicktheShowscreenshotsbuttononthetoprightcornerofthedisplayviewtoviewthescreenshotofeachoftheIECsregisteredwiththeIEM.
ThescreenshotsarepulledfromthelasttimethattheIECswerepolledbytheIEM.
TheREICdisplaysthewelcomescreenandthedualcontentTelePresencescreen.
Step5DefaultpollingtimeforeachIECimageis10minutes.
ThemostrecentimageforthespecificIECcanbeobtainedbyclickingtheRefreshbuttonbeneaththeIEC'sscreenshot.
Step6InordertoenlargethescreenshotoftheIEC,clicktheZoombuttonbeneaththeIEC'sscreenshot.
NowyouwillcheckthelogsintheIEM.
Step7IntheDevicesmenu(leftpane),choosetheIECbydouble-clickingontheiconforthatparticularIEC.
Step8ClicktheEventstab.
Step9YoucanfilterthelogsrelatedtoREICbycheckingthebrowsercheckboxintheFacilitieslist.
ClickApply.
CallRelatedErrorsWhencallisinitiatedfromREIC,allcallflowrelatedinformationisgeneratedinthedatabase.
Thedifferentflowsofacallbothsuccessfulandfailedare:CallConnectCallHoldCallResumeCallTransferCallConferenceCallDisconnectThedurationofeacheventcanalsobeseeninthequeries.
Allthecallrelatederrorsforeachfailedeventarelistedintheresc.
logunderthe$REM_HOME/resc/logsdirectory.
2-10CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringThefollowingareexamplesofhowsystemadministratorscansearchfortheerrorsusingUnixcommands:1.
Theerrorforaparticulareventcanbeextractedfromtheresc.
loglocatedunderthe$REM_HOME/resc/logsdirectory.
Gotothelogsfolderwithinthe$REM_HOMEdirectorybyexecutingthecommand:cd$REM_HOME/resc/logs/2.
UsethegrepcommandtofindallCallConnectrelatederrors.
3.
Tosearchforerrorsataspecifiedtime,usethecommandvi|grep.
Forexample,catresc.
log|grep00:53:184.
Tosearchforerrorsthatoccurredonaparticulardevice,usethecommandvi|grepandentertheIECorTPserialnumber.
Forexample,catresc.
log|grep6560150300305.
Tosearchforcallrelatederrorsthatoccurred,usethecommandvi|grep.
tofetchallthelinescontainingthestring"call".
Forexample,catresc.
log|grep-icallJTAPIIssuesAgentRegistrationWhenDoneFromREACShowsaRedStatus1.
CheckifthecorrectCUCMIPandcredentialswereprovidedwhentheREMwassetup.
2.
Foracallconnectiontoworkproperly,theJTAPIadaptershouldbeconfiguredcorrectly.
a.
TheIPaddressandTPdetailsshouldbeconfiguredproperlytoinitiatethecall.
b.
Toreceivethecall,theagentDNmustbeconfiguredinREAC.
TheregistrationstatusoftheagentDNisavailableintheExperttabofREAC.
RefertotheCiscoRemoteExpertManagerAdministrationGuideforinstructionsonhowtoconfiguretheagentDN.
Iftheparametersabovearenotconfiguredproperlythenlogmessagesareloggedintheresc.
logfileunder$REM_HOME/resc/logs.
TocheckifincorrectCUCMIPorcredentialshavebeensetup,checkfortheerrorintheresc.
log.
ChecktheIPaddressofCUCMintherem.
propertiesusingtheTUI.
GotoMainMenu>REMServerAdministration>EditREMProperties.
REICDisplaystheErrorMessage"Systemerror,pleasetryaftersometime"IftheREICdisplaystheerrormessage"Systemerror,pleasetryaftersometime",theTPdetails(TPDNorTPtype)inREACareincorrect.
Tofixthisissue,refertotheCiscoRemoteExpertManagerAdministrationGuidetoconfigureSSHinREAC.
CustomerPodDisplays"ExpertnotAvailable"MessageThecustomerpoddisplaysthe"ExpertnotAvailable"errormessagewhenthevideoendpointIPAddressisincorrect.
Asaresult,theVirtualAgentServicesfailedtofunctionproperly.
Tofixthisissue,refertotheCiscoRemoteExpertManagerAdministrationGuidetoconfigureSSHhostnameinREAC.
2-11CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringCustomerPodDisplays"Kioskphoneisoutofservice"MessageIfthecustomerpoddisplaysthe"Kioskphoneisoutofservice",thevideoendpointIPphoneterminalsarenotregisteredinCUCM.
ExpertRegistrationIssueTheexpertregistrationononeofthenodesorbothisshownasred.
ThisoccurswhentheREMdatabaseisnotinsync.
Tocorrectthisissue,performdatabasesynchronizationandre-startPostgresonthenodewheretheexpertregistrationhasfailedbyfollowingthesesteps:Step1SSHintoREMviatheTUI.
Step2IntheMainMenu,selectc)ServicesControl.
Step3IntheServicesControlmenu,selectc)DatabaseServer.
Step4IntheDatabaseServermenu,selectc)StopServiceandfollowtheon-screeninstructions.
Step5IntheDatabaseServermenu,selecta)StartServiceandfollowtheon-screeninstructions.
Step6GotoREACandclicktheDBClustertab.
Step7ChoosetheDBnodeandclicktheActivatebutton.
RsyncCommandFailureAnrsynccommandfailureoccurswhenaRSAfingerprintingissuehappens(e.
g.
itgoesoutofsync).
Thisoccursduringafiletransferwhenhandshakingbetweenthetwonodesfail.
@WARNING:REMOTEHOSTIDENTIFICATIONHASCHANGED!
@ITISPOSSIBLETHATSOMEONEISDOINGSOMETHINGNASTY!
Someonecouldbeeavesdroppingonyourightnow(man-in-the-middleattack)!
ItisalsopossiblethattheRSAhostkeyhasjustbeenchanged.
ThefingerprintfortheRSAkeysentbytheremotehostis30:2b:2c:e4:34:7d:3f:88:05:50:09:5b:5e:13:1f:fc.
Pleasecontactyoursystemadministrator.
Addcorrecthostkeyin/root/.
ssh/known_hoststogetridofthismessage.
Offendingkeyin/root/.
ssh/known_hosts:1RSAhostkeyfor10.
76.
8.
192haschangedandyouhaverequestedstrictchecking.
Hostkeyverificationfailed.
Tofixthissolution,removetheknownhostkeysfromalltheothernodesintheREMHAclusterandsetupthersyncutilityagainbyfollowingthesesteps:Step1SSHintoREMviatheTACaccount.
Step2Executethefollowingcommands:sudocp/root/.
ssh/known_hosts/root/.
ssh/known_hosts.
baksudovi/root/.
ssh/known_hostsStep3IntheVIeditor,deleteallentries.
2-12CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringStep4SavethefileandexittheVIeditor.
REMNodeFailureREMnodefailureoccurswhenthesequenceofREMnoderestorationisincorrect:Incorrectapproachofshuttingdownthenodes:IfNode1(Database1hasoutdateddata),whichisthefirstnodetogodown,isstartedfirstthenfollowedbythestartupofNode2,theonlinerecoverywillsyncthedatafromDatabase1(outdated)toDatabase2(updated).
Withthissequence,youwilllosetheupdateddatawhichwasavailableinDatabase2thoughbothnodeswillhavesynchedupdata.
NooperationalissuewillbeexperiencedbutanychangetoNode2whenNode1wasdownislost.
Correctapproachofshuttingdownthenodes:Thecorrectapproachofshuttingdownthenodestoensurenodatalossistofirststartthedatabasethatisthelasttogodown(Database2inthiscase)sinceitwillhaveextradata.
OnceNode2isup,thenstartNode1.
Afterbothnodesarerunning,online-recovery(i.
e.
clicktheActivatebuttonforthenodesintheDBClustertabofREAC)willsync(copy)datafromNode2:Database2toNode1:Database1.
Thiswillensurethatboththenodeshaveupdateddataandnodataislost.
CommunicationFailureCommunicationfailureoccurswhenthecommunicationbetweentwonodeshasfailedduetoanetworkissue.
Inthefollowingtext,youcanobservethefollowinginthehaz.
log.
2013-02-0709:38:58,646INFO[hz.
_hzInstance_1_dev.
ServiceThread]com.
hazelcast.
cluster.
ClusterManager-[192.
168.
2.
4]:5701[dev]Members[1]{Member[192.
168.
2.
4]:5701this}2013-02-0709:38:58,650DEBUG[hz.
_hzInstance_1_dev.
ServiceThread]com.
hazelcast.
impl.
ListenerManager-[192.
168.
2.
4]:5701[dev]AddListenerOperationfromAddress[192.
168.
2.
4]:5701,local=truekey:nullop:REMOVE_LISTENER2013-02-0709:38:59,637DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-5]com.
hazelcast.
impl.
SplitBrainHandler-[192.
168.
2.
4]:5701[dev]Searchingforotherclusters.
2013-02-0709:38:59,637DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-5]com.
hazelcast.
impl.
TcpIpJoiner-[192.
168.
2.
4]:5701[dev]Address[192.
168.
2.
3]:5701islocalFalse2013-02-0709:38:59,637DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-5]com.
hazelcast.
impl.
TcpIpJoiner-[192.
168.
2.
4]:5701[dev]Address[192.
168.
2.
4]:5701islocaltrue2013-02-0709:38:59,637DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-5]com.
hazelcast.
impl.
TcpIpJoiner-[192.
168.
2.
4]:5701[dev]Address[192.
168.
2.
4]:5701isconnectingtoAddress[192.
168.
2.
3]:57012013-02-0709:38:59,638DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-4]com.
hazelcast.
nio.
SocketConnector-[192.
168.
2.
4]:5701[dev]StartingtoconnecttoAddress[192.
168.
2.
3]:57012013-02-0709:38:59,638DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-4]com.
hazelcast.
nio.
SocketConnector-[192.
168.
2.
4]:5701[dev]Connectingto/192.
168.
2.
3:5701,timeout:0,bind-any:true2013-02-0709:38:59,638DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-4]com.
hazelcast.
nio.
SocketConnector-[192.
168.
2.
4]:5701[dev]Couldnotconnectto:/192.
168.
2.
3:5701.
Reason:ConnectException[Connectionrefused]2013-02-0709:38:59,638DEBUG[hz.
_hzInstance_1_dev.
cached.
thread-4]com.
hazelcast.
nio.
SocketConnector-[192.
168.
2.
4]:5701[dev]Connectionrefusedjava.
net.
ConnectException:Connectionrefused2-13CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringatsun.
nio.
ch.
Net.
connect(NativeMethod)atsun.
nio.
ch.
SocketChannelImpl.
connect(UnknownSource)atcom.
hazelcast.
nio.
SocketConnector.
tryToConnect(SocketConnector.
java:106)atcom.
hazelcast.
nio.
SocketConnector.
run(SocketConnector.
java:55)atjava.
util.
concurrent.
ThreadPoolExecutor$Worker.
runTask(UnknownSource)CallConnectThefollowinglogshowsthatthecallfromthecustomerisconnectedsuccessfullytotheagent.
2013-02-0709:36:31,153INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3cdf672a)]com.
cisco.
big.
sm.
impl.
SessionManager-SessionActivated:215941142013-02-0709:36:31,153INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3cdf672a)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004AddingGCID:26002013-02-0709:36:31,158INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3cdf672a)]com.
cisco.
big.
call.
cm.
observer.
BaseObserver-LockAcquired2013-02-0709:36:31,158INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3cdf672a)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004SessionaddedtoAgentandLocktriedCallDisconnectThefollowinglogshowsthatthecallisdisconnectedsuccessfully.
2013-02-0709:11:05,996INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@7ab2f62)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1005Event:CallObservationEndedEv2013-02-0709:11:05,996INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@7ab2f62)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1005Call:1036-->1005CallId:25912013-02-0709:11:05,996INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@7ab2f62)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1005CallKey:1036CallHoldandReleasefromHoldThefollowinglogshowsthatthecallisplacedonholdandisreleasedfromholdsuccessfully.
2013-02-0709:13:12,037INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3612afd8)]com.
cisco.
big.
sm.
impl.
SessionManager-SessiononHoldSession:215941132013-02-0709:13:12,037INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3612afd8)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-CallPutonHold2013-02-0709:14:11,849INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3612afd8)]com.
cisco.
big.
sm.
impl.
SessionManager-SessionresumeSession:215941132013-02-0709:14:11,849INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@3612afd8)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1005CallPutonTalking2-14CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoringCallTransferThefollowinglogshowsthatthecallistransferredsuccessfully.
2013-02-0709:33:41,836INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallTransferEventHandler-1004TransferSequenceHandlerEvent:CallObservationEndedEv2013-02-0709:33:41,836INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallTransferEventHandler–10042013-02-0709:33:41,836INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserverAgentObserver-DN:10042013-02-0709:33:41,836INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004Event:CiscoTransferEndEv2013-02-0709:33:41,836INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004Call:1036-->1005CallId:25932013-02-0709:33:41,836INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004Connections:103610052013-02-0709:33:41,851INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004CallKey:10362013-02-0709:33:41,851INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004SequenceHandlerexists2013-02-0709:33:41,851INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallTransferEventHandler-1004-TransferSequenceDestructionHandler2013-02-0709:33:41,851INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallTransferEventHandler-1004DetectedParts:TransferDn:1005Kioskdn:null2013-02-0709:33:41,863INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallTransferEventHandler-KioskDnAssignedCallConferenceThefollowinglogshowsthatthecallissuccessfullyaddedtoaconferencecall.
2013-02-0709:31:01,637INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserverAgentObserver-DN:10042013-02-0709:31:01,637INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004Event:CiscoConferenceEndEv2013-02-0709:31:01,637INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004Call:1005-->UnknownCallId:25932013-02-0709:31:01,637INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004Connections:1036100410052-15CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoring2013-02-0709:31:01,642INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004CallKey:null2013-02-0709:31:01,642INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
AgentObserver-1004SequenceHandlerexists2013-02-0709:31:01,642INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallConferenceEventHandler-ReceivedhandleDestructionEvent2013-02-0709:31:01,642INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallConferenceEventHandler-Destroyingthishandler2013-02-0709:31:01,642INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallConferenceEventHandler-1004-ConferenceSequenceDestructionHandler2013-02-0709:31:01,642INFO[ObserverThread(com.
cisco.
big.
call.
cm.
observer.
AgentObserver@75775dde)]com.
cisco.
big.
call.
cm.
observer.
sequencehandler.
CallConferenceEventHandler-1004DetectedParts:TargetDN:1005KioskDN:1036WebServiceThewebservicecorrectlyloadsiftheconfigurationsforwebserviceinrem.
propertiesarecorrect.
Thecontentsofrem.
propertiesarethefollowing:$*[REM_VIRTUAL_IP]*=VirtualIPdenotestheCiscoACELoadBalancerVirtualIP.
IncaseofSingleNodeprovidethesameIPasthatof$*[RESC_IP]*$*[RESC_IP]*=ProvidetheIPaddressofREM#ForaCallFlowviaCCXsetthebelowpropertytotrue$*[IS_CCX]*=True,ifthecallflowisviaUCCX.
False,ifthecallflowisviaUCCE#ForaCallFlowviaCVPsetthebelowpropertytotrue$*[IS_CVP]*=False,ifthecallflowisviaUCCX.
True,ifthecallflowisviaUCCE#CUCMCredentials$*[CUCM_HOST]*=ProvideCUCMHostnameorIPaddress.
Itcanhavemultipleaddressesseparatedbycomma,ifyouhaveCUCMHAsetupHAProperties$*[Total_Nodes_In_Cluster]*=1ifyouhavesinglenode;2ifyouhaveREMHA#REMServerIP&databasedetails$*[NODE_IP_1]*=sameasRESC_IPifitisforasinglenode;privateIPofnode1ifitisforaHAsetup……$*[NODE_IP_2]*=ignorethisoneifitisforasinglenode;privateIPofnode2ifitisforaHAsetup#RSYNCtrueinclusteredsetup,falseinsinglenodesetup$*[RSYNC_ENABLED]*=falseifitisforasinglenode;trueifitisforaREMHA#REICCustomerlogoandbackgroundimagecanbeconfiguredusingbelowpropertiesfile$*[REIC_BIG_CUSTOMER_LOGO]*=Thisisrequiredtocustomizethekioskscreen.
ReferREICUIconfiguration$*[REIC_SMALL_CUSTOMER_LOGO]*=Thisisrequiredtocustomizethekioskscreen.
ReferREICUIconfiguration$*[REIC_BACKGROUND_IMAGE]*=Thisisrequiredtocustomizethekioskscreen.
ReferREICUIconfiguration2-16CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationMonitoring$*[EXPERT_TYPE_WIDTH]*=Thisisrequiredtocustomizethekioskscreen.
ReferREICUIconfiguration$*[REIC_SCREEN_MODULE]*=nullisforregularRE;keyboardisforkeypadsupport;magstripeisforcardreader$*[EXTENSION_MOBILITY]*=falseisforregularRE;trueisforConnectedJusticeWarningProvidingincorrectREACcredentialswillresultinafiletransfererror.
WarningProvidingincorrectCUCMcredentialswillresultinacallconnecterrorduetoaJTAPIissue.
DualContentIssueRefertothe"EventsTabintheIEM"sectiontomonitorwhetherdualcontentisdisplayingproperlyinthesecondaryscreenattachedtotheIEC.
Ifdualcontentisnotloaded,followthestepsbelowtotroubleshoottheissue.
Step1CheckifthedualscreenimageandtimeareconfiguredproperlyintheREAC.
(RefertotheCiscoRemoteExpertManagerAdministrationGuide.
)Step2Enterhttps://:8443/resc/services/AdminService/getContentByTimeIntervaltime=inabrowser(e.
g.
https://10.
76.
8.
171:8443/resc/services/AdminService/getContentByTimeIntervaltime=22-06-201209:52:00).
Step3Verifythatthestatuscodeindicates"SUCCESS".
Step4Enterhttps://:8443/reic/dualcontent.
htmlintheFirefoxbrowser.
Step5Checkwhethertheimageloads.
NoteYoucanusetheFirebugtoolinFirefoxtotracktheflowinloadingthedualcontentURL.
Step6Iftheimageloadedinthebrowser,gototheIEMandchoosetheIECinquestion.
Step7LookatthescreenshotoftheIEC.
TheREICapplicationscreenandthedualscreenimageshouldbedisplayedintheIECscreenshotpane.
FailureofRESCWeb(API)ServicesSometimesaftertherebootoftheREMVM,servicesprovidedbytheREMServerfailtostartduetodelayinsettingupofthe$REM_HOMEenvironmentvariablepath.
Inordertocircumventtheproblemabove,performthefollowingstepsaftertheREMVMserverisrebooted:Step1Runthemain.
shfilefromthebashshellusingthefollowingcommand:#.
$REM_HOME/tools/ias/scripts/main.
sh2-17CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationVerifyingVMSettingsStep2CheckwhetheralltheservicesareupandrunningbyaccessingthelistofservicesprovidedbytheRESCserverbynavigatingtothefollowingURL:https://:8443/resc/services/AdminService/getVersionInfoStep3Ifwebservicesarerunningproperly,thenyouwillseearesultsimilarto:REM-1.
9.
6VerifyingVMSettingsInstallationofavirtualmachineisdoneusingthevShpereclient.
TheREMVMsetuprequiressomeofthesettingstobeconfiguredandmaintained,suchastimezonesetup,masterconfigurationsetup,NTPandSMTPserverconfiguration,etc.
ForinstallationoftheVM,refertotheCiscoRemoteExpertManagerInstallationGuide.
Totroubleshoottime-relatederrors(e.
g.
dualcontentisnotworkingorfiletimestampmismatch),REMcomponentsconfiguration-relatederrors,connectivityerrors,ormail-relatederrors,reviewthefollowingtopics.
MasterConfigurationThemasterconfigurationfilecontainsalltherequiredsettingsthatarerequiredtogetthesystemupandrunning.
AtthetimeofinstallationoftheREMISO,somebasicinputsarerequiredtobeconfiguredbeforetheenvironmentcouldbepreparedforthesuccessfulexecutionofthevariousREMapplications(i.
e.
REIC,RESC,READ,andREAC)andthird-partyserviceslikePostgresandTomcat.
AllthevariablesorinputsrequiredareenlistedintheMastertemplatewhichneedstobepopulatedbeforehandorbytheadministratoratthetimeofinstallation.
TheMastertemplateislocatedat$REM_HOME/tools/ias/templates/rem.
properties.
RefertotheCiscoRemoteExpertManagerInstallationGuide.
NTPNTPsetupispartoftheREMISOinstallation.
NTPsynchronizestheclockofthelocalserverwiththeNTPserver.
Thisisrequiredforsynchronizationoffileswithtimestamps.
AllNTP-relatedsettingscanbefoundin/etc/ntp.
conf.
IfyouplantouseinternalNTPservers,followthestepsbelow.
Otherwise,youmayskipthisstep.
Step1UsingtheTUI,choosea)SystemSettingsintheMainMenu.
Step2IntheSystemSettingsmenu,chooseb)DateandTimeSettings.
Step3IntheDateandTimeSettings,choosea)SetupNTPSource.
Step4AVIwindowwillopen.
ReplacetheIPsorhostnameofyourNTPserversinthefollowingentries.
IfyouonlyuseoneortwoNTPservers,commentouttheunusedentriesbyadding#atthebeginningoflines.
server0.
rhel.
pool.
ntp.
orgserver1.
rhel.
pool.
ntp.
org2-18CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationVerifyingVMSettingsserver2.
rhel.
pool.
ntp.
orgStep5SaveandexittheVI.
NetworkConfigurationStep1Tosetupthenetwork,SSHintoREMviaTUI.
Step2IntheMainMenu,typeatochoosetheSystemSettingsmenu.
Step3IntheSystemSettingsmenu,typeatochooseNetworkSettings.
Step4IntheNetworkSettingsmenu,typeatochooseSetupNetworkInformation.
Step5IntheSetupNetworkInformationscreen,pressanykeytocontinue.
Step6ConfigureforeitherSingleNodeorREMHAsetup.
a.
SingleNode:1.
WithintheSelectActionscreen,chooseEditDevices.
2.
Selecteth0toconfigure.
PresstheEnterkey.
3.
UnchecktheUseDHCPcheckbox.
4.
EnterthedesiredIPaddressesfortheStaticIPaddress,NetmaskIPaddress,andDefaultgatewayIPaddressfields.
SelectOk.
5.
ChooseSaveandpresstheEnterkey.
6.
ChooseEditDNSConfiguration.
7.
EntertheHostname,PrimaryDNS,SecondaryDNS,TertiaryDNS,andSearchfields.
SelectOk.
8.
SelectSaveandQuit.
9.
GobacktotheMainMenu.
10.
Torestartnetworkservice,selectc)ServicesControlintheMainMenu.
Thenchoosea)NetworkingintheServicesControlmenu.
Finally,choosea)RestartnetworkingintheNetworkingmenu.
Ifnetworkissetupproperly,youshouldseeamessagesimilarto"UpdatingREMDBwithIPAddress:172.
20.
20.
20".
b.
REMHA(DualNode):1.
ChooseEditDevices.
2.
Selecteth0toconfigure.
PresstheEnterkey.
3.
UnchecktheUseDHCPcheckbox.
4.
EnterapublicIPaddressintheStaticIPaddress,NetmaskIPaddress,andDefaultgatewayIPaddressfields.
SelectOk.
5.
Selecteth1toconfigure.
PresstheEnterkey.
6.
UnchecktheUseDHCPcheckbox.
2-19CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationVerifyingVMSettings7.
EnteraprivateIPaddress(e.
g.
192.
168.
10.
100)intheStaticIPaddressfield.
IntheNetmaskIPaddressfield,enter255.
255.
255.
0.
LeavetheDefaultgatewayIPaddressfieldempty.
SelectOk.
8.
ChooseSaveandpresstheEnterkey.
9.
ChooseEditDNSConfigurationandentertheHostname.
LeavethePrimaryDNS,SecondaryDNS,TertiaryDNS,andSearchfieldsempty.
SelectOk.
NoteEachnodeshouldbegivenauniquehostname.
10.
SelectSaveandQuit.
11.
GobacktotheMainMenu.
12.
Torestartnetworkservice,selectc)ServicesControlintheMainMenu.
Thenchoosea)NetworkingintheServicesControlmenu.
Finally,choosea)RestartnetworkingintheNetworkingmenu.
13.
Ifnetworkissetupproperly,youshouldseeamessagesimilarto"UpdatingREMDBwithIPAddress:172.
20.
20.
20".
14.
Repeatsubsteps1-13forthesecondnode.
Step7Toverifyifthenetworkisconfiguredproperly:a.
IntheMainMenu,typeatochoosetheSystemSettingsmenu.
b.
IntheSystemSettingsmenu,typedtochooseSystemInformation.
NoteIftheIPaddressneedstobechangedlater,youmayneedtofollowthissectiontoreconfigureyournetwork.
EffectonDatabaseFirst,makesurethatyouSSHtoREMviatheTACaccount.
ExecutethecommandsbelowontheShellpromptofREMservertochecktheREMDBDetails:Step1OntheShellpromptoftheREMserver,execute/etc/init.
d/postgresql-statustocheckPostgresstatus.
Step2LogintothenodewhereREMDBisrunning.
Step3AttheDBpromptoftheREMserver,executethefollowing:[REM1-9]$sudosu-lpostgres-bash-3.
2$psqlpostgresStep4CheckthelistoftablesavailableinREM_DB.
Step5AttheDBprompt,execute\dtolisttheREMdatabasetables.
Step6Totakeadatabaseschemabackup,enterpg_dump-Upostgres-hlocalhost--schema-only-fREM_DB.
2-20CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationVerifyingVMSettingsStep7Totakeadatabasedatabackup,enterpg_dump-Upostgres-hlocalhost--data-only-fREM_DB.
Step8Restoretheschemafromthesqlfilebyenteringthepsql-Upostgres-hlocalhost-dREM_DB-fcommand.
Step9Logintothedatabase.
Step10Executethe\icommandtorestorethedatafromthesqldumpfile.
Step11DropthedatabaseusingtheDBdropdb-Upostgres-hlocalhostREM_DBcommand.
Step12Createthedatabaseusingthecreatedb-Upostgres-hlocalhost'REM_DB'command.
Step13Checkthelistofdataavailableinthetable.
Step14AttheDBprompt,executeSelect*from.
Step15TologoutfromREMdatabase,enter\qattheREM_DBprompt.
DocumentAfteraddingthedocumentfromREAC,thedocumentwillbeaddedinthedatabasewithcreationdate(inthedoc_tableitiscalled'added_date').
Iftimezoneisnotconfiguredproperly,itisverydifficulttomaintainthedatabase.
CallSessionWhencustomertouchesanexperttypebuttonontheREIC,acallisinitiated.
Thiscallwillcometotheremoteexpertdesk.
Afteransweringthecall,asessionhasbeenestablished.
Thiscallsessioninformationcanbefoundinthesessiontableofthedatabase.
NoteThesessioninformationcontainsthetimewhenthesessionwasgenerated.
ThisinformationisgeneratedfromRESC.
Ifthetimezoneisnotconfiguredproperly,thenthiscallstarttimewillnotbecorrectcomparedtothelocaltimewheretheserverislocated.
Ifthatisthecase,itisverydifficulttoinspectthetimewhenthesessionwasgenerated.
VideoAllthevideosaddedbyREACareaddedintotheREMdatabase.
ThevideoisaddedtothedatabasewiththetimewhenitwasaddedtotheREAC.
Formaintainingthedatabasetimezone,theVMshouldbeconfiguredproperly.
REMEventsWhenacallisinitiatedfromtheREIC,allthecallrelatedinformationsuchascallconnect,callonhold,orcalldisconnectarevisibleinthedatabase.
Thestarttimeforeachcallstatusisalsoaddedtothedatabase.
Anincorrectlyconfiguredtimezoneaffectsthistime.
2-21CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationTUITroubleshootingToolsLogFilesThefollowinglogsareavailable:TomcatCatilanalogREAClogREADlogREMBackuplogREMRestorelogNoteTimezoneaffectsthegenerationoflogfiles.
Logsaregeneratedwithcurrenttimeanddateinthelogfiles.
TUITroubleshootingToolsThissectionexplainshowtousetheTroubleshootingtoolsintheTUIthatisaccessedbyaSSHclient.
PingUsethePingutilitytoverifythattheREMserversareup.
ThisisparticularlyusefulforHAwheretherearefourIPaddressesthatshouldbeverified.
Step1IntheMainmenu,typeetoaccesstheTroubleshootingmenu.
Step2IntheTroubleshootingmenu,typeatopingahost.
Step3EntertheIPaddressoftheREMserver.
Thehostwillbepingedandtheresultsaredisplayed.
Step4Whenfinished,pressanykeytoreturntothepreviousmenu.
System,RE,andApplicationServerLogsThereareanumberoflogsthataregeneratedthatcanhelpyouwithtroubleshootingissues.
Systemlogs:Thesearebasicallyhardware-relatedlogsincludingsystemconsolemessages,authenticationandauthorizationlogs,anddrivermessages.
RemoteExpertlogs:ThemajorityoflogsthatyouwillusearelocatedinthismenuwhichincludeslogsforalltheREMcomponents(REAC,READ,RESC),IAStool,andReportGenerationtoolaswellaslogsforBackupandRestore.
ApplicationServerlogs:ThesearetheTomcatandPostgreslogs.
Step1IntheTroubleshootingmenu,typebtoaccesslogs.
2-22CiscoRemoteExpertManagerTroubleshootingandServiceabilityGuideChapter2ServiceabilityAdministrationTroubleshootingToolsinREACStep2Choosealogmenu:System,RemoteExpert,orApplicationServer.
NoteForsecurityreasons,theTUIwilldisconnectafterthreeminutesofnoactivityonthepartoftheadministrator.
Iftheadministratorwantstoviewlogactivityformorethanthreeminutes,theadministratorneedstointeractwiththeTUIinsomewaysuchaspresstheEnterkeyinordertokeepaconnection.
Step3PressESC:q!
tostopwatchingalog.
Step4Whenfinished,presstheR,<,or,keytoreturntothepreviousmenu.
TroubleshootingToolsinREACTheDownloadLogstabinREACcontainslistsoflogsthatcanbeusedfortroubleshootingissues:1.
AdminConsole:Containsthereac.
logfromthisREAC2.
AgentDesktop:Containstheread.
logfromREAD3.
SessionManager:Containstheresc.
logfromtheRESC4.
REEvents:CaptureREeventsduringaspecifiedtime5.
Tomcat:ContainsthecatalinalogfortheTomcatserviceRefertothe"DownloadingLogs"sectionoftheCiscoRemoteExpertManagerAdministrationGuideforinstructionsonhowtodownloadthoselogs.