databaseUselotus

IBMLotusDominoServerandApplicationPerformanceintheRealWorldAndrewPollack,PresidentNorthernCollaborativeTechnologiesandrewp@thenorth.
comhttp://www.
thenorth.
comWhoAmIAdministrator&Developersinceversion2.
0IBMLotusBeaconAwardWinnerServicesSitePerformanceReviewsLegalCaseConsultingApplicationDevelopmentAdministrativeOverhaulSecurityReview&PenetrationTestingProductsNCTSearchNCTComplianceSearchNCTSimpleSignOnStructuralFirefighterKeyFocusPointsPerformancewithaBigPictureapproachDefiningPerformanceInUserTermsKeyPerformanceChokePointsGeneralConsiderationsCommonGeneralTweaksMakeYourWebSiteFaster!
Developers,Developers,DevelopersServersandSANSandVMs–ohMy!
VirtuallyPerfectionWhengoodINIsettingsgoBad!
FindingYourOwnChokePointsSummaryCanPreventPerformanceProblemsPERFORMANCEWITHABIGPICTUREAPPROACHBigPicture:ThereIsNoMagicNoSingleINIVariable--#1ServerFixFocusOnTheBasics!
NoSuperStorageNetworkNoUltimateNetworkSwitchNoOmnipotentThirdPartyApplicationNober-ConsultantNotEvenMe!
BigPicture:SmallIssuesStackUpPerformanceProblemsArelikesnowflakesIndividually,theydon'tmattermuchatallYounoticethemonlyoncetheystackupForexample:PoorlyPerformingDiskI/O+AgentsChangingManyDocuments+ManyViews(orBADviews)toUpdate==VerySlowSystemThesekindsofproblemscreateafeedbackloop,whichamplifiestheproblemsBewareofINIChangesMyNumberOneServerCrashResponseINIChangesComeFromWellmeaningtipsLowleveltech-supportToomuchtimeonpublicforumsHowtofixmostservercrashesCleanoutALLnon-defaultINIsettingsUnlessyoucanspecificallydocumentwhyit'scriticalCleanoutALLnon-shippingCodeGetridofthosefix-packsthatdidn'tfixtheproblemYes,therearesomegoodchangestomaketotheINIfileDEFININGPERFORMANCEINUSERTERMSIt'snothowyoufeel,itshowyoulook.
Darling,youlookmarvelous!
BillyCrystalPerformanceinUserTermsIftheusermustwaitforsomething,itwillalwaysseemslow–nomatterhowfastyoumakeit.
Nothingisworsethananhourglasscursorandabarslowlymovingacrossthescreen…ExceptNOThavingthebarPerformanceinUserTerms:TipsMoveanythingnotimmediatelyrequiredbytheusertoabackgroundprocessBatchprocessupdatesofdatathatusersdonotneedinstantlyCacheCommonlyReferencedDataHowoftendoyourcommonlookupschangeCountryNamesEscalationLevelsDocumentCategoriesLookuponcewhenthedatabaseisopened,andstorethevaluesasenvironmentvariableslocallyDon'tpop-upmodaldialogboxeswithnochoices!
KEYPERFORMANCECHOKEPOINTSWe'regoingthewrongway,butwe'remakingexcellenttime!
ChokePoints:TheNetworkBandwidthvs.
LatencyBandwidthHowbigaroundisthepipeLatencyHowlongisthepipefromendtoendEvenlighttakesseveralminutestoreachusfromtheSun.
Latencyimpacts"Chatty"connectionsNotesDatabaseOpenMultipleViewLookupsAJAXonWebApplicationsWheredoesLatencyComefromPingtimeslargerthan100msare"high"latency.
WANlinks,Satellitelinks,Modems,andVPN'sareallpronetolatencyissuesMulti-HopconnectionsacrossbufferedroutersandfirewallscanintroducelatencyEncryptionsoftwarecanintroducelatencyDealingwithHighLatencyAvoidopeningandclosingmanydocumentsAvoidDBLookupsbycachingcommonvaluesExample:UseadbopenscripttowritecommonlookupvaluestoalocalenvironmentvariableeachtimetheuseropensthedatabaseUse"RunOnServer"tomovecomplexagentworktotheserver,thereadtheresultfromaprofiledocumentConsiderJSONembeddedonthedocumentinsteadofAJAXlookupsStopusing"NoCache"onyourDBLookupsChokePoints:DiskI/OThisisthe#1,#2,and#3RootperformanceproblemonDominoServerNearlyanyotherperformanceproblemismademanytimesworseiftheDiskI/OisoverwhelmedMostDominoServersarenotwelloptimizedforDiskI/OCommonSourcesofDiskPerformanceProblemsFailuretouseDAOS!
One"Data"driveisusedfortoomuchdatabases,indexrebuilds,temporaryfiles,swapfiles,andeventransactionloggingPoorSANconfigurationforDominovolumesTooheavyarelianceonStorageAreaNetworksPoorchoiceofRAIDconfigurationsForFSMSake,StartUsingDAOS!
DAOSissafe.
Itwilllikelysaveyou50%ormoreofyourstoragespaceReally-Itissafe.
Thatmeans50%ormoresavingsinDISKI/OaswellIt'snotlike"SharedMail"–IpromiseItalsomeans50%lessspaceoneverybackupAndit'ssafe,too!
InthenewestserverversionsitwillalsosavenetworktrafficWheneverPossibleUseMultipleDrivesPutyourtransactionloggingfilesonaseparatedriveMoveyourviewindexingtemporaryfilestoanotherdriveConsidermovingdisk-intensiveapplicationstotheirowndriveIfyoumusthavememoryswapping,giveititsowndriveActiveLogFilesforWebServers,SMTP,etc.
canalsobeoffloadedtotheirowndrivesNotEverythingNeedsItsownDriveThingsthatloadonceandarenotre-accessedfrequentlydonotneedtobeonhighperformanceresourcesTheOperatingSystemApplicationProgramFilesArchivedLogFilesUniqueDrivelettersmaynotbedifferentdrivesInVirtualizedenvironmentsandonaSAN,multiplevirtualdrivesonthesamephysicalvolumedonothelpOneDiskmayhavemultiplepartitionsDifferentpartitionsareNOTdifferentspindlesAllthepartitionsonthesamedrive,sharethesameread-writeheadandareimpactedbydataaccessasasingleentity.
MultipledrivesinaRAIDarraydon'tcountARAIDarrayistreatedbythesystemasasingledrive.
Bydefinition,dataiswrittenacrossthewholearrayThe"Best-Case"ismultipledrivesondifferentdrivecontrollersTooheavyarelianceonStorageAreaNetworksTheSANisnotEvil–Butitisn'tperfecteitherHighSpeed,butHighLatencyASANisaCompromiseTradesthespeedandsimplicityoflocaldrivesforenterprisemanageabilityandflexibilityGoodforBackupDataGoodforBig,SequentialFilesMediaFilesInstallationKitsArchivalDataChokepointforactivedatabaseworkDominowithaSANConsiderthebenefitsofaSANHighlyredundantstorageSinglebackuppointConsolidatedfreespacePerformanceIhaveyettoseeaSANthattrulyoutperformslocalhighspeeddisksNotallDominoDataneedsthesefeaturesTransactionLogs–ConsiderlocalRAIDifpossibleIndexingScratchSpace–UseCheap,Local,FastDrivesIfyou'realreadyclusteringDomino,onlyoneoftheclusteredmachinesmayneedtobeontheSANIfyoudouseaSANWorkwiththeSANteamtoconfigureyourvolumesDedicatedLUN&Disksforeachoftheseifpossible:DominoDataTransactionLogsTempSpaceforviewindexrebuildingOperatingsystem"TEMP"variableNotes.
INI"View_Rebuild_Dir="TelltheSANteamtotreatitlikearelationaldatabaseHighlyReadIntensiveWherecanyoucompromiseCheaplocaldrivesforlow-riskuseMemorySwapFileTemporaryScratchSpaceforViewRebuildsWebServerCacheFilesLogFilesVirtualizationandDominoDominorunsjustfineinVMWARESomeofmybestfriendsarevirtualserversAllmyproduction&developmentserversareinVMsPerformanceissuesareVERYsimilartoSANsDiskI/OisagaincriticaltoDominoperformanceVirtualenvironmentsoftensharediskresourcesVirtualenvironmentsoftenutilizeSANsFollowtheguidelinesforusingDominoonaSANLocal,dedicatedstoragespindleswhereverpossibleDedicatedLUN&DiskswhereveryoucanPoorchoiceofRAIDconfigurationsRAIDisnotALWAYSthebestperformancechoiceSomeCommonTypesofRAIDRAID0Increaseperformance,DecreaseReliability(xnumberofdrives)RAID1IncreaseReliability,NoPerformanceDifferenceRAID5(TheMostCommon)–Uses3ormoredrivesBalanceofredundancyplussomeperformancegainRAID1+0(akaRAID10)TwopairsofRAID1readasaRAID0(Hybrid)RAID0+1TwopairsofRAID0writtenasRAID1(Hybrid)WhyRAIDisn'tALWAYSthebestwayCompetitionforResourcesAnoverallperformancegainwithRAID5of30%(typically)isspreadacrossallthediskI/Oontheserver.
YouaresmarterthanRAIDYoucanputhighlyintensiveresourcesonspecificdrivesorarrays,balancingtheloadmoreeffectivelyMultipleRAIDArraysarenotalwayspossibleExpensiveMultipleDriveBaysPowerHungryDrivesDiskI/O:RethinkingRAIDMostRAIDarraysareconfiguredtoimproveredundancy,notnecessarilyspeed.
NotalldatarequiresredundancyLossofsomedataisverylowriskMemorySwapFilesIndexingscratchspaceTemporaryfilesCachefilesInexpensiveSATAdrivescanbeusedforarealperformancegainSolidStateDrives–VeryfastbutoftennotideaSolidStateDrives(SSD)Alsoknownas"FLASH"drivesGettingmorecommononLaptops,NetbooksReliabilityIssuesareLargelyResolvedVERYFastREADTimesWritePerformanceQuicklyDegradesThisischangingquickly,butstillthecaseformostusesWindows7&WindowsServer2008R2Support"TRIM"http://en.
wikipedia.
org/wiki/TRIMGoodforProgramFiles,JavaLibrariesBadforNSFDatabases,Indexing,TranslogsChokePoints:SystemResourcesTheseshouldbeobviousMoreRAMisbetter–UptowhatissupportedDependingontheOS,youmayneedtopartitionyourservertotakefulladvantageDriveCache–IfyourOSletsyoumanageit,youshouldworktoreallyoptimizethisMostAnti-VirusSoftwareisEVILwhenitrunsagainstDominoDatabasesMakesureyourAVisDominoaware!
DoyoureallyneedAVsoftwarerunningonaDominoServerHint:No,youusuallydon'tMAKEYOURWEBSITEFASTER!
Fasterfaster!
Thelightsareturningred…LetthebrowsercachecommonitemsResourcesthatdon'tchangefrequentlycanbecachedJPGPNGGIFMOVMP3MSIMPGZIPEXEAPPLICATIONDESIGNSTRATEGIESDevelopersreallyLOVEwhenadministratorsgivethemfeedbackChokePoints:ViewsForapplicationperformancetuning,viewsarethefirst,second,andthirdplacetolookViewindexingisverydiskintensive–andcanamplifydiskI/OshortcommingsToupdateaview,afulldatabasescanoftenneedstohappen.
ThatcanbeveryveryslowonlargedatabasesAnyviewperformanceproblemgrowsexponentiallywiththevolumeofdataTheseproblemsareoftennotcaughtintestWHENGOODVIEWSGOBADWhatKillsViewPerformanceUsethe"ManageViews"AdminClientFeatureBadViewDesign:TooMuchDataSwitch@Responesesto@AllDescendantsNOvisibibledifferencetousersCanreduceviewsizesdrasticallyCanYouSetaCUTOFFdateForm="Request"&@ModifiedDominoConfigurationTunerDeliveredasadatabasetemplate(DCT.
NTF)Availableforfree–downloadfromIBMhttp://www-01.
ibm.
com/support/docview.
wssuid=swg24019358EvaluatestheserverandcomparestoknownbestpracticesMakesrecommendationsforchangesRecommendationsaregenerallygood–butnotuniversalDonotfollowblindly–UnderstandtherecommendationfirstDocumentanychangesyoumakesotheycanbeundoneIfsomethingshould"alwaysbeset"acertainway,itwouldbethedefault.
INISETTINGTWEAKSYoureallycameherelookingforcoolINIsettingslikeDominoRunFaster=11Remember–Manyofthesehavebecomethedefaultovertime.
YouareusuallyBetteroffusingthedefaultsettings.
Beespeciallycarefulofoldinisettingsafteryouhaveupgradedtheservertoanewversion.
ObsoleteINIsettingsHURTperformanceSomeNOTES.
INItweaksCOMMENTNOTES.
INIChanges!
Here'ssomethatIuseMailLeaveSessionsOpen=1Forbusymailservers,canspeedupdeliveryUpdate_Fulltext_Thread=1Movefulltextindexingtoitsownthread,distinctfromtheindexer–Thisistheclosestto"runfaster"IhavefoundFtg_use_sys_memory=1UsememoryoutsidetheDominoserverHttpQueueMethod=2LikehavingonelineformultiplecashregistersDefaultin8.
5.
1andlaterAfewmorenotes.
initweaksUseTheseTogether:SERVER_NAME_LOOKUP_NO_UPDATE=1TellstheservertousetheoldindexwhilethenewonecatchesupStartingwith8.
0thisshouldbethedefaultDEBUG_ENABLE_UPDATE_FIX=8191FinetuneswhenthedirectoryindexesgetrefreshedStartingwith8.
0.
1thisshouldbethedefaultSERVER_MAX_CONCURRENT_TRANS=-1Defaultasof8.
03and8.
0–usedtobe20SERVER_POOL_TASKS=100Defaultasof7.
03an8.
0isnow40Usedtobe2timesserver_max_concurrent_trans100isusedinIBMPerferformanceTestingSetlowerifprocessorismaxedAndofcourse…NSF_Buffer_Pool_Size_MBNSF_Buffer_Pool_Size_MB=Verypowerful,butverycomplexChecktheLotusNotesKnowledgebaseStartsataround300NotascriticalasitusedtobeDocumentationSaysitisnowsetAUTOMATICALLYfornon-partitionedServersMyTestingSaysitisalsonowsetAUTOMATICALLYevenforpartitionedServersin8.
5.
xCheckyoursuccesswiththisconsolecommandshowstatdatabase.
database.
b*Don'tchecktoosoonafterachange,itsonlyvalidovertimeNotes8ClientTweakTomaketheEclipsebasedclientloadfasterOpenthisfolder:{NotesProgramDirectory}\framework\rcp\deployPriorto8.
5.
1usethisfolderinstead:{NotesProgramDirectory}\framework\rcp\eclipse\plugins\com.
ibm.
rcp.
j2se.
{Version}Editthefile:jvm.
propertiesChangetheline:vmarg.
Xmx=-Xmx256mSothatitreads:vmarg.
Xmx=-Xmx512mNote:Youcansetithigher,butaimfornomorethanhalfofyouravailableRAMReadersonmyblogoverwhelminglyreportfantasticresultswiththisoneSummaryRepeatAfterme:ThereisNo"RUN_FASTER=1"IwillcleanupmyNOTES.
INIIwillCOMMENTmyNOTES.
INIchangesPerformanceIsn'tMagic,itsPlanningSavetheDiskI/O,SavetheServerLatencyisascriticalasBandwidthWhenindoubt,BlamethedeveloperQuestionsAsknow,don'twaitfortheendandaskquietlyatthepodiumThemostuptodatecopyofthispresentationwillbeonmyblogsite:http://www.
thenorth.
com/apblogAndrewPollack–NorthernCollaborativeTechnologiesandrewp@thenorth.
comhttp://www.
TheNorth.
com*