platformsblaze
blaze 时间:2021-01-12 阅读:()
ExportMemblazePBlaze4NVMeSSDbyNVMeOverFabricOrNVMeOverFabricinSPDKEnvWHITEPAPERExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
- platformsblaze相关文档
- Assembliesblaze
- 请浏览网站获取更多信息:www.memblaze.com
- hitblaze
- averageblaze
- Ellefsenblaze
- 系列blaze
tmhhost：全场VPS低至6.4折，香港BGP200M日本软银美国cn2 gia 200G高防美国三网cn2 gia韩国CN2
tmhhost放出了2021年的端午佳节+618年中大促的优惠活动：日本软银、洛杉矶200G高防cn2 gia、洛杉矶三网cn2 gia、香港200M直连BGP、韩国cn2，全都是高端优化线路，所有这些VPS直接8折，部分已经做了季付8折然后再在此基础上继续8折（也就是6.4折）。 官方网站：https://www.tmhhost.com 香港BGP线路VPS ，200M带宽 200M带...
HostYun全场9折,韩国VPS月付13.5元起,日本东京IIJ线路月付22.5元起
HostYun是一家成立于2008年的VPS主机品牌，原主机分享组织(hostshare.cn)，商家以提供低端廉价VPS产品而广为人知，是小成本投入学习练手首选，主要提供基于XEN和KVM架构VPS主机，数据中心包括中国香港、日本、德国、韩国和美国的多个地区，大部分机房为国内直连或者CN2等优质线路。本月商家全场9折优惠码仍然有效，以KVM架构产品为例，优惠后韩国VPS月付13.5元起，日本东京...
萤光云（13.25元）香港CN2 新购首月6.5折
萤光云怎么样？萤光云是一家国人云厂商，总部位于福建福州。其成立于2002年，主打高防云服务器产品，主要提供福州、北京、上海BGP和香港CN2节点。萤光云的高防云服务器自带50G防御，适合高防建站、游戏高防等业务。目前萤光云推出北京云服务器优惠活动，机房为北京BGP机房，购买北京云服务器可享受6.5折优惠+51元代金券（折扣和代金券可叠加使用）。活动期间还支持申请免费试用，需提交工单开通免费试用体验...
blaze为你推荐
-
域名备案查询如何查网站备案信息vpsVPS是什么？vps主机vps主机是什么?中文域名注册查询哪里有可以查询中文域名是否被注册的地方？海外域名求国外域名商列表域名备案域名怎么进行备案？重庆虚拟空间重庆合川宝龙城市广场有前途么虚拟主机管理系统我也想和你学虚拟主机管理系统的操作合肥虚拟主机虚拟主机怎么弄！山东虚拟主机能否在虚拟机与主机之间建立局域网，让主机与虚拟机同时上网？