platformsblaze
blaze 时间:2021-01-12 阅读:()
ExportMemblazePBlaze4NVMeSSDbyNVMeOverFabricOrNVMeOverFabricinSPDKEnvWHITEPAPERExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
- platformsblaze相关文档
- Assembliesblaze
- 请浏览网站获取更多信息:www.memblaze.com
- hitblaze
- averageblaze
- Ellefsenblaze
- 系列blaze
wordpress外贸企业主题 wordpress经典外贸企业建站主题
WordPress经典外贸企业建站主题，经典配色扁平化简约设计+跨屏自适应移动端设备，特色外贸企业建站功能模块+在线Inquiry询单功能，更有利于Google等英文搜索优化和站点收录。采用标准的HTML5+CSS3语言开发，兼容当下的各种主流浏览器： IE 6+(以及类似360、遨游等基于IE内核的)、Firefox、Google Chrome、Safari、Opera等;同时支持移动终端的常用...
Spinservers：美国独立服务器(圣何塞),$111/月
spinservers是Majestic Hosting Solutions，LLC旗下站点，主营美国独立服务器租用和Hybrid Dedicated等，spinservers这次提供的大硬盘、大内存服务器很多人很喜欢。TheServerStore自1994年以来，它是一家成熟的企业 IT 设备供应商，专门从事二手服务器和工作站业务，在德克萨斯州拥有40,000 平方英尺的仓库，库存中始终有数千台...
极光KVM美国美国洛杉矶元/极光kvmCN7月促销,美国CN2 GIA大带宽vps,洛杉矶联通CUVIP,14元/月起
极光KVM怎么样?极光KVM本月主打产品：美西CN2双向，1H1G100M，189/年!在美西CN2资源“一兆难求”的大环境下，CN2+大带宽 是很多用户的福音，也是商家实力的象征。目前，极光KVM在7月份的促销，7月促销,美国CN2 GIA大带宽vps,洛杉矶联通cuvip,14元/月起;香港CN2+BGP仅19元/月起，这次补货，机会，不要错过了。点击进入：极光KVM官方网站地址极光KVM七月...
blaze为你推荐
-
买虚拟主机如何选择、购买虚拟主机域名注册商最具权威的域名注册商me域名注册为什么现在无法注册.me的域名了全能虚拟主机旗舰型全能主机500m(x500.特惠虚拟主机)要什么数据库中国互联网域名注册负责我国境内internet用户域名注册是什么机构免费com域名注册有没有完全免费的域名？asp主机asp.net虚拟主机怎么样，它和asp虚拟主机是不是一样的，求解释免费国外空间那个国外空间好啊啊 价格便宜 急需域名服务域名服务器是什么？有什么作用美国vps主机美国VPS好？还是香港VPS好？