platformsblaze
blaze 时间:2021-01-12 阅读:()
ExportMemblazePBlaze4NVMeSSDbyNVMeOverFabricOrNVMeOverFabricinSPDKEnvWHITEPAPERExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
- platformsblaze相关文档
- Assembliesblaze
- 请浏览网站获取更多信息:www.memblaze.com
- hitblaze
- averageblaze
- Ellefsenblaze
- 系列blaze
简单测评melbicom俄罗斯莫斯科数据中心的VPS，三网CN2回国，电信双程cn2
melbicom从2015年就开始运作了，在国内也是有一定的粉丝群，站长最早是从2017年开始介绍melbicom。上一次测评melbicom是在2018年，由于期间有不少人持续关注这个品牌，而且站长貌似也听说过路由什么的有变动的迹象。为此，今天重新对莫斯科数据中心的VPS进行一次简单测评，数据仅供参考。官方网站： https://melbicom.net比特币、信用卡、PayPal、支付宝、银联...
统计一下racknerd正在卖的超便宜VPS，值得推荐的便宜美国VPS
racknerd从成立到现在发展是相当迅速，用最低的价格霸占了大部分低端便宜vps市场，虽然VPS价格便宜，但是VPS的质量和服务一点儿都不拉跨，服务器稳定、性能给力，尤其是售后方面时间短技术解决能力强，估计这也是racknerd这个品牌能如此成功的原因吧！ 官方网站：https://www.racknerd.com 多种加密数字货币、信用卡、PayPal、支付宝、银联、webmoney，可...
virmach：AMD平台小鸡，赌一把，单车变摩托？$7.2/年-512M内存/1核/10gSSD/1T流量，多机房可选
virmach送来了夏季促销，价格低到爆炸，而且在低价的基础上还搞首年8折，也就是说VPS低至7.2美元/年。不过，这里有一点要说明：你所购买的当前的VPS将会在09/30/2021 ~ 04/30/2022进行服务器转移，而且IP还会改变，当前的Intel平台会换成AMD平台，机房也会变动（目前来看以后会从colocrossing切换到INAP和Psychz），采取的是就近原则，原来的水牛城可能...
blaze为你推荐
-
域名主机电脑域名是什么com域名注册com域名注册要注意哪些情况啊?我想现在注册一个com域名~免费网站域名申请哪里可以申请到免费网站域名？免费vps服务器免费服务器有哪些论坛虚拟主机最适合做论坛的虚拟主机是什么？论坛虚拟主机虚拟主机禁止放论坛河南虚拟主机新乡在哪个网站买虚拟主机好？虚拟主机99idc如何选择虚拟主机的的操作系统以及更换操作系统是注意事项西安虚拟主机谁能推荐一个比较好西安网络公司?www二级域名一级域名 二级域名 三级域名什么区别