intendedpunkbuster

UsabilityHeuristicsforNetworkedMultiplayerGamesDavidPinelle1,NelsonWong2,TadeuszStach3,CarlGutwin21NationalResearchCouncil46DineenDriveFredericton,NB,CanadaE3B9W42UniversityofSaskatchewan110SciencePlaceSaskatoon,SK,CanadaS7N5C93Queen'sUniversity25UnionStreet,GoodwinHallKingston,ON,CanadaK7L3N6david.
pinelle@nrc.
gc.
ca,nelson.
wong@usask.
ca,tstach@cs.
queensu.
ca,gutwin@cs.
usask.
caABSTRACTNetworkedmultiplayergamesmustsupportamuchwidervarietyofinteractionsthansingle-playergamesbecausenetworkedgamesinvolvecommunicationandcoordinationbetweenplayers.
Thismeansthatdesignersmustconsideradditionalusabilityissuesthatrelatetogroupplay–buttherearecurrentlynousabilityengineeringmethodsthatarespecificallyorientedtowardstheneedsofmultiplayergames.
Toaddressthisproblem,wedevelopedanewsetofusabilityheuristics,calledNetworkedGameHeuristics(NGH),whichcanbeusedinthedesignandevaluationofnetworkedmultiplayergames.
Thenewheuristicswereidentifiedbyanalyzingproblemreportsfrom382reviewsofnetworkedPCgames,coveringsixmaingenres.
Weaggregatedproblemreportsintotenproblemcategories(coveringissuesfromsessionmanagementtocheatingtotrainingfornoviceplayers)anddevelopedheuristicsthatdescribehowtheseusabilityproblemscanbeavoided.
Wetestedthenewheuristicsbyhavingevaluatorsusethemandanexistingsettoassesstheusabilityoftwonetworkedgames.
EvaluatorsfoundmoreusabilityproblemswithNGH,andstatedthatthenewheuristicswerebetterforevaluatingmultiplayergameusability.
Ourresearchisthefirsttopresentnetworkedgameheuristicsthatarederivedfromrealproblemreports,andthefirsttoevaluatetheheuristics'effectivenessinarealisticusabilitytest.
CategoriesandSubjectDescriptorsH5.
2[UserInterfaces]:Evaluation/methodologyGeneralTermsDesign,HumanFactors.
KeywordsUsability,networkedgames,multiplayergames,gameusability,heuristicevaluation,NetworkedGameHeuristics,NGH.
1.
INTRODUCTIONMostofthecommercialvideogamesthathavebeenreleasedinrecentyearssupportnetworkedplay,inwhichgroupsofpeoplecanplaytogetherfromdifferentlocations.
Currentnetworkedgamesareextremelyvariedandcoverseveralgenresincludingfirst-personshooters,strategygames,sportssimulations,androle-playinggames.
Manyofthegamessupportseveraldifferentstylesofcooperativeandcompetitiveplayoverthenetwork,includingfree-for-allswhereplayerscompeteagainsteachother,andteam-basedplaywherepeopleworktogethertowardacommongoal(e.
g.
,capturetheflag).
Networkedgamesinvolveinteractionsbetweenplayersaswellasinteractionwiththegameenvironmentitself,whichmeansthatthesesystemsmustprovidesupportforcommunication,coordination,awareness,andsocialinteractionsbetweenpeople.
Asaresult,thereisamuchwidervarietyofusabilityconcernsthatmustbeaddressedforthesesystemsthaninsingle-playergames.
Theseadditionalusabilityissuesthataffectgroupgamingcancollectivelybecalledmultiplayergameusability,whichwedefineasthedegreetowhichaplayerisabletosuccessfullyinteractwithotherplayersandunderstandtheiractions.
Multiplayerusabilityisnottheonlyimportantissueinanetworkedgame–butitisdistinctfromotheraspectsofgamedesignthathavebeenstudiedpreviously.
First,agame'sentertainmentvalueisawell-knownbutseparateconcern,andinvolvesissuessuchasengagement,challenge,fun,andstoryline.
Second,multiplayerusabilityisdifferentfromsingle-userusabilityissuesthatareimportantforallgames,suchasinputmappings,controlsensitivity,visualrepresentations,andtrainingsupport[35].
Evaluatingmultiplayerusabilityisanimportantissue,sincepoorusabilitycangreatlydetractfromtheoverallplayexperience,evenifotheraspectsofthegamearewelldesigned.
However,therearecurrentlyveryfewmethodsforevaluatingmultiplayerusability.
Playtestingisoneofthemostcommonwaystouncoverdesignproblems[12],butthismethodneedsplayableprototypesthatusuallyonlyexistinthelaterstagesofthedevelopmentprocess.
Fewmethodsexisttoallowdesignerstocarryoutlessexpensiveusabilityinspectionsofgames,andtoalsoevaluateearly,non-functionalprototypes.
Recentusabilityinspectiontechniqueslookatsingle-userusabilityissuesingames[35,11,7],buttherearecurrentlynoestablishedmethodsforevaluatingmultiplayerusabilityissuesinnetworkedgames.
Inthispaper,weaddressthisproblembypresentinganewsetofheuristicsforevaluatingmultiplayergameusability.
Theheuristics,calledNetworkedGameHeuristics(NGH),canbeusedduringheuristicevaluations,whereevaluatorsexploreaninterfacewhilelookingforusabilityproblems[31,32].
Theheuristicapproachhasbeensuccessfullyusedinpreviousgameevaluationsbecauseitdoesnotmakeassumptionsabouttaskstructure,anditisflexibleenoughtobeadaptedtospecializeddomains[32,10].
WedevelopedourheuristicsusingaPermissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationonthefirstpage.
Tocopyotherwise,orrepublish,topostonserversortoredistributetolists,requirespriorspecificpermissionand/orafee.
GROUP'09,May10–13,2009,SanibelIsland,Florida,USA.
Copyright2009ACM978-1-60558-500-0/09/05.
.
.
$5.
00.
methodologyintroducedbyPinelleandcolleagues[35],inwhichheuristicsarederivedfromactualexperiencereports–inthiscase,publicreviewsofnetworkedPCgames.
Fromthesereviews,wederivedasetofusabilityheuristicsforassessingmultiplayergameusability.
Ournewheuristics,calledtheNetworkedGameHeuristics(NGH),arethefirstdesignprinciplestofocusexclusivelyonnetworkedgameusability,andarethefirsttobebasedonastructuredanalysisofreportedusabilityproblemsfromalargenumberofgamesandgamegenres.
AshortenedformoftheheuristicsisshowninTable1,andtheyareshowninmoredetailinTable4.
Inthefollowingsections,wedescribethedevelopmentofthenewheuristics,andthenreportonastudyinwhichtenevaluatorsusedtwodifferentmethods(NGHandBaker'sgroupwareheuristics[5])toevaluatetwonetworkedPCgames.
EvaluatorswereabletofindmoremultiplayerusabilityproblemswithNGH,andalsoreportedthattheprincipleswerewell-suitedtothegoalsoftheevaluation,wereeasytolearn,andwerestraightforwardtoapply.
Table1.
ShortsummaryoftheNetworkGameHeuristics.
1.
Providesimplesessionmanagement.
2.
Provideflexiblematchmaking.
3.
Provideappropriatecommunicationtools.
4.
Supportcoordination.
5.
Providemeaningfulawarenessinformation.
6.
Giveplayersidentifiableavatars.
7.
Provideprotectedtrainingforbeginners.
8.
Supportsocialinteraction.
9.
Reducegame-baseddelays.
10.
Managebadbehaviour.
2.
USABILITYINSPECTIONANDNETWORKEDGAMESUsabilityinspectiontechniquescanplayanimportantroleinthedesignofnetworkedvideogames.
Mostgamedesignprocessesuseplaytestingtoidentifyproblems,butthisisonlypossibleduringlaterstagesofdesignanddevelopmentwhenafunctionalprototypeexists.
Usabilityinspectiontechniques,whereskilledevaluatorsinspecttheprototypetofindproblems,areapotentiallyvaluableadditiontothegamedesignprocess.
Theyareinexpensive,andtheycanbeusedtoevaluateprototypesatallstagesofdevelopment.
Mostcurrentusabilityinspectiontechniquesarenotappropriateforgames,sincemostcurrentmethodsassumeadifferentsetofdesigngoalsthanwhataretypicallyseeningames[35].
Forexample,manygamesarerelativelyunstructuredcomparedwithproductivityapplications,andpromoteexplorationandvariedplayingstylesratherthanthecompletionofspecifictasks.
Further,oneofthemaingoalsofgamedesignersistoengageandentertaintheuser–videogamesemphasizemusic,voiceacting,storyline,andartwork,sotheydonotfollowtheconventionsseeninotherapplications.
Thismeansthatevaluationtechniquesfocusingonconceptssuchastasksequences(e.
g.
,[4])anddesktopdesign(e.
g.
,[31])areoftennotappropriateforgames.
Literatureongroupwareevaluationprovidesinsightintohowusabilityinspectiontechniquescanbedevelopedfornetworkedgames.
Groupwareapplicationssupportmultipleusers,usuallywhilecarryingoutcooperativeactivities.
Groupwareapplicationsareincrediblyvaried,butthereareseveralpersistentissuesthatdesignershavetoaddressinmostapplicationareas.
Theseincludesupportforcommunicationandcoordination,andsupportforhelpingteammemberstomaintainanawarenessofothers'actions,locations,activitylevels,etc.
[16,37].
GutwinandGreenberg[15]callthesecommoninteractionprimitivesthe"mechanicsofcollaboration.
"GutwinandGreenberg[15]statethatevaluationsofmulti-userapplicationsmustaddressbothtaskwork,whichcoverssupportforsingleuseractions,andteamwork,whichcoversthe"workofworkingtogether.
"Mosttechniquesforevaluatingtheusabilityofgroupwaresystemshaveaddressedtheteamworkaspectsofdesign,andhavelefttheevaluationoftaskworktosingleuserevaluationtechniques.
Severaltechniqueshavebeenproposed,manyofwhichhavebeenbasedonthemechanicsofcollaboration.
Theseincludeasetofusabilityheuristicsforgroupware[5],ataskanalysistechniqueformulti-userscenarios[37],andamulti-userwalkthroughtechnique[36].
However,mostofthesetechniquesarebasedonthenotionofstructuredtasksequencesandfocusoncooperationbetweenusers,sodonotmapwelltogames.
Theprinciplesthatwedescribeinthispaperaddress"teamwork"(i.
e.
multiuser)issuesinnetworkedgamesandrelyontheheuristicevaluationtechnique.
Heuristicevaluationhasbeenfavoredinrecentworkongameevaluation[e.
g.
7,11,35]becauseitisflexible—itdoesnotmakeassumptionsabouttaskstructure,anditcanbeusedineitherformalorinformalinspections.
Inheuristicevaluation,evaluatorsexploretheinterfacewhilelookingforproblemsbasedonasetofusabilityprinciplescalledheuristics[31].
Theylookforinstanceswherethereisamismatchbetweentheprinciplesandthedesign,andrecordmismatcheswhentheyarerealusabilityproblems.
Inthenextsection,wedescribedesignprinciples(i.
e.
heuristics)thathavebeendevelopedforevaluatingnetworkedvideogames.
3.
DESIGNPRINCIPLESFORNETWORKEDGAMESInrecentyears,severalresearchershavedevelopedsetsofheuristicsthatcanbeusedtoimprovethedesignofvideogames.
Someresearchershavefocusedonsingle-useraspectsofdesign,andhavenotaddressedmultiplayerissuesindetail.
Forexample,bothFederoff[11]andDesurvireetal.
[7]developedheuristicsthatincludecoverageofseveraldesignissues,includingplayability,gamemechanics,andusability.
Morerecently,Pinelleetal.
[35]developedaheuristicsetthatprovidesdetailedcoverageofusabilityissuesseeninsingleplayergames,andthataddressesarangeofissuesincludingcontrolsensitivity,inputmappings,andtrainingandhelp.
Manyotherresearchershaveinvestigatedissuesrelatedtothedesignofmultiplayergames.
Muchofthisworkhasaddressedtechnicalissuesthatareimportanttotheperformanceofnetworkedgames,includingapproachesforhandlingnetworktraffic[39]andfordesigningdistributedarchitectures[8].
Otherresearchershavecoveredissuesrelatedtomakinggamescompetitiveandengaging[3,17],andsomehaveinvestigatedhowgamescanbedesignedtofosteronlinecommunities[9,21].
Fewresearchershaveconsideredhowusabilityissuesshouldbeaddressedwhendesigningmultiplayernetworkedgames.
Researchersthathavestudiedusabilityhavefocusedondesignforspecificplatformsorspecifictypesofgames,suchasmobilegamesandmassivelymultiplayergames.
Forexample,Cornett[6]performedanexploratorystudytoidentifyusabilityissuesseeninMMORPGs.
Hedevelopedalistof17usabilityproblemsthatwerefoundwhileobservingusersplayinggames,andhepresentspotentialsolutionsforeachproblem.
Theproblemlistcoversmanyissuesthatdealwithsingle-playerusability,andothersarespecifictotheroleplayinggenre(e.
g.
"Attemptstobuy/sellitemswereoftenunsuccessful").
KorhonenandKoivistopointoutthatdesigninggamesformobileplatformsposesauniquesetofchallengesnotseenelsewhere[23],andtheydevelopedasetofheuristicsthataddresshownetworkedmultiplayergamescanbedesignedformobiledevices[22].
Eventhoughtheirheuristicsfocusonmobilegames,manyofthemcoverusabilityissuesthatarerelevanttoothertypesofnetworkedgames,includingtheneedtosupportcommunication,theneedtocontroladdressspoofing,andtheneedtohelppeoplefindotherplayers.
KorhonenandKoivistoalsoaddresstechnicalissuesrelatedtomanaginglagandnetworkconnections[22].
Insummary,currentliteraturedoesnotaddresstheproblemofevaluatingtheusabilityofgeneralnetworkedgames.
Inthenextsection,wedescribeoursolutiontothisproblem–anewsetofheuristicsspecificallyformultiplayerusability.
4.
HEURISTICEVALUATIONFORNETWORKEDGAMESWecreatedasetofheuristicsthatcanbeusedtocarryoutusabilityinspectionsofnetworkedmultiplayergames.
WedevelopedtheheuristicsusinganapproachthatisverysimilartotheoneusedbyPinelleetal.
[35]todevelopheuristicsforsingle-playergames.
Wecreatedtheheuristicsbyidentifyingusabilityproblemsreportedin382reviewsofnetworkedmultiplayergames,andthendevelopingstatementsthatencapsulatehowtheproblemscanbeavoidedbydesigners.
Usingactualproblemreports(fromthegamereviews)asthebasisfortheheuristicsisvital,asitensuresthattheheuristicswillreflectrealusabilityconcerns.
Gamesareincrediblyvaried,andsignificantdesigndifferencesareseenbetweengamesandbetweengamegenres.
Therefore,wefeltthatitwasimportantthattheheuristicsrepresenttherangeofproblemsfoundincurrentnetworkedgames,andthatthattheyprovidebothbreadthanddepthcoverageofthedesignspace.
Weaddressedthisbyanalyzingproblemsfoundinreviewsofalargenumberofcommercialgames,coveringsixmajornetworkedgamegenres.
Thenewheuristicsfocusonmulti-useraspectsofthedesign,andarenotintendedtoprovidecompletecoverageofallusabilityproblemsfoundinnetworkedgames.
Theycanbepairedwithheuristicsthatcoversingle-useraspectsofdesign(e.
g.
,visualrepresentations,controlsensitivity,andinputmappings)orevaluationsthatdealwithfun,playability,andentertainmentvalue.
Thenextsectionsoutlinethestepsweusedtodeveloptheheuristics.
4.
1Step1:IdentifyProblemsinReviewsOurfirstactivitywastoidentifyusabilityproblemsthatweredescribedinreviewsofnetworkedgames.
Thereviewswereobtainedfromtwopopulargamingwebsites–GameSpyandGameSpot–thatreviewalargenumberofgamesandthatmaintainarchivesofgamereviews.
Reviewsonbothsitesarewrittenbyprofessionaleditorsanddescribethequalityofgames,enumeratingtheirstrengthsandlimitations.
Inmanycases,theycoverusabilityproblems(bothsingle-userandmultiplayer),providingausefulsourceofinformationondesignproblemsthatarefoundingames.
WeanalyzedallreviewsofnetworkedPCgamesthatwerepostedonGameSpotin2006and2007andonGameSpyfrom2005to2007.
Toguaranteethatbroadcoverageofmultiplayerusabilityproblems,weincludedalargenumberofreviewswithcoverageofseveralmajorgenres.
Thereviewsinthesetwerewrittenby42differentauthors.
Weincluded382reviews:29ofthereviews(8%)coveredpersistentonlinegames(includingmassivelymultiplayeronlinegames),andtheremainingreviewscoveredtransientgames(whereplayerssetupandcontrolthegameserversthemselves).
Thedistributionofgenresinthegamereviewswere:Strategy34%,Shooter23%,RPG17%,Sports16%,Simulation5%,Action3%,andOther2%(seeFigure1).
Figure1.
Distributionofreviewsbygenre.
Reviewersreportedproblemsaboutmultiuseraspectsofthegamein213ofthe382reviews(56%).
Fromafirstpassthroughthisdata,wefurthersubdividedproblemsintothosethatarespecificallyconcernedwithaspectsofthegame'sdesign,andthosethatarecausedatleastpartiallybyexternalfactorssuchasnetworklagorserverresponsiveness.
Afterthisdivision,wehad112reviews(29%ofthetotal)thatdiscussedproblemsclearlyrelatedtomultiplayergameusability.
AtypicalexampleofausabilityproblemreportcanbeseeninMadigan'sreviewofthegameColdWarConflicts.
Indiscussingthematchmakingfeatures,thereviewerstates:CWCdoesincludeamultiplayeroption,butonceyoutakeacloserlookyouseethatthedevelopersarejusttauntingyou.
Thematchmakingmechanismisallbutnonexistent,requiringyoutoeitherbeonaLANorenteracomputer'sIPaddresstoconnect.
There'ssupposedlyarudimentarysearchsystem,butitdidn'twork[30].
Furtheranalysisofthese213reviewswascarriedoutbythreeoftheauthorswhohavesubstantialexperiencewithplayinggamesandwithconductingusabilityevaluations.
Eachpersonworkedseparatelyonthereviewsets,sothattheycouldindependentlydrawconclusionsabouttheproblemclassesthatwerefound.
Inthisanalysis,weidentifiedthemultiplayerusabilityproblemsineachreview,andalsocodedotherproblemsrelatedtomultiplayerfeatures.
Weexcludedproblemsrelatedtootheraspectsofthegame(i.
e.
,single-userusability,andfunandengagement).
Eachresearcherdevisedinformalcategoriesbasedontheproblemsfound,andindividuallycodedproblemsfromthereviewsusingtheseinitialcategories.
4.
2Step2:DevelopProblemCategoriesAfterallreviewswerecoded,wediscussedtheproblemclassificationstodevelopacommonclassificationscheme.
Therewereinitially61problemcategories,but18wereremovedduringthediscussions.
Weaggregatedtheremainingcategoriesbasedonsimilaritiesbetweenproblemdescriptions,andidentifiedtennewproblemcategories.
Wealsocreatedfivecategoriesforthecommonproblemsthatwererelatedtonetworkedplaybutthatwerenottruemultiplayerusabilityissues.
Wethenrecodedthe213reviewsusingthenewproblemcategories,andidentified147multiplayerusabilityproblemsand187relatedproblems.
TheusabilityproblemcategoriesareshowninTable2,whichbrieflydescribestheproblemcoveredbyeachcategoryandlistsasetofkeyissuesthatprovidemoredetailonhowtheproblemoccurredingamesfromthereviewset.
Forexample,thefirstcategorydealswithsessionmanagement,andthekeyissuesinvolve"problemswithfinding,joining,orstartinggames.
"Theproblemexamplelistedinthelastsectionwasclassifiedasasessionmanagementproblemsinceitdealswithdifficultiesencounteredwhentryingtosearchforandjoingameservers.
Table3showstheadditionalproblemareasthatwerefrequentlymentioned,butthatarenotdirectlyrelatedtousability.
Weincludethesebecausetheproblemswerecommon,becausetheycanhaveaseriousimpactonplayability,andbecausetheyarerelatedtomultiplayeraspectsofthegames.
TheproblemareascoverissuesthathavebeenpreviouslydiscussedinCSCW,suchasnetworklatency,criticalmass,andcompatibilityissues.
Weseparatedthemfromourmainsetbecausetheproblemsarecausedbyelementssuchasthequalityofanetworkconnection,ratherthanthegamedesignitself.
Thedistributionofgamegenresseeninourreviewsetwasskewed,withfourgenresaccountingfor90%ofthereviews.
Problemsfromalmostallcategorieswerereportedforeachofthefourmostcommongenres:Strategy(35%ofreviews):problemsfoundinalltencategories,andinallfivesecondaryareas.
Shooters(23%ofreviews):problemsfoundinnineoftencategories,andinallfivesecondaryareas.
Roleplaying(17%ofreviews):problemsfoundinnineoftencategories,andinallfivesecondaryareas.
Sports(15%ofreviews):problemsfoundinsevenoftencategories,andinfouroffivesecondaryareas.
Therewerefewerproblemsfoundforsomeproblemcategoriesthanforothers.
Forexample,threeproblemcategories(6,9,and10)havefewerthantenproblems.
Thesecategorieswereincludedbecausetheywerefoundinmultiplegenres,andbecausetheyhadasubstantialimpactontheusabilityofthegames.
4.
3Step3:DevelopHeuristicsWeusedtheproblemcategoriesandproblemdescriptionstodeveloptenusabilityheuristicsfornetworkedgames.
Theheuristics,calledNetworkedGameHeuristics(NGH),describedesignprinciplesthatareintendedtohelpdesignersavoidcommonproblemsseeningames.
EachheuristicisbasedontheproblemcategorieslistedinTable2.
Forexample,problem7,whichdealswithinadequatesupportfortraining,becametheheuristic"providetrainingopportunitieswherenoviceplayersarenotsubjecttopressuresfromexperts.
"TheheuristicsarelistedinTable4.
WedidnotcreateheuristicsfortheproblemsofTable3.
Eachheuristiccontainsaparagraphthatprovidesmoredetailonwaysthatrelatedusabilityproblemscanbeavoided.
Theparagraphsaregroundedindetailsfoundinthegamereviews.
Theyelaborateonthebasicconceptstatedintheheuristicitself,andwereincludedtohelpdesignersmaptheheuristicstofeaturesfoundingameinterfaces.
Ourheuristicshavesomesimilaritiestootherheuristicsetspresentedinpreviouswork(e.
g.
,[5,22]);however,asdescribedbelow,thereareseveraldifferences,anditisclearthatderivingtheheuristicsdirectlyfromproblemreportsingamereviewsprovidesconsiderablespecificitytothedomain.
Bakeretal.
[5]presentedasetofgroupwareheuristicsthatarebasedonpatternsofcollaborativeinteractionseenintherealworld.
Theheuristicscoverthreemainareas:supportforverbalandgesturalcommunication,supportformaintainingawareness,andsupportforgroupcoordination.
Ourheuristicsaddressmanyofthesesameissues,butaremorespecifictogames,andmakethemappingbetweentheseissuesandthegameinterfacemoreexplicit.
Further,ourheuristicsaddressseveralissuesthatarenotcoveredbyBaker'sheuristics(suchasmatchmakingorcheating).
Severalsetsofgame-specificheuristicshavealsobeenpresentedinearlierliterature.
OurheuristicshavefewsimilaritieswiththeheuristicscreatedbyDesurvireetal.
[7]andFederoff[11].
Ourheuristicsemphasizemultiplayerusability,whereaspreviousheuristicsprimarilyfocusonplayabilityandemphasizesingleplayeraspectsofdesign.
TherearesimilaritiesbetweensomeofourheuristicsandthosecreatedbyKorhonenandKoivisto[22].
Theirheuristicsaddressplayabilityissuesrelatedtomultiplayermobilegames,soalthoughthefocusissomewhatdifferent,thereissomeoverlapinfouroftheheuristicsrelatedtocommunicationsupport,matchmaking,awareness,anddeviantbehavior.
Thereissomedivergenceinthedetailsoftheheuristicdescriptions,however,sincetheirprinciplesfocusongamesformobileplatforms;forexample,theirheuristicsdealwithdesignconstraintsrelatedtomobilephones,suchastextentrylimitations.
WedevelopedNGHusingamethodthatwassimilartotheoneusedbyPinelleetal.
[35].
Theyderivedtheirheuristicsbyfirstanalyzingusabilityproblemsfoundinpublicgamereviews,buttheyincludedadifferenttypeofgame(primarilysingle-player)andcoveredadifferenttypeofusabilityproblem.
Pinelleandcolleagues'heuristicsaddressfundamentalusabilityissuesseeninvideogames,includingconsistency,customizability,inputmapping,andcontrolsensitivity.
Ourheuristicsfocusexclusivelyonmultiplayerissues,sothereisverylittleoverlapbetweenthetwoheuristicsets.
Severalofourheuristicsarecompletelynewandarenotcoveredinanyotherheuristicsets.
Forexample,wedefineheuristicsthataddressavatardesign,protectedtrainingforbeginners,andtheneedtominimizeinteractiondelays.
Theseissueshavenot,toourknowledge,beencoveredinanyothergameevaluationmethodologies.
Table2.
Usabilityproblemcategories.
Categoriesarelistedwithkeyissues,exampleproblems,andtotalproblemsfound.
ProblemcategoryKeyissuesExampleTotal1.
SessionmanagementProblemswithfinding,joining,orstartinggames"PerhapsI'mnotasbrightasthefolkswhodesignedit,butallIwanttodoistoseealistofserverstojoinandIcan'tevenaccomplishthat.
"[29]342.
MatchmakingProblemswithfindingfriends,orwithfindingappropriatepeopletoplaywith"Evenworse,pick-upgroupsonlywanttodothehighexperiencevaluedungeons,whichmeansthatmanyplayerswon'tevenbeabletoexperiencehalfthecontentthatistherebecausetheycan'tgetaproperpartytogether.
"[14]193.
CommunicationPoorsupportfortextorvoicechat;poorcommunicationinterface;notenoughbreaksingameplaytousesupportedcommunicationchannel"Anotherproblemisthechatbox:aflawmakesitimpossibletotypemorethanashortsentenceintoit,solvedonlybyresizingtheboxtoalargersize.
"[18]134.
CoordinationDifficulttocoordinateactionswithotherplayers"Playerswhocanformupintoorganizedteams…mighthavebetterluckwithactuallyfollowingsomesortofnavalattackplan(maintainingalineofbattle,etc.
),butthegamedoesn'tlenditselftoorganizingsuchtactics.
"[25]165.
AwarenessDoesnotprovideenoughinformationonothers'actions,locations,statuses,oravailabilities"Sometimescrazythingsarehappeningtoyouandyoudon'tknowwhat'shappening…It'sasifyou'replayingMagic:TheGatheringinreal-timewithnochancetoreadthecardsasthey'rebeingplayed.
[26]126.
EmbodimentsAvatarsdonotprovideanadequatesenseofpresence,oraretoosimilarandareeasytoconfuse"Youcanracebothpoint-to-pointandmulti-lapraces,thoughyourcompetitors'carsareghoststhatyoucandriverightthrough,soitdoesn'tmuchfeellikeyou'reracinganyone.
"[34]87.
TrainingEliteplayersmakeplaydifficultfornewplayers;noofflinetraining;noserversforbeginners"Mattersgetworseonlinewhereyou'llplayagainsttracksdesignedbyTMUmasterswhohavecreatedmind-bogglingchallengesthatwillbreakthewillofanygamerwhoisn'taneliteTMUplayer.
"[27]158.
SocialopportunitiesSizeofgameworldistoosmallortoolargefordesiredinteractions;limitedsocialopportunities"…thetinyTubestationsarejusttoosmalltodisplaymorethanafractionoftheplayerbase,reducingtheopportunityforcasualmeetingsandmakingnewfriends.
"[38]159.
InteractiontimeframeSlowonlineplay(notrelatedtolatency);playersforcedtotaketurnsorwaitforotherplayers"Playingaturn-basedgameviamultiplayermodeisalwaysatoughsell…InHeroesVyouhave'GhostMode,'whichinaroundaboutwayallowsyoutomoveaghostheroaroundanddostuff,butit'snotreallythesameassimultaneousturns.
"[1]710.
CheatingandunsavorybehaviorInadequatefeaturestodiscouragecheatingornegativegameplay"OnlyabouthalfoftheserversonlinearerunningthePunkbusteranti-cheatingmeasures,andplayersseemtohavediscoveredanunlimitednukecheatforthenon-Punkbusterservers.
"[24]8Combinedtotal147Table3.
Othercommonproblemsfoundinnetworkedmultiplayergames.
ProblemcategoryKeyissuesExampleTotal1.
NetworklatencyNetworklag,whereresponsetimesinterferewithgameplay"Latencywasprettyhighforanonlinegame(thelowestpingIevergotwas200),andtherewereconsistentissueswithghosting,whereanenemywouldsimplyvanishrightbeforeyoudouble-tappedhim.
"[20]492.
NetworkconnectionsProblemsestablishingandmaintainingconnections,firewallproblems"OnceIreachedtheserverlist,Irepeatedlygotcrypticerrormessagestryingtoconnecttoserversthatseeminglyhadplayersfraggingaway,andhadsimilarhiccupstryingtousethebuilt-inQuickMatchsystem.
"[2]353.
CriticalmassNotenoughplayersorserversonline;gamedoesnotallowplayerstocreatebotstofilloutteams"Youcanhaveupto28playersatatimebutIcouldhardlyevenfind28servers.
MostoftheserversIfoundwereemptyandthebiggestraceImanagedtojoinonlyhadsixpeopleplaying.
"[28]714.
GameplaybalanceSometeams,weapons,orunitsaretoopowerful;mapsgiveunfairadvantagestosomeplayers"Mybiggestconcerns,though,restwithhowmultiplayerisbalanced.
Theheroesaresimplyfartoopowerful;inonegame,threeheroesutterlydestroyedanentireenemyarmy.
"[19]155.
BugsandcompatibilityTheapplicationcrashes;compatibilityproblemswherepeoplecannotplaytogetherbecauseofdifferencesingameversion"OuronlymajorissuewiththePCversionisthatthere'saknownandnastybugthatcausestheentiregametocrashifyouattempttosearchforagameusingthedefaultsettings.
"[13]17Combinedtotal187Table4.
Networkedgameheuristics.
Eachheuristicislistedwithaparagraphdescribinghowcommonproblemscanbeavoided.
1.
Simplesessionmanagement:providesessionmanagementsupportthatallowsplayerstostartnewgames,andthatallowsthemtofindandjoinappropriategames.
Non-persistentgamesrequiresessionmanagementsupportsothatplayerscanstartandpublicizenewgames,andsothattheycanfilterandjoinappropriategames.
Sessionmanagementfeaturesshouldallowplayerstobroadcastinformationaboutthegametheyarestarting,includingthetypeofplayertheyareinterestedin(e.
g.
expertornovice)andthetypeofgameplaytheyaresupporting(e.
g.
cooperative,competitive,capture-the-flag,etc.
).
Whenplayersaresearchingforgamestojoin,theyshouldbeabletoselectivelyfiltergameslistingssothattheycanfindonethatissuitableforthem,andtheyshouldbeabletoobtainadditionalrelevantinformationaboutgameservers(e.
g.
ping,numberofplayersonline).
2.
Flexiblematchmaking:providematchmakingfeaturestohelppeoplefindplayerswithsimilarinterests.
Gamescanprovidetwotypesofmatchmakingsupport:theycanhelppeoplefindothersinsidelargepersistentgameworlds,ortheycanhelppeoplefindcompatiblepeoplepriortostartinganewgame(usuallyingamelobbies).
Designsshouldsupportthetypeofmatchmakingthatisappropriateforthegame(forpersistentortransientgames,respectively).
Inbothcases,supportshouldbeprovidedfordiscoveringwhetheraplayer'sfriendsareonlineandforhelpingtolocatethem.
Gamesshouldalsoprovidecommunicationsupportandotherfeaturesthathelpplayersfindstrangerswhohavesimilargameplaygoalsorexperiencesothattheycanputtogetheranad-hocgrouporgame.
3.
Appropriatecommunicationtools:providecommunicationfeaturesthataccommodatethedemandsofgameplay.
Mostmultiplayergamesallowplayertocommunicateusingtextorvoicechannels.
Thechoiceofchannelandthedesignofthecommunicationinterfaceshouldbeappropriatefortheinputdemandsofthegame.
Ingamesthatrequireconstantinteraction,itisoftendifficulttoaccesscommunicationfeatures,sotextcommunicationtoolsthatrequiremodeswitchingcanbedifficulttouse.
Possiblewaystoreducecommunicationoverheadincludeshortcutkeysthattriggercommonlyusedtextphrases,openvoicechannels,andwell-designedinterfacesthatdonottaketheuser'sattentionawayfromeventsinthegame.
4.
Supportcoordination:providefeaturesthatallowplayerstocoordinatetheiractionsduringcooperativegameplay.
Whengamessupportcooperativeplay,playersusuallymustcoordinatetheiractionstosucceed.
Designersshouldprovidecoordinationfeaturesthatallowplayerstoseamlesslyconductsharedactivitiesinthegame.
Commonexamplesofcoordinationfeaturesincludeinterfacesfor:distributingtreasureamongteammates,planningandcommunicatingintentions,andmanagingcommandstructuresinteams.
5.
Meaningfulawarenessinformation:providemeaningfulinformationaboutplayers,includinginformationaboutaction,location,andstatus.
Gamesshouldsupplyplayerswithenoughinformationtoallowthemtounderstandtheactivitiesofothers,andtheinformationshouldbepresentedsothatitiseasytoaccessandunderstand.
Awarenessinformationcanbepresentedeitherasynchronouslyorsynchronously.
Asynchronousawarenessinformationincludesinformationaboutplayers'statistics(e.
g.
wins,losses,pointsscored,etc.
)orabouttheirranking.
Synchronousawarenessinformationhelpspeopleunderstandwhatiscurrentlyhappeninginagame,andexamplesincludefeaturestoinspectothercharacters'inventoriesandfeaturestohelptrackteammates(e.
g.
mini-mapsandothervisualindicators).
6.
Identifiableavatars:usenoticeableanddistinctavatarsthathaveintuitiveinformationmappings.
Manygamesuseavatarstoconveyinformationaboutplayerpresence,location,andmovement.
Avatarsusuallyhaveseveralvisualindicatorsthatencodeinformationabouteachplayer(e.
g.
avatarcolorcanbeusedtorepresentteam,andsizecanbeusedtorepresentthecharacter'sstrength),andselectedmappingsshouldbesimpleandeasytointerpret.
Avatarsshouldalsobenoticeable,providingastrongsenseofpresencesothatplayersknowwhenothersarenearby,andtheyshouldbevisuallydistinctsothatplayersdonotconfuseteammembershipandplayeridentity.
7.
Trainingforbeginners:providetrainingopportunitieswherenoviceplayersarenotsubjecttopressuresfromexperts.
Incooperativemultiplayergames,playersusuallyexpecttheirteammatestoactivelycontributetothesuccessoftheteam.
Whenplayersarenewtoagame,theymaynothavetheexpertiseneededtosucceed,causingthemtobebadlyoutmatchedandopeningthemuptocriticismfrommoreexperiencedplayers.
Gamesshouldbedesignedtoprovidesafetrainingopportunitiesfornewplayers,allowingthemtodevelopgameskillseitherofflineoronserversforbeginners,sothattheycanavoidcriticismandhostilityfromexpertplayers.
8.
Supportsocialinteraction:providesupportforplannedandopportunisticsocialinteractions.
Severalfactorshelptopromotesocialinteractionsingames,includingtheuseofspaceingameworldsandthepresenceoffeaturesthatencouragecooperation.
Gameworldsshouldbedesignedsothattheyarenottoosmallortoolargefortheexpectednumberofplayers,makingitdifficulttotrackconversationsortofindotherplayers,respectively.
Gamesshouldalsoprovidefeaturesthatencourageconversationandcooperationbetweenplayers.
Forexample,featuresthatallowpeopletoauctiontheiritemscanpromotetradingbetweenplayers,andfeaturesthatallowpeopletobroadcastmessagestootherplayersinthevicinitycanhelptoencourageteamworkbetweenstrangers.
9.
Reducegame-baseddelays:minimizeinteractiondelaysbyreducingtemporaldependenciesbetweenplayers.
Multiplayergamescanintroducetemporaldependenciesbetweenplayersintwomainways.
First,theyforceturn-taking,whereoneplayermustwaitforanothertocompletetheirturn,andsecond,theydecreaseperformanceforallplayerswhenoneplayerhasaslownetworkconnection.
Whentemporaldependenciesinterferewiththeflowofthegame,designersshouldadoptstrategiestominimizethem.
Possiblewaystodealwithturn-takingdelaysaretousesimultaneousturnsortolimittheamountoftimeplayerscanspendonaturn.
Possiblewaystoimprovenetworkdependenciesaretouseclientsideprediction,ortoonlypenalizetheplayerwithnetworkproblems,nottheentiresetofusersthatareloggedintothegame.
10.
Managebadbehaviour:providetechnicalandsocialsolutionsformanagingcheatingandunsavorybehavior.
Onlinegameplayoftensuffersfromproblematicbehavior,suchaswhenplayersmodifycodeonthelocalclienttogainanunfairadvantage,workagainstteammates,violategameplayconventions(e.
g.
spawnkilling),oruseinappropriatelanguage.
Technicalsolutionsareoftenimplementedtopreventcheatingortofilterobscenelanguageintext-basedcommunication,andtheyshouldbeusedtoreduceoccurrencesoftheseproblems.
Gamesshouldalsoprovidesocialsolutions,byhelpingpeoplerecognizecheatingandotherunacceptablebehavior,andbyprovidingmechanismssothattheycanremoveorsanctionoffendingplayers.
5.
EvaluationWeconductedanevaluationwheretenpeopleusedNGHtoevaluatetwonetworkedmultiplayergames.
Ourgoalsweretodeterminewhetherknowledgeableevaluatorscouldunderstandandoperationalizetheheuristicsandwhethertheheuristicswouldbeusefulatidentifyingmultiplayerusabilityproblems.
Wealsowantedtoevaluatetheareasofcoverageseeninthesettoseewhethertheheuristicswereeffectiveatspecializingtheevaluationprocessfornetworkedgames.
Last,wewantedtodeterminewhethertheheuristicsimprovedoncurrentevaluationapproachesbycomparingthemtoanexistingset.
WecomparedourheuristicstoBakeret.
al's[5]groupwareusabilityheuristics.
Baker'sheuristicsarebasedonthe'mechanicsofcollaboration'[15]andaddresscommonissuesinsharedtasks,includingtheneedforverbalandgesturalcommunication,andtheneedtocoordinatecooperativeactions.
WechoseBaker'sheuristicsasacomparisoncasesincetheyaddressgenericcollaborationissuesthatareimportantinmostmulti-userapplications.
5.
1MethodWerecruitedtenparticipantswhohadpreviousexperiencewithusabilityevaluations,andexperiencewithmultiplayernetworkedgames.
Theparticipantsevaluatedtwonetworkedvideogames.
Participantsweredividedintotwogroupsoffive.
OnegroupusedourheuristicstoevaluatethefirstgameandthenBaker'sheuristicstoevaluatethesecond.
Theothergroupusedtheheuristicsetsinreverseorder.
Webalancedthegroupsaccordingtotheirexperiencewithusabilityevaluation,withanequalnumberofexpertandnoviceevaluatorsineachgroup.
Participantsevaluatedtwomultiplayergames:BZFlag(www.
bzflag.
org),a3Dtankgame,andTeeWars(www.
teeworlds.
com),a2Dside-scrollingshooter.
Weusedopensourcegamesintheevaluationbecausetheywerenotprofessionallydevelopedandwerecontinuallyunderrevisionbymembersofthedevelopmentcommunity.
Thismeansthattheyhaddesignflawsthatweresimilartothoseseeninfunctionalprototypes,makingthemwellsuitedtoevaluationwithusabilityinspectiontechniques.
Priortoeachevaluation,wetrainedparticipantsontheheuristicsetsthattheywoulduseduringthatsession,andonclassifyingproblemseverityusingNielsen'sscale[33].
Participantsweregivenprintedcopiesoftheheuristicsthattheywouldusetoevaluatethegame,andtheyweregivenformsforrecordingusabilityproblems.
Duringeachsession,evaluatorswereaskedtospend15minutesplayingandfamiliarizingthemselveswiththegame,usinganyofthepublicserversavailablethroughmatchmakingfeatures.
Wethensetupatestingserversothatparticipantswouldbeabletoassessbothcompetitiveandcooperativeaspectsofplay,andweaskedthemtologintoourserverfortheremainderofthesession.
Weaskedparticipantstoindividuallyinspectthegameinterfaceusingtheheuristics.
Whentheyfoundamismatchbetweentheheuristicsandthegameinterface,theywereaskedtorecordthemismatchiftheyfeltthatitwasausabilityproblem.
Theywereinstructedtodescribetheproblem,recordtheheuristicthattheyusedtofindtheproblem,andgiveeachproblemaseverityratingusingNielsen'sseverityscale[33]:1-Cosmeticproblem,2-Minorproblem,3-Majorproblem,and4-Usabilitycatastrophe.
Thestudylastedapproximately3hours,andparticipantshadroughly90minutestoevaluateeachgame.
Attheendofeachsession,participantscompletedaquestionnairethataskedthemtodescribethestrengthsandweaknessesoftheheuristicsthattheyused.
Onceparticipantscompletedbothevaluations,theyweregivenafinalquestionnairethataskedthemwhethertheycouldidentifyproblemsmoreeffectivelywithoneheuristicsetortheother,andtheywereaskedwhichsetwaseasiesttouse.
5.
2ResultsWeanalyzedtheproblemdescriptions,removingproblemsthatwererepeatedbythesameevaluator,orthatdidnotdescriberealmulti-playerusabilityproblems(e.
g.
,commentaryongameplay).
Wecountedthenumberofproblemsfoundusingeachheuristic,andwetrackedtheseverityratingsthatwereassignedtoeachproblem.
Participantsfoundacombinedtotalof49usabilityproblemsusingBaker'sheuristics(meanof4.
9problemsperevaluator),andfound67problemswithNGH(mean6.
7).
Theseverityratingofproblemsforbothsetswassimilar,withanaverageseverityforBaker'sheuristicsof2.
39,and2.
30fortheNGHset.
5.
2.
1SummaryofproblemsfoundEvaluatorsfoundalargenumberofproblemsinbothgames,anditisclearevenfromoursimplestudythatmultiplayerusabilityevaluationscouldhaveamajorimpactonthesuccessofanetworkedgame.
Althoughitisnotourgoaltoreportonthespecificevaluationsofthegamesthemselves,webrieflysummarizethemainproblemsfoundbeforeanalyzingthedifferencesbetweentheheuristicsets.
BZFlag.
BZFlagisa3Dtankgamewherepeoplemaneuverthroughthegameworldwhileshootingotherplayers.
Participantsfoundavarietyofproblems,buttheirgreatestfrustrationcamefromtheabsenceofofflinetrainingandprotectedservers,sinceexpertplayersloggedintotheevaluationserveranddominatedgameplay.
Peoplealsofoundproblemsrelatedtocommunicationsupport,andhadparticulardifficultyusingthetextchatduetothereal-timedemandsofthegame.
Otherproblemswerefoundwiththegame'ssupportforcoordinatingstrategieswithteammembers,understandingwhatishappinginthegame,trackingteammates,andinterpretingembodimentsinthemini-map.
TeeWars.
TeeWarsisa2Dgamewhereplayersmovetheircharacterthroughthegameworld,attackingotherplayerswithanassortmentofweapons.
Peoplefoundawiderangeofproblems,includingissuesrelatedtofindingappropriateserversandplayerswithsimilarexpertiselevels,andlimitedopportunitiestotrainoffline.
AsinBZFlag,peoplefounditdifficulttofindtimetousetextchat.
Theyfoundarangeofproblemsrelatingtoawarenessandcoordinationsupport,includingproblemswithidentifyingteammates,theabsenceofamini-map,andaninabilitytocoordinatestrategywithteammates.
5.
2.
2DifferencesintheproblemsfoundwiththetwoheuristicsetsTheproblemsfoundbytheevaluatorsdifferedaccordingtowhichheuristicsettheyused.
ProblemsfoundusingBaker'sheuristicsreflectedtheemphasisplacedonreal-worldcollaboration,includingissuessuchasverbalandgesturalcommunication,andthissetappearstohaveconstrainedtherangeofissuesthatpeopleconsidered.
Elevenproblemswerefoundusingtheverbalcommunicationheuristic,buttheymissedproblemswiththetextchatentirely;instead,theyfocusedontheneedforvoicechatsupport:"Cannotengageinvoicechat.
Thiswouldbegoodforthisgamesoyoucaneasilyplayandtalksimultaneously.
"Peoplealsodescribedeightproblemsrelatedtothegesturalcommunicationheuristic,althoughthedescribedproblemsrelatedtotheappearanceofplayers'embodiments.
Participantsdescribedeightproblemsrelatedtothecoordinationheuristics,forexample,"nowaytomakeanalternatestrategyorreorganizeefforts.
"Problemswerefoundforallotherheuristicsintheset,buttheseproblemswerelessfrequent(fiveorfewer).
Problemswerefoundusingnineofourtenheuristics(seeTable5).
Noproblemswerefoundforheuristic9(interactiontimeframe),butthisheuristicwasnotveryrelevanttothereal-timegamesincludedinthestudy.
AswithBaker'sheuristics,evaluatorsfoundalargenumberofproblemsrelatedtocommunication,awareness,andcoordination.
Insomeoftheareasofoverlapbetweenthetwoheuristicsets,thefocusoftheproblemdescriptionswasdifferent.
Additionally,somesubstantiveproblemsthatwerefoundusingourheuristicswerenotfoundusingBaker's(training,matchmaking,unsavorybehavior,sessionmanagement).
ThereweresomeproblemsfoundusingBaker'sheuristicsthatwerenotfoundusingours(atotalof8);however,weinspectedeachoftheseanditwasclearthattheproblemscouldhavebeenfoundwithNGH.
Forexample,aparticipantfoundaproblemrelatedtomini-mapdesignusingBaker'sconsequentialcommunicationheuristic:"Thedirectionatankisfacingcannotbedeterminedontheminimap.
"Coverageforthisproblemisprovidedinoursetbytheawarenessheuristic(H5),whichexplicitlyaddressesmini-mapdesign.
Webelievethatthesedifferenceswereduetonormalvariationamongevaluators.
Table5.
SummaryofresultsforNGH.
Showsheuristic,totalproblems,andmeanseverity.
HeuristicTotalMeanseverity1.
Sessionmanagement72.
142.
Matchmaking82.
503.
Communication82.
384.
Coordination122.
185.
Awareness82.
386.
Embodiments52.
007.
Training102.
408.
Socialopportunities41.
679.
Interactiontimeframe0-10.
Cheating,unsavorybehavior62.
20Evaluatorsfoundtwelveproblemsusingourcoordinationheuristic(heuristic4),andtheproblemsaresimilartothosefoundwhenBaker'sheuristicswereused.
Peoplealsofoundtenproblemsrelatedtotrainingfornovices(H7);forexample:"Sincegamesseemtobeopentoanybodyitishardtokeepoutexpertplayers.
"Therewereeightproblemsrelatedtoawarenessissues(H5),andtherewassignificantoverlapwiththosefoundwithBaker'sheuristics.
Participantsidentifiedeightmatchmakingproblems(H2),forexample:"nowaytosearchforaparticularplayeroraparticularexpertiselevel.
"Peoplealsofoundeightcommunicationproblems(H3),andtheproblemdescriptionscoveredabroaderrangeofissuesthanthoseseeninBaker'sheuristics.
Forexample,theydealtwithbothchatproblemsandverbalcommunicationissues:"Itishardtodifferentiatebetweenstatusmessagesandchat"and"Sincethegameisquitefast,voicechatwouldbeawesome.
Idiedonafewoccasionswhentexting.
"5.
2.
3Subjectivedifferences:benefits,disadvantagesAftereachsession,wegaveparticipantsquestionnairesandaskedthemtodescribethebenefitsanddisadvantagesofusingeachheuristicset.
ForBaker'sheuristics,threepeopleindicatedthattherewerenoadvantages.
Theotherparticipantsindicatedthattheyhelpedtofocustheevaluationoncollaborationissues,withonepersonwritingthatthey"emphasizedcoordinationandcommunicationwhichthisgamelacked"andanotherstatingthatthey"focusoncommunicationandinteractionamongplayers.
"Allpeoplelisteddisadvantages,andfocusedondifficultiesthattheyencounteredwhenmappingtheheuristicstothegames:"tootaskorientedwhereasthegameisveryrealtime/actionoriented","thegroupwareissuesonlylooselyrelatetogaming",and"didnotreallyfitthedomain.
"Wealsoaskedwhetherpeoplewereabletofindproblemsusingtheheuristicsthattheywouldhavemissedotherwise.
Onlythreepeopleagreed,andindicatedthattheyfoundproblemsrelatedtosupportforinformalencounters,andcommunicationsupport.
ParticipantslistedtwomainbenefitsforusingtheNGHset.
First,sixpeopleindicatedthattheheuristicsprovideaclearpathtofollowduringtheevaluation:"letsyoufocusontheimportantissues.
Providesguidance.
"Second,fivepeoplewrotethattheheuristicshelptofocustheevaluationonissuesthatareimportantingames:"Clearlytargetedatgamespecificissues.
Surprisinglyeffectiveatfocusingtheevaluationonsalientissues.
"Sixpeoplelisteddisadvantages,andthesefellintotwomaincategories.
First,fourpeoplewrotethatallheuristicsmaynotapplytoallgame:"afewheuristicsaregamespecificandwouldnotapplytoallmultiplayergames.
"Second,twopeoplewrotethatusingheuristicsconstrainstheproblemsthattheycanfindingames,andthattheycouldmissproblemsnotcoveredbytheheuristics.
Eightpeopleindicatedthattheyfoundproblemswiththeheuristicsthattheywouldhavemissedotherwise,andthesecoverawiderangeofissuesincludingcommunication,cheating,cooperativestrategies,session-finding,andmatchmaking.
5.
2.
4SubjectivepreferencesAttheendofthestudy,wegaveparticipantsafinalquestionnaireandaskedthemtocomparetheeffectivenessandeaseofuseoftheheuristicsets.
Mostpeople(9/10)foundthatourheuristicsweremoreeffectiveatdiscoveringproblemsthanBaker's;onepersonfeltthattherewasnodifference.
MostpeoplestatedthatNGHprovidedbettercoverageofmultiplayerusabilityproblems.
Forexample,onepersonwrote:"NGHtotallytrumpedBaker's.
NGHspecificallyidentifiedmultiplayergame-sensitiveproblemsandBaker'sdidn'tcomeclosetoidentifyingthem.
"Anotherparticipantwrote:"NGHbyawidemarginbecauseitdoeslookatspecificthingslikesessionmanagement,training,cheating,andthingsthatyouwouldn'thavewithmoregeneralheuristics.
"MostpeoplealsofeltthatourheuristicswereeasiertousethanBaker's(9/10),againwithonepersonindicatingnodifference.
PeopleindicatedthattheyfoundthatBaker'sheuristicstobeabstractanddifficulttomaptothegameinterface:e.
g.
,"NGHwaseasierandsimplerbecauseitwasmorespecificandmoreeasilymappedtothegameelements.
Baker'swasalittleabstract"Anotherwrote:"NGHwasbetteradaptedtothetaskthatI'mjudging.
WithBaker's,itwasjusttoohardtotrytofigureouthowtoapplyaheuristicinthisparticularcontext.
"Finally,weaskedparticipantswhichheuristicsettheypreferredforevaluatingmultiplayergames.
Almostallofthem(9/10)preferredNGH,butonepersonindicatedthattheyfoundbothsetstobeuseful.
6.
DISCUSSION6.
1SummaryofResultsThegoaloftheevaluationwastodeterminewhetherpeoplecouldoperationalizetheheuristicsandfindrealusabilityproblemswhenusingthemtoevaluatenetworkedgames.
Wealsowantedtoassesswhethertheheuristicsprovidedadequatecoverageoftheproblemspaceseeninnetworkedvideogames.
Further,wewantedtodeterminewhethertheheuristicsimprovedoncurrentevaluationapproaches.
Theevaluationshowsthatthenewheuristicswereeffectiveatspecializingtheevaluationprocessformultiplayernetworkedgames.
Almostallparticipantsfoundthattheywereagoodmatchforevaluatingthegamesusedinthestudy,andtheypreferredusingNGHinsteadofBaker'sheuristicswhencarryingoutgameevaluations.
PeopleindicatedthatitcoveredgamespecificproblemsthatwerenotaddressedbyBaker'sheuristics,andthattheywereeasiertoapplybecausethemappingsbetweenfeaturesinthegameinterfacewereeasiertoestablish.
PeoplefoundmoreproblemsusingourheuristicsthantheydidusingBaker'sheuristics.
TheareaofcoverageseeninproblemsfoundwithBaker'ssetshowssomeofthelimitationsofapplyinggenericheuristicstogames.
Theheuristicsemphasizeverbalcommunication,whichlargelyledpeopletooverlookproblemswithchatinterfacesthatwerefoundusingourheuristics.
Further,thereweremanyareasofcoveragefoundusingourheuristicsthatwerenotfoundusingBaker's,includingtraining,matchmaking,unsavorybehavior,andsessionmanagement.
6.
2ScopeandGeneralizabilityOurheuristicsweredevelopedfromproblemdescriptionsfoundinreviewsofcommercialvideogames.
Weconsideredalargenumberofgames,fromseveralmajorgenres,andwebelievethattheheuristicswillgeneralizetoothersimilargames.
However,itispossiblethattheywillnotgeneralizetogamesthataresignificantlydifferentthantheonescoveredinthereviewset.
Forexample,someoftheheuristicsmaynotbeapplicabletosimpleonlinegameswhereuserinteractionisverylimited,suchasonlinecardgamesandotherWeb-basedgames.
WebasedtheheuristicsetonPCgamesratherthanconsolegames;however,webelievethattheheuristicswilltransfertomostnetworkedconsolegamessincemanycommercialgamesarereleasedconcurrentlyforbothplatforms.
Wedonotbelievethattheheuristicswilltransferwelltosingle-displaygameswhereseveralplayersusethesameconsole.
Singledisplaygamesdonotneedthesamelevelofinteractionsupportthatisseeninnetworkedgamessincepeoplecancommunicatefacetoface.
Therefore,whilesomeofourheuristicsmayberelevant,itispossiblethatadifferentsetisneededforsingle-displaysettings.
Ourheuristicsfocusexclusivelyonthemultiplayerusabilityissuesseeninnetworkedgames.
Theydonotaddresssingleplayerusability,andneedtobepairedwithsingleuserheuristics,suchasthosedescribedbyPinelleetal.
[35],toprovidefullcoverage.
Further,theheuristicsdonotaddressotheraspectsofdesignthatareimportanttothesuccessofgames,suchasthechallengelevelandentertainmentvalue.
Otherheuristics,suchasthosedescribedbyFederoff[11]andDesurvireetal.
[7],addresstheseissues,andtheycouldpotentiallybeusedinconjunctionwithourset.
NGH,likeallheuristicsets,mustbeviewedasguidelinesratherthanrules;anditispossiblethatdesignerswillintentionallyviolatevariousheuristicsinsupportofdifferenttypesofgameplay.
Forexample,inaspygamedesignersmaywantfriendsandenemiestobedifficulttodistinguish,whichwouldviolateourheuristicthatemphasizestheneedforidentifiableavatars.
Thepointoftheheuristicsistoprovidedesignerswithasetofprinciplesthattheycanusetoclearlydecidewhetherachallengeordifficultyinthegameisanintentionalpartofthegameplay,orwhetheritisanunnecessaryusabilityproblem.
Itisalsolikelythatexpertevaluatorscanidentifymanymultiplayerusabilityproblemswithouttheneedforaheuristicset.
However,usingtheheuristicsprovidesastructuretohelpthemconsidereachmajordesigndimensioninturn,andtopreventthemfrombecomingdistractedbyotheraspectsofthedesign,suchassingleuserorentertainmentissues,whichcouldcausethemtomissimportantmultiplayerproblems.
Wealsonotethattheheuristicsetwascompiledfromproblemsthatwerefoundincommercialgamesthatwerereleasedtothemarket,meaningthatmanyoftheissuescoveredinthesetwerenotaddressedbydesignerspriortotherelease.
Ourheuristicsarethefirstgeneralsetthatcoversgeneralmultiplayerproblemsfoundinnetworkedgames.
Webelievethereviewsweusedprovidethoroughcoverageofmajorusabilityproblemsseeninnetworkedgames.
However,thereviewswerenotwrittenbyusabilityprofessionals,anddidnotfocusonusabilityissuesasthemainevaluationcriteria.
Therefore,itispossiblethatourheuristicsetmissedsubtleusabilityproblems,orproblemsthatoccurinearlyformativestages.
Weexpecttheheuristicstocontinuetogrowandevolveinthefutureaswegainmoreexperienceandinsightsfromusingthemtoevaluatemoregames.
7.
CONCLUSIONTherearefewmethodsforevaluatingtheusabilityofmultiplayernetworkedgames.
Wedevelopedanewsetofheuristics(NGH)toaddressmultiplayerissuesduringnetworkgamedesign;theheuristicscanbeusedtoevaluatebothearlymockupsandfunctionalprototypes.
Ourheuristicsarethefirstsettofocusexclusivelyonusabilityproblemsfoundinmultiplayerfeaturesofnetworkgames.
Theheuristicsarebasedonproblemreportsfromalargenumberofgamesandprovidebroadcoverageofmajorgamegenres.
Evaluationresultsshowthatevaluatorsareabletousetheheuristicsafterabrieftrainingsession,thattheyareablefindrealusabilityproblemsrelatedtomultiplayerfeatures,thattheyprovidebroadercoverageofproblemsseeningamesthanBaker'sgroupwareheuristics[5],andthatpeoplepreferredthemoverBaker'sheuristics.
Thenewheuristicscontributeavaluablenewtoolforassessingandimprovingmultiplayerusability.
Inaddition,theyalsoshowthevalueofdomain-specificheuristicsets,andshowthepotentialforusingreviewsasasourceofusabilitydata.
8.
REFERENCES[1]Abner,W.
ReviewofHeroesofMightandMagicV,pc.
gamespy.
com/pc/heroes-of-might-magic-5/710168p3.
html,2006.
[2]Accardo,S.
ReviewofMedalofHonor:Airborne,pc.
gamespy.
com/pc/medal-of-honor-airborne/820348p1.
html,2007.
[3]Alexander,T.
MassivelyMultiplayerGameDevelopment.
CharlesRiverMedia,2003.
[4]Annett,J.
andDuncan,K.
D.
Taskanalysisandtrainingdesign.
OccupationalPsychology12,1967,211-221.
[5]Baker,K.
,Greenberg,S.
,Gutwin,C.
Empiricaldevelopmentofaheuristicevaluationmethodologyforsharedworkspacegroupware.
ProceedingsCSCW2002,96-105.
[6]Cornett,S.
Theusabilityofmassivelymultiplayeronlineroleplayinggames:designingfornewusers.
ProceedingsCHI2004.
703-710.
[7]Desurvire,H.
,Caplan,M.
,Toth,J.
A.
Usingheuristicstoevaluatetheplayabilityofgames.
Proc.
CHI2004,1509-12.
[8]Diot,C.
,Gautie,L.
ADistributedArchitectureforMultiplayerInteractiveApplicationsontheInternet.
IEEENetwork,13(4),6-15,1999.
[9]Ducheneaut,N.
,Moore,R.
J.
,&Nickell,E.
DesigningforsociabilityinMassivelyMultiplayerGames:Anexaminationofthe"thirdplaces"ofSWG.
Proc.
oftheOtherPlayersConference,2004.
[10]Dykstra,D.
J.
AComparisonofHeuristicEvaluationandUsabilityTesting:TheEfficacyofaDomain-SpecificHeuristicChecklist.
Ph.
D.
dissertation,DepartmentofIndustrialEngineering,TexasA&MUniversity,1993.
[11]Federoff,M.
HeuristicsandUsabilityGuidelinesfortheCreationandEvaluationofFuninVideoGames.
IndianaUniversityMasterofScienceThesis,2002.
[12]Fullerton,T.
,Swain,C.
,Hoffman,S.
GameDesignWorkshop:Designing,Prototyping,andPlaytestingGames,CMPBooks,2004.
[13]Gamespotstaff.
ReviewofTomClancy'sRainbowSixVegas,www.
gamespot.
com/pc/action/rainbowsixvegas/review.
htmlsid=6163307,2006.
[14]GameSpystaff.
ReviewofDungeons&DragonsOnline:Stormreach,pc.
gamespy.
com/pc/dungeons-dragons-online/696856p4.
html,2006.
[15]Gutwin,C.
,Greenberg,S.
Themechanicsofcollaboration:developinglowcostusabilityevaluationmethodsforsharedworkspaces.
ProceedingsWETICE2000.
[16]Gutwin,C.
andGreenberg,S.
Theeffectsofworkspaceawarenesssupportontheusabilityofreal-timedistributedgroupware.
TOCHI,6(3),1999,243-281.
[17]Habgood,J.
,Overmars,M.
GameMaker'sApprentice:GameDevelopmentforBeginners.
Apress,2006.
[18]Harker,C.
ReviewofRYL:PathoftheEmperor,pc.
gamespy.
com/pc/risk-your-life-2-path-of-the-emperor/635953p1.
html,2005.
[19]Harms,W.
ReviewofTheLordoftheRings:TheBattleforMiddle-earthII,pc.
gamespy.
com/pc/the-lord-of-the-rings-the-battle-for-middle-earth-2/693176p3,2006.
[20]Harms,W.
ReviewofRainbowSix:Lockdown,pc.
gamespy.
com/pc/rainbow-six-4-tentative-title/690399p2.
html,2006.
[21]Koivisto,E.
M.
SupportingCommunitiesinMassivelyMultiplayerOnlineRole-PlayingGamesbyGameDesign.
Proc.
LevelUp,2003.
[22]Korhonen,H.
Koivisto,E.
M.
Playabilityheuristicsformobilemulti-playergames.
Proc.
DIMEA2007,28-35.
[23]Korhonen,H.
andKoivisto,E.
M.
Playabilityheuristicsformobilegames.
Proc.
MobileHCI2006,9-16.
[24]Kosak,D.
ReviewofCrysis,pc.
gamespy.
com/pc/ea-crytek-title-untitled-project/839785p1.
html,2007.
[25]Kosak,D.
ReviewofAgeofPirates:CaribbeanTales,pc.
gamespy.
com/pc/age-of-pirates/737252p2,2006.
[26]Kosak,D.
andLopez,M.
ReviewofGuildWars,pc.
gamespy.
com/pc/guild-wars/613203p1.
html,2005.
[27]Kuo,L.
C.
ReviewofTrackmaniaUnited,pc.
gamespy.
com/pc/trackmania-united/799496p1.
html,2007.
[28]Kuo,L.
C.
ReviewofGTR2,pc.
gamespy.
com/pc/gtr-2/751848p1.
html,2006.
[29]Kuo,L.
C.
ReviewofTomClancy'sSplinterCellDoubleAgent,pc.
gamespy.
com/pc/tom-clancys-splinter-cell-4/751217p1.
html,2006.
[30]Madigan,J.
ReviewofColdWarConflicts:DaysintheField1950-1973,http://pc.
gamespy.
com/pc/cold-war-conflict/626650p1.
html,2005.
[31]Nielsen,J.
,andMack,R.
L.
(Eds.
),UsabilityInspectionMethods,JohnWiley&Sons,NewYork,NY,1994.
[32]Nielsen,J.
HowtoConductaHeuristicEvaluation.
www.
useit.
com/papers/heuristic/,2007.
[33]Nielsen,J.
SeverityRatingsforUsabilityProblems.
www.
useit.
com/papers/heuristic/severityrating,2007.
[34]Osborne,S.
ReviewofTrackManiaSunrise,pc.
gamespy.
com/pc/trackmania-sunrise/620071p1,2005.
[35]Pinelle,D.
,Wong,N,Stach,T.
HeuristicEvaluationforGames:UsabilityPrinciplesforVideoGameDesign.
ProceedingsCHI2008,1453-1462.
[36]Pinelle,D.
andGutwin,C.
GroupwareWalkthrough:AddingContexttoGroupwareUsabilityEvaluation.
ProceedingsCHI2002,455-462.
[37]Pinelle,D.
,Gutwin,C.
,Greenberg,S.
TaskAnalysisforGroupwareUsabilityEvaluation:ModelingShared-WorkspaceTaskswiththeMechanicsofCollaboration.
TOCHI,10(4),2003,281-311.
[38]Rausch,A.
ReviewofHellgateLondon,pc.
gamespy.
com/pc/hellgate-london/832252p3.
html,2007.
[39]Smed,,J.
,Kaukoranta,T.
,andHakonen,H.
AspectsofNetworkinginMultiplayerComputerGames.
TheElectronicLibrary,20(2):87–97,2002.