varietybuddypress

1openreview:AStudyofContextsandPracticesOpenReview:AStudyofContextsandPracticesTheAndrewW.
MellonFoundationWhitePaperWrittenbyKathleenFitzpatrick,ModernLanguageAssociationAviSanto,OldDominionUniversityfortheMellonOpenReviewGrantteamMichaelStoller,NYULibraries,PrincipalInvestigatorMonicaMcCormick,NYULibrariesandNYUPressEricZinner,NYUPressandtheMellonOpenReviewGrantAdvisoryGroupCherylBall,IllinoisStateUniversityDanCohen,GeorgeMasonUniversityCathyDavidson,DukeUniversityLisaGitelman,NewYorkUniversityNicholasMirzoeff,NewYorkUniversitySidonieSmith,UniversityofMichiganAppendicesbyDanViselandPeterBrantleyDecember20122openreview:AStudyofContextsandPracticesExecutiveSummaryOverthecourseofthe2011-2012academicyear,MediaCommonsandNYUPressjointlyundertookastudyoftechnologies,practices,anddesiresforopen,onlinepeer-to-peerreviewinhumanities-basedscholarlycommunication.
Weassembledanadvisorypanelthatmetthreetimesoverthecourseoftheyeartodiscuss:themeritsandpitfallsassociatedwithopenreview,thedesirabilityofopenreviewforcertaintypesofcommunitiesandworks,criteriaandparametersneededtoorganizeandconductsuccessfulopenre-view,technologicalrequirementsformeetingopenreviewcriteria,andtechnologiescurrentlyavailablethatcanhelpmeetcriteriasetforthbyschol-arlycommunities.
Theadvisorypanelweassembledconsistedofsixscholarswithdivergentinterestsandinvestmentsinthedigitalhumanities,includingbothchampionsofopenre-viewandskepticsoftheseonlineprocesses.
Themembersofthepanelalsorepre-senteddiversedisciplinarybackgroundswithinthehumanities:CherylBall,associateprofessorofnewmediastudies,IllinoisStateUniversity;DanCohen,associateprofessorofhistoryanddirectoroftheRoyRosenzweigCenterforHistoryandNewMedia,GeorgeMasonUniversity;CathyDavidson,RuthF.
DeVarneyProfessorofEnglish,DukeUniversity;LisaGitelman,associateprofessorofmediaandEnglish,NewYorkUniversity;NicholasMirzoeff,professorofmedia,culture,andcommunication,NewYorkUniversity;SidonieSmith,professorofEnglishandwomen'sstudies,UniversityofMichi-gan.
Inaddition,meetingswereledbyMediaCommons'co-creatorsKathleenFitzpat-rickandAviSantoaswellasbyNYUPresseditor-in-chief,EricZinnerandNYUDigitalScholarlyPublishingOfficer,andliaisonbetweenNYULibrariesandNYUPress,MonicaMcCormick.
Ourhopeaswebeganthesemeetingswasthattheywouldhelpusarticulateasetofcommunityprotocolsandtechnicalspecificationsinordertohelpsystematizeopenpeerreviewpractices.
Creatingsuchasystematicapproachtoopenpeerreview,wehoped,wouldensurethatthesepracticeswouldmeetacademicexpectationsforrigorwhilenonethelessembracingtheopennessmadepossiblebysocialnetworksandotherdigitalplatforms.
Wediscovered,however–andperhapsunsurprisingly3openreview:AStudyofContextsandPractices–thatdifferentpublicationsanddifferentknowledgecommunitieshavestrikinglydifferentgoalsforthereviewprocessesthattheyputinplace,aswellasdifferentnormsforcollegialinteractionaroundscholarlywork.
Asaresult,theoutcomeofourmeetings–thiswhitepaper–containsfeweran-swersthanitdoesquestions.
Weconsiderthecriticalquestionsraisedinthisdocumentaroundshiftingpeerreviewnormsandstrategiesformovingforwardwithopenpeerreviewtoconstituteimportantprogressinbothdelineatingthecoredebatesaswellasfleshingoutspecificareaswithinthemthatrequiremorenuancedassessmentmodes.
FitzpatrickandSantodraftedthewhitepaperafterthemeetingswereconcluded,andsubsequentlyposteditasasharedGoogleDrivedocumentfordiscussionwithandeditingbytheadvisorygroup.
TheeditedversionofthewhitepaperdraftwasthenpostedatMediaCommonsPressforopendiscussionandreviewwithabroaderrangeofinterestedscholars,publishers,librarians,andothers.
1Thewhitepaperwasfinallyrevisedwithattentiontotheconcernsraisedintheonlinecomments,allofwhichpointedtowardthegrowingrangeofinterestsinopenreviewprocesses.
Ourroleinthisprocesshasnotbeenadvocatingfortheacceptanceandadoptionofopenreview;despiteourobviousinvestmentsinthesedevelopingprocesses,wehaveinsteadunderstoodourgoalasbeingthedevelopmentofasetof"bestprac-tices"forthoseinterestedinimplementingopenonlinereviewasacomponentofascholarlypublicationseriesorcommunity.
Overthecourseofourdiscussions,however,wecametobelievethatnosinglesetofbestpracticescouldsufficetocoverallofthepotentialconcernsandneedsofdiversescholarlycommunities,evenwithinthecomparativelyrestrictedsetoffieldsthatcomprisethehumanities.
Asaresult,ourconclusionsandrecommendationsfocusonwaysthatcommunitiesmightbegintodetermineforthemselveswhattheirbestpracticesmightbe,andhowtheymightputthemintopractice.
Wethusarguethat:a)Openreviewisnotaradicallynewscholarlypracticedrivenbytechnologi-calinnovation;rather,ithasdevelopedoutofalonghistoryofhumanities-basedscholarlyendeavors,takingformsincludingthepresentationofconferencepapers,theformationofworkinggroups,andotherformsofprocess-orientedscholar-to-scholarcommunication.
b)However,thetreatmentofsuchscholar-to-scholarcommunicationaspartofaformalizedmodeofpeerreviewchangestheaudiencesforandthestakesofsuch1"OpenReview:AStudyofContextsandPractices,"MediaCommonsPress.
4openreview:AStudyofContextsandPracticescommunication,andaccountingforthesechangeswillbecrucialforthesuccessofanyopenreviewprocess.
c)Theformandfunctionofopenreviewpractices,likeanypeerreviewprocess,shouldbedictatedbycommunitygoalsandneeds,whichshouldinturndeterminethetechnologiesemployed.
d)Nosinglesetoftoolsorrulesforopenreviewcanmeetthediverseneedsofallscholarlycommunities.
Imposinganysinglesystemormodewouldbedetrimentaltotheethosof"openness"thatisoneofopenreview'sprimarystrengths.
e)Instead,parametersandplatformsforopenreviewmustbedevelopedwithaneyetoward"structuredflexibility,"allowingcommunitiesofpracticetosetandcommunicatetheirownstandardsforreview,dependingonthedesiredoutcomeoftheirwork.
Thesestandardsmightbeestablishedbyworkingthroughasetofques-tionsaboutprocess,reviewcriteria,andcommunityinteractions,andthenselect-ingplatformfunctionalitiesthatallowthosestandardstobeapplied.
f)Openreview's"rigor"comesinpartfromanassessmentofthescrupulousnesswithwhichcommunitymembers,includingeditors,authors,andreviewers,engagewithandadheretothecommunity'sexpressedstandards.
g)Openreviewpracticescanbeappliedsuccessfullytoarangeofkindsofscholarlyproduction,includingmonographs,journalarticles,multimediaessays,archivalprojects,andmore.
Justascommunitiesofpracticewilldifferintheirneedsanddesiresforopenreviewpractices,however,differentmodesofscholarlyproduc-tionwillcallfordifferentformsofreviewerengagement.
Therearenumerousopenreviewmodelsthathaveemergedoverthepastdecadeorso(andcontinuetoemerge)thatdifferentscholarlycommunitiescanlatchonto,builduponandlearnfrom.
Eachnewmodelmustbeassessedinrelationtothepre-establishedgoalsandneedsofthescholarlycommunitiesengaginginopenpeerreview.
Inarguingfor"structuredflexibility"inopenreviewpracticesandtooldevelop-ment,wehopetoindicatethattheconversationsthatproducedthisreportarefarfromfinished;indeed,likethereviewprocessesthatwehavesoughttostudy,ourownprocessremainsopen-ended.
Thisinvitationtoongoingdialoguecharacteriz-esthebestkindsofhumanitiesscholarship,andsowehopethatthisreport,andthekindsofreviewprocessesthatitmightinspire,willworktopromotefurthercriti-caldiscussionamongcommunitiesofpractice.
Werecognizethatbyembracingflexibilityandopen-endednessweareadvocatingforsomethingfarmorecomplexthanwouldbeavailableinacentralizedormonolithicsystem.
Yetonlysuchcom-plexitycanprovideforthenuancethatwebelievefunctionalopenreviewsystemswillrequire.
5openreview:AStudyofContextsandPracticesAcknowledgmentsThanksareduetotheextraordinarycastofreviewersforthedraftofthispaper,including(innoorderbutthealphabetical)AdelineKoh,DaynaGoldstein,DianeHarley,DirkRemley,DorotheaSalo,EricBakovic,GertvanVugt,JackDougherty,JasonM.
Kelly,JasonMittell,KariKraus,KatinaRogers,PeterFrishauf,RogerWhitson,SandyThatcher,SarahWerner,TedUnderwood,TimMcCormick,Vir-giniaKuhn,andXeniavanEdig.
Thehelpfulcriticalitywithwhichtheyapproachedthedrafthashelpedtobearoutmanyofourconclusionsaboutthepotentialben-efitsofopenreviewprocesses;wesharethesuccessofthisdocumentwiththem.
PrefaceThiswhitepaperfocusesonthewaysthatnetworkedenvironmentsmightincorpo-rateandadapttheprocessesofscholarlypeerreviewinthehumanities.
Itismeanttohelpauthors,editors,reviewersandotherconstituenciesdefineforthemselvesthebestpracticestheymightfollowindesigninganopenreviewprocess.
Inthisfocusontheprocessofdevelopingbestpractices,wehopethisdocumentwillservenotasasetofanswers,butratherasasetofquestionsthatgroupsofscholarsmightusetohelpclarifythestagesoftheirreviewprocesses,theparametersforeachofthesestages,andthepurposesandvaluesthattheyserve.
WehopeinthissensetofollowinthepathslaidoutbytheSocietyforCinemaandMediaStudies'fairusepolicies,2theCollegeArtAssociation'sguidelinesforfacultyinnewmedia,3theModernLanguageAssociation'sguidelinesforevaluatingworkindigitalhumani-tiesanddigitalmedia,4andtheConferenceonCollegeCompositionandCom-munication'spromotionandtenureguidelinesforworkwithtechnology,5aseachofthesestatementsrevealsahumanitiesfieldintheprocessofincorporatingandaccountingforthedigital,recognizingtheincreasingcomplexityofthelandscapeforcontemporaryscholarship.
Themeetingsanddiscussionsthatweheldinworkingtowardthiswhitepaperwerenotcharacterizedbyuninterruptedagreementonallissues.
Weuncoveredstrik-inglydifferentviewpointsamongthemembersoftheadvisorygroupaboutsomekeytermsandissues,andaccordinglysomeofourexpectationsforaclear,unani-moussetofrecommendationswereconfoundedbythecomplexitiesweunearthed.
2SocietyforCinemaandMediaStudies,"FairUsePolicies".
3CollegeArtAssociation,"GuidelinesforFacultyTeachinginNew-MediaArts"4ModernLanguageAssociation,"GuidelinesforEvaluatingWorkinDigitalHumanitiesandDigitalMedia".
5ConferenceonCollegeCompositionandCommunication,PromotionandTenureGuidelinesforWorkwithTechnology.
6openreview:AStudyofContextsandPracticesThesecomplexities,however,areverymuchtothepoint:openreviewprocessesareunlikelyevertoproduceunanimityaboutanygivenpieceofscholarlywork,assumingthatscholarshipisengagedwithbyadiversesetofscholarswithequallydiverseinterests.
Similarly,noone-size-fits-allprocesscouldbedesignedthatcouldaccountfortheneedsofallscholarlycommunities.
Ratherthanproducingun-blemishedconsensus,thebestopenreviewprocesseswillworktobringdiversityofopinion,interpretation,andexperiencetothesurface.
Inordertodoso,thoseprocessesmustthemselvesbeflexibleenoughtoworkwithmanydifferentcommu-nitiesofpracticewithmanydifferentrequirements.
Thisdocument,asaresult,reflectsthesumofourthinkingoverthecourseofayear'sworthofdiscussionandexploration.
Itrepresentsnotasingularpath,butarangeofpossibilitiesandsomerecommendationsforfutureexploration,experi-mentation,andstudy.
ContextualizingQuestionsOurworkonopenreviewdrewontheexistingcontextofhumanitiesscholarship,leadingustobeginourdeliberationsbyconsideringanumberofgroundingques-tions:WhatIsPeerReviewThisseemsaverysimplequestionwithaverysimpleanswer:peerreviewisthereviewofscholarshipandotherformsofscholarlyactivitybyone'speers.
Thefoundationalrolethatpeerreviewplaysinthedeterminationofscholarlyauthor-ityindicatesthatitis"theprimaryavenueofqualityassessmentandcontrolintheacademicworld.
"6Yetmanyscholars,acrossmanyfields,aretodayraisingquestionsaboutthepurposesthatpeerreviewserves,andwhetherthosepurposes,par-ticularlywithrespecttonewformsofdigitalscholarlycommunication,arebeingservedaswellastheymightbebyourcurrentreviewsystems.
Peerreviewismeanttoaccomplishanumberofthings:forinstance,itprovidesameansofcriticalfeedbackforscholarsinthedevelopmentoftheirwork,anditprovidesameansforselectionamongtheworkofmanyscholars.
Attimesreviewprocessesaremeantprimarilytoserveoneortheotherofthosepurposes,butmostoftenpeerreviewisintendedtoserveboth,simultaneouslyhelpingindividual6DianeHarleyandSophiaKrzysAcord,"PeerReviewinAcademicPromotionandPublishing:ItsMeaning,Locus,andFuture.
"UniversityofCalifornia,Berkeley:CenterforStudiesinHigherEducation,2011.
7openreview:AStudyofContextsandPracticesscholarsimprovetheirworkandenablingtheselectionforqualityinpublications,fellowshipsandgrants,andemployment.
Peerreviewismeanttorepresentthebestofscholarlyvaluesastheyoughttobeespoused.
Blindpeerreview,forinstance,hashistoricallyservedasaninstrumentofmeritocracy,cuttingacrossdivisionsofrank,gender,class,andraceinordertoenablecommunitiesofpracticetodiscoverthebestnewknowledgebeingcreatedintheirfields.
Peerreviewinthehumanities,inparticular,functionstoforwardthevaluesofhumanelettersinproducingoriginalthoughtlocatedinandagainstrele-vantexistingliteratures.
Incontrastwithpeerreviewinthesciences,whichostensi-blyservesasameansofverificationofresultsorvalidationofmethodologies,peerreviewinthehumanitiesoftenfocusesonoriginality,creativity,depthandcogencyofargument,andtheabilitytodevelopandcommunicatenewconnectionsacrossandadditionstoexistingtextsandideas.
However,scholarsinanumberofdifferentfieldshaveleviedseveralimportantcritiquesofconventionalpeerreviewprocesses,includingthedegreetowhichanonymousreviewersaregranted"powerwithoutresponsibility"7andthepotentialfailuresofreviewerandinter-reviewerreliability.
8Moreover,severalcoreassump-tionsaboutpeerreviewanditsfunctionhavebeenallowedtoremainunques-tioned,assumptionsthatlimittherolethatpeerreviewcanplayevendespitetheadmirablemotivesbehindit.
Theideaofthe"peer,"mostnotably,hasinrecentdecadesbeenrestrictedtocredentialedscholars,andevenfurther,tothosecreden-tialedinone'sspecificfieldorsubfield,averynarrowandusuallyverticalcom-munityorganizationinwhichjuniorscholarsmustprovetheirworthtothosewhoprecedethem.
Asaresult,fieldscanoftenbecomeself-replicating,astheylimittheinputthatmorehorizontally-organizedpeergroups–suchasscholarsfromrelateddisciplinesandinterdisciplines,andevenmembersofmorebroadlyunderstoodpublics–mightplayinthedevelopmentofscholarlythought.
Intheageoftheinternet,however,asauthorsincludingChrisAnderson9andKathleenFitzpatrick10haveargued,thedefinitionofa"peer"isshiftingfromthemeritocraticnotionof7FionaGodlee,"TheEthicsofPeerReview,"inEthicalIssuesinBiomedicalPublication,ed.
AnneHudsonJonesandFaithMcLellan.
Baltimore:JohnsHopkinsUniversityPress,2000,p.
65.
8SeeDouglasP.
PetersandStephenJ.
Ceci,"PeerReviewPracticesofPsychologicalJournals:TheFateofPublishedArticles,SubmittedAgain.
"ReprintedinDavidShatz,PeerReview:ACriticalInquiry,191–214.
Lanham,MD:RowmanandLittlefield,pp.
191-214.
9ChrisAnderson,"Wisdomofthecrowds,"NaturePeerReviewDebate.
10KathleenFitzpatrick,PlannedObsolescence:Publishing,Technology,andtheFutureoftheAcad-emy(NYUPress,2011),p.
32.
8openreview:AStudyofContextsandPractices"credentialedcolleague"toamoretechnically-derivedsenseofapeerasanynodeonanetwork.
Thisisnottosaythat,intheageofopennetworks,a"peer"isbecoming"justanyone";rather,itindicatesthatpeerstatusmightonlyemergethroughparticipa-tioninreviewprocesses.
Apeerinthisnewformhasthepotentialtobecomeapeerthroughthequalityofhisorherparticipationinnetworkedknowledgeexchanges.
Giventhispotential,asPeterFrishaufhasnoted,peerscanandshouldbeselectedonthebasisof"expertiseandtrustworthiness,notcredentials,"allowingcom-munitiestodevelop"atrustmarkforeveryeditor,author,andreviewerbasedonexpertiseinthesubjectmatterunderconsideration,andontheiractualworkasareviewer"ratherthanassumingexpertisebasedonparticularcredentials.
11Thisshiftintheunderstandingofthe"peer"pointstotheneedtorethinkthedominantpracticesofpeerreview,particularlywithrespecttoscholarshipthatoriginatesorispublishedonline.
WhyOpenReviewInexploringthepossibilitiesforwhatweoriginallythoughtofas"peer-to-peerreview"–reviewpracticesandtoolsthatwouldenablethedirectcommunicationamonganetworkofexistingpeersaroundscholarlypublications–webegantofocusonthewaysthatopeningupreviewpracticestonewkindsofpeersmightfurthersomecrucialvaluesandgoalsinthehumanities.
Weaspire,asscholars,toengageourstudents,ourcolleagues,andarangeofbroaderpublicsinexploringaspectsofourcomplexhistoriesandcultures.
WealsoseektomodeltheemergenceofcriticalthinkingandintellectualpursuitthroughSocraticformsofinquiry,stressingtheessentialroleplayedbydiscussionanddebateinknowledgeforma-tion.
Muchofthisworkisalreadydonewithvaryinglevelsofpublicness;wepresentworkatconferences,discussitinworkshops,shareitwithourcolleagues,andsoforth.
Buttypicallyourpublicationprocesseshaveoperatedoff-stage.
Inrecentyears,however,moreandmoreimportantnewworkinthehumanitiesissimplybeingpublishedonline,withoutthepresumedbenefitsofpre-publicationreview;scholarsarekeepingindividualblogsandparticipatingingroupblogs,andmoreandmoreconferencepresentationsandworkingpapersarebeingpostedonline.
Further,manyscholarsarefindingthatthefeedbacktheyreceivethroughtheseformatsisassubstantiveandproductiveastraditionalpeerreviewshavebeen.
Giventhewaysthatopenpracticesareenablingscholars'worktodevelop,itseems11PeterFrishauf,"ReputationSystems:ANewVisionforPublishingandPeerReview,"JournalofParticipatoryMedicine1.
1(Oct2009)9openreview:AStudyofContextsandPracticesincreasinglyimportantforthehumanitiestotakeaccountofsuchpractices,andtoexplorethepossibilitiesthattheypresentforourfieldsmorebroadly.
Whatdowemeanbyouruseof"open"inthinkingaboutopenreview,howeverMusteverythingbefullyopentoeveryone,oraretheredegreesofopennessthatmightbeusefultodifferentcommunitiesofpracticeatdifferenttimesMight,forinstance,afrankdiscussionamongadefinedclusterofscholarsbeparticularlyimportantatcertaintimes,whileadiscussionopenedtobroaderpublicswouldbecrucialatothersArethereinstancesinwhichareviewprocessmightbeopentovolunteerparticipantswhilenonethelessbeingconductedinprivateMustallreviewsbesubmittedunderreviewers'realnames,oraretheresituationsinwhichsomedegreeofanonymityorpseudonymityremainsusefulMoreover,arethesetwoformsofopenness–opennessofaccesstothereviewprocessandopennessofrevieweridentity–related,oraretheyseparableWedonotwanttoconflatedistinctfunctionsorformationsinthescholarlycommunicationprocess,nordowewanttoforeclosechoicesthatparticularcommunitiesofpracticemightfindbeneficial.
Atroot,wevalueopennessasadefaultvalue,whichmightattimesbeoverridden,forthewaysthatitcanenablethemembersofacommunityofpracticetoperformanddevelopincollaborationwithoneanother,ratherthantoassumeorprescribepre-existingstandardsforcontribution.
Openprocessessuchasthoseweexploreinthisdocumentcangobadly,ofcourse,ortheycangowell–butofcoursethesameistrueoftraditional,closedreviewprocesses.
Openprocessesrequirecarefulcultivationwithinacommunity,aswellascarefulattentiontotheheterogeneityofthatcommunity.
Awell-craftedopenre-viewprocess,however,canturnthe"problem"ofadiversecommunityintoavalue,creatingaself-consciousnesswithinthecommunityaboutitspresuppositionsandassumptions,andfacilitatingthedevelopmentofarangeofnewperspectivesandvoices.
WhatExperimentsHaveBeenConductedinOpenReviewPublishers,scholars,andacademiccollectiveshaveconductedanumberofre-centexperimentswithopenreviewpractices,witharangeofresults.
Perhapsthebest-knownoftheseexperimentswasthatconductedbyNaturein2006,anopenreviewtrialthathasbecomefamousforitspublicdeclarationoffailure.
However,asFitzpatrickhasargued,itislikelythattheexperiment'sdesignmadeitsfailurein-evitable12;thejournalproposedasysteminwhichtherewerenoperceivedbenefitstobederivedfromparticipation.
Otherscientificjournaleditors,however,posteddescriptionsofmoresuccessfulopenreviewprocessesintheforumthatwasposted12Fitzpatrick,PlannedObsolescence,pp.
25-26.
10openreview:AStudyofContextsandPracticesalongsidetheopenreviewtrial,andnumerousotherpublishershavelongusedhybridorfullyopenreviewprocesses.
13Severalexperimentswithopenreviewinthehumanitieshavereceivedbothschol-arlyandjournalisticattention.
TheInstitutefortheFutureoftheBookworkedwithMcKenzieWarktopostthedraftofhisbook,GamerTheory,onlineincom-mentableform;thoughthisexperimentwasnotexplicitlypartofapeerreviewprocess,itnonethelessresultedinsubstantivefeedbackthatWarkemployedinhisrevisions.
14ThemodificationsthattheInstituteengineeredintoWordPressforGamerTheorywerelatergeneralizedandreleasedasaplugin,Comment-Press15;CommentPresswasinitsearlystagesemployedbyCathyDavidsonandDavidTheoGoldbergintheprocessofreviewingandrevisingtheirMacArthurreport,"TheFutureofLearningInstitutionsinaDigitalAge,"16aswellasbyNoahWardrip-Fruin,inseekingfeedbackonhismanuscriptforExpressiveProcessing.
17FurthersuchexperimentsinopenreviewhavebeenconductedatMediaCommonsPress,includingtheopenreviewofKathleenFitzpatrick'sPlannedObsolescenceandthetwoopenreviewexperimentsconductedincollaborationwithShakespeareQuarterly.
18Allofthesetextswereatthestageatwhichtheywouldbesubmittedfortraditionalpeerreview,butwereintheseexperimentsopenedtocommunitydiscussion.
ThediscussionofPlannedObsolescence,likethatofExpressiveProcessing,tookplace13See,forinstance,Sandewall,"Openinguptheprocess",onthehybridprocessemployedbyElectronicTransactionsinAr-tificialIntelligence;KoopandPschl,"Anopen,two-stagepeerreviewjournal",ontheprocessusedbyAtmosphericChemistryandPhysics;Kooninetal,"Revivingacultureofscientificdebate"14SeeMcKenzieWark,GamerTheoryPortal,InstitutefortheFutureoftheBook.
15InstitutefortheFutureoftheBook,"CommentPress".
16CathyN.
DavidsonandDavidTheoGoldberg,"TheFutureofLearningInstitutionsinaDigitalAge,"InstitutefortheFutureoftheBook,laterreleasedinprintandPDFformbyMITPress(2009),aswellasinanexpandedbookform,asTheFutureofThinking:LearningInstitutionsinaDigitalAge(2010).
17NoahWardrip-Fruin,"ExpressiveProcessing:AnExperimentinBlog-BasedPeerReview,"GrandTextAuto18SeeMediaCommonsPress;Fitzpat-rick,PlannedObsolescence;andtheShakespeareQuarterlyreviewprocessesconductedforSQ61.
3,"ShakespeareandNewMedia"andSQ62.
3,"ShakespeareinPerformance".
11openreview:AStudyofContextsandPracticesalongsidetraditionalpeerreviews,whiletheShakespeareQuarterlyreviewstookplaceasthecentralpartofamulti-stageprocess.
Ineachcase,thetextswerereadandcommenteduponbymanyofthesamescholarswhowouldhavebeencalledupontoconducttraditionalreviews,butalsobyreaderswhoseexpertisemighthavebeenoverlookedinsuchaprocess(librarians,inthecaseofPlannedObsoles-cence;performersanddirectorsinthecaseofShakespeareQuarterly).
Thelocallytargeted,threadedcommentingfacilitatedbyCommentPress,alongwiththeun-derlyingsocialfeaturesofWordPress,resultedinrobustdiscussionsaimedathelp-ingtheauthorsinvolvedrevisetheirworkbeforefinalprintpublication.
Moreover,theCommentPressformatallowedreviewersandauthorsnotsimplytorespondtothetextbuttorespondtooneanotheraswell,andtheauthorshavereportedonthehelpfulnessofhavingacontextwithinwhichtounderstandandinterpretreviewercomments.
19Theopenreviewprocessthusservedadevelopmentaleditingrole,butinthecaseofShakespeareQuarterly,thediscussionsalsohelpedtheeditorialboardmakefinaldecisionsaboutwhethertoacceptthearticlesforinclusionintheprintjournal.
JackDoughertyandKristenNawrotzkisimilarlyusedCommentPresstofacili-tatetheopenreviewoftheessayscontainedintheirforthcomingvolume,WritingHistoryintheDigitalAge,usingtheplatforminordertohelpmake"thenormallybehind-the-scenesdevelopmentofthebookmoretransparent.
"20MattGoldlike-wiseusedCommentPressinthereviewprocessfortheessaysinDebatesintheDigitalHumanities,asdidLouisaSteinandKristinaBusseforSherlockandTrans-mediaFandom.
Inthesetwocasesthereviewprocesswasprimarilycommunallyorganized,withessaydraftsopenedtotheauthorsincludedinthecollectionsforcomment;inthisway,theauthorsworkedtogetherasacommunitytoimprovethevolumeasawhole.
21BeyondtheseCommentPress-basedprojects,however,anumberofhumanitiespublicationshaveputvariouskindsofopenreviewprocessesintopractice.
Thejournalpostmedievalconductedacrowdreviewforitsspecialissueentitled"Be-comingMedia,"usingWordPresstopresentablog-likestructureforitsdiscussions,withthearticlesappearingatthetopofeachpage,andthecommentsbeloweach19SeeWardrip-Fruin,"Blog-BasedPeerReview:FourSurprises,"GrandTextAuto;Fitzpatrick,PlannedObsoles-cence,pp.
189-90.
20JackDoughtertyandKristenNawrotzki,"Introduction,"WritingHistoryintheDigitalAge.
21SteinandBussealsoinvitedtwoexternal,non-anonymousreaderstoparticipateinthereviewprocessforSherlockandTransmediaFandom,engagingdirectlywiththeauthorsastheydiscussedthevolume'sessays.
12openreview:AStudyofContextsandPracticesarticle.
22ThejournalKairosusesanextensivemulti-tiereditorialreviewprocess,whichincludesseveralphasesofopencommunicationamongsteditorialboardmembersandbetweeneditorsandauthors.
23ThesiteDigitalHumanitiesNowusesPressForward'scombinationofcrowd-andeditorial-filteringmethodstohighlightsomeofthebestworkbeingdoneindigitalhumanitiesacrosstheopenweb;thosehighlightsarethenreviewedforrepublicationintheJournalofDigitalHumani-ties.
24Publicationssuchastheseareusingopencommentingatarangeofstageswithinthereviewprocessandwithavarietyofdegreesofpublicness.
Determiningwhetherareviewprocesshasbeen"successful"presentscertainchal-lenges,whichmayhighlightunspokenassumptionsabouttraditionalpeerreview:weassumethatareviewprocesshasbeensuccessful–thatreviewersrespondedtothetextsunderconsiderationinaforthright,scrupulous,criticalmanner,andthatauthorsmadeuseofthiscriticisminrevision–whengoodworkresultsfromit.
Inanopenreviewprocess,wehavethatsamemarkeravailable–istheworkresultingfromtheprocessgood–butwealsohavethehistoryoftheprocessitselfavailableforexamination.
Thatavailabilitycreatesanumberofcomplicationsinassessingthesuccessofareviewprocess,aswe'resuddenlyabletoaskmorequestions:Howmanycommentswouldbe"enough"HowmanycommentersArethecom-mentersestablishedorprestigiousenoughIsthecriticaldiscussioninwhichthosecommentersengagesufficientlyrigorousBeyondthequalityoftheend-product,openreviewprocessesalsoraisequestionsaboutthevalueweassigntothereviewprocess,whichmayresultinimportantscholarlydiscussionsamongreviewers,au-thorsandotherparticipantsthatdonotmakeitintothe"final"product,butnever-thelesscontributetoscholarlydiscourse(aswellaspossiblefuturewritings).
Ifwefocusonwhethertheprocessesdescribedaboveproducedgoodwork,itiscleartoseethatallweresuccessful:alloftheprojectsthatresultedfromtheseprocessesarewell-thought-of.
Examiningthereviewprocessesthemselvesdemon-stratesthatthereviewersandauthorsinvolvedcommentedseriouslyontheworkunderreview,andthatthereviewprocessesinalmostallcaseshelpedtomaketheworkbetter.
However,publishersandauthorsengaginginopenreviewprocessesmayneedtodevelopmoreparticularcriteriaforassessingtheirprocessesonan22postmedieval,"CrowdReview".
23Kairos,"TheKairosEditorialReviewProcess".
24SeeDigitalHumanitiesNowandJournalofDigitalHumani-ties;seealsoPressForward13openreview:AStudyofContextsandPracticesongoingbasis,toensurethatblindspotsdonotarise;asanexample,onemightseethesix-monthreviewofDigitalHumanitiesNowconductedbyPressForward.
25However,theseexampleswehavegatheredsuggestthatthereisnosingularpathtosuccess;successfulreviewprocessesmaydifferfromoneanotherinanumberofways.
Somemaybeentirelyopen,whileothersmaybeonlypartlyso;somemaybesingle-stageprocesseswhileothersusemultiplestagesofreview;somemaybewhollyopenwhileothersareopenwithinacommunity;somemayentirelyeschewanonymitywhileotherspermitit.
Whatallsuchsuccessfulexperimentsbearincommon,however,isaself-consciousconsiderationofthevaluesofthecommunitythatthereviewprocessismeanttoserve,andaflexiblebutnonethelessrigorousattempttoreflectthosevaluesinthemechanicsoftheprocessitself.
WhoElseIsExploringTheseIssuesInadditiontothewebforumheldalongsideNature's2006openreviewexperiment,26muchhasrecentlybeenwrittenaboutopenreviewpracticesandrelatedissues.
DianeHarley,SophiaKrzysAcord,andtheCenterforStudiesinHigherEducationatUCBerkeleyhavepublishedanextensivereportontheusesofpeerreviewintheacademicpublishingandpromotionprocesses.
27Whilethereport'sfocusisonthepracticeofpeerreviewingeneral,thusonlytreatingopenreviewprocessesasasmallsubset,28itincludesasubstantialassessmentofthediffi-cultiespresentedbycurrentpeerreviewpractices.
Otherpublicationshavefocusedmoreexplicitlyonopenreview,includingthepossibilitythatnewalternativescouldupholdqualitycontrolandsupportbetterscholarlycommunicationwhilemorevisiblyandequitablydistributingtheburdenofpeerreviewunderwhichscholars25"SixMonthReviewofDigitalHumanitiesNow,"DigitalHumanitiesNow,RoyRosenzweigCenterforHistoryandNewMedia.
26Nature'speerreviewdebate.
27DianeHarleyandSophiaKrzysAcord,"PeerReviewinAcademicPromotionandPublishing.
"28WhereHarleyandAcorddodiscussthoseprocesses,theirdiscussionatmomentstakesforgrantedtheconventionalwisdom,ascommunicatedbythescholarswithwhomtheyconductedinterviews,thatopenprocessesarelessrigorousthantraditionalpeerreview,andthattheyruntheriskofdevolvingintopopularitycontests:"Thismodelsuggests(intheextreme)thatweshouldabandonformalpublishingvenuescompletelyandsimplyallowscholarstopublishanywhere–frompersonalwebpagestoblogstoinstitutionalrepositories–andletthe'market'begintorankandcommentonthenon-peer-reviewedpublicationstodeterminetheirimpactandpopularityandattention-grabbingnature"(HarleyandAcord39);seealsopp.
45-48,whichdiscussesthepotentialforopenreview,butfocusesonresistancetoitingreaterdepth.
Aforthcomingpaper,"Credit,Time,andPersonality,"(NewMediaandSociety)willexploreopencommentarysystemsingreaterdepth,inthecontextofthemanywaysthatscholarshavelongsharedpre-publicationworkwithoneanother.
14openreview:AStudyofContextsandPracticeslabor.
PeterFrishauf,founderofMedscape,haswrittenextensivelyabouttheben-efitsthatopenreviewmightprovideinthemedicalsciences.
29KathleenFitzpatrickhaswrittenaboutthehistoryandfutureofpeerreview,arguingforthedevelop-mentofnewopenreviewpractices,inPlannedObsolescence.
30Fitzpatrickhasalsoco-writtenanarticlewithKatherineRoweexploringtheprocessandresultsoftheShakespeareQuarterlyopenreviewexperiment,31whichhasbeencoveredbytheChronicleofHigherEducation,theNewYorkTimes,andUniversityAffairs/Affairesuniversitaires.
32Morerecently,thejournalpostmedievalpublishedanopenwebforumtodiscussopenreviewprocesses,33andJackDougherty,KristenNawrotzki,CharlotteRochez,andTimothyBurkepublishedathoughtfulconclusiontotheironlinereviewprocessforWritingHistoryintheDigitalAge.
34Discussionssuchasthesearenotuniversallylaudatory;theirfrankevaluationsofopenreviewpro-cessesidentifybothstrengthsandchallengesinwaysintendedtoprovokecarefuldeliberationbyothereditorialandcollaborativegroupsconsideringconductingtheirownexperimentswithopenreview.
Additionally,anumberofeffortsrelatedtoopenpeerreviewareunderway,includ-ingprojectssuchastheOpenAnnotationCollaboration(OAC),anefforttocreatetechnicalstandardsandtoolstoenablethecreationofwebannotationsthatcanbebesharedinmultiplecontexts,andtheOpenResearcherandContributorIDproj-ect(ORCID),whichisworkingtodevelopastandardfortheuniqueidentificationofscholarlyauthors;bothprojectsintersectwithworkbeingdonebythoseseekingalternativemeansofaccountingfortheimpactofscholarlyresearch,suchasthealtmetricsgroup.
35ProjectssuchasImpactStoryandHypothesisseektobringthese29SeeFrishauf,"ReputationSystems";seealso"ATroubledTrifecta:PeerReview,Academia,&Tenure,"e-patients.
netand"TheEndofPeerReviewandTraditionalPublishingasWeKnowIt,"MedscapeJournalofMedicine10.
11(2008):267.
30Fitzpatrick,PlannedObsolescence,pp.
15-49.
31KathleenFitzpatrickandKatherineRowe,"KeywordsforOpenReview,"Logos21.
3-4(2010):pp133-41.
32JenniferHoward,"LeadingHumanitiesJournalDebuts'Open'PeerReview,andLikesIt,"ChronicleofHigherEducation(26July2010);PatriciaCohen,"ScholarsTestWebAlternativetoPeerReview,"NewYorkTimes(24August2010);RosannaTamburri,"Openinguppeerreview,"UniversityAffairs/Affairesuniversitaires(5March2012)33postmedievalForum,"ForumII:TheState(s)ofReview"(March2012)34"Conclusions:WhatWeLearnedfromWritingHistoryintheDigitalAge(Spring2012),"Writ-ingHistoryintheDigitalAge.
35OpenAnnotationCollaboration;OpenResearcherand15openreview:AStudyofContextsandPracticeskindsofinformationtogether,linkingopenwebannotationwithreputationman-agement,inwaysthatmightbeusefultoopenreview.
36PressForwardwillprovidetoolsthroughwhichgroupsofscholarscancapturethebestworkbeingpublishedacrosstheopenweb,throughacombinationofcrowd-sourcingandeditorialmanagement.
37SocialreadingplatformslikeBookGluttonandSocialBookconnecttextsandreadersinflexible,community-orienteddiscussions.
38AndsitessuchasAcademia.
eduareworkingtocreatecommunitiesofscholarssharingtheirworkwithoneanother.
39Inaddition,ourworkwithopenreviewbearsmuchincommonwiththatofothergroupsseekingtointroduceotherformsofopennessintoschol-arlyresearchsuchastheopenaccessandopendatamovements.
Therecommenda-tionsthatwedevelopbelowpresupposethatopenreviewprojectswilllearnfromandcollaboratewithprojectssuchastheseinthecomingyears.
RecommendationsInthecourseofourexaminationofthecontextsintowhichopenreviewpracticesenter,aswellastheexistingexamplesandstudiesofthesepractices,wedevelopedaclusterofrecommendationsfortheacademicandeditorialsystemsthatmightbestfacilitateopenreview,aswellasforthetechnologicalsystemsthatmightbestsupportit.
RecommendationsforCommunitiesofPracticeThoughdiscussionsaboutopenreviewtypicallyrevolvearoundtechnologicalin-novations,oneofthekeyobservationsrepeatedlymadebytheadvisoryboardwasthathumansystemswereprimaryindevelopingcriteriaforsuccessfullycarryingoutanyreviewprocess,closedoropen.
Byhumansystemswemeantherecogni-tionthatrealpeopleneedtoworktogethertowardacommonpurposewithinestablishedinstitutionsandorganizations,andwithdefinedrolesandobjectives,inordertoundertakeanevaluativeandcriticalengagementwithscholarlywork,regardlessofplatformsortools.
Moreover,humanrelationshipsandinvestments–morethantechnologies–areessentialforattractingparticipationinopenreviewprocesses,fordevelopingandmodelingnormsforparticipation,forteachingandpracticingprinciplesofmutualresponsibilityandgoodcitizenship,andforenact-ingaparticipatoryethos,allofwhichareessentialtoasuccessfulopenreview.
Creatingasuccessfulopenreviewprocessbeginswithclearlyestablishingrolesandexpectationsforparticipants.
SuchrolesandexpectationsmustbecommunicatedContributorID;altmetrics.
36ImpactStory;Hypothesis.
37PressForward.
38BookGlutton;SocialBook.
39Academia.
edu.
16openreview:AStudyofContextsandPracticesrepeatedly,inordertoensurethatallpartiesworkinconcertthroughoutaprocess.
Moreover,aworkingopenreviewprocessrequireslaborfromorganizers,editors,authorsandreviewers.
Evenenthusiastsmustbemobilizedtoparticipate.
De-spiteutopianrhetoricaboutcollectiveintelligenceandculturalconvergence,openreviewprocessescannotrelyonviralityorbuzztosustaincommunityengagementanymorethantheycanbesuccessfullycarriedoutthroughpurecrowdsourcing.
Instead,openreviewprocessesrequirededicatedguides–likelyassumingtherolesofeditors–whoactasstewardsforaproject,leadingpotentialparticipantstoandthroughtheopenreviewprocess.
Developingaclearsetofexpectationsofwhat"participation"entailsisessentialforprocuringandretainingparticipants,thougheventhisisnotaguarantorofparticipation.
Further,adequateincentivesforpar-ticipationmustbecreatedandmaintained.
Thoughthelikelihoodofparticipationisgreaterwhenparticipantssharecommoninterestsandaffinities,suchcommonalitiesdonotassureasuccessfulopenreviewprocess,aspre-existingcommunitiescansometimesreproduceaformof"group-think"thatlimitshowaworkcanbeevaluated.
Asonereviewerofthedraftofthiswhitepapernoted,openreviewprojectsmayneedto"consciouslystageconfronta-tionacrosscommunityboundaries"inordertoensurethatcompetingperspectivesspeaktooneanother.
40However,openprocesseshavethepotentialtorevealthefaultlinesthatotherwiseexistunremarkedwithinafield.
Ofcourse,acommunitymadeupofindividualswithdivergentandincongruousdisciplinaryandschol-arlybackgroundscanfrustrateareviewprocess,producingcompetingevaluatorypracticesthatspeakpastoneanother.
Stewardsforopenreviewprocessesmustactastranslatorsacrosscommunities,helpingparticipantsrecognizeaffinitiesinspiteofinstitutionalorrhetoricaldifferences.
Fromtheoutset,stewardsmustworkwithcommunitymemberstoestablishmutu-ally-agreeduponcriteriaforconductingopenpeerreview.
Thesecriterianeedtobebothflexibleandrigorous.
Duetothemanypossibledesiredoutcomesandmodesofreviewavailable,arigidsetofruleswouldnotonlybeimpossibletodevelopwithoutseverelylimitingthepossibilitiesforopenpeer-to-peerreview,butwouldactuallybedetrimentaltotheprocess.
Atthesametime,rigorisessentialpreciselybecausetheflexibilityopenreviewrequirescanquicklydevolveintoadhoc,unruly,oruncriticalformsofassessmentandfeedback.
Openreviewistoooftenassumedbyskepticstoproduceinferiorqualityevaluationsderivedfromlaxand/ornon-ex-istentstandards;clearstatementsofacommunity'sstandardsandexpectationsfor40TedUnderwood,untitledcomment,"OpenReview:AStudyofContextsandPractices".
17openreview:AStudyofContextsandPracticesitsopenreviewpracticesmayhelptoalleviatethisassumption.
(Itshouldbenoted,ofcourse,thatclosedpeerreviewprocessescanalsosufferfromalloftheaboveandareequallydependentonhumanagentsinvestinginarigorousevaluationsystem.
)Ahealthyopenreviewprocessisoneinwhichpeersmodelandtherebyreinforcesalient,productivenormsforparticipation.
Whileflexibilitynecessitatesthateachopenpeer-to-peerreviewundertakingestablishandfollowitsownrigorousevaluationcriteria,therearesomeidentifiableparametersapplicabletoallopenreviewcommunitiesandcontextsthatshouldguidecriteriadeterminations:1)Desiredoutcomes:Asdescribedabove,differentformsofpeerreviewcanservedifferentpurposes,fromgatekeepingandcredentialingtopromotingdialogandofferingadvice,fromcriticalengagementtoevaluationtoimprovementofongo-ingwork.
Itcanbebothmacroandmicroinscope,addressingconceptual,orga-nizational,evidentiary,attributional,methodologicalandstylisticmattersatlevelsthatrangefromcompletedwork(and,inthecaseofpromotionandtenure,acrosscompletedworks)allthewaydowntoindividualwordsandsentences.
Itisbothamechanismforauthenticatingproductsandaprocessofintellectualcollaboration(thoughonethatisrarelyvisible).
Peerreviewisoftenallofthesethingsatonce,thoughcertaingoalsmighttakeprecedenceoverothers.
Withthisinmind,openreviewcommunitiesmustbeveryclearaboutthegoalsanddesiredoutcomesofanyundertakinginadvanceoftheprocess:Howareworksselectedforevaluation(throughopensubmissionorvettingbyeditorsorcommu-nity,etc.
)Whatisbeingevaluated(anessay,amonograph,ablogpost,amulti-mediaproject,atenurefile,thereviewprocessitself,etc.
)andforwhatpurpose(foreventualpublicationineithertraditionaloremergingpublicationformsandvenues;forawork'sscholarly,pedagogical,analytical,prescriptive,polemicand/orcreative/innovative/experimentalmerit;forthepurposesofbrainstorming,foster-ingdialogwithinacommunityorbetweenacommunityandawork'sauthor(s);forcredentialing,etc.
)Whataspectsofaworkaretobeevaluated(rangingfromqualityofargumenttoabilitytoengageimaginedaudiences),atwhatlevel(s)andthroughwhatmeans(rangingfromholisticevaluationofacompletedworkdowntothechapter,section,paragraph,sentenceandevenwordlevel,takingformsrang-ingfromcopyeditingadocumenttoembeddingcommentsto"liking"or"dislik-ing"variouscomponents)Determiningthemeansofevaluationrequiresopen18openreview:AStudyofContextsandPracticespeer-to-peerreviewcommunitiestoreachconsensusaboutthetypesanddegreesofopennesstheywishtoembrace.
2)Openness:Aspreviouslystated,opennesscantakeseveralforms.
Optionsin-cludepublicaccesstoandparticipationinthereviewprocess;removingtheano-nymityamongstauthorsandreviewers;establishingameansofgreaterback-and-forthbetweenauthorsandreviewersandamongstreviewers.
Eachoptionpresentsbenefitsandchallenges.
Forexample,makingthereviewprocesspubliccanhelprenderscholarlyprocessestransparent,butmayalsoblurdistinctionsbetweenpeergroups,asindividualswithvaryingdegreesandformsofexpertisebecomepar-ticipants.
Ofcourse,thelatterisonlya"downside"ifthisisnotadesiredoutcomeofopenness.
Ultimately,openreviewcommunitiesmustdeterminethetypesanddegreesofopennesstheywillpursueinrelationtothedesiredoutcomesofthepeerreviewprocess.
Broadly,decisionsaboutopennessencompass:thechoicebetweenanonymity,pseudonymity,andtransparencyinrepresentingreviewerandauthoridentities;thechoicetoopenupthereviewprocesstopublicviewingand/orpar-ticipation;andthechoicetoallowandencouragereciprocitybetweenauthorsandreviewers,aswellasamongstreviewers.
3)Etiquette:Extendingthechoicesmadewithregardtoopenness,openreviewcommunitiesmustalsolayoutgroundruleswhenitcomestoexpectationsofcivil-ity,reciprocity,andrevisionwhenitcomestoprovidingandrespondingtocon-structivecriticism.
41Suchnormsforparticipationarecommonplaceineverytypeofreviewprocess,blindoropen,thoughtherearesometimesconcernsraisedaboutopenreview'sabilitytobothmaintaincivilityandoffersubstantivefeedback.
Inactuality,blindreviewalsorequiresparticipantstolearnandpracticepropernormsofengagementand,unfortunately,unhelpful,dismissive,andmean-spiritedfeed-backcanproliferatejustasfrequentlyasconstructivecriticismthroughtheclosedreviewprocess.
Oneoftheconcernsrepeatedlyraisedaboutopenpeerreviewisthatreviewersmaybelesslikelytooffercandidassessmentsorharshcritiqueswhentheirnamesareattachedtotheirevaluations.
Thisproblemisseenasparticularlyacuteforjuniorscholarswaryofupsettingseniorcolleagues.
Concernssuchasthesearenoteasilyovercomeandmightcontributetoacommunityadoptinganethosofopennessthatembraceseitheranonymityorpseudonymity.
Webelieve,however,thataclearly41AndyFamigliettihasexploredarelatedsetofnormsforparticipationwithrespecttowhathecallsthe"moraleconomy"ofWikipedia'sfamousNPOVorneutralpointofviewrule.
See"Nego-tiatingtheNeutralPointofView:PoliticsandtheMoralEconomyofWikipedia".
19openreview:AStudyofContextsandPracticesestablishedsetofnormsforacceptableandunacceptablemodesofdiscourseoffersacommunitygroundsforaddressing,managing,andpolicinguncivilbehavior.
Italsoestablishesmechanismsformediatingharshcriticism,including,forexample,theabilitytoswitchbetweenpublic,semi-public,andprivatechannelsdependingontheseverityofthecritiqueortheabilitytohaveeditorsactasintermediariesbetweenauthorsandreviewerswhenconflictsarise.
Beyondconcernsaboutcivility,openreviewraisesuniquechallengesforcommu-nityetiquettewhenitcomestointeractionsbetweenauthorsandreviewers,aswellasamongstreviewers.
Inatraditionalreviewprocess,authorshavenoroleinsolic-itingreviewersfortheirwork,butinanopenreview,creatingasuccessfulprocessoftenrequiresauthorstakeanactiveroleinencouragingdiscussion,thusplacingtheneutralityoftheauthorunderacertaindegreeofpressure.
Suchactivesolici-tationofreviewersisacomfortableprocessforscholarswhoareaccustomedtointeractionswiththeirpeersviasocialmediaformssuchasblogsandTwitter,butforscholarswithlessexperienceinonlinecommunication,requestingcommentmayfeelindiscreet.
Moreover,opendiscussionbetweenauthorsandreviewersmaynotcomenaturallytothoseaccustomedtotraditionalprocesses.
Intryingtocreateanenvironmentconducivetothegreatestpossiblecomfortwiththemutualityofparticipationfosteredonline,anopenreviewcommunitymaywishtodetermine:(a)whattheroleofauthorsinencouragingdiscussionshouldbe,(b)whetherauthorsandreviewersshouldbeallowedtointeracteitherdirectlyorindirectly,(c)whetherinteractionisrequiredormerelyencouraged,(d)whatre-sponsetimeforinteractionsisappropriate,(e)whetherallreviewerfeedbackmustbeacknowledged,andifso,inwhatmanner;orifnot,howchoicesshouldbemadeaboutwhichcommentsmustbeaddressed,(f)howdisagreementsamongstreview-ersshouldberesolvedintherevisionprocess,and(g)whenscholarlyconversationhasstrayedtoofarofftopictobeconsideredpartofthereviewprocess.
Itshouldalsobekeptinmindthatifcommentingguidelinesaretoorestrictive,elaborate,orpredetermined,thistoomightdissuadepotentialparticipants.
Commentingguide-linesmuststrikeabalancebetweenidentifyingdesiredobjectivesandprovidingflexiblemeansforparticipantstoachievethoseobjectivesthroughtheircomments.
4)Labor/Reward/Authority:Theworkinvolvedintraditionalpeerreviewhasoftenbeeninvisible,butithasneverbeeninsubstantial.
Editorsmustmobilizereviewers.
Reviewersevaluateandmakerevisionrecommendations.
Authorseitherreviseaccordingtoreviewerfeedbackoraddresstheirdecisiontosetasidecertainsuggestions.
Editorsdecidewhetherornotauthorshavesufficientlyaddressedreviewerconcernsandrequestsbeforeacceptingorrejectingawork.
Inthetradi-20openreview:AStudyofContextsandPracticestionalprocess,reviewerlaborcanbemostintensivebutitisalsotheleastvisible.
Inopeningthepeerreviewprocess,wehavetheopportunitytomakethemmoretransparent,thusmakingthislaborandthecontributionsthatitmakestothede-velopmentofscholarlythoughtmorevisible.
Reviewingtheworkofcolleagues,whetherformally(forajournalorpresssubmis-sion)orinformally(aspartofawritinggroup)isconsideredpartoftheacademicgifteconomy,aserviceperformedwithminimalreward,savepossiblyananony-mousthankyouwhenaworkiseventuallypublished(assumingthereviewerrecommends"accept")orarequestfromaneditortoreviewanotherworkasacknowledgementthatpreviouseffortsareindeedappreciated.
Scholarsreviewoneanother'sworkforavarietyofreasons,rangingfromthedesiretobegoodcitizenstothewishtostayontopof(andpossiblyshape)newscholarshipemergingwithintheirfields.
Whileanonymitymaypermitreviewerstheabilitytooccasionallyoffernecessaryharshcriticalfeedbackwithoutfearofrepercussion,italsoallowssomereviewerstoofferunconstructiveanduncivilresponsesthatneitherhelpauthorstoimprovetheirworknorforwardscholarlydiscourse.
Inanopenreview,theworkdone–andnotdone–byreviewersisvisible(evenifthereviewersremainanonymous).
Morethanthis,theworkofreviewmayalsobecomethesubjectofreview,aseditorsandauthorscancalluponareviewcom-munitytoaffirmordismissparticularrecommendations,deliberateoncontradic-toryrevisionrequests,orsocializeparticipantsintocommunityetiquette.
Atthesametime,thevisibilityofreviewcanbecomethemeansbywhichreviewersattainpeerstatus,garneringrecognitionforboththesubstanceoftheircontributionsandtheirinvestmentoftime,energyandthoughtintheprocess.
Finally,thevisiblecontributionsthatreviewerscanmaketobothawork-in-progressandtoscholarlydiscoursecanclarifytheoriginanddevelopmentprocessofscholarlyideas,high-lightingthecollaborativenatureofallscholarlywork.
Giventhepotentialweightthatsuchreviewsmightthuscarry,openreviewcom-munitiesmightdecideinadvancehowreviewercontributionsaretobe"counted"andevaluated.
Theformercanrangefromhavingeachcommentpostedconstituteitsownuniquecitation(whichthetechnicalworkoftheOpenAnnotationCol-laborationisdesignedtoenable)tograntingaformofco-authorshipintheover-allwork.
Whateverdecisionsaremadewithrespecttotherelationshipbetweenreviewsandauthorship,openreviewcommunitiesshouldnonethelesshavepoli-ciesinplacethataccountforthepotentialvalueofideassharedwithinafieldthatrewardsintellectualcapital.
ThismayincludetheuseofCreativeCommonslicensesallowingreviewersandauthorstodefinetermsforincorporatingreviewersugges-21openreview:AStudyofContextsandPracticestionsintoarevisedwork,oracollectivelyacceptedstatementaboutparticipationinanopengifteconomy.
Alternately,recognitionofreviewerscansimplytakeasimi-larformtotraditionalprintpublications,throughfootnotesoracknowledgementpages.
Again,thespecificformsofrecognitionareultimatelylessessentialthanthecommunity'sclearunderstandingandagreementonastandardforrecognitionpriortocommencementofthereview.
Similarly,evaluatingtheworkdonebyreviewerscantakemanyforms,rangingfromsimplevotingmechanismsthatupgradeordowngradethestatusofparticularcomments,tomid-tiersystemsthatallowmemberstomarkapost'srecommen-dationsas"required/recommended/provocative/tobedisregarded,"totheuseofmoresubstantivemeta-commentingtoolsthatallowcommunitiestoassessreview-erpostsinmuchthesamemannerthattheyassesstheoriginalworkunderreview.
Whateveroptionisdecidedupon,openreviewcommunitiesmusthavecriteriainplaceforevaluatingbothaworkunderreviewandthereviewersparticipatingintheprocess.
Whilecriteriaforevaluatingreviewsandreviewerswillvaryacrosscommunitiesandprojects,broadareasforconsiderationinclude:tone,contribu-tiontoimprovingworkunderreview(whetheratthemacroconceptuallevelorthemicroorganizational/grammaticallevel),contributiontofurtheringscholarlydis-course,andgoodcitizenship.
Thatlatterconsiderationmightraisequestionsabouthowactiveandengagedarevieweris,howwillingarevieweristoshareresources,andsoforth.
Finally,openreviewcommunitiesmayalsowishtoestablishwhereauthorityultimatelyresidesinmakingfinaldecisionsaboutreviewerrecommen-dationsandstatus.
Istheprocessconsensus-based,ordoeditors(orapre-selectedsubgroupofpeerevaluators)takecommunitysuggestionsintoaccountwhenmak-ingdecisionsaboutthereviewprocessWhilethepublish-the-filtermodefosteredbyopenreviewprocessescancertainlyspeedupthetimebetweenthecompletionofscholarlyworkanditsreceptionbyanaudience,itdoesnoteliminatethelaborinvolvedinreview.
Indeed,whenthereviewprocessisopened,laborisnotonlymademorevisible,itcanalsobecomemorelaborious,particularlyforauthorstryingtorespondtomultipleasynchro-nousrevisionrequestsandseekingconsensusapprovalonawork'sprogressionthroughthepublicationprocess.
Whileopennesscanproduceabroaderandricherreview,itcanalsoproducealongerone,asanexpandingmiddleforestallsfinality.
Thevalueofongoingrevisioncanbeprogressive,allowingargumentstobere-thoughtandre-presentedinrelationtoemergingdiscoverieswhileprivilegingtheprocessofintellectualengagementoveritsendproduct.
However,withinanaca-demicsystemthatstillusespublicationasameasuringstickfortenureandpromo-tion,andwithinanenvironmentinwhichscholarsarealwayspursuingmultiple22openreview:AStudyofContextsandPracticesprojects,finalityalsohasitsmerits.
Hence,openreviewcommunitiesneedtohaveprotocolsinplaceformovingaworkforwardinthereviewsystem,potentiallyincludingtimelimitsonthereviewprocess,filteringmechanismsthatallowau-thorstoprioritizerevisionrecommendations,distinctionsbetweenpre-and-post-revisionassessmentcriteria,andeasilyrecognizabledifferentiationsbetweenstagesofreview.
Finally,openreviewcommunitiesmustmakedeterminationsaboutarchivingordocumentingthereviewprocessinamannerthatpreservesanddemonstratesthelaborinvolvedinmovingaworkthroughvariousstages.
Thelatterisparticu-larlyimportantwhenpresentingopenlyreviewedworkstoskepticalcolleagues,whoseassumptionsaboutlackofrigorareoftenusedtodismisssuchworks'merit.
However,decisionsaboutthepersistenceofareviewprocessbearimportanceforthescholarlyrecord.
Authorsmayhaveaninterestinensuringthatonlythefinal,revisedversionoftheirworkisaccessible,buttheerasureofcommenterlaborshouldnotbeundertakenlightly,asitcancreateanegativeincentiveforparticipa-tioninopenreviewprocesses.
Scholarlycommunitiesmusttogetherdecideaboutthebalancebetweenthepersistenceofthereviewprocessandthedesiresforaclear"versionofrecord"ofscholarlywork.
WeatheringtheCurrentClimateforOpenPeerReviewWhiletheparameterslistedaboveareintendedtoensurethatallopenpeerre-viewsadoptarigorousandstructured–ifalsonon-standardized–setofnormsforparticipationthatguidehowagivencommunitygoesaboutitstasks,noneoftheseideaswillnecessarilyensurethatanopenreviewprocesswillbeviewedbyskepticsasaviablealternativetoaclosedsystem.
Whilewedonotseethepurposeofthisdocumentasofferingstrategiesforcombatingskeptics,wedorecognizetheimportanceofhavingopenpeerreviewcountasalegitimatemodeofevaluationforscholarly,pedagogicalandotherformsofacademicwork.
Thisrequiresstrik-ingabalancebetween,ontheonehand,meticulouslydocumentingtheprocessformoving(andnotmoving)aworkthroughpublicationstages(aswellastheprocessfordefiningandrefiningthepeerreviewcriteria)and,ontheother,articulatingavisionforpeerreviewthatrecognizesthedegreetowhichfluidityincriticalen-gagement,debate,discussion,anddialogarecentraltoproducingbetterhumani-ties-basedscholarship.
Inotherwords,proponentsofopenpeerreviewmustmovebeyondclaimsthatsuchprocessesareeitherjustasrigorousastraditionalblindrevieworintendedtoaccomplishdifferentresults,andinsteadmusthighlightthewaysthatopenpeerreviewadoptsandenhancesthebestaspectsofhumanities-basedscholarlypractices.
AsJackDoughertyhasnoted,openreviewprocesseshavethepotentialtodocumenttheirsuccess;"bymakingdraftsandcommentary23openreview:AStudyofContextsandPracticesvisible,wecantracehowdifferentauthor-readerexchangesinfluencedthefinalmanuscript.
"42The"openness"ofopenreviewcanprovedaunting,however,whenitcomestoevaluatingscholarshipforthepurposesoftenureandpromotion,whereeventheslightesthintofcriticismcansinkascholar'schances.
43Ithasbecomesomethingofaruefulin-joketoacknowledgethatalmosteverytenurereviewletterrankscandidatesamongthetop5%ofscholarsintheirfield–andthatanydeviationmayberegardedasa"redflag"byreviewcommittees.
Whileunrelatedtotheactualpracticesofopenreview,suchaninflationinstandardsforevaluatingscholarscanhaveachillingeffectonparticipationinaprocessthatmakescritiqueanddisagree-mentvisible.
PersonnelreviewprocesseswillneedtoacknowledgethatwhatJohnGuilloryhasreferredtoastheimmanentsceneofjudgement44isoneofdebateanddisagreement,andthatthisdissentisasignofengagementinafield–thatthemoreengagedanopenreviewcommunityiswithamember'swork,thegreateritssig-nificance.
Whileopenreviewcommunitiesmustadoptprotocolsfordistinguishingbetweenengageddebateand"rejection"intheseenvironments,concernsoverhowopenreviewwillbeperceivedbytenureandpromotioncommitteessuggestthattheworkcurrentlybeingdonetovalueopenpeerreviewmustbemadepartofalargerprocessofreformingacademicstandardsforevaluation.
Proponentsofopenpeerreviewmusttakeaholisticview,situatingargumentsaboutopennessinrela-tiontobroaderquestionsaboutthefutureofscholarlydiscourseandtherolesofthe21stcenturyacademic.
Wethusfeelanimperativetoacknowledgethecontextinwhichopenreviewcurrentlyoccursandrecognizethatdiscussionsaboutopenreviewshouldideallybepartofabroaderconversation(onebeyondthescopeofthisproject)aboutreformingotheracademicpracticesandprocedures.
Whiletechnologycanbeatoolinhelpingtoaccomplishmanyofthesephilosophi-calandpragmaticgoals,thissectionhashighlightedsomeofthecoreconcernsand42JackDougherty,untitledcomment,"OpenPeerReview";seealsoDougherty,etal,"Conclusions:WhatWeLearnedfromWritingHistoryintheDigitalAge,"inwhichtheprojecteditorsexplorethedatafromtheirreviewprocessinarguingforitssuccess.
43OurdiscussionhereisspecifictothedominanttenurepracticesandpoliciesintheUnitedStates;Europeanandothernationalsystemsarequitedifferent,ofcourse,andpresenttheirownuniquechallenges.
44SeeJohnGuillory,"EvaluatingScholarshipintheHumanities:PrinciplesandProcedures.
"ADEBulletin137(Spring2005):18–33.
24openreview:AStudyofContextsandPracticesdesiresthatmustdrivetechnologicalinnovations.
Inthenextsection,wediscusstechnologicalsystemsthatcanbedeployedordesignedtoaccomplishtheseends.
RecommendationsforTechnologicalSystemsGiventhedesireforflexibilitythatourdiscussionsunearthed,aswellastherecog-nitionthatdifferentcommunitiesofpracticewillhavenotonlydifferentgoalsfortheirreviewprocessesbutalsofundamentallydifferentwaysofworking,wearenotledtoproposethedevelopmentofanew,monolithicplatformorreviewtool.
DevelopingfromthegroundupaplatformwiththerobustnessandeaseofuseofWordPress,theflexibilityofDrupal,thepossibilitiesformultimodalpublishingofOmekaorScalar,andtheworkflowmanagementofOpenJournalSystemswouldnotonlybeaninordinatelyexpensiveproposition,andwouldnotonlyhavetofightthealreadyextensivebuy-inthatthoseplatformshavedeveloped,butitwouldalsopresentrequirementsforongoingsupportthatareunlikelytobemetinthecurrentenvironment.
Wethusintendinwhatfollowstoidentifyasuiteofpossibilities,an"ecology"oftoolsthatcanworktogetherindifferentcombinations.
MuchofwhatfollowsisdrawnfromworkdonefortheadvisorygroupbyDanViselandPeterBrantley;theirfullreportsareincludedasappendicestothispaper.
Openreviewtoolsneedtoserveanumberofbasicfunctions,includingbutnotlimitedtothefollowing:1.
UsermanagementaIdentity:Aflexibleopenreviewsystemmustallowforanonymityorpseud-onymityandforself-identification,andmustallowaparticularpublicationorcommunityofpracticetodeterminewhethertoenforceareal-namepolicy,orwhetheranonymouscommentingisdesirable,andinwhatcircumstances.
bRoles:Anopensystemshouldallowforthecreationofdifferentusertypeswithdifferentprivilegesandpermissions,suchasEditor,Moderator,Author,Reviewer,andsoforth,andthoserolesmustbeconnectedtothecontextofaparticularpublicationcAggregation:Anopenreviewsystemmustbeabletogatheranddisplaythecompletecontributionsofanindividualuser.
dReputation:Thissystemshouldpermittheassessmentnotjustofprimarytexts,butofthequalityofcontributionsofindividualreviewers.
eModeration:Thesystemshouldallowbothspampreventionandforcom-mentstobevoteddown(whenunhelpful)ordeleted(whentrolling).
fNuance:Whileareviewershouldbeabletoregisterlightweightapprovalor25openreview:AStudyofContextsandPracticesdisapprovalofcontent("likes"and"dislikes"),anemphasisshouldbeplacedonmorequalitativeassessment.
gCommunitynorms:Thesystemshouldprovidemeanstoaddressscore-settlingbehaviorandtokeepanygivenuserfromdominatingaconversation,amongothersuchissues.
2.
ContentmanagementaMedia:Thesystemshouldprovidetheoptiontoembedoruploadimage,audio,orvideocomments;similarly,thesystemshouldpermitcommentingonmediaobjects.
bMobility:Thesystemshouldpermitreviewerstoreadthepublishedcontentintheirpreferredinterface,evenifcommentingmustbeundertakeninaparticularsite.
cGranularity:Thesystemshouldpermitcommentstobeattachedtothepage/paragraph/sentence/wordlevelofatext,aswellastotheentiretyofatext.
dVersioning:Thesystemshouldideallyallowatexttoberevised,andshouldaccountformultipleeditionsoftextsandcomments.
Readersandreviewersshouldbeabletocomparedrafts(viaa"trackchanges"mechanism).
eCitation:Thesystemshouldallowthelineageofideastobetracedbothforwardandbackwardintimethroughbothtextsandcomments,usinglinksthatfunctioninbothdirections(thatis,acitationoftextAintextBshouldnotonlyresultinalinkfromBtoA,butalsoacorrespondinglinkonAtoB).
fFilters:Thesystemshouldallowanauthororeditortocustomizethekindsorqualityofcommentstheyseeatanygiventime.
Reviewsmightbefilteredbytheirfocus(argumentation,evidence,style),forinstance,orbythecom-munity'sratingoftheirhelpfulness,bytheidentityofthereviewer,orbyotherfactors.
3.
WorkflowmanagementaReviewerselection:Thesystemshouldalloweditorstosolicitspecificreview-ersfromanavailablepool,whilenotexcludingothersfromtheprocess;itcouldalsoallowauthorstorecommendparticularreviewerswhosefeedbackwouldbeparticularlydesirable.
bNotification:Thesystemshouldalertmemberstothepresenceofnewtextsavailableforreview,especiallywherenewtextsspeaktodeclaredmemberinterests,andshouldalertauthorstonewcommentsontheirtexts.
cPrompts:Thesystemshouldalloweditorstocreatespecificandeasypromptstoguidereviewersintheirtaskswhilealsoencouragingbestpractices.
dPrivacy:Thesystemshouldallowbothpublicandprivateconversationstotakeplacearoundtexts,inkeepingwithcommunitypreferencesandpolicies.
26openreview:AStudyofContextsandPracticeseClosure:Thesystemshouldpermitreviewperiodstohaveafixedtermwheredesired,aswellaspermittingopen-endedconversation.
fExport:Thesystemshouldallowreviews(includingaggregationsandfiltersfornuance,granularity,reputation,andsoforth)tobecollatedandremixedinwaysusefultoeditors,authors,andreaders.
Buildingaplatformfromthegrounduptosupportsuchawiderangeofauthor,editor,reviewer,andreaderneedswouldbeacomplexandcostlyprocess,andwouldruntheriskofproducingafragile,difficulttosupportpieceofsoftware.
Farmoreproductivewouldbetoworkwithexistingplatforms,toseewhatcanbedonewithcombinationsofexistingtools,andtoexplorehowongoingprojectdevelopersmightbeledtoincorporatetheneedsofopenreviewintotheirprojectroadmaps.
Whiletherearemanycontentmanagementsystemsontopofwhicharobustopenreviewprocessmightbedeveloped,forpurposesofexample,toconveyhowonesystemmightfunction,wewillfocusinthissectiononthepossibilitiespresentedbyWordPress,whichhasarobustpluginarchitecture,arelativeeaseofuse,andavibrant,organizeddevelopercommunity.
WordPress'scorearchitectureincludesanumberoffeaturesthatareusefultoformalpublications,includingtheabilitytosupportmultiplesiteswithinoneinstallation,built-inversioningthatincludestheeditorwhomadetherevisions,customizableuserroles,threadedcomments,andthelike.
AninstallationofWordPressmightbecombinedwithanumberofpluginstoservemanyoftheopenreviewfunctionsnamedabove;suchpluginsinclude:BuddyPress(http://www.
buddypress.
org):asocial-networkpluginthatfacili-tatestheformationofandcommunicationwithingroupsformedinalargernetwork.
CommentPress(http://www.
futureofthebook.
org/commentpress)andDigress.
it(http://digress.
it):facilitateparagraph-,page-,anddocument-levelcom-menting.
Annotum(http://annotum.
org/):aplugindesignedtoextendWordPressasaplatformforscientificpublishing,byaddingsupportformultipleauthors,cita-tions,versioning,andrevisioncomparison,amongotherfeatures.
EditFlow(http://editflow.
org/):aneditorialworkflowpluginfacilitatingedito-rialteamcollaboration,reviewtracking,reviewernotifications,andmore.
Ofcourseanumberofthefunctionsdesiredinanopenreviewplatformremainunservedbytheseplugins;whileCommentPressandDigress.
itbothprovidecommentingofagreatergranularitythandoesWordPressoutofthebox,eachisrestrictedtotheparagraphorpagelevel,andrevisionstoanunderlyingtextcan27openreview:AStudyofContextsandPracticesbreaktherelationshipsbetweenthattextanditscomments.
Similarly,whileBud-dyPressfacilitatesthecreationofcommunitymemberprofilesandcapturesamem-ber'sparticipationwithinanetwork,itdoesn'tcurrentlyprovideameanstorevieworratethatactivity,ortotransformthoseratingsintosomemeasureofreputation.
Needssuchasthesewillhavetobemetbyextendingthefeaturesofexistingpluginsorcreatingadditionalplugins.
Thecomplexityofsuchapluginarchitectureofferssomereasonforcaution;giventhemanydisperseddevelopersworkingindependentlyonthesecomponents,thereiscausetoexpectthatcertainpluginsmightnotinteroperatewellwithoth-ers,thatupdatestotheWordPresscoremaybreaksomepluginfunctions,andthatsomeprojectsmayceasetobeupdatedastheirdevelopersmoveontootherproj-ects.
However,workingtogether,aconsortiumofinterestedparties(whichmightincludepublishers,libraries,scholarlysocieties,individualscholars,andfunders),withanappropriateagreementonasetofcentralizedorstandardizedprocesses,couldassembleaneasilyinstallablepackageofpluginsandcommittothedevelop-mentandmaintenanceofthatpackageonbehalfofitsuserbase;theCUNYAca-demicCommons'sCommons-in-a-Boxprojectmightprovideamodel,andinfactsuchanopenreviewplatformprojectmightbeimaginedasaforkofthisproject.
45Thereare,ofcourse,othernotableplatformsthatmightbebuiltuponorlearnedfrom,includingothercontentmanagementsystemssuchasDrupalandJoomla,aswellasplatformsdesignedspecificallyforscholarlycommunication,suchasOpenJournalSystemsfromthePublicKnowledgeProject,orAmbra,theplatformonwhichthePLoSjournalsarepublished.
Anidealscenariowouldofcoursebethedevelopmentofanopenreviewarchitecturethatcouldbeimplementedinaplatform-agnosticfashion,allowingreviewercommentstobeaggregatedacrosstheweb,regardlessofthesystemhostingtheprimarytext.
(ServicessuchasDis-qusprovidesuchacomment-aggregationfunction,butsomeconcernsabouttheirbusinessmodelsandproprietarydatastructuresmakethemless-than-idealassolutions.
)Thecrucialelementinanycaseisflexibility;inordertomeettheverydifferentrequirementsthatdifferentcommunitieswillbringtoopenreview,andinordertoremainsustainableastechnologieschange,thesystemsthatsupportsuchpracticeswillrequiresignificantmalleability.
45SeeMatthewK.
Gold,"TheCUNYAcademicCommonsAnnouncestheCommonsinaBoxProject".
28openreview:AStudyofContextsandPracticesConclusionWhatyouhavejustreadisapreliminarysetofsuggestionsforwaysthatopenpeerreviewinthehumanitiesmightbeorganizedaroundaseriesofhumaninterac-tionsandtechnologicalspecificationsthatpromotebothflexibilityandrigorinthereviewprocess.
Thenextstepsaretotesttheserecommendations.
Throughoutthisdocument,wehavesuggestedthatnosinglesetoftoolsorrulescanworkforallopenreview,butratherthattheprocessmustbesubjecttostructuredflexibilitywherebyeachcommunityfirstestablishescriteriabasedontheneedsofparticularprojectsandthenselectsplatformfunctionalitiesthatallowthemtorealizethosecriteria.
Tofurtherthat,weproposethatvariouscommunitiesofpracticebegintoworkwithtechnologiststomapoutanarrayoftoolsortooloptionsthatmightbedeployed,adapted,orbuiltandthatcanprovideahealthyrangeofreviewsettingsandtemplates.
Bringingtogetherscholarlyanddevelopercommunitiestocustom-izeexistingtoolslikeDrupal,WordPressandOJSwouldprovidealternativesforopenreviewthatneitherrequireare-inventionoftheproverbialwheelnorforceacademicstosimplymakedowithwhatisalreadyoutthereregardlessofhowwellitmeetscommunityparameters.
Whiledifferenttoolswillbeneededandusedtomeetdifferentcommunityobjectives,ifaconsistentsetofquestionsisaskedabouteachreviewprocess,thenthetechnologicaldevelopmentswouldremainboundbyaflexible-butnotinfinite-rangeofoptions,ratherthanexistingasself-containedanddisparateexperiments.
Thoughmoreexperimentationisneededbeforeanyconclusionsmightbedrawnabouttheeffectivenessoftheflexibleparametersandtechnicalspecificationswe'veproposed,wenonethelessbelievethatopenreviewisessentialformodelingaconversational,collaborativediscoursethatnotonlyharkensbacktothehumani-ties'longinvestmentincriticaldialogueasessentialtointellectuallabor,butalsomodelsaforward-lookingapproachtoscholarlyproductioninanetworkedera.
Moreover,wefirmlyendorsethenotionthatopenreviewcanandshouldfacilitatethebestkindsofhumanitiesscholarshipbyvirtueofitsfocusontheprocessofscholarlyreviewasmuchasitsendproduct.
Indeed,intheSocratictradition,openreviewmodelsthedegreetowhichprocessisintegraltoproduct,demandingalevelofaccountabilityandreciprocitytypicallyobscuredbytraditionalreviewpractices.
Inopeningupnotonlythereviewprocessbutalsotheprocessesforcultivatingandrecognizing"peers,"openreviewalsoholdsthepotentialtoexpandknowledgenodeshorizontally(asopposedtotraditionalreview'spenchantforverticalpeer29openreview:AStudyofContextsandPracticesconferral),encouraginggreaterinterdisciplinarityandgreaterpublicengagement,which,again,arecentraltothevaluesofhumanities-basedscholarlypractice.
Whileideasaboutopenreviewaresteepedinahumanitiestradition,wecontendthatnewreviewpracticesmightalsocontributetoaredefinitionoftheroleofthehumanitiesscholarinthe21stcenturyaswellasofthenatureofscholarlypublish-ing.
Openreview'sembraceoftransparencyanddialoguecanpotentiallytransformscholarsfromknowledgepurveyorstofacilitators,byplacingequalimportanceonthe(normallyeitherinvisibleordevalued)"scribblesinthemargins"–thediscus-sionsaroundaparticularwork.
Inturn,openreviewcanhelpvalidatenewtypesofscholarlyoutput,asannotations,commentsandothershort-formpostsbecomebothcitedandcitable,whilewrittenworksgothroughmultiplepre-andpost-pub-licationiterationswithversionsreplacingnumberededitions.
Finally,openreviewcanhelpre-conceptualizethewaysthatscholarsaretrainedandmentored,widen-ingthecommunitythatjuniorscholarscanlearnfromwhilepromotinganethosofopennessandtransparencywhenitcomestoparticipatingincollaborativeformsofintellectuallabor.
Therippleeffectsofsuchtransformationswouldbefeltwellbeyondthereviewprocess,possiblycontributingtoarethinkingofotheracademicgoldstandardsandritesofpassage,fromhowgraduateseminarsengagewithscholarlyworkstohowdissertationsarewrittenanddefended,fromthematerialsevaluatedbypromotionandtenurecommitteestotheveryprocessofevaluatingtenurefiles.
Beforesuchtransformationscanoccur,however,wemustfirstfindawaytostrengthentheopenreviewprocessesforhumanitiesscholarship.
Webelievethattheparameterslaidoutinthisdocumentareimportantfirststepstowardachievingthatgoal.
30openreview:AStudyofContextsandPracticesappendix1OpenPeerReviewsoftware:areportbyDanVisel22January2012.
(modified6June2012)0.
Overview:Whatthisreportcovers1.
Theproblemtobesolved1.
1.
Dealingwithusers1.
2.
Dealingwithcontent:basics1.
3.
Creatinganeditorialsystem1.
4.
Dealingwithintellectualproperty2.
Modelsforwhatcouldbedone2.
1.
Moderation:usefulmodels2.
2.
Sitesbasedaroundcommentedreadingorpresentation2.
3.
Sitesbasedaroundcollaborativewriting2.
4.
Othersites2.
5.
Prioronlineexperimentswithpeerreview3.
Potentialpiecesofanimplementation3.
1.
Commentingplugins3.
2.
Revisioning&documentmanagementplugins3.
3.
Dedicatedjournalcontentmanagementsystems3.
4.
Otherframeworks&ideasthatcouldbeuseful4.
Suggestionsforasolution4.
1.
Buildingfromscratch4.
2.
Buildingfromlittlepieces4.
3.
Buildingfrombigpieces4.
4.
What'snotinsoftware31openreview:AStudyofContextsandPractices0.
Overview:whatthisreportcoversApplicationURLCurrentstatusTypeofsystemPrimaryvalueSlashdothttp://slashdot.
comPubliclyavailableCommercialinstal-lationofopensourceCMSModelformodera-tionDigghttp://digg.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionReddithttp://reddit.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionDisqushttp://disqus.
comPubliclyavailableCommentingpluginfordifferentCMSesModelformodera-tionHackerNewshttp://news.
ycombi-nator.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionMetafilterhttp://metafilter.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionYahoo!
Answershttp://answers.
yahoo.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionStackOverflowhttp://stackoverflow.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionQuorahttp://quora.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionFacebookhttp://facebook.
comPubliclyavailableCommercialinstal-lationofproprietaryCMSModelformodera-tionBookGluttonhttp://bookglutton.
comPubliclyavailableWebapplicationforannotatingdocu-mentsModelforcom-mentedreadingNowCommenthttp://nowcom-ment.
comPubliclyavailableWebapplicationforannotatingdocu-mentsModelforcom-mentedreadingElihttp://www.
elire-view.
comPubliclyavailableWebapplicationforannotatingdocu-mentsModelforcom-mentedreadingHighlighterhttp://www.
high-lighter.
comPubliclyavailableWeb/mobileappli-cationforannotat-ingdocumentsModelforcom-mentedreading32openreview:AStudyofContextsandPracticesSocialBookhttp://livemargin.
comPrivatebetaCommercialpub-lishingandannota-tionplatformModelforcom-mentedreadingGoogleDocshttp://docs.
google.
comPubliclyavailableWebapplicationsforeditingdocumentsModelforcollab-orativewritingApacheWavehttp://incubator.
apache.
org/wave/IndevelopmentDocument-baseddiscussionenviron-mentModelforcollab-orativewritingScholarPediahttp://scholarpedia.
orgPubliclyavailableWikibasedaroundscholarshipModelforcollab-orativewritingAcademia.
eduhttp://www.
aca-demia.
eduPubliclyavailableDocument-sharingsocialnetworkModelfordocu-mentsharingScribdhttp://scribd.
comPubliclyavailableDocument-sharingsocialnetworkModelfordocu-mentsharingO'ReillyRoughCutshttp://PubliclyavailableOnlinepublishingplatformModelfordocu-mentsharingPeerEmedhttp://peeremed.
comPubliclyavailablePeer-reviewedjournalExampleofonlinepeerreviewPhilicahttp://philica.
comPubliclyavailablePeer-reviewedjournalExampleofonlinepeerreviewCommentPresshttp://www.
future-ofthebook.
org/com-mentpressPubliclyavailableWordpresstheme/pluginPluginthatcouldbeusefulDigress.
ithttp://digress.
itPubliclyavailableWordpresspluginPluginthatcouldbeusefulDocumentRevi-sionshttp://wordpress.
org/extend/plugins/wp-document-revisions/PubliclyavailableWordpresspluginPluginthatcouldbeusefulAnnotumhttp://annotum.
orgPubliclyavailableWordpressthemePluginthatcouldbeusefulOpenJournalSys-tems(OJS)http://pkp.
sfu.
ca/q=ojsPubliclyavailableCMSforjournalsDedicatedCMSforjournalsOpenAnnotationCollaborationhttp://www.
openan-notation.
orgPubliclyavailableFrameworkforsharingannotationUsefulframeworks&ideasHypothes.
ishttp://hypothes.
isAnnouncedDistributedopensourceannotationlayerforwebUsefulframeworks&ideasPressForwardhttp://digitalhu-manitiesnow.
org/PubliclyavailableInformationsourceUsefulframeworks&ideasPLoSarticle-levelmetricshttp://blogs.
plos.
org/plos/2009/09/article-level-met-rics-at-plos-addi-tion-of-usage-data/PubliclyavailableFeatureofpropri-etaryCMSUsefulframeworks&ideas33openreview:AStudyofContextsandPractices1.
TheproblemtobesolvedThisreportislookingatcurrent(andnearfuture)toolsandplatformsthatcouldbeusedforonlinepeerreviewwiththelong-rangegoalofmakingasustainableopensourcesystem.
Thereareanumberofcriteriaandquestionsthatneedtobeaddressedoransweredbyanysolution.
1.
1.
DealingwithusersThesequestionsarecommontoanysocialnetwork.
Theexamplesofthelinkag-gregatorsdescribedinsection2.
1mightprovideusefulexamplesforthedesignofthissystem.
Asuccessfulsystemmustallowanonymity(orpseudonymity)aswellasself-identification.
Whilearealnamepolicymightbeused,therearecaseswhereanonymityisrequired.
Thereshouldbeawaytoseeinasinglepagewhatauserhascontributedacrossthesite.
Thequalityofreviewersmustbeassessed.
Thiscouldbetiedtothequalityoftheircontributions;itmightalsobeaseparatefield.
Asuccessfulsystemmustadequatelymoderatenewcontent,preventingtrollsandspam.
Itshouldbepossibletodownvoteunhelpfulcommentssothattheybecomeinvisible.
Reviewingneedstobelinear,notbinary:whileareviewershouldbeabletosaythatsomethingisgoodorbad,morenuanceisalsoneeded.
Howdoesthesystemmanagescore-settling,ortheproblemofthepersonwhoshoutsloudestgettingthemostattention1.
2.
Dealingwithcontent:basicsThesequestionsarefocusedonthecontentthat'sinthesystem:the"texts"tobere-viewedandthereviewsthemselves.
Section3ofthisreportdealswithsoftwarethatcouldbeusefulinmanagingcontent.
Doesthesystemenableembedding/uploadingvideooraudioreviewsDoesthesystemallowforcommentingatthepage/paragraph/sentence/wordlevelonaudio,imageorvideofilesVersioningneedstobeaccommodatedandeasilyaccessible:multipleeditionsoftexts(andcomments)needtobeaccountedfor.
Thesystemshouldallowreadersandreviewerstocomparedraftsbeforeandafterreview.
Whatkindsofcitationpracticesorothermeansoftracingthelineageofideas34openreview:AStudyofContextsandPracticesdoesthesystemallowforAretheremeansavailableofcitingorlinkingtoindividualcommentsCanfiltersbecustomizedbyauthor/editorCanthesystemfilterreviewsbycategories(i.
e.
,distinguishbetweenreviewsdealingwithdifferentfacetsoftheproject(argument,evidence,writingstyle,etc.
)ordifferentlevelsofrevisionsuggestions(recommendedversusrequiredrevisions,etc.
)1.
3.
CreatinganeditorialsystemThequestionsinthissectionarelesstechnicalandmoredependentuponhowthesystemisimplementedforeaseofuse.
AbasiceditorialworkflowwillneedtobeimaginedtoconstructasystemWhatkindsofpromptsareavailabletoguidereviewersWhatoptionsdoesthesystemprovideforbothpublicandprivatediscussionDoesthesystemoffermechanismsforalertingmemberstomaterialsinneedofreviewCansuchmechanismsbecustomizedbasedontagging,keywords,orauthor/editor"invitation"Howarequestionsof"finality"addressedwithinthesystem:doesarevieweverend1.
4.
DealingwithintellectualpropertyFinally,somequestionsaboutintellectualpropertyneedtobespelledoutaspartoftheeditorialprocess.
WhataretheunderstandingsunderwhichuserscontributetothesiteHowarethoseunderstandingsspelledoutinthetermsofuseWho"owns"thecommentsmadewithinthesystemWhattermsofuseorotherprovisionsaremadeforintellectualpropertyandthereuseofmaterialproducedwithinthesystem2.
ModelsforwhatcouldbedoneAnumberofmodelsforwhatcouldbedonewithonlinepeerreviewalreadyexist.
Whilethemajorityofwhat'sdescribedinthissectionareproprietary(andthusnotsuitable)solutions,theseexamplesareworthlearningfrom.
2.
1.
Moderation:usefulmodelsAnumberofproposedsystemsaren'tdirectlyusefulasasolutiontotheproblem,butmayprovideusefulmodelsforhowcommentingand/ormoderationcouldwork.
Theseinclude,inorderofdiscussion:35openreview:AStudyofContextsandPracticesSlashdot(http://slashdot.
org;theCMShasbeenreleasedasanopensourceproj-ectcalledSlash(http://www.
slashcode.
com).
Digg(http://digg.
com)Reddit(http://reddit.
com)HackerNews(http://news.
ycombinator.
com)Disqus(http://disqus.
com)Metafilter(http://www.
metafilter.
com)AskYahoo!
(http://ask.
yahoo.
com)StackOverflow(http://stackoverflow.
com)Quora(http://quora.
com)Facebook(http://facebook.
com)Slashdotistheoldestofthesesites,havingbeenaroundsince1997;itpioneeredmanyofthebasicideasusedincommentingandmoderationontheweb.
Slashdotisessentiallyablogthatpresentstechnews;mostpostsareapproximatelyapara-graphlongandbasedaroundalink.
Tofullyusethesite,usersmusthaveanac-count.
Postsareuser-submitted,thougheditorschoosewhichpostsappearonthesite.
Slashdot'smoderationhaschangedanenormousamountovertime(largelyonaccountofincreasedvolumeofusers),butitbecamenotableforasystemwhichalloweduserstoup-anddown-votearticlesandcomments.
Thiswasoriginallydesignedtohighlightthemostinterestingcomments;asvolumegrew,thetaskofmoderationincreasinglybecameoneofhidingspamandtrolls.
Eachcommenthasanumericalscorefrom–1to5;commentsstartoutat1,thoughifotherusersmarkthemasinsightful,informative,orfunny,theyreceivemorepointsandifusersmarkthemasunhelpfulorjunktheylosepoints.
Higher-scoredcommentsareshownbydefault;lower-scoredcommentsarehidden.
Inaddition,thescoresofcommentsaretiedtothetrustmetricofusers,calledkarma(thetermoriginatedhereandhasbeenpickedupbyotherplatforms):ifyousubmitcommentsorpoststhatothersjudgetobegood,yourkarmascoregoesup.
Postsorcommentsbyuserswithhigherkarmastartwithhigherscoresthanpostsorcommentsbyuserswithlowkarma.
WhileSlashdothasdeclinedinpopularityovertime,inpartbecauseofaper-ceiveddropinquality(the"EternalSeptember"problem),thecentralideaisinusebymostblog-likesiteswithlargereaderships.
Diggmightbeseenasastripped-downiterationoftheSlashdotconceptthatarrivedin2004:userscouldsubmitlinks,commentonthem,andvotethelinksorcommentsupordown.
There'snoeditoriallayer.
Untilaredesignin2010(basedonaperceiveddrop-offinquality),karmawasextremelyimportant,whichledtotheemergenceofaclassofpoweruserswhocouldessentiallycontrolwhatappearedonthesitewiththeirupand36openreview:AStudyofContextsandPracticesdownvotes.
Reddit,foundedin2005,presentedamorerefinedversionofDigg'sapproach:there'smoreofanemphasisonsub-communitiesandcomments.
UserkarmaseemstomatterlessthanitdoesonDigg;therearealsomoreclearlydefinedculturalnormsforwhatmakesagood(asopposedtoasimplyfunny)comment.
Atthemoment,thesiteismoresuccessful;agooddealofanti-SOPAactivism,forexample,camefromtheRedditcommunities.
HackerNews,startedin2007,worksthesameterritorythatSlashdotdid;however,theinterfaceismuchmorestrippeddown.
Postscanonlybevotedup;ifusershaveenoughkarma,theycanvotecom-mentsdown,butthesitegenerallyworksonpromotion.
HackerNewsisgenerallysuccessfulatfosteringsmartconversations,althoughmuchofthismaybeduetoitscomparativelysmalluserbase.
Disqusisapluginformostcontentmanagementsystemsthatallowscommentingusingrelativelysimilarfeaturestothesesites.
Ingeneral,Disqus'sfeaturesarebetterthanthecommentingsystemsthatcomewithmostCMSesoutofthebox;theyal-lowuserstobesignedintoasinglecommentingsystemacrossmultiplesites,whichtendstoencouragemorecommentingbecausetherearefewerbarrierstoentry.
Karmaoperatesbothinsideasitethat'susingDisqusandacrossallsitesthatthecommenteriscommentingon,reducingspam.
AdifferentapproachtotheproblemoffosteringsmartconversationsisbetheroutetakenbyMetafilter,whichpresentslinksandcomments,likeSlashdot,Digg,Reddit,andHackerNews.
Foundedin1999,ittakesadifferentapproachtotheproblemofdiscoursequality:from2004,newusershavebeenchargedaone-time$5registrationfee.
Thisweededoutagreatdealofthetrollingthatothersiteshavetodealwith;becausetheusersareself-selected,there'slessproblemwithtrollingandpoor-qualitycommenting.
Socialnormsandpeerpressuretendtokeepthingscleanhere(asisthecasewithHackerNews),thoughtherearehumanmoderatorsaswell.
Metafilternotablydiffersfromtheabovesitesinthatitusesnokarmasys-tem.
Becauseoftheregistrationfee,however,it'soperatingatamuchlowervolume;aswithHackerNews,thismightmakeitamoreusefulexample.
Allofthesesites,ofcourse,requireanaccount;purelyanonymouscommentsaren'tallowed,althoughthere'snothingtostoppeoplefromcreatinganonymousac-counts.
Asmalljumpoverfromthelinks-and-commentssitesarequestion-and-answersites.
(Anumberoflinkssitesalsohostquestionsinvariousforms–AskSlashDot,AskMetaFilter,etc.
)AbasictemplateofthissortofsitemightbeseenatYahoo!
Answers,startedin2005,whichallowsuserstoaskandanswerquestions;however,37openreview:AStudyofContextsandPracticesthere'snometricforauthority,andthecontentisn'tespeciallygood.
StackOverflowbeganasamorespecializedsite,basedaroundcomputerprogrammingquestions,in2007;sincethen,it'sgrownandbeguntocoverotherareas.
StackOverflowusesareputationscore:agoodanswertoaquestion(orotheractionsthataffectthecom-munityandcontentpositively)resultsinmorereputationpoints.
Thishasgenerallybeensuccessful.
Quoraisamorerecentquestion-and-answersite;it'ssimilartoStackOverflow,butemphasizesreal-worldnamesandqualificationsasexperts(tak-ingadvantageofusers'socialnetworksfromFacebookorTwitter).
It'stoosoontosaywhetherthisreputationmodelwillbesuccessfuloutsidethetechnology/SiliconValleyworldthatiscurrentlyitsprimaryfocus;withinthatlimitedcommunity,itseemstowork.
Finally,it'shardtoavoidFacebook,whichhasliberallyborrowedfromallofthesesitesandintroducedsomeofitsowninnovationsinmoderation.
UsersonFace-bookcanchoosewhooftheirfriendstheywanttoseemoreorlessof;Facebookisinternallyusingitsownkarmasystemwhichdetermineshowimportantparticularusers'updatesare,thoughthisisalmostentirelyopaquetotheenduser.
Commentsonpostsdon'tcurrentlyreceiveanyinternalmoderation,thoughthisseemslikeitmaychangesothatthemostlikedorsharedcommentsareshownmostpromi-nently.
What'sworthtakingawayfromtheseexamplesAcoupleofkeypointsmightbeusefullyremembered:Volume.
Theproblemsofmoderationaredirectlyrelatedtotheproblemofsitevolume:asitewithamillionusershasverydifferentproblemsfromasitewithathousandusers.
WhatworksonFacebookisn'tlikelytoworkonasitedesignedfor10,000users.
Smallercommunitiestendtohavebetterconversa-tionsbecauseuserstendtoknoweachother;thisdeclinesasthesizeofthecommunityincreases.
Reputation.
Theproblemofreputationisonethatneedstobedealtwith:notallcommentersareequal.
Karmamightbepartofasolutiontothis,thoughittendstoleadtousersgamingthesystem;intheacademicworld(wherethenumberofpotentialcommentersisfinite),thereareprobablymoreusefulsolutions.
Identity.
Qualityinmoderationandcommunityappearstobedirectlycor-relatedtoconsistentuseofuseraccounts,thoughthosemaybecomparativelyanonymous.
Recentlyamovehasbeenmadetocorrelatequalityofcommu-38openreview:AStudyofContextsandPracticesnitytolackofanonymity,butthiscorrelationismoreambiguous.
2.
2.
SitesbasedaroundcommentedreadingorpresentationBookglutton(http://www.
bookglutton.
com)NowComment(http://nowcomment.
com)Eli(http://www.
elireview.
com)Highlighter(http://www.
highlighter.
com)SocialBook(http://livemargin.
com)Anumberofsiteshavebeenconstructedaroundtheideaofsocialreading,gen-erallywithouttheconceptofpeerreview.
Digress.
it,discussedinsection3.
1,isprimarilyaWordPressplugin,butalsofunctionsasaplatformforthedistributionofcommentedtexts;itcouldbeseenasbeingpartofthisgroup.
Bookgluttonwasoneofthefirstofthededicatedsitesoutofthegate,in2008;itallowsuserstouploaddocumentsandtocommentonthemonagranularlevel.
Asocialnetworkallowsforgroupstobeconstructedaroundtexts;integrationwithFacebookisalsopresent.
Bookscanbeembeddedinothersites(likeblogs)aswid-gets.
RudimentarypeerreviewmightbepossiblewithBookgluttonbyconstruct-ingagroupforreviewersandgivingoutanonymousaccountsasnecessary;nooneseemstohavedonethisyet.
Bookgluttononlyworkswithsimpletextfiles,whichmightruleoutacademicdocuments;thisisalsoadownsidetoNowComment,averysimilarproject.
Now-CommentseemstohavemoreeducationalusersthanBookGlutton;BookGlutton'sinterfaceismuchbetterthanNowComment's.
Bothareself-containedsites,whichcomplicatesthepossibilityofextendingthem.
WhilethesesitesfunctionsimilarlytopluginslikeCommentPress,themodularnatureofpluginsmeansthattheycanbeusedinradicallydifferentenvironments;usersofBookGluttonorNowCom-mentneedtousethesite.
Eliworksverysimilarly;theplatformistightlytiedtotheideaofteachingwrit-ingandteachersgivingwritingfeedbacktostudents.
Highlightertakesthesameideaandmovesittoapredominantlymobileplatform;there'ssomethingmoreofafocusonanalytics,sothatpublisherscanseewhatreadersaremostinterestedin.
Highlighterisalsointerestinginthattheyprovidecodethatcanbeaddedtoanywebsiteaddingasocialannotationlayer.
Neitheroftheseisanespeciallygoodfitforpeerreview,thoughHighlighterseemstohavemorefundingthantheother39openreview:AStudyofContextsandPracticessitesandabitmoredrive;theymightbeworthkeepinganeyeon.
(Worthnoting:they'vehiredpeoplefromtheSakaiproject.
)SocialBookisBobStein'snewplatformforsocialreading,whichhasn'tyetbeenreleased;it'sapublisher-centricmodel,allowingthesaleofgroupaccesstotexts.
Thefocusisstronglyonhowdifferentgroupscanusethesametext;itseemspos-siblethatwithsomeeditorialstructures,onlinepeerreviewcouldbedoneusingthesoftware.
Thisisnot,however,somethingthatitisdesignedforoutofthebox.
2.
3.
SitesbasedaroundcollaborativewritingGoogleDocs/Drive(http://docs.
google.
com)Google/ApacheWave(http://incubator.
apache.
org/wave/)variouswikis,includingScholarpedia(http://www.
scholarpedia.
org)Collaborativewritingenvironmentsmightpresentapathforward;it'spossibletoimagine,forexample,usingGoogleDocsforrudimentarypeerreview.
Adocu-mentcouldbeforwardedtoareviewerwhoaddscommentstoit;thesecomebacktoaneditor,whocouldforwardthem.
Oneproblemisthelackofawaytoano-nymizecomments;andwhileGoogleDocsdoestrackchanges,theuserinterfaceforthisistoopoortoconsiderasaserioussolution.
(It'salsodifficulttotellwhatGoogle'slongtermplanisforGoogleDocs,whichtheyseemtobede-prioritizing.
)GoogleWavepresentedwhatmighthavebeenasolution:asinitiallypresented,itwasanenvironmentforconversationsarounddocuments.
Whilefullofpromise,itwasconfusing,userswereslowtopickitup,andGooglekilledtheproject;it'sbeenpickedupbytheApacheSoftwareFoundationandrebrandedApacheWave.
Sched-uledforreleasein2011,itisstillnotout.
Itseemspossiblethatthiscouldworkasanenvironmentforonlinepeerreview,butthiscannotbedetermineduntilrelease.
Google/ApacheWaveisreminiscentofwikisinthatitallowsfortheeditingofdocumentsbymanyusersovertime.
Whiletheuseofwikisinpeerreviewhasbeentried–Scholarpedia,basedonMediaWiki,isanexample–theformishardtouseandrequiresagreatdealofeditorialself-disciplinetobeuseful.
ICE,aWordPressplugin,makescollaborationpossibleineditingWordPresscon-tent;itisdiscussedbelow.
2.
4.
OthersitesAcademia.
edu(http://www.
academia.
edu)Scribd(http://scribd.
com)40openreview:AStudyofContextsandPracticesCurēus(http://www.
cureus.
com/,formerlyPeerEmed)O'ReillyRoughCuts(http://my.
safaribooksonline.
com/roughcuts)Philica(http://philica.
com)Academia.
eduisasocialnetworkfoundedin2008basedaroundtheunitoftheacademicpaper.
Userscancreateaccountsandfolloweachotherlikeanysocialnetwork;theycanalsouploadpapers.
Despitethedomainname,thisisafor-profitcorporation.
Therearecurrentlynofacilitiesfordetailedcommentingondocu-ments;thisprobablyisn'tthatuseful.
Scribdisimmenselypopular;thesiteallowsuserstouploadtextinavarietyofformats(Word,PDF,EPUB)tobeviewedonline.
Whileit'snotinconceivablethatScribdcouldmoveinthedirectionofcollaborativereading(byallowingtargetedcomments,forexample),thishasn'thappenedsofar.
Userscancommentondocu-ments,butthere'sonlyasinglecommentstreamfordocuments.
Scribdisnotableinitsintegrationintoothersocialnetworks(especiallyFacebook)anditsattemptstobuildcommunitiesaroundtexts,butthishasn'tgottenfar.
ItisworthtakingalookatPeerEmed,anattempttomakeanonlinepeerreviewsystemformedicalpapersbasedontheScribdengine.
Thisdoesn'tseemtohavemuchtractiondespitebeingstartedin2010;peerreviewislimitedtoasinglestreamofcommentsandasinglenumericscore.
Themostinterestingelementofthisistheeditoriallayer,whichdoesn'tseemtremendouslyengaged;itdoespointtotheideathatasimplepeerreviewsystemcouldbesetupusingablogandem-beddingdocumentsfromsomewherelikeScribd.
(PeerEmediscurrentlyinthemidstofrebrandingitselfasCurēus;it'sdifficulttotellwherethey'llendup.
)O'Reilly'sRoughCutsisanonlinebookreadingenvironmentfromO'Reilly,thetechnicalpublisher;itisdesignedtogetbooksonnewordevelopingsubjectstoreadersbeforetheyhavebeenthoroughedited.
Readers–whopaytousetheser-vice–areinvitedtoleavecommentsonwhatworksanddoesn'twork.
Comment-ingisgranular,soreaderscanhighlightpassagesforcommenting.
Onthewhole,however,thisdoesn'tseemtohavetakenoffonline,eventhoughthisisaimedatahighlytechnicalaudience;mostbookshavefewifanycomments.
Thesamebooksareavailableinlessinteractiveformats(EPUB,PDF),whichmaybecompromisingtheonlineenvironment.
Philicaisasomewhatquixoticattempttomakeanonlinepeerreviewedjournal;articlesareacceptedonanytopic.
Qualifiedexpertsareallowedtowritereviewsforsubmittedarticles.
Commentsaren'tgranular.
It'sasomewhatinterestingimple-41openreview:AStudyofContextsandPracticesmentation–theyhavespecificallytriedtoreimaginepeerreviewfortheonlineworld–butit'shardtoimaginethisbeingmoregenerallyusedorgainingmuchtraction.
2.
5.
PrioronlineexperimentswithpeerreviewPlentyoftheseexist.
Adetailedlistisoutsideofthescopeofthisreport,butimpor-tantforanyconsiderationofthesubject.
3.
PotentialpiecesofanimplementationNothingofftheshelfisgoingtoprovideaperfectsolutionfortheproblemathand.
Butsomeexistingcontentmanagementsystems&plug-inscouldconceivablybepartofasolution.
BothDrupalandWordPressarerobustenoughthattheycouldperhapsbeusedasabasisforthisproject.
Bothareopensourceandhavefairlyrobustcommunities;ingeneral,WordPressiseasiertouse,whileDrupal'sarchi-tectureismorepowerful.
It'salsoworthkeepinginmindOpenJournalSystems,aCMSbasedaroundjournalpublication.
ForbothWordPressandDrupal,anysolutionisgoingtoinvolveplugins(Word-press)ormodules(thewordDrupalusesforthesamething),chunksofcodethatenablenewfunctionality.
Pluginsareavailabletodoalmostanythingconceivableforthisproject.
However,pluginscanalsocomplicatethedevelopmentprocess:generallythey'remadebydifferentprogrammersandmaynotbeupdatedasfrequentlyasthemainCMSis.
Gettingpluginstoworktogethercanalsopres-entproblems:addinganewplugincancausenewproblemswhichwillneedtobedebugged.
Foranyoneofthesesystems,useraccountsaregoingtobeimportant:theCMSneedstofunctionlikeasocialnetwork,sothatindividualsusingthesitehaveaneasilyaccessiblehistoryoftheircontributions.
Drupalhandlesthisoutofthebox;WordPresscanbeturnedintomoresocialsoftwareusingBuddyPress,apopularplugin.
3.
1.
CommentingpluginsCommentPress(http://www.
futureofthebook.
org/commentpress)Digress.
it(http://digress.
it)CommentPressisaWordPressplugindevelopedbytheInstitutefortheFutureoftheBook;Digress.
itisaforkedversionoftheprojectbyoriginaldeveloperEddieTejedathatworkssimilarly.
Bothallowparagraph-by-paragraphlevelcommentingontextinWordPress.
Digress.
italsooffersahostingplatform,sothatpotentialus-42openreview:AStudyofContextsandPracticesersdon'tneedtheirowninstallationofWordPress;CommentPressisapurepluginthattheusermustinstallonaWordPressinstallation.
Bothhavebeenusedonlineextensively(Cornell'sRegulationRoomprojectisprobablythemostseriousinstallofDigress.
it);thedifferencesbetweenthetwopiecesofsoftwarearen'textremelyserious.
Eitherofthesepluginscouldfunctionaspartofapeerreviewservice.
Onedraw-backisthebackendoftheseprojects:gettingtextintothemrequirestheeditortounderstandWordPressmanagement.
Abetterbackendwouldbeausefulavenueforfurtherdevelopment.
Versioningisalsoaproblem:ifacommentsuggestsachangeinthetext,theeditorchangesthetext;there'snowayofmarkingthesug-gestedchangeashavingbeendone(orpointingbacktotheversionofthetextthatthecommentreferredto).
Inaddition,it'sdifficultforreviewerstoeditcommentsthatthey'vemade.
Theseproblemscanbesolved,possiblybyusingotherplugins.
3.
2.
Revisioning&documentmanagementpluginsDocumentRevisions(http://wordpress.
org/extend/plugins/wp-document-revisions/)EditFlow(http://wordpress.
org/extend/plugins/edit-flow/)Annotum(http://annotum.
wordpress.
com)ICE(http://nytimes.
github.
com/ice/demo/)DocumentRevisionsisaWordPressplugindesignedtoallowmultiplerevisionsoffilesbeingusedinWordPress.
Thisisnot,however,designedtobeusedwithtextentereddirectlyinWordPress.
Rather,itwillworkwithdocuments(likePDFsorWorddocuments)thatarebeingmanagedbyWordPress.
ThiscouldbeusedaspartofarudimentarypeerreviewsystemifthedocumentsbeingusedwereinWordformat;DocumentRevisionscanalsobeusedinconjunctionwithEditFlow,whichisdesignedforcollaborativeeditingandmanagementofaWordPresssite.
EditFlowallowsinternaleditorialcommentsondifferentversionsofdocuments;itbringsfullercontroloverversioningtoWordPress.
It'spossibletoimaginethesetwopluginsbeingusedtogethertomakeapeerreviewsystem,althoughtheeditorswouldbeworkingonthebackendofthesystem,notpublicly,whichseemslikeadrawback.
Thesepluginsaredesignedforwebmagazines,notreallyforpeerreviewonjournals;however,theycouldprobablybeputtouse.
AnnotumisanewWordPresstheme(whichincludesavarietyofplugins)designedtofunctionasanopen-accessscholarlypublishingplatform,comingfromthescienceworld.
It'stooearlytosaymuchabouthowwellAnnotumwillwork,asitdoesn'tseemtohavemuchactualuse;however,itisdesignedtosupportanarticle43openreview:AStudyofContextsandPracticesreviewworkflowandversioncomparison,aswellasstructureddocumentsandsmartcitations.
Thisisaprojectthatshouldbekeptinmind.
ICEisanewWordPressplugindevelopedbyTheNewYorkTimesthatenablescollaborativeeditingofpostsakintoTrackChangesinMicrosoftWord.
Thiscouldprovidebasicversioningforediting.
3.
3.
DedicatedjournalcontentmanagementsystemsOpenJournalSystem(http://pkp.
sfu.
ca/q=ojs)OpenJournalSystemisaCMSforonlinejournals,releasedbythePublicKnowl-edgeProject,aconsortiumoftheUniversityofBritishColumbia,SimonFraserUniversity,andStanford,underthedirectionofJohnWillinsky.
Thesoftwareisopensourceanddedicatedtoopenaccess;over5000journalsuseOJS.
Avarietyofmodulesexist,amongthemonededicatedtopublishingmonographs(OpenMonographPress).
OJSdoessupportonlinepeerreview,thoughnotinanespeciallyautomatedfash-ion.
ThesubmittersendsaWordfileorPDFtotheeditor;forablindpeerreview,theeditorremovesthesubmitter'snameandsendsit(throughOJS)tothereviewer.
ThereviewereditsthedocumentinWordandsendsitback;ifapproved,finalchangesaremadeinWordandthepieceispublishedthroughOJS.
OnestrengthofOJSisthatit'sessentiallyaframeworkandiscontentagnostic.
OJScanbeusedwithsomethinglikeCommentPressasadisplayfrontend;it'scon-ceivablethatamoduleformoreinteractivepeerreviewcouldbebuiltontheOJSformat.
AdistinctadvantageofOJSishowwidelyit'sused.
TheredoesnotappeartobeahugenumberofOJSdevelopers(asthereareforWordpressorDrupal);however,thePublicKnowledgeProjectwilldoworkforhire.
It'sworthkeepingOpenJour-nalSysteminmind.
AcurrentweaknessofOJSisthatitisnotparticularlysocial:useraccountsdon'tformamajorpartoftheinterface,althoughitispossiblethatmoredevelopment(orintegrationwithanotherCMS)mightsolvethisproblem.
3.
4.
Otherframeworks&ideasthatcouldbeusefulOpenAnnotationCollaborationhttp://www.
openannotation.
orgHypothes.
ishttp://hypothes.
is44openreview:AStudyofContextsandPracticesPressForwardhttp://pressforward.
orgPLoSarticle-levelmetricshttp://blogs.
plos.
org/plos/2009/09/article-level-metrics-at-plos-addition-of-usage-data/DocumentCloudhttp://www.
documentcloud.
org/;exampleuse:http://www.
propubli-ca.
org/article/why-cant-linda-carswell-get-her-husbands-heart-backOpenAnnotationCollaborationisdevelopingaframeworkforsharingannota-tionsacrosstheweb.
Whilethisisn'tspecificallyfocusedontheproblemofpeerreview–andisgenerallyoperatingatafairlydeeplevel–developmentofasystemforopenpeerreviewshouldtakethissystemintoaccount.
Hypothes.
isisanewandrelativelyprominentattemptataddingareputationlayertotheweb,similartotheoldsemanticwebidea.
Itisveryhardtotellifthiswillgetanywhere;historically,thissortofideafailsquicklyandignominiously,thoughtheslateofpeopleinvolvedisimpressive.
Theprojectessentiallyseekstobringopenpeerreviewtotheentireweb;expertiseandreputationfactorintoit.
Itmightbeworthmeetingtoseeifthere'sanintersectionwiththeacademicproblemofpeerreview.
PressForward'scriticalworkonthepeerreviewprocess(includingsiteslikeDigitalHumanitiesNow)shouldcertainlybetakenintoaccount.
PLoSprovidesarticle-levelmetricsforeveryarticlepublishedintheirjournal;thisisanexamplethatshouldbetakenintoaccount.
Thesemetricsallowreaderstoseewho'scitedthearticleandwherementionshavebeenmade.
Thisisworthdoing;ifanewsystemisbeingbuilt,buildingthisinwouldnotbeespeciallytricky.
DocumentCloudisanewprojectforjournalistsbasedaroundtheideaofmakingprimarysourcesmoreaccessible.
AsdemonstratedatProPublica,itenablesdy-namiclinkingtothedocumentsthatweretheoriginalsourcesforinformationusedinnewsreports.
It'simaginablethatasimilarsystem,putintousetheacademicworld,mightfunctionasawayofmakingcitationsmoreimmediate.
Whilesuchasystemmightbedesirable,itisprobablyoutsidethescopeofthisproject.
4.
SuggestionsforasolutionNoperfectsolutioncurrentlyexistsforcomprehensivepeertopeerreview.
How-ever,thenumberoftoolsandplatformsthatcouldbeusedformorelimitedimple-mentationsisgrowingfairlyrapidly.
Anincreasingnumberofthesetoolsand45openreview:AStudyofContextsandPracticessoftwarearefor-profit,limitingtheirutilityintheacademy;andnonebringtobeareverythingthatwouldbeneeded.
Ahandfulofdifferentstrategiesmightbeemployed.
4.
1.
BuildingfromscratchItwouldbepossibletobuildabespokeplatformforonlinepeerreview.
Thisisn'tquitetheimmensejobitwouldhavebeenafewyearsago;librariesexistformostofthemajorfunctionalitythatwouldbeneeded(asocialnetworkcomponent,com-menting,versioning,moderation).
Thiswouldprovidethebenefitthatallofthecodecouldbemanagedbyasingleteam,makingchangesrelativelysimple.
Thedownside,however,iswhenthesoftwarerequirementschange:theprogram-merswhodevelopedthesystemwouldbetheonesmostcompetenttomakechang-es,whichmightleadtobottlenecks.
Developmenttimeingeneralwouldbelikelytotakemuchlongerthananyotherstrategy.
Anumberofcomponentsofthisproject–forexample,anautomatedsystemthatwillissueinvitationstoreviewerswhenapieceisreadytobereviewed,whichthencreatessemi-anonymousaccountsforthem–willcertainlyneedtobebuiltfromscratch,justbecausenobodyiscurrentlydoinganythinglikethis.
Thiswillbetrueinthenexttwoscenariosaswell.
4.
2.
BuildingfromlittlepiecesAnumberofpiecesarecurrentlyavailablethatcouldbecobbledtogethertocreateaworkablesystem.
Onecanimagine,forexample,aWordPress-basedsystemthatusesBuddyPresstoprovideasocialnetwork,Annotumtoprovideanarticlereviewworkflow,CommentPresstoenableparagraph-by-paragraphlevelcommenting,andDisqusforsuperiorcommentmoderationpowers.
Additionalpluginsmighthandlebibliographiccitations(likeMendeley),audioorvideocontent(likePod-Press),orversioning(DocumentRevisions).
Thereisalargepotentialdownsidetothisstrategy:ifmanydifferentpiecesofsoft-ware(eight,inthisexample)arebeingused,thesystemisdependentuponmanydifferentcodebases.
UpdatestoWordPress,forexample,mightbreaksomethinginCommentPress.
Ifrelativelystandardpluginsarebeingused,itwon'tbehardtofindaprogrammerwhoknowswhatneedstobechanged;butmoreobscurepluginsmayrequiredealingwiththepeopleresponsiblefortheirdevelopment.
AcompetentWordPressprogrammermightbeabletohandlethesechangesovertime;however,thisisgoingtobeanongoingtask,assoftwareonthewebisrarely46openreview:AStudyofContextsandPracticesstatic.
(WhileDrupalismoreproteanthanWordPress,usingitislikelytoexacer-batethispotentialproblem:aDrupalinstallationwillalmostcertainlyneedmoremodulesthanaWordPressinstallationwillneedplugins.
)Itisalsoworthrememberingthatpluginsalmostcertainlywon'tworknicelytogetheroutofthebox:sometweakingbyaprogrammerwillbenecessary.
Thismeansthataprogrammerstaffingtheprojectislikelytobenecessary.
Inaddition,ofcourse,designerswillneedtomakeacustomuserinterfacethatmakestheedito-rialstructuresclear.
Thisstrategyisworthconsidering.
4.
3.
BuildingfrombigpiecesAthirdstrategymightbetosplitthedifferencebetweenthefirsttwobycreatingasystemthatusesexistingsoftwarealongwithafairamountofcustomcoding.
Forexample,wemighttakeOpenJournalSystems,whichtakescareofmostoftheproblemsofeditorialmanagementforjournalcontentasthebackend,anduseWordPressasafrontend,todisplaythecontentandallowreviewersandauthorstointeract.
Thiswillrequireafairamountofprogrammingwork:thesesystemsareverydifferent,andagooddealofcustomfunctionalitymustbeadded.
Butthisstrategydoesnothaveasmanydependenciesonothercodebasesas4.
2does;anditwouldbeconsiderablylessworkthan4.
1.
Becausewhatisbeingdoneisveryspecific,forkingtheOJScodebasemightbeareasonablesolution,removingthatdependencyentirely;ineffect,thatwouldmakethisaseparatepieceofsoftware.
4.
4.
What'snotinsoftwareFinally,somewordsshouldbesaidaboutwhatsoftwarewon'tdobyitself.
Whilesoftwarecancertainlygoalongwaytowardseasingworkflow,itwon'tmakethisproblemgoawayentirely.
Noristheroleoftheeditorgoingtodisappear.
Editorswillneedtoknowhowtoguidecontentthroughthesystem;reviewersaregoingtoneedsomeinstructioninhowtointeractwiththesystem.
Andwhileautomaticmoderationcandealwithspam,dealingwithtrollswillrequirethegoodwillofthecommunity.
Dealingwithscore-settlingislikelytostillbethejoboftheeditor.
It'salsoworthnotingthatdeployingthissoftwareislikelytobeiterative:whenit'sactuallybeingused,editorsandreviewerswillfindthatsomeofthesoftwaredoesn'tworkthewaythey'dlike,andthatchangesshouldbemade.
Ifthesystemistobeinstalledbyotherpeople,detailedinstructionsonbothinstall-ingandmaintainingthesystemwillhavetobeprovided;therewillalsoneedtobe47openreview:AStudyofContextsandPracticescoordinationonupdatingthesystemorplanstoaddnewfunctionality.
Thisworkiscrucialifthesystemistobebroadlyused;withoutit,plentyofopensourceproj-ectshavefounderedeventhoughthesoftwarewassuitable.
48openreview:AStudyofContextsandPracticesappendix2FunctionalrequirementsforanopenpeerreviewsystembyPeterBrantleyApril19,2012BasicsAnyopenpeerreview(OPR)systemwillpresentasetofcompromisesindesignandfunctionality.
Difficultiesarisenotonlyintechnicalcapabilitybutalsointheheterogeneousnatureofworkflowpreferencesacrossawiderangeofscholarlycommunities,withvaryingchoicesforparticipation,editorialfunction,andpubli-cationgoals.
Inanyconsiderationofappropriatechoicesforsoftwareimplementation,itiscriti-caltodistinguishbetweenandprioritizedifferentfunctionsforthesoftwareplat-form;personaadministration;andworkflowmanagement.
Examplesofpersonaadministrativeoptionsincludesupportforreviewerpro-fileattributes(anonymousorauthenticated);managementofreputationmetrics;identitymanagementforauthors,editors,andreviewers;andtheabilitytomanageuseraccountsforprivileges,inappropriatebehavior,andotherpurposes.
Examplesofworkflowcharacteristicsincludetherelativedegreeofopennessforsolicitationofreviewers;originsofreviewsolicitation(author,editor,oropen);commentandannotationmoderation;revisioncontrolsupport;progressiontracking;schedulingrequirements;group-basedpermissions,andsoforth.
Considerationsforsoftwareplatformchoiceincludethepreferencefordistributedinstallation,suchthateachjournalorworkteamhasitsownlocalinstanceoftheOPRsoftware;hosted,suchthatindividualprojectscanbespawnedfromacom-monsoftwarebaseatanorganizationallevel,whichmightbeoptimalforapublish-erorpublishingservicesprovider;cloud-based,inwhichthesoftwareisresidentonthenetworkandiscapableofsupportingvaryingsizeprojectsandworkgroups;orsomehybridoftheabove.
Itisworthnotingthatonthisfactor,Wordpresssoft-warehasanadmirablestructure,replicatingitscorefunctionalityacrossavarietyofmodalities,includinglocallyhostedandcloud-based.
MechanicsAuthoring,editing,commenting,reviewing,revising,andpublishingallseemtobevitalelementsofwhatwemustincorporate.
Yetsomeofthesedescribewhatweseektoaccomplish–e.
g.
,peer-reviewed,publishedmaterial–andsomedescribe49openreview:AStudyofContextsandPracticeshowitiswemightultimatelyaccomplishit.
Authoring,reviewing,andpublishingarediscrete,essentialprocessesthathelpusachieveourgoal,whereasediting,com-menting,andrevisingdescribethefunctionsthatenablethatgoaltobeobtained.
Whenweconsiderfunctionalrequirementsforasoftwaresystem,wemustbeclearonthedistinctionbetweentheprocesseswearetryingtoautomateandthefunc-tionsthroughwhichweachievethem.
Forexample,inmostvariationsofOPR,itisvitalthatwemanagereviewersadroitlyandfacilitatetheircommentingandanno-tating,aswellassummarizingandcoalescingtheiroutput;reviewingcomprisesallofthese.
EachoftheactivitiesthatcomprisethehowofanyOPRprocessischar-acteristicallysmallanddiscrete.
Oneofourkeydesigngoalsistoassemblethesefunctionswithoutinducingextraneouscomplexitythroughtheircompilation.
AmemberofthecommitteenotedintheJanuary2012meetingthattheprimaryworkfocuswouldhavetobebookchapters(vs.
anentirebook),ordiscretearticles.
Absorbingthemanagementofsomethinggreaterintoareasonablereviewprocesswouldbetooeasilydauntingforbothreviewersandeditors.
However,thisdoesnotyieldasummaryperspectiveofthelargerwork:whoreviewsthemanuscript,whenonlinecommentaryiscouchedonlyatthechapterlevelThisnexusbetweenourabilitytoattendandfocus,combinedwithhowwedesiretodotheworkathand,mustbereflectedthroughthelensofsoftwareengineering.
Softwareshouldnotaskmoreofusthanwearecapableof.
Asimpleandimperfectsolutionispreferabletoanattempttoarticulateamorecomplexandfull-featuredsoftwaresystemthatwillbeinfinitelymorefragile.
Insum,OPRsystemsmustbeabletosegregatetheirinevitabledesignshortcomingsintoarenaswheretheycanbecapablyhandledthroughseparatehumanormachineinterventions.
Letthema-chinehandlewhatitcanbesthandle;therestwillrequirehumanintervention.
Beingwiseaboutwherefailureispermittedisacriticalcomponentofdesign.
IfitistoomuchtoassumethatOPRsoftwaresystemscouldreasonablyeffectthereviewofbothchapterandbookasawhole,thenthereviewofthegreatermanuscriptcaneithertakeplaceinawhollyseparateworkflow,orbeconductedoutofbandofthesoftwareitself.
Inelegant,yetfunctional,suchanapproachpermitsustoincorpo-rateasmuchrobustnessaspossibleintotheOPRsystemitself.
Designingforflexibilityisfundamental.
Itisconceivablethatonecouldarchitectasuperblyworkingsystemforonefunctionalgoal,e.
g.
,asystemthatwouldpro-videforthesubmissionandreviewofmanuscriptsgearedtowardpromotionandtenure.
ButtherewillbemyriadgoalsforOPR,andmanydifferentkindsofwork-50openreview:AStudyofContextsandPracticesgroups.
Theusecasesenumeratedrangefromevanescentworkteamsassemblingonlytodeliveraone-offproduct,tomid-durationprojectssuchasprocessingbookmanuscripts,tolonger-termendeavorssuchasopenjournalpublicationordeposit.
Wearebetteroffrestrictingthegoalsofoursoftwareratherthanthepurposestowhichitmaybeput.
RequirementsAsetofcorerequirementsisidentifiable;takenfromDanVisel'sdocumentonsoftwarereview.
1.
1.
PersonaManagementThesystemmustallowanonymity(orpseudonymity)aswellasself-identifica-tion;Thesystemshouldpermitthetogglingofvariouslevelsofidentificationde-pendingonparticipantroleandreviewcontext;Thereshouldbeawaytoexamineauser'scontributionsgloballyacrossthesiteforsiteowners;dependingonpermissions,editorsshouldhavecommensurateaccesstousercontributions;Thequalityofreviewsmustbecapableofassessmenttoaidtheevaluationofanygivenreviewer;Thesystemshouldpermitassessmentofreviewerqualificationstoaidtheevaluationofanygivenreview;Theabilitytoadequatelymoderatenewcontent,preventingtrollsandspam,isessential;Thesystemshouldmakeitpossibletodown-voteordeprecateunhelpfulcom-ments,andtoup-voteand"praise"helpfulones.
1.
2.
ContentManagementThesystemshouldbeabletomanagein-textconversationsamongauthors,editors,andreviewersinamannerallowingforvaryingdegreesofpublic/private/ormediatedaccessamonganycombinationoftheparties.
Thesystemshouldpermitcommentingatthepage/paragraph/sentence/orwordlevel;Versioningneedstobeaccommodatedandeasilyaccessible–multipledraftsandeditionsoftextsandcommentsneedtobemaintained;Thesystemshouldallowreadersandreviewerstocomparedraftspre-andpost-review,orbasedonversioning;Thesystemshouldsupportthecitationorlinkingofindividualcomments;Theeditingordeletionofreviewercommentsshouldbegracefullyhandled;Thesystemshouldbeabletocategorizeandfilterreviewsbyprimarytype(i.
e.
,51openreview:AStudyofContextsandPracticescontentvs.
editorial),andbysubcategories,suchasargument,evidence,cita-tion,orgrammar;Thesystemshouldbeabletodistinguishbetweenreviewersuggestionsorrec-ommendations,andrequiredrevisions.
1.
3.
EditorialBasicsThesystemshouldhaveeasypromptstoguidereviewersintheirtasks;Thesystemshouldenablebothpublicandprivatediscussions;Editorsshouldbeableto"enlist"reviewersoutofanavailablepool;Iftheplatformpermits,authorsshouldbeabletorecommendreviewers;Reviewersshouldbeabletoreceivenotificationsofnewtasks;Thesystemshouldbeabletoaccommodatebothclosedandopen-ended(ev-ergreen)publication.
1.
4.
IntellectualPropertyThesystemshouldpermittheapplicationofCreativeCommonslicensestoindividualcontentitems.
ObservationsCorecomponentsofworkflowmanagementaresubjecttonearlyinfinitecustom-ization.
Inmanyscenarios,editorswillhaveaccesstoreviewsandreviewercom-mentary.
Dependingonwhetherthesecontributionsareconsideredtobepartofthefinalpublishedwork,andwhethertheworkisactuallyleftinanykindoffinalstateitself,editorsmayhavetheabilitytoedit,annotate,orcensor.
Itisalsopossibletoestablishbyzantinelevelsofrequiredauthorizationforcertainactivities,suchaspostpublicationremovalofcomments,ortheblacklistingofaspecificreviewer.
Softwareisinfinitelymalleable.
Muchmorethoughtmustbeplacedonpolicyandprocedurethansoftwarerequirements.
Ratherthanover-specifypreferredfunc-tionality,itisfarbettertobequiteclearaboutwhatthepublishinggoalsare,andthepoliciesandproceduresaroundsubmissions,reviews,andpublication.
Mostaspectsofpublishingmanagementhavelittletodowiththenatureofpeerreview.
Arobustpublishingplatformislikelytosupportsophisticatedcontentmanagement,revisioncontrol,andsomeaspectsoftaskassignment.
Marriedtoarealisticunderstandingofthepublishingventure'sgoals,anappropriateplatformwillmakepeerreviewmorestraightforwardandeasytomanage.
Incontrast,build-inganentirepublishingplatformfromthegroundupinordertosupportanewopenpeerreviewsystemputsanenormousobligationonsoftwareengineering;it'sabitlikebuildingaskyscraperinordertoobtainanewmarblebathtub.
52openreview:AStudyofContextsandPracticesSampleworkstreamsAfewsampleworkstreamshelpelucidatetherangeoffunctionsforreviewers,edi-tors,andotherroles.
Twophaseopenreview.
Inatwo-phasereview,aninitialgroupof"closereaders"annotatesparticularpassagesorparagraphs(orgroupingsofthese).
Asecondgroup,suchasjournaleditors,wouldthenwriteasetofreviewsthatsynthesizeandextendtheobservationsoftheclose-readergroup.
Asanexampleofcontributedworkflow,thesecondaryreviewgroupcouldalsoevaluatethefirstgroup'sworkbyrecordingthecontributedvalueofeachclosereadingtothehigher-levelreview.
Tagscouldincludetermslike'formativeof,''ex-emplifies,''elicitsrefutation,'orothertermsthatcharacterizetheinfluenceoffirst-roundcommentonhigher-levelreview.
Thisprocesswouldenablecommentaryonfirst-stagereviewers:e.
g.
,'UserX'scommentshaveelicited100directrefutations';'UserY'sobservationshavemadesignificantcontributionstowarddevelopmentofscholarshipontopicZ,'etc.
Threephaseopenreview.
Inathree-phasereview,apreliminarygatingappraisalofsubmissionsbythehouseeditorstakesplace.
Uponprovisionalacceptance,thesubmissionisreleasedforanopenpeerreviewprocesstothecommunityofre-viewerswhohaveregisteredwith,andbeenacceptedby,thepublisher.
Afterthereviewperiodcloses,theauthorsre-submittheirpapertothepublisherwhothenundertakesafinalacceptancereviewofthework.
Inthelargemajorityofcases,mostpaperssuccessfullytransitingtheopenpeerreviewprocesswouldbeac-cepted.
Scoringreview.
Inonepossiblesinglestepreviewscenario,journalsubmissionsflowautomaticallyintoapool,perhapscategorizedbymetadatasuchassubjectclassifications;thesecouldbederivedthroughmachineanalysiswithouttheneedformanualintervention.
Reviewerscan"checkout"orregisterforanysubmissionoftheirchoosing,annotatingandratingthedocument.
Sitethresholdsdeterminethevisibilityofthepaper,oralternatively,ifitreceivesenoughvotingpoints,itisautomaticallyapprovedand"published".
Gatingreview.
Someonlinepublishingforumsmayhaveverylowcriteriaforac-ceptance,e.
g.
incorporatingrulesagainstinappropriatenarrativeprofanity,defa-mation,orotherpersonalattacks;propercitationofliterature;originality;andgeneralawarenessofthedomain.
Insuchacase,theeditorialfunctionmaybelim-itedtodesignatingorsolicitingathresholdnumberofreviewsthatevaluatethese53openreview:AStudyofContextsandPracticescriteriawithoutpayingsignificantattentiontothecontent.
Disagreementswouldhavetobemediatedbyeditorsbutotherwisereviewerapprovalwouldelevatethesubmissiononthepublishingsiteintoapubliclyaccessiblestatus.
Commentaryreview.
Insomepublishingstreams,thedesiredoutcomeistocreateapublicationcombiningexpertcommentarywithasubmitted,oftenrequisitioned,narrative.
Insuchacase,thesubmittedworkwillneedaninitialreviewpasstoevaluaterhetoricalstrengthandstructure,andtoelucidateusefuledits.
Thisinitialanalysiscouldbeperformedbyappointed"pool"reviews.
Inthesecondstageoftheprocess,intellectualpeersofthesubmitterwouldthenoverlayorintegratetheirinformedcommentaryandcritiqueintothestructureofthework,constructinganewintegratedwhole.
Inapotentialthirdstage,thisworkpackagewoulditselfpotentiallybeavailabletoasecondopenreview,ifdesired.
Monographreview.
Amonographwillneedparallelandpossiblymulti-stagedreview.
Inonetrack,readersmustbeassignedtoperformacomprehensiveandholisticcritiqueoftheentirework.
Thesereviewersmustincludesubjectmatterexpertscapableofplacingtheworkinitslargercontext;theirreviewswouldpoten-tiallybeavailableforsecondaryopencomment.
Inparallel,manuscriptchapterswouldhavetobepulledoffforindividualreviewandcomment.
Authorswillneedtoaccommodateandmergebothdiscreteandholisticcomments.
Itislikelythatthiswillrequiresupportforatwo-passreviewprocess,withafinalcuratorialdeci-sionbyhousestaff.
NextStepsPeerreviewisafastmovingareainscholarlypublishing,andthereissubstantialin-stitutionalinterestinsoftwareplatformsthatsupportitsfunctions.
Oneofthecru-cialchallengesisthateveryeditorialgrouporpublisheriscertaintohaveitsownpreferredworkflowsteps,bothinthespecificsandinhigher-levelfunctionsaswell.
Forasoftwaretooltobewidelyutilized,itwillhavetosupportadisparaterangeofflexibleconfigurations.
Forittofitasinglepurpose,itcanbenarrowlydefined.
Anynewopenpeerreviewprojectwillhavetomakeadecisionwhethertosupportanalreadyactiveeffortortosynthesizeitsown.
Atthepointwherethisdecisionneedstobemade,engagementwithgroupsbuildingoutsoftwareplatformssuchasAnnotum,OpenJournalSystems,andPLoS'Ambramustbemadeinordertocon-siderthepotentialformeetingprojectaims.
Ideally,aneweffortcouldjoinwiththe54openreview:AStudyofContextsandPracticesaimsofanexistingoneandintegrateitsrequirementsintotheparent'sengineeringworkplan.
Thealternative–tobuildoutanewsoftwareplatform–shouldbeapproachedwithcautionandtrepidation.
Itisnottrivialtodesignfromscratchacomplete,robust,scalablepublishingsystemthatsupportsopenpeerreview,contentmanagement,versioning,andotherfeatures.
Itcertainlycanbebuilt–butitwouldneedtobeconstructedbyacompetentengineeringteam.
Partneringwithanexistingdevel-opment-readyshop,suchasGeorgeMason'sCenterforHistoryandNewMedia,wouldbeanoption.
Ineitherpath,anopenpeerreviewprojectwouldneedtoacquireaproductman-agerwithsufficienttechnicalskilltobeabletorepresentprojectgoalswhileinter-actingcompetentlyandcriticallywithexternalsoftwareengineeringteams.
Anopenpeerreviewprojectwouldalsohavetobeveryclearinitsownaims:whethertolaunchaproofofconceptjournalofitsowndevice,ortoRFPorsup-portanexistingonlinejournalseekingamorerobustplatformorgreaterfunction-ality.
Toomuchdispersalofgoalsattooearlyastagewouldbedisastrous.
Forthisreason,itwouldbewisetoavoidhavingfull-timeacademicprofessionalswiththecapacitytoreviseprojectaimsinasupervisorycapacityunlessprojectoutcomescarrieddirectcareerconsequences.
Insum,anopenpeerreviewprojectwillneedtoscopeitsdesirestightly;performanenvironmentalscan;andproceedontheleastexpensivepathpossibleintermsofengineeringoutlayandmaintenance.
Openpeerreviewisanotarevolutionaryidea;itisareactiontohowthingshavebeendoneinthepast,recastinthedieofdistributed,web-basedtechnology.
Itisareasonablesuppositionthattheongoingredefinitionofacademicpublishing'spurposeandoperationalmechanicswillmanifestitselfquietlybutinexorably.
Thisdictates:don'tcreatesomethingtooambitious,becausethefuturewillmakeitobsolete.