called001my.com

Weightedlog-odds-ratio,informativeDirichletpriormethodtocomparepeerreviewfeedbackfortopandbottomquartilecollegestudentsinafirst-yearwritingprogramValerieRossUniversityofPennsylvania3808WalnutStreetPhiladelphia,PA19104215-573-2729vross@writing.
upenn.
eduMarkLibermanUniversityofPennsylvaniaWilliamsHall,Room619Philadelphia,PA19104215-898-0363myl@cis.
upenn.
eduLanNgoUniversityofPennsylvania3808WalnutStreetPhiladelphia,PA19104215-573-2729lngo@writing.
upenn.
eduRodgerLeGrandUniversityofPennsylvania3808WalnutStreetPhiladelphia,PA19104215-573-2729legrand@writing.
upenn.
eduABSTRACTThepurposeofthispaperistouseaweightedlog-odds-ratio,informativeDirichletpriormethod("bagofwords"approach)toanalyzestudentcommentsandscoresposedtoMyReviewers,aweb-basedtooldesignedforcollectingstudentwritingaswellastheirpeers'commentsandscoresoftheircolleagues'drafts.
Ourpreliminaryfindingssuggestthatstudentswhoreceivelowerscoresmayalsobereceivingsignificantlydifferentkindsoffeedbackthatsomeinthefieldofwritingstudieshavesuggestedmayhaveanegativeimpactonstudentlearningandmotivation.
Findingspointtothepossibilityofidentifyingthetheeffectivenessofdifferentkindsoffeedbackonlowerandhigherperformingstudentwriters;evaluatingtheimpactoffeedbackonstudentrevisionandgradingpractices;andidentifyingandanalyzingsymmetriesandasymmetriesinteacherandstudentfeedbackcommentaryandscores.
1.
INTRODUCTIONAskingstudentstogivewritingfeedbacktotheirpeersisacommonpracticeandmaybeintroducedtostudentsasearlyaskindergartenandcontinueintograduateschoolbasedontheunderstandingthatevaluatingandsharingfeedbackwithpeersmayacceleratethelearningofthewriterandpeerrevieweralike.
However,whilemuchattentionhasbeenpaidtothequalityandnatureoffeedbackgiventostudentwritersbyteachersandwritingtutors,verylittleexplorationhasbeendoneoftheimpactofthefeedbackgivenbystudentstotheirpeers'work(Kelly2016,Poe2016,Bouzidi,L.
&JailletA2009,Stanley1992).
Thispaperrepresentsapreliminaryinvestigationofdatagatheredaspartofalarger,cross-institutionalstudyofpeerreviewofwritingassignmentsinundergraduatecourses.
Employingtheweb-basedtoolMyReviewerswecollectedcollectedstudentwriting,commentary,andscoresfromonesemesterofafirst-yearwriting-in-the-disciplinesprogramattheUniversityofPennsylvania.
Weusedasimple"bagofwords"approachtoexplorewhetherthetypeofcommentsstudentsreceivecorrelateswiththescoretheygivetothestudents'assignment.
Resultssuggestthatstudentsinthelowerquartilereceivesignificantlydifferentcommentary--moreprescriptiveandnegative--fromthatgiventostudentsintheupperquartile.
Thishasconsiderableimplications,foritsuggeststhatgoodwritersreceivethekindofpositivereinforcementfrompeersthatmanyinthefieldofwritingstudiesconsidermosteffectiveforadvancingwritingskills,whileunderperformingwritersreceivethesortofcommentaryfrompeersthataregenerallyregardedasthesortthathindersdevelopmentofwritingskills.
Writingfeedbackistypicallydividedintotwocategories:directandindirectinstruction.
Directinstructionincludestelling,suggesting,explaining,andexemplifying(Mackiewicz&Thompson)andisoftencontrastedwithopenproblemsolvingordiscoverylearning(see,forexample,Kirschner,Sweller,&Clark,2006).
Indiscussingtheiranalysisofdiscoursebetweenwritingcentertutorsandwritingstudents,MackiewiczandThompson(2015)describedirectivenessasworrisomebutadmittedlynecessary.
Thoughdirectivenessprovidesstudentswithessentialknowledge,itmaycurtailopportunitiesforlearnerstogeneratesolutionsontheirownandmaynotfostermotivationandcuriosity(Glogger-Frey,Fleischer,Gruny,Kappich,&Renkl,2015).
However,MackiewiczandRiley's(2003)analysisofatechnicaleditor'sroleinprovidingfeedbacktowritersshowsthatindirectstrategiesarelessclearincommunicatingthehearer'sobligationtoimplementtheimplieddirective,therebypotentiallycreatingambiguity.
ResearchondirectivenessinvariousK-12settingshasalsohighlightedtheimportantroleofdirectinstructioninstudentlearning.
Glogger-Freyetal.
(2015)studiedtheeffectofanopenproblemsolvingapproach(i.
e.
indirectapproach)toteachingphysicstoeighthgradestudents.
Contrarytotheirexpectationthatopenproblemsolvingwouldcultivateknowledgetransfer,theyfoundthatthestudentswerelesspreparedforlearningandtransferinphysicsthanstudentswhohadreceiveddirectinstruction.
TheirfindingsechoKirschneretal.
'sobservationthatthereislittleresearchtosupporttheeffectivenessofsolelyusingminimalguidancewhenteaching.
Assuch,MackiewiczandThompson(2015)suggestabalancebetweendirectnessandindirectnessinprovidingwritingsupporttostudents.
Inthecontextofwritingcentertutoring,MackiewiczandThompson(2015)suggestthatmotivationplaysakeyroleintheamountoftimeandeffortthatstudentsdevotetowritingtasks.
Theyexplaintheimportanceofmotivationalscaffoldingstrategiesinencouragingstudentsthrough"praise,assurancesofcaring,andstatementsreinforcingstudentwriters'ownershipoftheirwork"(MackiewiczandThompson,2015,p.
5).
DrawingonstudiesbyHarris(1992)andLunsford(1991),MackiewiczandThompson(2015)particularlyemphasizetheeffectivenessofhelpingstudentstomaintaincontroloftheirownwriting,andtheirresearchunderscorestutors'useofmotivationalscaffoldinginfosteringstudents'abilitytomonitortheirownlearning.
However,theeffectofmotivationalscaffoldingmaydependonvariousfactors,includingastudent'sself-efficacylevel.
Forexample,Boyer,Phillips,andWallis(2008)examinedtutorialdialogueinthecontextofcomputersciencelearningandfoundthatdirectencouragementappearedtoaidstudentswithlowself-efficacy,thoughitmaynothavebeenhelpfulforhighself-efficacylearners.
AsBoyeretal.
(2008)suggest,balancingmotivationalscaffoldingandcognitivescaffolding--whichencouragesstudentstoreflectontheirownthinkingandreasoning(Mackiewicz&Thompson,2015)--remainsanissuetobestudied.
Thoughstudentsmayliketoreceivepositivefeedback,includingpraise,researchhighlightsthecomplexityoffeedbackintermsofcircumstanceandeffect.
Straub(1997)surveyed142first-yearcollegewritingstudentstoinvestigatetheirperceptionsaboutteachercommentsontheirwritingandfoundthatstudentspreferredpraise,evenwhenitwasmerelyintheformoftheword"good"adjacenttocriticism.
However,studentsmostpreferredpraisethatwasaccompaniedbyreasonsforthepositiveevaluation(Straub1997).
HattieandTimperley(2007)reviewedresearch,includingmeta-analyses,onfeedbackandconsiderittobeoneofthestrongestinfluencesonlearningandachievement.
Accordingtotheirreview,positivefeedbackmayincreaseastudent'spersistenceand,forhighself-efficacystudents,positivefeedbackbuildstheirabilitytocopewithfuturenegativefeedback(Deci,Koestner,&Ryan,1999;Hattie&Timperley,2007;Swann,Pelham,&Chidester;1988).
Ontheotherhand,studentswithalowlevelofself-efficacymayreacttopositivefeedbackbyavoidingtaskstolimittheriskofreceivingfuturenegativefeedback(Hattie&Timperley,2007).
Researchonsecondlanguagelearningindicatesthatlow-performingstudentsmaycontinuetounderperformiftheyareconsistentlygivenpositivefeedbackratherthaninformationonhowtomoveforward(Hiver,2014).
Negativefeedback,whichisequallyascomplicatedaspositivefeedback,mayeitherhinderorbolsterlearning,dependingonthestudentandcontext.
Highself-efficacylearnersviewtheirperformanceoptimistically,andtherefore,mayseeknegativefeedbacktooutperformontasks(Hattie&Timperley,2007).
Forlowself-efficacystudents,disconfirmationoftheirperformancemayadverselyimpacttheirmotivationandfutureperformance(Brockner,Derr,&Laing,1987;Hattie&Timperley,2007;Moreland&Sweeney,1984).
Negativefeedbackfromeitherateacherorpeermayhurtastudent'sconfidence.
Aftersurveying200studentsofEnglishasaforeignlanguage,Kaivanpanah,Alavi,andSepehrinia(2015)notethatnegativefeedbackfromclassmatesmaybeconfusingandharmfultoastudent'sconfidence.
However,disconfirmationofperformancemaybewelcomedifpresentedintermsofguidance(Straub,1997).
AstudybyMuis,Ranellucci,Trevors,andDuffy(2015)emphasizesthecomplexnatureofnegativefeedbackanditsimpact.
Inexaminingtheattitudes,emotions,engagement,andlearningoutcomesofkindergartenstudentswhoreceivedimmediatefeedbackfromaliteracylearningapponiPad,theresearchershadexpectednegativeevaluativefeedbacktodecreaseenjoymentandincreaseboredom(Pekrun,2006).
Ultimately,resultsfromtheirstudydemonstratedthattheimpactofnegativefeedbackonthestudentswasmixed.
2.
DATASETTheCriticalWritingProgramhasoverthepastdecadedevelopedandrefinedagenre-,discourse-anddiscipline-basedsharedcurriculumforintroducingstudentstowhatiscalled"authentic"writingsituationsthatinvolverealgenres,audiences,motives,andsubjectknowledge,aswellasintroduceandprovidestudentswithpracticeinusingasharedvocabularyandconceptsaboutwriting,fromknowledgedomainssuchasgenreandprocesstomorespecificaspectsofrhetoricalanalysisandproduction.
Thecurriculumemphasizespeerreviewandreflectionthroughout.
Thedatasetconsistsoftheworkof1,183undergraduates,mostlyfirst-year,whocompletedawritingseminarattheUniversityofPennsylvaniainSpring2016.
Thisdatasetincludesuptofivedraftsofaliteraturereviewaswellasthepeerreviewseachdraftreceived.
Peerreviewsconsistofrubric-guidedcommentsandnumericscores.
Mostdraftsareaccompaniedbythestudentwriter'spre-orpost-outlinethatprovidesarhetoricalanalysisaswellaslineofreasoningforthedraft.
Alldraftsreceivefromonetosixpeerreviews(commentsandscores)aswellascommentsandscoresfromtheirinstructor.
Inaddition,moststudentswillprovidearevisionplanthatrespondstothefeedbacktheyhavereceivedfromtheirpeersandinstructor.
AllundergraduatestudentsatPennacrossthefourundergraduateschools--CollegeofArts&Sciences,Wharton,Engineering,andNursing--arerequiredtocompleteawritingseminar.
Penn'swritingseminarsareadministeredbytheCriticalWritingProgram;mostseminarswithintheprogramaresituatedwithinaspecificdiscipline,boundedbyaparticular,discipline-basedinquiry,andtaughtbyaPh.
D.
inthatdisciplinewhofrequentlyisengagedinthelineofinquiryfocuseduponintheseminar.
Thusthetopicsanddisciplinesvaryandalongtheselineswhileallstudentsareassignedthegenreoftheliteraturereview,werecognizethatliteraturereviewswritteninthebenchsciencesareoftensubstantiallydifferentformthosewrittenin,say,thehumanities.
Someinstructorsteachtwosectionsofthesametopic.
Inadditiontodiscipline-basedseminars,wehavetwosetsofseminarsthatshareasingletopicacrossdisciplines:CraftofProse(14sectionsrepresenting161students)andUpperDivisionseminars(8sectionsrepresenting104students).
Students,sometimeswiththehelpoftheiradvisors,choosethetypeofseminarthatbestsuitstheirneeds,includingtheirself-assessmentoftheircompetenceandconfidenceaswriters.
CraftofProseseminarsaredesignedforstudentswhomayhavelesspreparationinwritingorhaveconfidenceissuesorotherconcernsaboutwriting.
UpperDivisionseminarsaredesignedforupperclassmenandtransferstudentswhowereuncomfortableinseminarsdesignedmainlyforfreshmen.
WealsohavesingletopicGlobalEnglishclassesforinternationalstudentswhoareconcernedwithlearninghowtowriteforanAmericanacademicaudience.
Formoreinformationonourdirectedself-placementcriteria,visit:http://writing.
upenn.
edu/critical/seminars/choosing_the_right_seminar.
phpThespecificdatainthissetincludespeerreviewscoresandcommentsproducedby1,183undergraduatesenrolledin90writingseminarsduringthespring2016semester.
TheExcelfilehousingthedataisorganizedinto19columnsand14,010rows.
Thecolumnheadingsinclude:ColumnHeadingDefinitionClassCodeSignifiesthedisciplineforthewritingseminartopicandthecoursetitle.
Thedatasetincludeswritingseminarsin21disciplines.
SectionNumberUniquenumericalidentifierforeachwritingseminar.
InstructorFullnameofthecourseinstructor.
DateDateonwhichapeerreviewwascompleted.
ProjectThewritingassignmentanddraftnumber.
Draft1oftheliteraturereviewwasaone-on-onereview.
Draft2wasamultiple-reciprocalpeerreviewwith1-6peers.
Draftthreeoftheliteraturereviewwasamultiplereciprocalreview.
Drafts4and5wereoptional.
Thefirstdraftofthepublicargumentpeerreviewwasone-on-one,andtheseconddraftwasmultiple-reciprocal.
Thestudentsalsocompletedamultiple-reciprocalpeerreviewfortheirfinalportfoliodrafts.
StudentNameFullnameofthestudentwriter(anonymized)GraderNameFullnameofthestudentpeerreviewer(anonymized)RubricScoreRepresentsthetotalscoreforacrossthe4scoringcategoriesidentifiedinourrubricofCognitiveandHeuristicProcesses,Invention,Reasoning,andPresentation(seebelow).
FinalGradeThelettergradeconversionofthecumulativerubricscore.
GradeCognitiveandHeuristicProcessesPeerreviewer'snumericalassessmentofstudentwriter'sknowledgeofwritingandrhetoricalawareness.
GradeInventionPeerreviewer'snumericalassessmentofstudentwriter'snoveltyandpersuasivenessforatargetedaudience.
GradeReasoningPeerreviewer'snumericalassessmentofstudentwriter'sreasonablenessandlogicalcoherence.
GradePresentationPeerreviewer'snumericalassessmentofstudentwriter'sabilitytoproducevoice,vocabulary,syntax,sentencestructure,punctuation,andtoneappropriatetothegenreandaudience.
CommentonCognitiveandHeuristicProcessesPeerreviewer'swrittenassessmentofstudentwriter'sknowledgeofwritingandrhetoricalawareness.
CommentonInventionPeerreviewer'swrittenassessmentofstudentwriter'screativity,novelty,andinventivenessinwhattheyselecttopersuadetheirtargetaudience.
CommentonReasoningPeerreviewer'swrittenassessmentofstudentwriter'sreasonablenessandlogicalcoherence.
CommentonPresentationPeerreviewer'swrittenassessmentofstudentwriter'sabilitytoproducevoice,vocabulary,syntax,sentencestructure,punctuation,tone,sourcehandling,etc.
,appropriatetothegenreandaudience.
GeneralCommentsPeerreviewer'sfinalcomments,insights,andobservationsofstudentwriter'swriting.
CombinedCompilesthepeerreviewer'swrittenassessmentsofCommentonCognitiveandHeuristicProcesses,CommentsonInvention,CommentsonReasoning,CommentsonPresentation,andGeneralCommentsintoonefield.
2.
1RubricStudentsaregivenadetailedrubric,thesameoneusedbyinstructorsinourprogramtoassessindividualstudentsintheirclassesaswellastoassessmid-termandfinalportfolios.
Therubricactsasaguideforformativeaswellas,atsemester'send,summativeassessment.
Cognition/Metacognition:KnowledgeofWritingRecognizesthepurposeoftheassignmentConceivesofaprocedureforfulfillingitPerceivestheproblem(s)tobesolvedintheassignmentFollowsdirectionsthroughallstagesoftheassignmentAbletodetectflawsinreasoninginone'sownorother'sreasoning(outlinesandpeerreviews)Abletoidentifyandevaluate(inplan,outlines,peerreviews,coverletters,otherartifacts):oRhetoricalStrategiesoAudienceoPurposeoGenreoPlan/ArrangementoComplexSynthesisoPresentationInvention:Idea/Audience(testofnovelty,creativity,persuasion)SelectionofanappropriateandengagingsubjectwithinthetopicAbilitytoselectandworksuccessfulwithinagenreSelectionofanappropriatepropositionandreasonstosupportit,attunedtotheaudienceandpurposeSelectionoftheappropriateamountandtypeofevidenceandmaterialstosupporttheproposition,attunedtotheaudienceandpurposeArrangementandstyleattunedtotheaudience,purpose,andgenre,includingabilitytoevaluatethestrengthofreasonsandevidenceIdentificationofsharedpremisestoenableaneffectiveintroductionandconclusionAbilitytograspfeedbackordetectproblemswithinventionandreviseaccordinglyAbilitytovaryvoiceandstyletoaccommodatedifferentaudiencesandgenresReasoning:Development/Coherence(testofreasonableness)CreationorselectionofanappropriatelyjustificatoryorexplanatorypropositionCreationorselectionofreasonsthatdirectlysupportthepropositionSelectionofevidencethatconfirms,illuminatesorotherwisedevelopsthereasonsAbilitytotestargumentthroughstrategiesofcounterargumentDemonstrationoflogicalcoherence:allreasonssupporttheproposition,allevidencesupportsthereasons,andtotheextentpossible,reasonsdonotcontradicteachotherDemonstrationofsemanticcoherence:sentencesandparagraphssticktogetherPresentationAbilitytoproduceavoiceandstyleappropriatetothegenreandaudienceControlofvocabulary,syntax,sentencestructure,punctuation,toneAbilitytointegraterhetoricalstrategiesandsourcessothattheycreateaconsistencyofstyleappropriatetothegenreandaudienceDemonstratedabilitytoproofreadandpolishworkforanoutsidereaderCreationanduseofgrammarchecklisttoidentifycontextandpatternsoferrorinmechanicsandusage,aswellastocorrectthemAppropriateformatting,citation,documentationofsources3.
METHODWeexcludepeerreviewswherethenumericalRubricScoreislistedas0,sincemanyoftheseappeartobecaseswherethescoreissimplymissing,althoughthereviewerevaluatedthedraftpositively.
Wealsoexcludereviewswherethescorewasgreaterthan0butlessthan2outof4--theserepresentevaluationswithfailinggrades,andinsuchcasesthereviewersoftendidn'tbothertosupplydetailedcomments.
Amongtheremainingexamples,thebottomquartileincludes3046reviewswithscoresbetween2and3.
3outof4,andthetopquartileincludes3054reviewswithscoresabove3.
78.
Thecombinedcommentsinthebottom-quartilereviewscomprise1,022,709words,andthecombinedcommentsinthetop-quartilereviewscomprise759,637words.
Inordertoevaluatethedegreeofassociationbetweenindividualwords*[FN:here"word"simplymeansastringofalphabeticletterssetoffbyspace,punctuation,orothersymbols]andscorequartiles,weusedthe"algorithmfromsection3.
5.
1ofMonroeetal.
2008.
Thismethod,originallydevelopedforastudyofpoliticalwriting,startswithasimpleratioofestimatedwordfrequenciesintwocollectionsoftext.
Theproblemisthatwhentheoverallfrequencyofawordislow,soisourconfidencethattheratioisnotastatisticalaccident--andsoisthevalueofthatwordasapredictivemarkerofthedistinctionunderstudy.
Thusinthiscollection,thewordjudgedoccursfivetimesinthebottom-quartilereviews,andonlyonceinthetopquartile;giventhedifferentoverallwordcountsinthetwogroups,themaximum-likelihoodestimateisthatjudgedisabout4timesascommoninbottom-quartilereviewsasintop-quartilereviews.
Butwecan'tbeveryconfidentthatinthenextbatchofreviews,thisrationmightnotbequitedifferent,orevenreversed.
Andinanycase,judgeddoesn'toccuroftenenoughtobeaverystrongindicatorofareviewer'ssentiment.
Incontrast,thewordshouldoccurs3,780timesinthebottom-quartilecomments,and1,914timesinthetop-quartilecomments.
Allowingforthegroups'overallwordcounts,thistellsusthatthefrequencyofshouldisabout1.
5timesgreaterinthebottom-quartilecommentsthaninthetop-quartilecomments.
Butinthiscase,theoverallfrequencyishighenoughthatwecanbefairlyconfidentthatshouldwillalsobeabout50%morefrequentinthelow-quartilecommentsinnextsemester'ssample--andshouldiscommonenoughtobeausefulindicatorofoverallreviewsentiment.
Inordertodealwiththeseissues,thecitedmethodshrinkstheoddsratioforeachwordbasedonafactorderivedfromasimplestatisticalestimateoftheprocessgeneratingthecounts,alongwithanestimateofthatword'soverallfrequencyinarelevantmoregeneralsource.
Theresultisanumber,the"weightedlog-oddsratio",thatwecanusetorankwordsaccordingtotheirapparentaffinityforonetextsampleortheother.
4.
PRELIMINARYRESULTSThebottomquartilehasmorewordsper(combined)commentthanthetopquartile:336vs.
249.
Thewordsmostreliablyassociatedwiththebottomquartileinclude:WordLowQCountLowQFreqPerMillionHighQCountHighQFreqPerMillionWeightedLogOddsRatiobe102199992.
0957107516.
757.
261sentence92789071.
9850926703.
27.
221more76647493.
8235054614.
0510.
11paragraph70016845.
5434434532.
438.
445not64246281.
3635164628.
536.
309but51235009.
2527423609.
625.
949should37803696.
0719142519.
625.
687some29842917.
7415292012.
84.
925however27012641.
0212551652.
15.
617than19381894.
979451244.
024.
536seems17191680.
83720947.
8215.
626sure12681239.
84549722.
7144.
78rather10521028.
64425559.
4784.
708try888868.
282316415.
9884.
962needs793775.
392253333.
0545.
439media731714.
768208273.
8155.
235pass300293.
3393546.
07465.
514chaplin205200.
4481925.
01194.
765mid168164.
27810.
53134.
835Thewordsmostreliablyassociatedwiththetopquartileinclude:WordLowQCountLowQFreqPerMillionHighQCountHighQFreqPerMillionWeightedLogOddsRatiothe7141869832.
25690374908.
1-5.
51and2677826183.
42380831341.
3-8.
526is2010319656.
61668021957.
9-4.
52very33913315.
752696936.
21-14.
319well34743396.
8647636270.
1-11.
643was31853114.
2838425057.
68-8.
508good32223150.
4631694171.
73-4.
742topic27382677.
227513621.
47-4.
619piece26472588.
2225913410.
84-4.
236clear22062157.
0222112910.
6-4.
13all17731733.
6320832682.
86-5.
637job14401408.
0319842611.
77-7.
544great11491123.
4918112384.
03-8.
504really13301300.
4716822214.
22-6.
091interesting14471414.
8716532176.
04-4.
985easy676660.
9910961442.
79-6.
835strong824805.
70310911436.
21-5.
281read906885.
88210591394.
09-4.
177written593579.
8337701013.
64-4.
303liked130127.
113324426.
52-5.
147enjoyed9996.
8017257338.
319-4.
64picasso2726.
4005112147.
439-4.
294twins10.
977795111146.
122-5.
658identical54.
8889876100.
048-4.
47poincare006382.
9343-4.
81There'sobviouslysomemixtureofcontentwithcommentaryhere,butoverallitmakessense--mostifnotallofthecontentadmixturecouldberemovedbylimitingthelisttowordsthatoccuratleastatarateof100permillioninbothsetsaswellasremovingwordsthatarerelatedtothespecificcontentofaparticularseminar,suchas"Picasso.
""But"isthe6thmostnegativeword:but5123(5009.
25)2742(3609.
62)5.
949Theevaluationsinthetopquartilebyrubricscorearemoreheavilyweightedwithevaluativewordsofalltypes.
Theevaluationsinthelowerquartilearemoreheavilyweightedwithprescriptivetermssuchas"should,""try,""more,""needs,"unnecessary.
"Onesymptom:onthenegativeside,there'sjustoneevaluativewordthat'sbothreasonablycommonoverall(frequency>15/100k)andmorethantwiceascommoninthelowerquartileofevaluations:"unclear".
Therestofthatendofthelistofthelowerquartileincludes:WORDRATIOQ4/Q1unclear2.
004incorrect1.
969unnecessary1.
825needs1.
729clearer1.
688Attheupperendofthequartile,morethan20positivetermsareemployed:WORDRATIOQ4/Q1easy2.
939great2.
857very2.
816nice2.
716flows2.
553logically2.
547organized2.
500job2.
497well2.
485supported2.
456fits2.
419strong2.
400really2.
292nicely2.
251convincing2.
211presentation2.
155persuasive2.
122coherent2.
118engaging2.
111interesting2.
071consistent1.
983supports1.
949clearly1.
932helps1.
927appropriate1.
925ThismaybebecausetheQ4(lower-scoring)evaluationsarefullofspecificcomplaints/suggestionssuchthateventhenegativeevaluativewordsaredilutedinfrequency.
4.
FUTUREDIRECTIONSOurnextstepswillincludeidentifyingcaseswherethecommentaryandgradeareasymmetrical.
Wewillalsoanalyzeinstructorcommentaryandscorestoseeifsimilarpatternsobtain.
Ofparticularinterestwillbetoexplorecorrelationsbetweenthetypesandlengthofcommentaryaswellasscoresfrompeersinrelationtoimprovementindraftsandfinalscores.
WeanticipateacorpusofseveralthousandpapersfromourownprogramaswellasfromtheotherinstitutionsparticipatinginabroaderNSF-fundedstudyofpeerreview:USF,MIT,UCNS,andDartmouth,andwillengageinarangeofcorpus-basedapproachestotextanalysis.
Wehopetocontributetoresearchontheeffectivenessofdifferentkindsoffeedback,particularlyasconcernsstrugglingwriters.
5.
ACKNOWLEDGMENTSThisresearchissupportedbytheNationalScienceFoundationunderAward#1544239,"CollaborativeResearch:TheRoleofInstructorandPeerFeedbackinImprovingtheCognitive,Interpersonal,andInterpersonalCompetenciesofStudentWritersinSTEMcourses.
OurthanksforthevitalcontributionsofJoeMoxley,PrincipalInvestigator,AlexRudniyandtheUSFteamRajeevReddyRachamalla,DatBaLe,BrookeDowney,andNatalieKass;NorbertElliot,ProfessorEmeritusofEnglishatNewJerseyInstituteofTechnologyandourNSFgrantconsultant;fellowinvestigatorsChrisAnson,SuzanneLane,andChristianeDonahue;andBrighidKelly,whosolvesallproblems,bigandsmall.
6.
REFERENCES[1]Boyer,K.
E.
,Phillips,R.
,Wallis,M.
,Vouk,M.
,&Lester,J.
(2008).
Balancingcognitiveandmotivationalscaffoldingintutorialdialogue.
IntelligentTutoringSystems,5091,239-249.
DOI=https://link.
springer.
com[2]Bouzidi,L.
,&Jaillet,A.
(2009).
CanOnlinePeerAssessmentbeTrusted.
EducationalTechnology&Society12(4),257–268.
[3]Brockner,J.
,Derr,W.
R.
,&Laing,W.
N.
(1987).
Self-esteemandreactionstonegativefeedback:Towardsgreatergeneralizability.
JournalofResearchinPersonality,21(3),318-334.
DOI=10.
1016/0092-6566(87)90014-6[4]Deci,E.
L.
,Koestner,R.
,&Ryan,M.
R.
(1999).
Ameta-analyticreviewofexperimentsexaminingtheeffectsofextrinsicrewardsonintrinsicmotivation.
PsychologicalBulletin,125(6),627-668.
DOI=10.
1037/0033-2909.
125.
6.
627[5]Glogger-Frey,Fleischer,C,Gruny,L.
,Kappich,J.
,&Renkl,A.
(2015).
Inventingasolutionandstudyingaworkedsolutionpreparedifferentlyforlearningfromdirectinstruction.
LearningandInstruction,39,72-87.
DOI=10.
1016/j.
learninstruc.
2015.
05.
001[6]Harris,M.
(1992).
Collaborationisnotcollaborationisnotcollaboration:Writingcentertutorialsvs.
peer-responsegroups.
CollegeCompositionandCommunication,43(3),369-383.
DOI=https://www.
jstor.
org[7]Hattie,J.
,&Timperley,H.
(2007).
Thepoweroffeedback.
ReviewofEducationalResearch,77(1),81-112.
DOI=10.
3102/003465430298487[8]Hiver,P.
(2014).
Attractorstates.
InZ.
Dornyei,A.
Henry,&P.
D.
MacIntyre(Eds.
),Motivationaldynamicsinlanguagelearning(pp.
20-28).
MultilingualMatters:Bristol,UK.
[9]Kaivanpanah,S.
,Alavi,S.
M.
,&Sepehrinia,S.
(2015).
TheLanguageLearningJournal,43(1),74-93.
DOI=10.
1080/09571736.
2012.
705571[10]Kelly,L.
EffectivenssofGuidedPeerReviewofStudentEssaysinaLargeUndergraduateBiologyCourse.
InternationalJournalofTeachingandLearninginHigherEducation2015,Volume27,Number1,56-68DOI=http://www.
isetl.
org/ijtlhe/ISSN1812-9129[11]Kirschner,P.
A.
,Sweller,J.
,&Clark,R.
E.
(2006).
Whyminimalguidanceduringinstructiondoesnotwork:ananalysisofthefailureofconstructivist,discovery,problem-basedexperiential,andinquiry-basedteaching.
EducationalPsychologist,41(2),75–86.
http://dx.
doi.
org/10.
1207/s15326985ep4102_1[12]Lunsford,A.
(1991).
Collaboration,controlandtheideaofawritingcenter.
TheWritingCenterJournal,12(1),3-10.
https://www.
ebscohost.
com/academic/education-source[13]Mackiewicz,J.
,&Thompson,I.
K.
(2015).
Talkaboutwriting:Thetutoringstrategiesofexperiencedwritingcentertutors.
NewYork,NYandLondon,UK:Routledge.
[14]Moreland,R.
L.
,&Sweeney,P.
D.
(1984).
Self-expectanciesandreactionstoevalua-tionsofpersonalperformance.
JournalofPersonality,52(2),156-176.
DOI=10.
1111/j.
1467-6494.
1984.
tb00350.
x[15]Muis,K.
R.
,Ranellucci,J.
,Trevors,G.
,Duffy,M.
C.
(2015).
Theeffectsoftechnology-mediatedimmediatefeedbackonkindergartenstudents'attitudes,emotions,engagementandlearningoutcomesduringliteracyskillsdevelopment.
LearningandInstruction,38,1-13.
DOI=http://dx.
doi.
org/10.
1016/j.
learninstruc.
2015.
02.
001[16]Pekrun,R.
(2006).
Thecontrol-valuetheoryofachievementemotions:Assumptions,corollaries,andimplicationsforeducationalresearchandpractice.
EducationalPsychologyReview,18(4),315-341.
DOI=10.
1007/s10648-006-9029-9[17]Poe,S.
andEmilyO.
Gravett(2016)AcknowledgingStudents'CollaborationsthroughPeerReview:AFootnotingPractice,CollegeTeaching,64:2,73-83,DOI=10.
1080/87567555.
2015.
1094441.
http://dx.
doi.
org/10.
1080/87567555.
2015.
1094441Publishedonline:08Mar2016.
[18]Stanley,CoachingStudentWriterstoBeEffectivePeerEvaluators,JOURNALOFSECONDLANGUAGEWRITING,1(3),217-233(1992)[19]Straub,R.
(1997).
Students'reactionstoteacherco9ments:Anexploratorystudy.
ResearchintheTeachingofEnglish,31(1),91-119.
DOI=http://www.
jstor.
org/stable/40171265[20]Swann,W.
B.
,Pelham,B.
W.
,&Chidester,T.
(1988).
Changethroughparadox:Usingself-verificationtoalterbeliefs.
JournalofPersonalityandSocialPsychology,54(2),268-273.
DOI=10.
1037/0022-3514.
54.
2.
268