SequenceMatters,ButHowDoIDiscoverHowTowardsaWorkflowforEvaluatingActivitySequencesfromDataShayanDoroudi1,KennethHolstein2,VincentAleven2,EmmaBrunskill11ComputerScienceDepartment,2Human-ComputerInteractionInstituteCarnegieMellonUniversity{shayand,kjholste,aleven,ebrun}@cs.
cmu.
eduABSTRACTHowshouldawidevarietyofeducationalactivitiesbesequencedinordertomaximizestudentlearningWerecentlyproposedtheSequencingConstraintViolationAnalysis(SCOVA)methodtohelpaddressthisquestion.
Inthispaper,weproposehowSCOVAcouldbetransformedintoaworkflowinLearnSpheresothatotherresearchersandpractitionerscanfindanswerstotheaforementionedquestionintheirowndatasets.
Wehopethatsuchaworkflowwillleadtomoreandbetterresearchintothisimportantquestion,aswellasinterestingnewfindingsforboththeeducationaldataminingandlearningsciencescommunities.
Keywordssequencing,ordering,IntelligentTutoringSystems,LearnSphere,DataShop,workflow.
1.
INTRODUCTIONHowtosequenceeducationalactivitiesisanimportantpedagogicalquestion[12].
Muchoftheexistingworkonsequencingactivitiesconsistsoftheoreticalanalyses[2,4,7]andempiricalstudies[1,13,5,11].
Whileempiricalstudiescanhelpaddressquestionsthatcomparetwoorthreedifferentwaystosequenceacurriculum(e.
g.
,whethertopicsshouldbeblockedorinterleaved),itcannoteffectivelyscaletoanalyzingthemyriadofpotentialsequencesthatcouldbeconsidered.
However,educationaldatamining(EDM)techniquescanenableonetosimultaneouslystudydifferenttypesofsequencesbasedonpastdata.
Werecentlyproposedonesuchmethod—SequencingConstraintViolationAnalysis(SCOVA)—forcomparingtheefficacyofdifferentsequencingconstraintsgivenadatasetthatisrichinthevarietyofsequencesitexplores[3].
SCOVAcanbeusedtoanalyzeawidevarietyofsequencingconstraints,suchasprerequisiterelationships,constraintsonwhendifferentlearningmechanismsshouldbeintroduced,blocking,interleaving,andspiraling.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,includingintelligenttutoringsystems(ITSs),aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorinwhatorderlearningmechanismsshouldbesupported).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,SCOVAwillmostlikelynotbeusedforanyofthesepurposesifitjustremainsinapaperthatafewresearchersmight,atbest,readandcite.
Rather,itsbenefitwilllikelyonlyoutlivetheconfinesofaone-offEDMpaperifitisreleasedasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Ifreleasedassuchaworkflow,SCOVAcanalsointroduceresearcherswhomaynothaveotherwiseconsideredthequestionofhowactivitiesshouldbesequencedintheirlearningenvironmentstofindanewfoundinterestinthisarea,whichwebelieveisbecomingincreasinglyimportanttoboththelearningsciencesandeducationaldataminingcommunities.
2.
WORKFLOWMETHOD2.
1DataInputsSCOVAisapplicabletodatasetswithsubstantialvariabilityinthetypesofactivitysequencesthatstudentscomplete.
Thisvariabilityistypicalofmanydatasets,includingonesthatincluderandomnessinhowproblemswerepresentedtostudents(e.
g.
,[9]),oneswhereadaptivepolicieswereusedforproblemselectionresultinginsequencesthatvaryfromstudenttostudent(e.
g.
,[10]),andoneswherestudentsareabletodochoosewhichproblemstoworkonthemselves(e.
g.
,[8]).
TheworkflowcanworkwithdatasetsinthePSLCDataShopformat.
GiventhatSCOVAisaverygeneral-purposemethod,whichcanbeusedtoanalyzehowawidevarietyofsequencingconstraintsimpactpotentiallydifferentmeasuresofstudentperformance(e.
g.
,within-tutorperformance,posttestscores,learninggains,timeontask,etc.
),itmaypotentiallyneedtoutilizeavarietyofthecolumnsinaDataShopdataset.
However,forsimplicitywewilldescribeaversionofSCOVAthatislimitedtoanalyzingsequencingconstraintsthatmayonlydependonwithin-tutorcorrectnessandpropertiesoftheactivitiespresentedtostudentsandcanonlymeasuretheimpactwithrespecttowithin-tutorperformanceandfunctionsofpretestandposttestscores(suchaslearninggains).
Infull,SCOVAneedsthreeinputfiles:1.
TheDataShoptransaction-levelfile.
Foreverystepinatransaction-leveldataset,SCOVAneedstoknowtheproblemnameandwhetherthestepwasansweredcorrectlyornot.
2.
Amappingofeveryproblemnametocategoriestowhichtheproblembelongs.
Forexample,whenusingSCOVAonourfractionsITS[3],welabeledeachproblemwithoneofthreetopiclabels(makingandnamingfractions,fractionequivalenceandordering,andfractionaddition)aswellasoneofthreeactivitytypescorrespondingtolearningmechanismsfromtheKnowledge-Learning-Instruction(KLI)framework(sense-making,inductionandrefinement,andfluency-building)[6].
Thesecategorylabelswillthenbeusedasthebuildingblocksofsequencingconstraints,asexplainedinSection2.
2.
3.
Afilethatgivesthepretestandposttestscoreforeachstudent.
2.
2WorkflowModelTheworkflowbeginswiththeresearcherselectingdifferentsetsofsequencingconstraintsthattheywanttoanalyze.
Eachsequencingconstraintcanbeselectedbyfirstchoosingacategory(e.
g.
,topicsoractivitytype)andthenselectingapatternthatcorrespondstothesequencingconstraint.
Thepatterncantakeononeofthreeforms:1.
Specifyingaparticularsequence(e.
g.
,ABCABCABC,whichmaycorrespondtointerleavingdifferentactivitytypesortopics).
2.
SpecifyingthatastudentshouldbeexposedtoaproblemwithlabelAbeforeaproblemoflabelB(e.
g.
,astudentshouldbeshownanumberlineproblembeforebeingshownafractionequivalenceproblem)3.
SpecifyingthatastudentshouldhavereachedsomeperformancethresholdonaproblemwithlabelAbeforeaproblemwithlabelB(e.
g.
,astudentshouldhave95%accuracyonfractionequivalenceproblemsbeforebeingexposedtofractionaddition)Theresearchercanselectasmanysequencingconstraintsofthethreeformsabove.
Thenforeachpossiblepermutationofcategorylabels(e.
g.
,A=fractionequivalence,B=fractionaddition,C=namingfractions),SCOVAcomputesascoreforhowwelleachstudent'ssequenceinthedatasetmatchesthegivensequencingconstraints.
Thescoreistheproportionofproblemsinthetrajectorywhereasequencingconstraintwasviolated.
SCOVAthenlearnsalinearregressionmodelthatusesthedegreetowhichastudentviolatesaparticularsetofsequencingconstraintstopredictsomechosenoutcomevariable(i.
e.
,somemeasureofwithin-tutorperformanceorsomefunctionoftheposttestandpretestscores).
Noticethatifthemodelhasanegativecorrelationthenthatimpliesthemoreastudentobeysaparticularsequencingconstraint,thebetterthatstudentlearns/performsinthetutoringsystem,i.
e.
negativecorrelationsareindicativeofbeneficialsequencingconstraints.
ThefinalstepofSCOVAistocomparethemodelfitsfordifferentsetsofsequencingconstraintstoguidethepractitioner/researchertowhichsequencingconstraintshavethelargestpositiveimpactonstudentlearning.
Formoredetailsonthemethodandparticularinstantiationsofsequencingconstraints,referto[3].
2.
3WorkflowOutputsTheprimaryoutputisatableofBICvaluesofmodelsforeverysetofsequencingconstraintsevaluated.
Thepractitionercanchoosefromasetofoptionshowtheywantthetableorganized.
Forexample,ifwewereevaluatingtheimpactofconstraintsoftheformtopicAshouldcomebeforetopicB,whichshouldcomebeforetopicCintandemwithconstraintsoftheformactivitytypeXshouldcomebeforeactivitytypeY,whichshouldcomebeforeactivitytypeZ,thiscouldberepresentedina6-by-6tablewheretherowscorrespondtothedifferentpermutationsovertopicsandthecolumnscorrespondtothedifferentpermutationsoveractivitytypes.
(Iftherewasathirdcategoryofinterestwiththreedifferentlabels,suchassaywhetherthedifficultyleveloftheproblemwaseasy,medium,orhard,thentheworkflowcoulddisplaysixdifferenttables,oneforeachpermutationofdifficultylevels.
)Foranexampleofsuchatable,seeTable3in[3].
InadditiontoshowingBICvalues,thetablewillhighlightthosecellswheretheviolationofsequencingconstraintscorrelatesnegativelywithperformance/learning(againanindicatorthatthesequencingconstraintisbeneficialforstudentsratherthanharmful),andwilldesignatethemodelwiththelowestBIC(i.
e.
,thebest-fittingmodel).
TherewillalsobeatoggletodisplayotherquantitiesofimportanceinplaceofBIC,suchasthecoefficientsofthepredictorsinthemodels.
Inthecaseofevaluatingsequencingconstraintsoverasinglecategory(e.
g.
,onlyhowactivitytypesshouldbesequenced),theusercanchoosetodisplaythescatterplotsusedtofiteachmodelandthebest-fitlinesthemselves.
Theusercanalsochoosetocolor-codeeachpointofthescatterplotswiththevalueofsomefeature(e.
g.
,howmanyproblemsthatstudentreceived).
Thiscolor-codingoftheplotscanhelpidentifypotentialconfounds(e.
g.
,studentswhodomoreproblemsmighttendtoviolatefewerofasequencingconstraintandalsodobettersimplybecausetheydidmoreproblems).
Finally,theworkflowwillallowdoingexploratoryanalysestodetectotherpotentialconfounds.
Forexample,ifthesequencesinthedataweregeneratedaccordingtoadaptivepolicies,onepotentialconfoundisthatastudent'sperformanceaffectsthedegreetowhichsequencingconstraintsareviolatedinadditiontotheintendedcausaldirectionofthedegreetowhichasequencingconstraintisviolatedinfluencingthestudent'sperformance.
Toanalyzethepresenceofsuchaconfound,modelscanbelearnedwheretheoutcomevariableisthestudent'spretestscore(ratherthansayposttestscore);sincethepretestscorecomesbeforethestudents'useofthetutor,weknowthattheonlyreasonitwouldcorrelatewithviolationsofcertainsequencingconstraintsisiftheadaptivepoliciesdiscriminatedbetweenstudentswithdifferentamountsofpriorknowledge.
InusingSCOVAonourfractionstutor,wefoundthatwhilethisreversecausaldirectiondidexist,itwasseeminglynegligibleandactuallybiasingagainsttheconclusionsthatourresultssupport[3].
SuchaworkflowshouldallowuserstheabilitytodoexploratoryanalysesbeforemakingfirmconclusionsusingSCOVA.
3.
DISCUSSIONHavingaworkflowforanalyzingtheimpactofdifferentsequencingconstraintscanhaveanumberofbenefitsforboththeEDMandlearningsciencecommunities.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorhowlearningmechanismsshouldbesequenced).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,forSCOVAtobeusedinsuchafashion,itwilllikelyhavetobereadilyavailableasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Additionally,byhavingsuchaworkflowonLearnSphere,moreresearchersmaybeattractedtothequestionofhowtosequenceproblemsintheirlearningenvironmentofinterest.
Furthermore,ifLearnSpherealsoincludesworkflowsforothermethodsofanalyzingsequencingconstraintssuchas[9],moreresearchcanbedoneincomparingthesemethods.
Currentlywhensuchamethodispublisheditisnotwidelyadoptedeitherinpracticeorbyotherresearchers,anditisnotcomparedtomethodsthatsucceedit.
Byputtingallmethodsthatdosimilarstylesofanalysesononeplatform,LearnSpherecanleadtomoreproductiveresearch,includinghopefullybetterwaysofunderstandinghowweshouldsequenceeducationalactivitiesindifferentlearningenvironments.
4.
ACKNOWLEDGMENTSTheresearchreportedherewassupportedbytheInstituteofEducationSciences,U.
S.
DepartmentofEducation,throughGrantsR305A130215andR305B150008toCarnegieMellonUniversity.
TheopinionsexpressedarethoseoftheauthorsanddonotrepresentviewsoftheInstituteortheU.
S.
Dept.
ofEducation.
5.
REFERENCES[1]W.
Battig.
Intrataskinterferenceasasourceoffacilitationintransferandretention.
Topicsinlearningandperformance,pages131–159,1972.
[2]R.
E.
Clark,D.
Feldon,J.
J.
vanMerrienboer,K.
Yates,andS.
Early.
Cognitivetaskanalysis.
Handbookofresearchoneducationalcommunicationsandtechnology,3:577–593,2008.
[3]S.
Doroudi,K.
Holstein,V.
Aleven,andE.
Brunskill.
SequenceMatters,ButHowExactlyAMethodforEvaluatingActivitySequencesfromData.
InEDM,2016.
[4]J.
-C.
Falmagne,M.
Koppen,M.
Villano,J.
-P.
Doignon,andL.
Johannesen.
Introductiontoknowledgespaces:Howtobuild,test,andsearchthem.
PsychologicalReview,97(2):201,1990.
[5]S.
Kalyuga.
Expertisereversaleffectanditsimplicationsforlearner-tailoredinstruction.
EducationalPsychologyReview,19(4):509–539,2007.
[6]K.
Koedinger,A.
Corbett,andC.
Perfetti.
TheKnowledge-Learning-Instructionframework:Bridgingthescience-practicechasmtoenhancerobuststudentlearning.
CognitiveScience,36(5):757-798,2012.
[7]K.
Korossy.
Modelingknowledgeascompetenceandperformance.
Knowledgespaces:Theories,empiricalresearch,andapplications,pages103–132,1999.
[8]Y.
LongandV.
Aleven.
Supportingstudents'self-regulatedlearningwithanopenlearnermodelinalinearequationtutor.
InAIED,2013.
[9]Z.
A.
PardosandN.
T.
Heffernan.
Determiningthesignificanceofitemorderinrandomizedproblemsets.
2009.
[10]M.
A.
Rau,V.
Aleven,andN.
Rummel.
Complementaryeffectsofsense-makingandfluency-buildingsupportforconnectionmaking:AmatterofsequenceInAIED,2013.
[11]A.
RenklandR.
K.
Atkinson.
Structuringthetransitionfromexamplestudytoproblemsolvingincognitiveskillacquisition:Acognitiveloadperspective.
Educationalpsychologist,38(1):15–22,2003.
[12]F.
E.
Ritter,J.
Nerb,E.
Lehtinen,andT.
M.
O'Shea,editors.
Inordertolearn:howthesequenceoftopicsinfluenceslearning.
OxfordUniversityPress,2007.
[13]D.
RohrerandK.
Taylor.
Theshufflingofmathematicsproblemsimproveslearning.
InstructionalScience,35(6):481–498,2007.
georgedatacenter怎么样?georgedatacenter这次其实是两个促销,一是促销一款特价洛杉矶E3-1220 V5独服,性价比其实最高;另外还促销三款特价vps,大家可以根据自己的需要入手。georgedatacenter是一家成立于2019年的美国vps商家,主营美国洛杉矶、芝加哥、达拉斯、新泽西、西雅图机房的VPS、邮件服务器和托管独立服务器业务。georgedatacen...
#年终感恩活动#华纳云海外物理机688元/月,续费同价,50M CN2 GIA/100M国际大带宽可选,超800G 防御,不限流华纳云成立于2015年,隶属于香港联合通讯国际有限公司。拥有香港政府颁发的商业登记证明,作为APNIC 和 ARIN 会员单位,现有香港、美国等多个地区数据中心资源,百G丰富带宽接入,坚持为海内外用户提供自研顶级硬件防火墙服务,支持T B级超大防护带宽,单IP防护最大可达...
易探云怎么样?易探云最早是主攻香港云服务器的品牌商家,由于之前香港云服务器性价比高、稳定性不错获得了不少用户的支持。易探云推出大量香港云服务器,采用BGP、CN2线路,机房有香港九龙、香港新界、香港沙田、香港葵湾等,香港1核1G低至18元/月,183.60元/年,老站长建站推荐香港2核4G5M+10G数据盘仅799元/年,性价比超强,关键是延迟全球为50ms左右,适合国内境外外贸行业网站等,如果需...
EDM为你推荐
滴滴软银合资亚滴新能源跑滴滴怎么样?可爱桌面背景图片浪漫桌面壁纸p图软件哪个好用新手用哪个软件p图较好?视频制作软件哪个好制作视频用哪个软件比较好闪迪和金士顿哪个好tf卡闪迪和金士顿哪个更好朱祁钰和朱祁镇哪个好朱高炽在位时间长还是朱祁钰在位时间长?炒股软件哪个好用用手机股票软件哪个好空间登录器qq空间登录电脑版qq网盘在哪里qq网盘在哪里360云存储360云盘最高多少内存
买域名 子域名查询 国际域名抢注 俄罗斯vps 荷兰vps la域名 godaddy优惠码 idc评测网 sockscap 地址大全 html空间 柚子舍官网 共享主机 国外代理服务器软件 佛山高防服务器 爱奇艺会员免费试用 安徽双线服务器 视频服务器是什么 稳定空间 阵亡将士纪念日 更多