SequenceMatters,ButHowDoIDiscoverHowTowardsaWorkflowforEvaluatingActivitySequencesfromDataShayanDoroudi1,KennethHolstein2,VincentAleven2,EmmaBrunskill11ComputerScienceDepartment,2Human-ComputerInteractionInstituteCarnegieMellonUniversity{shayand,kjholste,aleven,ebrun}@cs.
cmu.
eduABSTRACTHowshouldawidevarietyofeducationalactivitiesbesequencedinordertomaximizestudentlearningWerecentlyproposedtheSequencingConstraintViolationAnalysis(SCOVA)methodtohelpaddressthisquestion.
Inthispaper,weproposehowSCOVAcouldbetransformedintoaworkflowinLearnSpheresothatotherresearchersandpractitionerscanfindanswerstotheaforementionedquestionintheirowndatasets.
Wehopethatsuchaworkflowwillleadtomoreandbetterresearchintothisimportantquestion,aswellasinterestingnewfindingsforboththeeducationaldataminingandlearningsciencescommunities.
Keywordssequencing,ordering,IntelligentTutoringSystems,LearnSphere,DataShop,workflow.
1.
INTRODUCTIONHowtosequenceeducationalactivitiesisanimportantpedagogicalquestion[12].
Muchoftheexistingworkonsequencingactivitiesconsistsoftheoreticalanalyses[2,4,7]andempiricalstudies[1,13,5,11].
Whileempiricalstudiescanhelpaddressquestionsthatcomparetwoorthreedifferentwaystosequenceacurriculum(e.
g.
,whethertopicsshouldbeblockedorinterleaved),itcannoteffectivelyscaletoanalyzingthemyriadofpotentialsequencesthatcouldbeconsidered.
However,educationaldatamining(EDM)techniquescanenableonetosimultaneouslystudydifferenttypesofsequencesbasedonpastdata.
Werecentlyproposedonesuchmethod—SequencingConstraintViolationAnalysis(SCOVA)—forcomparingtheefficacyofdifferentsequencingconstraintsgivenadatasetthatisrichinthevarietyofsequencesitexplores[3].
SCOVAcanbeusedtoanalyzeawidevarietyofsequencingconstraints,suchasprerequisiterelationships,constraintsonwhendifferentlearningmechanismsshouldbeintroduced,blocking,interleaving,andspiraling.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,includingintelligenttutoringsystems(ITSs),aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorinwhatorderlearningmechanismsshouldbesupported).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,SCOVAwillmostlikelynotbeusedforanyofthesepurposesifitjustremainsinapaperthatafewresearchersmight,atbest,readandcite.
Rather,itsbenefitwilllikelyonlyoutlivetheconfinesofaone-offEDMpaperifitisreleasedasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Ifreleasedassuchaworkflow,SCOVAcanalsointroduceresearcherswhomaynothaveotherwiseconsideredthequestionofhowactivitiesshouldbesequencedintheirlearningenvironmentstofindanewfoundinterestinthisarea,whichwebelieveisbecomingincreasinglyimportanttoboththelearningsciencesandeducationaldataminingcommunities.
2.
WORKFLOWMETHOD2.
1DataInputsSCOVAisapplicabletodatasetswithsubstantialvariabilityinthetypesofactivitysequencesthatstudentscomplete.
Thisvariabilityistypicalofmanydatasets,includingonesthatincluderandomnessinhowproblemswerepresentedtostudents(e.
g.
,[9]),oneswhereadaptivepolicieswereusedforproblemselectionresultinginsequencesthatvaryfromstudenttostudent(e.
g.
,[10]),andoneswherestudentsareabletodochoosewhichproblemstoworkonthemselves(e.
g.
,[8]).
TheworkflowcanworkwithdatasetsinthePSLCDataShopformat.
GiventhatSCOVAisaverygeneral-purposemethod,whichcanbeusedtoanalyzehowawidevarietyofsequencingconstraintsimpactpotentiallydifferentmeasuresofstudentperformance(e.
g.
,within-tutorperformance,posttestscores,learninggains,timeontask,etc.
),itmaypotentiallyneedtoutilizeavarietyofthecolumnsinaDataShopdataset.
However,forsimplicitywewilldescribeaversionofSCOVAthatislimitedtoanalyzingsequencingconstraintsthatmayonlydependonwithin-tutorcorrectnessandpropertiesoftheactivitiespresentedtostudentsandcanonlymeasuretheimpactwithrespecttowithin-tutorperformanceandfunctionsofpretestandposttestscores(suchaslearninggains).
Infull,SCOVAneedsthreeinputfiles:1.
TheDataShoptransaction-levelfile.
Foreverystepinatransaction-leveldataset,SCOVAneedstoknowtheproblemnameandwhetherthestepwasansweredcorrectlyornot.
2.
Amappingofeveryproblemnametocategoriestowhichtheproblembelongs.
Forexample,whenusingSCOVAonourfractionsITS[3],welabeledeachproblemwithoneofthreetopiclabels(makingandnamingfractions,fractionequivalenceandordering,andfractionaddition)aswellasoneofthreeactivitytypescorrespondingtolearningmechanismsfromtheKnowledge-Learning-Instruction(KLI)framework(sense-making,inductionandrefinement,andfluency-building)[6].
Thesecategorylabelswillthenbeusedasthebuildingblocksofsequencingconstraints,asexplainedinSection2.
2.
3.
Afilethatgivesthepretestandposttestscoreforeachstudent.
2.
2WorkflowModelTheworkflowbeginswiththeresearcherselectingdifferentsetsofsequencingconstraintsthattheywanttoanalyze.
Eachsequencingconstraintcanbeselectedbyfirstchoosingacategory(e.
g.
,topicsoractivitytype)andthenselectingapatternthatcorrespondstothesequencingconstraint.
Thepatterncantakeononeofthreeforms:1.
Specifyingaparticularsequence(e.
g.
,ABCABCABC,whichmaycorrespondtointerleavingdifferentactivitytypesortopics).
2.
SpecifyingthatastudentshouldbeexposedtoaproblemwithlabelAbeforeaproblemoflabelB(e.
g.
,astudentshouldbeshownanumberlineproblembeforebeingshownafractionequivalenceproblem)3.
SpecifyingthatastudentshouldhavereachedsomeperformancethresholdonaproblemwithlabelAbeforeaproblemwithlabelB(e.
g.
,astudentshouldhave95%accuracyonfractionequivalenceproblemsbeforebeingexposedtofractionaddition)Theresearchercanselectasmanysequencingconstraintsofthethreeformsabove.
Thenforeachpossiblepermutationofcategorylabels(e.
g.
,A=fractionequivalence,B=fractionaddition,C=namingfractions),SCOVAcomputesascoreforhowwelleachstudent'ssequenceinthedatasetmatchesthegivensequencingconstraints.
Thescoreistheproportionofproblemsinthetrajectorywhereasequencingconstraintwasviolated.
SCOVAthenlearnsalinearregressionmodelthatusesthedegreetowhichastudentviolatesaparticularsetofsequencingconstraintstopredictsomechosenoutcomevariable(i.
e.
,somemeasureofwithin-tutorperformanceorsomefunctionoftheposttestandpretestscores).
Noticethatifthemodelhasanegativecorrelationthenthatimpliesthemoreastudentobeysaparticularsequencingconstraint,thebetterthatstudentlearns/performsinthetutoringsystem,i.
e.
negativecorrelationsareindicativeofbeneficialsequencingconstraints.
ThefinalstepofSCOVAistocomparethemodelfitsfordifferentsetsofsequencingconstraintstoguidethepractitioner/researchertowhichsequencingconstraintshavethelargestpositiveimpactonstudentlearning.
Formoredetailsonthemethodandparticularinstantiationsofsequencingconstraints,referto[3].
2.
3WorkflowOutputsTheprimaryoutputisatableofBICvaluesofmodelsforeverysetofsequencingconstraintsevaluated.
Thepractitionercanchoosefromasetofoptionshowtheywantthetableorganized.
Forexample,ifwewereevaluatingtheimpactofconstraintsoftheformtopicAshouldcomebeforetopicB,whichshouldcomebeforetopicCintandemwithconstraintsoftheformactivitytypeXshouldcomebeforeactivitytypeY,whichshouldcomebeforeactivitytypeZ,thiscouldberepresentedina6-by-6tablewheretherowscorrespondtothedifferentpermutationsovertopicsandthecolumnscorrespondtothedifferentpermutationsoveractivitytypes.
(Iftherewasathirdcategoryofinterestwiththreedifferentlabels,suchassaywhetherthedifficultyleveloftheproblemwaseasy,medium,orhard,thentheworkflowcoulddisplaysixdifferenttables,oneforeachpermutationofdifficultylevels.
)Foranexampleofsuchatable,seeTable3in[3].
InadditiontoshowingBICvalues,thetablewillhighlightthosecellswheretheviolationofsequencingconstraintscorrelatesnegativelywithperformance/learning(againanindicatorthatthesequencingconstraintisbeneficialforstudentsratherthanharmful),andwilldesignatethemodelwiththelowestBIC(i.
e.
,thebest-fittingmodel).
TherewillalsobeatoggletodisplayotherquantitiesofimportanceinplaceofBIC,suchasthecoefficientsofthepredictorsinthemodels.
Inthecaseofevaluatingsequencingconstraintsoverasinglecategory(e.
g.
,onlyhowactivitytypesshouldbesequenced),theusercanchoosetodisplaythescatterplotsusedtofiteachmodelandthebest-fitlinesthemselves.
Theusercanalsochoosetocolor-codeeachpointofthescatterplotswiththevalueofsomefeature(e.
g.
,howmanyproblemsthatstudentreceived).
Thiscolor-codingoftheplotscanhelpidentifypotentialconfounds(e.
g.
,studentswhodomoreproblemsmighttendtoviolatefewerofasequencingconstraintandalsodobettersimplybecausetheydidmoreproblems).
Finally,theworkflowwillallowdoingexploratoryanalysestodetectotherpotentialconfounds.
Forexample,ifthesequencesinthedataweregeneratedaccordingtoadaptivepolicies,onepotentialconfoundisthatastudent'sperformanceaffectsthedegreetowhichsequencingconstraintsareviolatedinadditiontotheintendedcausaldirectionofthedegreetowhichasequencingconstraintisviolatedinfluencingthestudent'sperformance.
Toanalyzethepresenceofsuchaconfound,modelscanbelearnedwheretheoutcomevariableisthestudent'spretestscore(ratherthansayposttestscore);sincethepretestscorecomesbeforethestudents'useofthetutor,weknowthattheonlyreasonitwouldcorrelatewithviolationsofcertainsequencingconstraintsisiftheadaptivepoliciesdiscriminatedbetweenstudentswithdifferentamountsofpriorknowledge.
InusingSCOVAonourfractionstutor,wefoundthatwhilethisreversecausaldirectiondidexist,itwasseeminglynegligibleandactuallybiasingagainsttheconclusionsthatourresultssupport[3].
SuchaworkflowshouldallowuserstheabilitytodoexploratoryanalysesbeforemakingfirmconclusionsusingSCOVA.
3.
DISCUSSIONHavingaworkflowforanalyzingtheimpactofdifferentsequencingconstraintscanhaveanumberofbenefitsforboththeEDMandlearningsciencecommunities.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorhowlearningmechanismsshouldbesequenced).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,forSCOVAtobeusedinsuchafashion,itwilllikelyhavetobereadilyavailableasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Additionally,byhavingsuchaworkflowonLearnSphere,moreresearchersmaybeattractedtothequestionofhowtosequenceproblemsintheirlearningenvironmentofinterest.
Furthermore,ifLearnSpherealsoincludesworkflowsforothermethodsofanalyzingsequencingconstraintssuchas[9],moreresearchcanbedoneincomparingthesemethods.
Currentlywhensuchamethodispublisheditisnotwidelyadoptedeitherinpracticeorbyotherresearchers,anditisnotcomparedtomethodsthatsucceedit.
Byputtingallmethodsthatdosimilarstylesofanalysesononeplatform,LearnSpherecanleadtomoreproductiveresearch,includinghopefullybetterwaysofunderstandinghowweshouldsequenceeducationalactivitiesindifferentlearningenvironments.
4.
ACKNOWLEDGMENTSTheresearchreportedherewassupportedbytheInstituteofEducationSciences,U.
S.
DepartmentofEducation,throughGrantsR305A130215andR305B150008toCarnegieMellonUniversity.
TheopinionsexpressedarethoseoftheauthorsanddonotrepresentviewsoftheInstituteortheU.
S.
Dept.
ofEducation.
5.
REFERENCES[1]W.
Battig.
Intrataskinterferenceasasourceoffacilitationintransferandretention.
Topicsinlearningandperformance,pages131–159,1972.
[2]R.
E.
Clark,D.
Feldon,J.
J.
vanMerrienboer,K.
Yates,andS.
Early.
Cognitivetaskanalysis.
Handbookofresearchoneducationalcommunicationsandtechnology,3:577–593,2008.
[3]S.
Doroudi,K.
Holstein,V.
Aleven,andE.
Brunskill.
SequenceMatters,ButHowExactlyAMethodforEvaluatingActivitySequencesfromData.
InEDM,2016.
[4]J.
-C.
Falmagne,M.
Koppen,M.
Villano,J.
-P.
Doignon,andL.
Johannesen.
Introductiontoknowledgespaces:Howtobuild,test,andsearchthem.
PsychologicalReview,97(2):201,1990.
[5]S.
Kalyuga.
Expertisereversaleffectanditsimplicationsforlearner-tailoredinstruction.
EducationalPsychologyReview,19(4):509–539,2007.
[6]K.
Koedinger,A.
Corbett,andC.
Perfetti.
TheKnowledge-Learning-Instructionframework:Bridgingthescience-practicechasmtoenhancerobuststudentlearning.
CognitiveScience,36(5):757-798,2012.
[7]K.
Korossy.
Modelingknowledgeascompetenceandperformance.
Knowledgespaces:Theories,empiricalresearch,andapplications,pages103–132,1999.
[8]Y.
LongandV.
Aleven.
Supportingstudents'self-regulatedlearningwithanopenlearnermodelinalinearequationtutor.
InAIED,2013.
[9]Z.
A.
PardosandN.
T.
Heffernan.
Determiningthesignificanceofitemorderinrandomizedproblemsets.
2009.
[10]M.
A.
Rau,V.
Aleven,andN.
Rummel.
Complementaryeffectsofsense-makingandfluency-buildingsupportforconnectionmaking:AmatterofsequenceInAIED,2013.
[11]A.
RenklandR.
K.
Atkinson.
Structuringthetransitionfromexamplestudytoproblemsolvingincognitiveskillacquisition:Acognitiveloadperspective.
Educationalpsychologist,38(1):15–22,2003.
[12]F.
E.
Ritter,J.
Nerb,E.
Lehtinen,andT.
M.
O'Shea,editors.
Inordertolearn:howthesequenceoftopicsinfluenceslearning.
OxfordUniversityPress,2007.
[13]D.
RohrerandK.
Taylor.
Theshufflingofmathematicsproblemsimproveslearning.
InstructionalScience,35(6):481–498,2007.
对于一般的用户来说,我们使用宝塔面板免费版本功能还是足够的,如果我们有需要付费插件和专业版的功能,且需要的插件比较多,实际上且长期使用的话,还是购买付费专业版或者企业版本划算一些。昨天也有在文章中分享年中促销活动。如今我们是否会发现,我们在安装宝塔面板后是必须强制我们登录账户的,否则一直有弹出登录界面,我们还是注册一个账户比较好。反正免费注册宝塔账户还有代金券赠送。 新注册宝塔账户送代金券我们注册...
大硬盘服务器、存储服务器、Chia矿机。RackNerd,2019年末成立的商家,主要提供各类KVM VPS主机、独立服务器和站群服务器等。当前RackNerd正在促销旗下几款美国大硬盘服务器,位于洛杉矶multacom数据中心,亚洲优化线路,非常适合存储、数据备份等应用场景,双路e5-2640v2,64G内存,56G SSD系统盘,160T SAS数据盘,流量是每月200T,1Gbps带宽,配5...
CUBECLOUD(魔方云)成立于2016年,亚太互联网络信息中心(APNIC)会员,全线产品均为完全自营,专业数据灾备冗余,全部产品均为SSD阵列,精品网络CN2(GIA) CU(10099VIP)接入,与当今主流云计算解决方案保持同步,为企业以及开发者用户实现灵活弹性自动化的基础设施。【夏日特促】全场产品88折优惠码:Summer_2021时间:2021年8月1日 — 2021年8月8日香港C...
EDM为你推荐
火影忍者644火影644集4分钟左右的背景音乐是什么软银孙正义日本的孙正义是阿里的最大股东,马云没有孙正义的股份多,为什么马云是亚洲首富而不是孙正义?浏览器哪个好什么浏览器最好用?手机浏览器哪个好手机什么浏览器最好用核芯显卡与独立显卡哪个好独立显卡和核心显卡哪个好电动牙刷哪个好飞利浦电动牙刷哪款好?求推荐网页qq空间登录为什么我用网页登录QQ空间时,没有登录界面,直接进去一个QQ空间,请问是怎么回事?东莞电信网上营业厅电信网上营业厅广东电信电信dns服务器地址电信宽带的DNS服务地址是多少360云盘360云盘有多大?
域名备案批量查询 westhost mediafire下载 360抢票助手 国外php空间 电子邮件服务器 傲盾官网 能外链的相册 in域名 空间首页登陆 备案空间 重庆电信服务器托管 supercache 碳云 phpinfo 塔式服务器 侦探online 木马检测 wordpress安装 vim命令 更多