impactedm

EDM  时间:2021-01-27  阅读:()
SequenceMatters,ButHowDoIDiscoverHowTowardsaWorkflowforEvaluatingActivitySequencesfromDataShayanDoroudi1,KennethHolstein2,VincentAleven2,EmmaBrunskill11ComputerScienceDepartment,2Human-ComputerInteractionInstituteCarnegieMellonUniversity{shayand,kjholste,aleven,ebrun}@cs.
cmu.
eduABSTRACTHowshouldawidevarietyofeducationalactivitiesbesequencedinordertomaximizestudentlearningWerecentlyproposedtheSequencingConstraintViolationAnalysis(SCOVA)methodtohelpaddressthisquestion.
Inthispaper,weproposehowSCOVAcouldbetransformedintoaworkflowinLearnSpheresothatotherresearchersandpractitionerscanfindanswerstotheaforementionedquestionintheirowndatasets.
Wehopethatsuchaworkflowwillleadtomoreandbetterresearchintothisimportantquestion,aswellasinterestingnewfindingsforboththeeducationaldataminingandlearningsciencescommunities.
Keywordssequencing,ordering,IntelligentTutoringSystems,LearnSphere,DataShop,workflow.
1.
INTRODUCTIONHowtosequenceeducationalactivitiesisanimportantpedagogicalquestion[12].
Muchoftheexistingworkonsequencingactivitiesconsistsoftheoreticalanalyses[2,4,7]andempiricalstudies[1,13,5,11].
Whileempiricalstudiescanhelpaddressquestionsthatcomparetwoorthreedifferentwaystosequenceacurriculum(e.
g.
,whethertopicsshouldbeblockedorinterleaved),itcannoteffectivelyscaletoanalyzingthemyriadofpotentialsequencesthatcouldbeconsidered.
However,educationaldatamining(EDM)techniquescanenableonetosimultaneouslystudydifferenttypesofsequencesbasedonpastdata.
Werecentlyproposedonesuchmethod—SequencingConstraintViolationAnalysis(SCOVA)—forcomparingtheefficacyofdifferentsequencingconstraintsgivenadatasetthatisrichinthevarietyofsequencesitexplores[3].
SCOVAcanbeusedtoanalyzeawidevarietyofsequencingconstraints,suchasprerequisiterelationships,constraintsonwhendifferentlearningmechanismsshouldbeintroduced,blocking,interleaving,andspiraling.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,includingintelligenttutoringsystems(ITSs),aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorinwhatorderlearningmechanismsshouldbesupported).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,SCOVAwillmostlikelynotbeusedforanyofthesepurposesifitjustremainsinapaperthatafewresearchersmight,atbest,readandcite.
Rather,itsbenefitwilllikelyonlyoutlivetheconfinesofaone-offEDMpaperifitisreleasedasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Ifreleasedassuchaworkflow,SCOVAcanalsointroduceresearcherswhomaynothaveotherwiseconsideredthequestionofhowactivitiesshouldbesequencedintheirlearningenvironmentstofindanewfoundinterestinthisarea,whichwebelieveisbecomingincreasinglyimportanttoboththelearningsciencesandeducationaldataminingcommunities.
2.
WORKFLOWMETHOD2.
1DataInputsSCOVAisapplicabletodatasetswithsubstantialvariabilityinthetypesofactivitysequencesthatstudentscomplete.
Thisvariabilityistypicalofmanydatasets,includingonesthatincluderandomnessinhowproblemswerepresentedtostudents(e.
g.
,[9]),oneswhereadaptivepolicieswereusedforproblemselectionresultinginsequencesthatvaryfromstudenttostudent(e.
g.
,[10]),andoneswherestudentsareabletodochoosewhichproblemstoworkonthemselves(e.
g.
,[8]).
TheworkflowcanworkwithdatasetsinthePSLCDataShopformat.
GiventhatSCOVAisaverygeneral-purposemethod,whichcanbeusedtoanalyzehowawidevarietyofsequencingconstraintsimpactpotentiallydifferentmeasuresofstudentperformance(e.
g.
,within-tutorperformance,posttestscores,learninggains,timeontask,etc.
),itmaypotentiallyneedtoutilizeavarietyofthecolumnsinaDataShopdataset.
However,forsimplicitywewilldescribeaversionofSCOVAthatislimitedtoanalyzingsequencingconstraintsthatmayonlydependonwithin-tutorcorrectnessandpropertiesoftheactivitiespresentedtostudentsandcanonlymeasuretheimpactwithrespecttowithin-tutorperformanceandfunctionsofpretestandposttestscores(suchaslearninggains).
Infull,SCOVAneedsthreeinputfiles:1.
TheDataShoptransaction-levelfile.
Foreverystepinatransaction-leveldataset,SCOVAneedstoknowtheproblemnameandwhetherthestepwasansweredcorrectlyornot.
2.
Amappingofeveryproblemnametocategoriestowhichtheproblembelongs.
Forexample,whenusingSCOVAonourfractionsITS[3],welabeledeachproblemwithoneofthreetopiclabels(makingandnamingfractions,fractionequivalenceandordering,andfractionaddition)aswellasoneofthreeactivitytypescorrespondingtolearningmechanismsfromtheKnowledge-Learning-Instruction(KLI)framework(sense-making,inductionandrefinement,andfluency-building)[6].
Thesecategorylabelswillthenbeusedasthebuildingblocksofsequencingconstraints,asexplainedinSection2.
2.
3.
Afilethatgivesthepretestandposttestscoreforeachstudent.
2.
2WorkflowModelTheworkflowbeginswiththeresearcherselectingdifferentsetsofsequencingconstraintsthattheywanttoanalyze.
Eachsequencingconstraintcanbeselectedbyfirstchoosingacategory(e.
g.
,topicsoractivitytype)andthenselectingapatternthatcorrespondstothesequencingconstraint.
Thepatterncantakeononeofthreeforms:1.
Specifyingaparticularsequence(e.
g.
,ABCABCABC,whichmaycorrespondtointerleavingdifferentactivitytypesortopics).
2.
SpecifyingthatastudentshouldbeexposedtoaproblemwithlabelAbeforeaproblemoflabelB(e.
g.
,astudentshouldbeshownanumberlineproblembeforebeingshownafractionequivalenceproblem)3.
SpecifyingthatastudentshouldhavereachedsomeperformancethresholdonaproblemwithlabelAbeforeaproblemwithlabelB(e.
g.
,astudentshouldhave95%accuracyonfractionequivalenceproblemsbeforebeingexposedtofractionaddition)Theresearchercanselectasmanysequencingconstraintsofthethreeformsabove.
Thenforeachpossiblepermutationofcategorylabels(e.
g.
,A=fractionequivalence,B=fractionaddition,C=namingfractions),SCOVAcomputesascoreforhowwelleachstudent'ssequenceinthedatasetmatchesthegivensequencingconstraints.
Thescoreistheproportionofproblemsinthetrajectorywhereasequencingconstraintwasviolated.
SCOVAthenlearnsalinearregressionmodelthatusesthedegreetowhichastudentviolatesaparticularsetofsequencingconstraintstopredictsomechosenoutcomevariable(i.
e.
,somemeasureofwithin-tutorperformanceorsomefunctionoftheposttestandpretestscores).
Noticethatifthemodelhasanegativecorrelationthenthatimpliesthemoreastudentobeysaparticularsequencingconstraint,thebetterthatstudentlearns/performsinthetutoringsystem,i.
e.
negativecorrelationsareindicativeofbeneficialsequencingconstraints.
ThefinalstepofSCOVAistocomparethemodelfitsfordifferentsetsofsequencingconstraintstoguidethepractitioner/researchertowhichsequencingconstraintshavethelargestpositiveimpactonstudentlearning.
Formoredetailsonthemethodandparticularinstantiationsofsequencingconstraints,referto[3].
2.
3WorkflowOutputsTheprimaryoutputisatableofBICvaluesofmodelsforeverysetofsequencingconstraintsevaluated.
Thepractitionercanchoosefromasetofoptionshowtheywantthetableorganized.
Forexample,ifwewereevaluatingtheimpactofconstraintsoftheformtopicAshouldcomebeforetopicB,whichshouldcomebeforetopicCintandemwithconstraintsoftheformactivitytypeXshouldcomebeforeactivitytypeY,whichshouldcomebeforeactivitytypeZ,thiscouldberepresentedina6-by-6tablewheretherowscorrespondtothedifferentpermutationsovertopicsandthecolumnscorrespondtothedifferentpermutationsoveractivitytypes.
(Iftherewasathirdcategoryofinterestwiththreedifferentlabels,suchassaywhetherthedifficultyleveloftheproblemwaseasy,medium,orhard,thentheworkflowcoulddisplaysixdifferenttables,oneforeachpermutationofdifficultylevels.
)Foranexampleofsuchatable,seeTable3in[3].
InadditiontoshowingBICvalues,thetablewillhighlightthosecellswheretheviolationofsequencingconstraintscorrelatesnegativelywithperformance/learning(againanindicatorthatthesequencingconstraintisbeneficialforstudentsratherthanharmful),andwilldesignatethemodelwiththelowestBIC(i.
e.
,thebest-fittingmodel).
TherewillalsobeatoggletodisplayotherquantitiesofimportanceinplaceofBIC,suchasthecoefficientsofthepredictorsinthemodels.
Inthecaseofevaluatingsequencingconstraintsoverasinglecategory(e.
g.
,onlyhowactivitytypesshouldbesequenced),theusercanchoosetodisplaythescatterplotsusedtofiteachmodelandthebest-fitlinesthemselves.
Theusercanalsochoosetocolor-codeeachpointofthescatterplotswiththevalueofsomefeature(e.
g.
,howmanyproblemsthatstudentreceived).
Thiscolor-codingoftheplotscanhelpidentifypotentialconfounds(e.
g.
,studentswhodomoreproblemsmighttendtoviolatefewerofasequencingconstraintandalsodobettersimplybecausetheydidmoreproblems).
Finally,theworkflowwillallowdoingexploratoryanalysestodetectotherpotentialconfounds.
Forexample,ifthesequencesinthedataweregeneratedaccordingtoadaptivepolicies,onepotentialconfoundisthatastudent'sperformanceaffectsthedegreetowhichsequencingconstraintsareviolatedinadditiontotheintendedcausaldirectionofthedegreetowhichasequencingconstraintisviolatedinfluencingthestudent'sperformance.
Toanalyzethepresenceofsuchaconfound,modelscanbelearnedwheretheoutcomevariableisthestudent'spretestscore(ratherthansayposttestscore);sincethepretestscorecomesbeforethestudents'useofthetutor,weknowthattheonlyreasonitwouldcorrelatewithviolationsofcertainsequencingconstraintsisiftheadaptivepoliciesdiscriminatedbetweenstudentswithdifferentamountsofpriorknowledge.
InusingSCOVAonourfractionstutor,wefoundthatwhilethisreversecausaldirectiondidexist,itwasseeminglynegligibleandactuallybiasingagainsttheconclusionsthatourresultssupport[3].
SuchaworkflowshouldallowuserstheabilitytodoexploratoryanalysesbeforemakingfirmconclusionsusingSCOVA.
3.
DISCUSSIONHavingaworkflowforanalyzingtheimpactofdifferentsequencingconstraintscanhaveanumberofbenefitsforboththeEDMandlearningsciencecommunities.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorhowlearningmechanismsshouldbesequenced).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,forSCOVAtobeusedinsuchafashion,itwilllikelyhavetobereadilyavailableasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Additionally,byhavingsuchaworkflowonLearnSphere,moreresearchersmaybeattractedtothequestionofhowtosequenceproblemsintheirlearningenvironmentofinterest.
Furthermore,ifLearnSpherealsoincludesworkflowsforothermethodsofanalyzingsequencingconstraintssuchas[9],moreresearchcanbedoneincomparingthesemethods.
Currentlywhensuchamethodispublisheditisnotwidelyadoptedeitherinpracticeorbyotherresearchers,anditisnotcomparedtomethodsthatsucceedit.
Byputtingallmethodsthatdosimilarstylesofanalysesononeplatform,LearnSpherecanleadtomoreproductiveresearch,includinghopefullybetterwaysofunderstandinghowweshouldsequenceeducationalactivitiesindifferentlearningenvironments.
4.
ACKNOWLEDGMENTSTheresearchreportedherewassupportedbytheInstituteofEducationSciences,U.
S.
DepartmentofEducation,throughGrantsR305A130215andR305B150008toCarnegieMellonUniversity.
TheopinionsexpressedarethoseoftheauthorsanddonotrepresentviewsoftheInstituteortheU.
S.
Dept.
ofEducation.
5.
REFERENCES[1]W.
Battig.
Intrataskinterferenceasasourceoffacilitationintransferandretention.
Topicsinlearningandperformance,pages131–159,1972.
[2]R.
E.
Clark,D.
Feldon,J.
J.
vanMerrienboer,K.
Yates,andS.
Early.
Cognitivetaskanalysis.
Handbookofresearchoneducationalcommunicationsandtechnology,3:577–593,2008.
[3]S.
Doroudi,K.
Holstein,V.
Aleven,andE.
Brunskill.
SequenceMatters,ButHowExactlyAMethodforEvaluatingActivitySequencesfromData.
InEDM,2016.
[4]J.
-C.
Falmagne,M.
Koppen,M.
Villano,J.
-P.
Doignon,andL.
Johannesen.
Introductiontoknowledgespaces:Howtobuild,test,andsearchthem.
PsychologicalReview,97(2):201,1990.
[5]S.
Kalyuga.
Expertisereversaleffectanditsimplicationsforlearner-tailoredinstruction.
EducationalPsychologyReview,19(4):509–539,2007.
[6]K.
Koedinger,A.
Corbett,andC.
Perfetti.
TheKnowledge-Learning-Instructionframework:Bridgingthescience-practicechasmtoenhancerobuststudentlearning.
CognitiveScience,36(5):757-798,2012.
[7]K.
Korossy.
Modelingknowledgeascompetenceandperformance.
Knowledgespaces:Theories,empiricalresearch,andapplications,pages103–132,1999.
[8]Y.
LongandV.
Aleven.
Supportingstudents'self-regulatedlearningwithanopenlearnermodelinalinearequationtutor.
InAIED,2013.
[9]Z.
A.
PardosandN.
T.
Heffernan.
Determiningthesignificanceofitemorderinrandomizedproblemsets.
2009.
[10]M.
A.
Rau,V.
Aleven,andN.
Rummel.
Complementaryeffectsofsense-makingandfluency-buildingsupportforconnectionmaking:AmatterofsequenceInAIED,2013.
[11]A.
RenklandR.
K.
Atkinson.
Structuringthetransitionfromexamplestudytoproblemsolvingincognitiveskillacquisition:Acognitiveloadperspective.
Educationalpsychologist,38(1):15–22,2003.
[12]F.
E.
Ritter,J.
Nerb,E.
Lehtinen,andT.
M.
O'Shea,editors.
Inordertolearn:howthesequenceoftopicsinfluenceslearning.
OxfordUniversityPress,2007.
[13]D.
RohrerandK.
Taylor.
Theshufflingofmathematicsproblemsimproveslearning.
InstructionalScience,35(6):481–498,2007.

数脉科技8月促销,新客减400港币,BGP、CN2+BGP、阿里云线路低至350元

数脉科技(shuhost)8月促销:香港独立服务器,自营BGP、CN2+BGP、阿里云线路,新客立减400港币/月,老用户按照优惠码减免!香港服务器带宽可选10Mbps、30Mbps、50Mbps、100Mbps带宽,支持中文本Windows、Linux等系统。官方网站:https://www.shuhost.com* 更大带宽可在选购时选择同样享受优惠。* 目前仅提供HKBGP、阿里云产品,香港...

CloudCone 新增洛杉矶优化线路 年付17.99美元且简单线路测试

CloudCone 商家在以前的篇幅中也有多次介绍到,这个商家也蛮有意思的。以前一直只有洛杉矶MC机房,而且在功能上和Linode、DO、Vultr一样可以随时删除采用按时计费模式。但是,他们没有学到人家的精华部分,要这样的小时计费,一定要机房多才有优势,否则压根没有多大用途。这不最近CloudCone商家有点小变化,有新人洛杉矶优化线路,具体是什么优化的等会我测试看看线路。内存CPU硬盘流量价格...

tmhhost(100元/季)自带windows系统,香港(三网)cn2 gia、日本cn2、韩国cn2、美国(三网)cn2 gia、美国cn2gia200G高防

tmhhost可谓是相当熟悉国内网络情况(资质方面:ISP\ICP\工商齐备),专业售卖海外高端优质线路的云服务器和独立服务器,包括了:香港的三网cn2 gia、日本 cn2、日本软银云服务器、韩国CN2、美国三网cn2 gia 云服务器、美国 cn2 gia +200G高防的。另外还有国内云服务器:镇江BGP 大连BGP数据盘和系统盘分开,自带windows系统,支持支付宝付款和微信,简直就是专...

EDM为你推荐
cf蜗牛外挂现 在 开 C F 蜗 牛 透 视 封 号 吗?浮动利率和固定利率哪个好银行贷款是选固定利率好还是浮动利率绝地求生加速器哪个好绝地求生用什么加速器比较好?红茶和绿茶哪个好红茶和绿茶哪个比较好?网络机顶盒哪个好什么牌子的网络机顶盒最好考生个人空间登录自学考试的“考生个人空间”密码忘记了……一定要本人带身份证和考籍证去有关部门吗?辽宁联通网上营业厅网中国联通的初始服务密码一般是多少dns服务器未响应电脑网络连接不到,DNS服务器未响应是什么意思?360云盘关闭360云盘已经关闭了 文件怎么下360云盘36t360云盘怎么扩容到36T?_(:зゝ∠)_
西安服务器租用 网页空间租用 合租服务器 花生壳域名贝锐 hkbn 国内永久免费云服务器 68.168.16.150 sub-process 服务器日志分析 一点优惠网 debian7 个人空间申请 小米数据库 网游服务器 电信托管 网通服务器 新加坡空间 永久免费空间 lamp怎么读 lamp什么意思 更多