impactedm

EDM  时间:2021-01-27  阅读:()
SequenceMatters,ButHowDoIDiscoverHowTowardsaWorkflowforEvaluatingActivitySequencesfromDataShayanDoroudi1,KennethHolstein2,VincentAleven2,EmmaBrunskill11ComputerScienceDepartment,2Human-ComputerInteractionInstituteCarnegieMellonUniversity{shayand,kjholste,aleven,ebrun}@cs.
cmu.
eduABSTRACTHowshouldawidevarietyofeducationalactivitiesbesequencedinordertomaximizestudentlearningWerecentlyproposedtheSequencingConstraintViolationAnalysis(SCOVA)methodtohelpaddressthisquestion.
Inthispaper,weproposehowSCOVAcouldbetransformedintoaworkflowinLearnSpheresothatotherresearchersandpractitionerscanfindanswerstotheaforementionedquestionintheirowndatasets.
Wehopethatsuchaworkflowwillleadtomoreandbetterresearchintothisimportantquestion,aswellasinterestingnewfindingsforboththeeducationaldataminingandlearningsciencescommunities.
Keywordssequencing,ordering,IntelligentTutoringSystems,LearnSphere,DataShop,workflow.
1.
INTRODUCTIONHowtosequenceeducationalactivitiesisanimportantpedagogicalquestion[12].
Muchoftheexistingworkonsequencingactivitiesconsistsoftheoreticalanalyses[2,4,7]andempiricalstudies[1,13,5,11].
Whileempiricalstudiescanhelpaddressquestionsthatcomparetwoorthreedifferentwaystosequenceacurriculum(e.
g.
,whethertopicsshouldbeblockedorinterleaved),itcannoteffectivelyscaletoanalyzingthemyriadofpotentialsequencesthatcouldbeconsidered.
However,educationaldatamining(EDM)techniquescanenableonetosimultaneouslystudydifferenttypesofsequencesbasedonpastdata.
Werecentlyproposedonesuchmethod—SequencingConstraintViolationAnalysis(SCOVA)—forcomparingtheefficacyofdifferentsequencingconstraintsgivenadatasetthatisrichinthevarietyofsequencesitexplores[3].
SCOVAcanbeusedtoanalyzeawidevarietyofsequencingconstraints,suchasprerequisiterelationships,constraintsonwhendifferentlearningmechanismsshouldbeintroduced,blocking,interleaving,andspiraling.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,includingintelligenttutoringsystems(ITSs),aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorinwhatorderlearningmechanismsshouldbesupported).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,SCOVAwillmostlikelynotbeusedforanyofthesepurposesifitjustremainsinapaperthatafewresearchersmight,atbest,readandcite.
Rather,itsbenefitwilllikelyonlyoutlivetheconfinesofaone-offEDMpaperifitisreleasedasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Ifreleasedassuchaworkflow,SCOVAcanalsointroduceresearcherswhomaynothaveotherwiseconsideredthequestionofhowactivitiesshouldbesequencedintheirlearningenvironmentstofindanewfoundinterestinthisarea,whichwebelieveisbecomingincreasinglyimportanttoboththelearningsciencesandeducationaldataminingcommunities.
2.
WORKFLOWMETHOD2.
1DataInputsSCOVAisapplicabletodatasetswithsubstantialvariabilityinthetypesofactivitysequencesthatstudentscomplete.
Thisvariabilityistypicalofmanydatasets,includingonesthatincluderandomnessinhowproblemswerepresentedtostudents(e.
g.
,[9]),oneswhereadaptivepolicieswereusedforproblemselectionresultinginsequencesthatvaryfromstudenttostudent(e.
g.
,[10]),andoneswherestudentsareabletodochoosewhichproblemstoworkonthemselves(e.
g.
,[8]).
TheworkflowcanworkwithdatasetsinthePSLCDataShopformat.
GiventhatSCOVAisaverygeneral-purposemethod,whichcanbeusedtoanalyzehowawidevarietyofsequencingconstraintsimpactpotentiallydifferentmeasuresofstudentperformance(e.
g.
,within-tutorperformance,posttestscores,learninggains,timeontask,etc.
),itmaypotentiallyneedtoutilizeavarietyofthecolumnsinaDataShopdataset.
However,forsimplicitywewilldescribeaversionofSCOVAthatislimitedtoanalyzingsequencingconstraintsthatmayonlydependonwithin-tutorcorrectnessandpropertiesoftheactivitiespresentedtostudentsandcanonlymeasuretheimpactwithrespecttowithin-tutorperformanceandfunctionsofpretestandposttestscores(suchaslearninggains).
Infull,SCOVAneedsthreeinputfiles:1.
TheDataShoptransaction-levelfile.
Foreverystepinatransaction-leveldataset,SCOVAneedstoknowtheproblemnameandwhetherthestepwasansweredcorrectlyornot.
2.
Amappingofeveryproblemnametocategoriestowhichtheproblembelongs.
Forexample,whenusingSCOVAonourfractionsITS[3],welabeledeachproblemwithoneofthreetopiclabels(makingandnamingfractions,fractionequivalenceandordering,andfractionaddition)aswellasoneofthreeactivitytypescorrespondingtolearningmechanismsfromtheKnowledge-Learning-Instruction(KLI)framework(sense-making,inductionandrefinement,andfluency-building)[6].
Thesecategorylabelswillthenbeusedasthebuildingblocksofsequencingconstraints,asexplainedinSection2.
2.
3.
Afilethatgivesthepretestandposttestscoreforeachstudent.
2.
2WorkflowModelTheworkflowbeginswiththeresearcherselectingdifferentsetsofsequencingconstraintsthattheywanttoanalyze.
Eachsequencingconstraintcanbeselectedbyfirstchoosingacategory(e.
g.
,topicsoractivitytype)andthenselectingapatternthatcorrespondstothesequencingconstraint.
Thepatterncantakeononeofthreeforms:1.
Specifyingaparticularsequence(e.
g.
,ABCABCABC,whichmaycorrespondtointerleavingdifferentactivitytypesortopics).
2.
SpecifyingthatastudentshouldbeexposedtoaproblemwithlabelAbeforeaproblemoflabelB(e.
g.
,astudentshouldbeshownanumberlineproblembeforebeingshownafractionequivalenceproblem)3.
SpecifyingthatastudentshouldhavereachedsomeperformancethresholdonaproblemwithlabelAbeforeaproblemwithlabelB(e.
g.
,astudentshouldhave95%accuracyonfractionequivalenceproblemsbeforebeingexposedtofractionaddition)Theresearchercanselectasmanysequencingconstraintsofthethreeformsabove.
Thenforeachpossiblepermutationofcategorylabels(e.
g.
,A=fractionequivalence,B=fractionaddition,C=namingfractions),SCOVAcomputesascoreforhowwelleachstudent'ssequenceinthedatasetmatchesthegivensequencingconstraints.
Thescoreistheproportionofproblemsinthetrajectorywhereasequencingconstraintwasviolated.
SCOVAthenlearnsalinearregressionmodelthatusesthedegreetowhichastudentviolatesaparticularsetofsequencingconstraintstopredictsomechosenoutcomevariable(i.
e.
,somemeasureofwithin-tutorperformanceorsomefunctionoftheposttestandpretestscores).
Noticethatifthemodelhasanegativecorrelationthenthatimpliesthemoreastudentobeysaparticularsequencingconstraint,thebetterthatstudentlearns/performsinthetutoringsystem,i.
e.
negativecorrelationsareindicativeofbeneficialsequencingconstraints.
ThefinalstepofSCOVAistocomparethemodelfitsfordifferentsetsofsequencingconstraintstoguidethepractitioner/researchertowhichsequencingconstraintshavethelargestpositiveimpactonstudentlearning.
Formoredetailsonthemethodandparticularinstantiationsofsequencingconstraints,referto[3].
2.
3WorkflowOutputsTheprimaryoutputisatableofBICvaluesofmodelsforeverysetofsequencingconstraintsevaluated.
Thepractitionercanchoosefromasetofoptionshowtheywantthetableorganized.
Forexample,ifwewereevaluatingtheimpactofconstraintsoftheformtopicAshouldcomebeforetopicB,whichshouldcomebeforetopicCintandemwithconstraintsoftheformactivitytypeXshouldcomebeforeactivitytypeY,whichshouldcomebeforeactivitytypeZ,thiscouldberepresentedina6-by-6tablewheretherowscorrespondtothedifferentpermutationsovertopicsandthecolumnscorrespondtothedifferentpermutationsoveractivitytypes.
(Iftherewasathirdcategoryofinterestwiththreedifferentlabels,suchassaywhetherthedifficultyleveloftheproblemwaseasy,medium,orhard,thentheworkflowcoulddisplaysixdifferenttables,oneforeachpermutationofdifficultylevels.
)Foranexampleofsuchatable,seeTable3in[3].
InadditiontoshowingBICvalues,thetablewillhighlightthosecellswheretheviolationofsequencingconstraintscorrelatesnegativelywithperformance/learning(againanindicatorthatthesequencingconstraintisbeneficialforstudentsratherthanharmful),andwilldesignatethemodelwiththelowestBIC(i.
e.
,thebest-fittingmodel).
TherewillalsobeatoggletodisplayotherquantitiesofimportanceinplaceofBIC,suchasthecoefficientsofthepredictorsinthemodels.
Inthecaseofevaluatingsequencingconstraintsoverasinglecategory(e.
g.
,onlyhowactivitytypesshouldbesequenced),theusercanchoosetodisplaythescatterplotsusedtofiteachmodelandthebest-fitlinesthemselves.
Theusercanalsochoosetocolor-codeeachpointofthescatterplotswiththevalueofsomefeature(e.
g.
,howmanyproblemsthatstudentreceived).
Thiscolor-codingoftheplotscanhelpidentifypotentialconfounds(e.
g.
,studentswhodomoreproblemsmighttendtoviolatefewerofasequencingconstraintandalsodobettersimplybecausetheydidmoreproblems).
Finally,theworkflowwillallowdoingexploratoryanalysestodetectotherpotentialconfounds.
Forexample,ifthesequencesinthedataweregeneratedaccordingtoadaptivepolicies,onepotentialconfoundisthatastudent'sperformanceaffectsthedegreetowhichsequencingconstraintsareviolatedinadditiontotheintendedcausaldirectionofthedegreetowhichasequencingconstraintisviolatedinfluencingthestudent'sperformance.
Toanalyzethepresenceofsuchaconfound,modelscanbelearnedwheretheoutcomevariableisthestudent'spretestscore(ratherthansayposttestscore);sincethepretestscorecomesbeforethestudents'useofthetutor,weknowthattheonlyreasonitwouldcorrelatewithviolationsofcertainsequencingconstraintsisiftheadaptivepoliciesdiscriminatedbetweenstudentswithdifferentamountsofpriorknowledge.
InusingSCOVAonourfractionstutor,wefoundthatwhilethisreversecausaldirectiondidexist,itwasseeminglynegligibleandactuallybiasingagainsttheconclusionsthatourresultssupport[3].
SuchaworkflowshouldallowuserstheabilitytodoexploratoryanalysesbeforemakingfirmconclusionsusingSCOVA.
3.
DISCUSSIONHavingaworkflowforanalyzingtheimpactofdifferentsequencingconstraintscanhaveanumberofbenefitsforboththeEDMandlearningsciencecommunities.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorhowlearningmechanismsshouldbesequenced).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,forSCOVAtobeusedinsuchafashion,itwilllikelyhavetobereadilyavailableasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Additionally,byhavingsuchaworkflowonLearnSphere,moreresearchersmaybeattractedtothequestionofhowtosequenceproblemsintheirlearningenvironmentofinterest.
Furthermore,ifLearnSpherealsoincludesworkflowsforothermethodsofanalyzingsequencingconstraintssuchas[9],moreresearchcanbedoneincomparingthesemethods.
Currentlywhensuchamethodispublisheditisnotwidelyadoptedeitherinpracticeorbyotherresearchers,anditisnotcomparedtomethodsthatsucceedit.
Byputtingallmethodsthatdosimilarstylesofanalysesononeplatform,LearnSpherecanleadtomoreproductiveresearch,includinghopefullybetterwaysofunderstandinghowweshouldsequenceeducationalactivitiesindifferentlearningenvironments.
4.
ACKNOWLEDGMENTSTheresearchreportedherewassupportedbytheInstituteofEducationSciences,U.
S.
DepartmentofEducation,throughGrantsR305A130215andR305B150008toCarnegieMellonUniversity.
TheopinionsexpressedarethoseoftheauthorsanddonotrepresentviewsoftheInstituteortheU.
S.
Dept.
ofEducation.
5.
REFERENCES[1]W.
Battig.
Intrataskinterferenceasasourceoffacilitationintransferandretention.
Topicsinlearningandperformance,pages131–159,1972.
[2]R.
E.
Clark,D.
Feldon,J.
J.
vanMerrienboer,K.
Yates,andS.
Early.
Cognitivetaskanalysis.
Handbookofresearchoneducationalcommunicationsandtechnology,3:577–593,2008.
[3]S.
Doroudi,K.
Holstein,V.
Aleven,andE.
Brunskill.
SequenceMatters,ButHowExactlyAMethodforEvaluatingActivitySequencesfromData.
InEDM,2016.
[4]J.
-C.
Falmagne,M.
Koppen,M.
Villano,J.
-P.
Doignon,andL.
Johannesen.
Introductiontoknowledgespaces:Howtobuild,test,andsearchthem.
PsychologicalReview,97(2):201,1990.
[5]S.
Kalyuga.
Expertisereversaleffectanditsimplicationsforlearner-tailoredinstruction.
EducationalPsychologyReview,19(4):509–539,2007.
[6]K.
Koedinger,A.
Corbett,andC.
Perfetti.
TheKnowledge-Learning-Instructionframework:Bridgingthescience-practicechasmtoenhancerobuststudentlearning.
CognitiveScience,36(5):757-798,2012.
[7]K.
Korossy.
Modelingknowledgeascompetenceandperformance.
Knowledgespaces:Theories,empiricalresearch,andapplications,pages103–132,1999.
[8]Y.
LongandV.
Aleven.
Supportingstudents'self-regulatedlearningwithanopenlearnermodelinalinearequationtutor.
InAIED,2013.
[9]Z.
A.
PardosandN.
T.
Heffernan.
Determiningthesignificanceofitemorderinrandomizedproblemsets.
2009.
[10]M.
A.
Rau,V.
Aleven,andN.
Rummel.
Complementaryeffectsofsense-makingandfluency-buildingsupportforconnectionmaking:AmatterofsequenceInAIED,2013.
[11]A.
RenklandR.
K.
Atkinson.
Structuringthetransitionfromexamplestudytoproblemsolvingincognitiveskillacquisition:Acognitiveloadperspective.
Educationalpsychologist,38(1):15–22,2003.
[12]F.
E.
Ritter,J.
Nerb,E.
Lehtinen,andT.
M.
O'Shea,editors.
Inordertolearn:howthesequenceoftopicsinfluenceslearning.
OxfordUniversityPress,2007.
[13]D.
RohrerandK.
Taylor.
Theshufflingofmathematicsproblemsimproveslearning.
InstructionalScience,35(6):481–498,2007.

ucloud国内云服务器2元/月起;香港云服务器4元/首月;台湾云服务器3元/首月

ucloud云服务器怎么样?ucloud为了扩大云服务器市场份额,给出了超低价云服务器的促销活动,活动仍然是此前的Ucloud全球大促活动页面。目前,ucloud国内云服务器2元/月起;香港云服务器4元/首月;台湾云服务器3元/首月。相当于2-4元就可以试用国内、中国香港、中国台湾这三个地域的云服务器1个月了。ucloud全球大促仅限新用户,国内云服务器个人用户低至56元/年起,香港云服务器也仅8...

iHostART:罗马尼亚VPS/无视DMCA抗投诉vps;2核4G/40GB SSD/100M端口月流量2TB,€20/年

ihostart怎么样?ihostart是一家国外新商家,主要提供cPanel主机、KVM VPS、大硬盘存储VPS和独立服务器,数据中心位于罗马尼亚,官方明确说明无视DMCA,对版权内容较为宽松。有需要的可以关注一下。目前,iHostART给出了罗马尼亚vps的优惠信息,罗马尼亚VPS无视DMCA、抗投诉vps/2核4G内存/40GB SSD/100M端口月流量2TB,€20/年。点击直达:ih...

HostYun 新上美国CN2 GIA VPS 月15元

HostYun 商家以前是玩具主机商,这两年好像发展还挺迅速的,有点在要做点事情的味道。在前面也有多次介绍到HostYun商家新增的多款机房方案,价格相对还是比较便宜的。到目前为止,我们可以看到商家提供的VPS主机包括KVM和XEN架构,数据中心可选日本、韩国、香港和美国的多个地区机房,电信双程CN2 GIA线路,香港和日本机房,均为国内直连线路。近期,HostYun上线低价版美国CN2 GIA ...

EDM为你推荐
软银支付烧钱,玩家多,生态复杂,印度的移动支付大战有多火爆p图软件哪个好用什么P图软件好用?输入法哪个好用输入法哪种比较好用?华为p40和mate30哪个好荣耀30pro和华为p40对比。,哪个更值得入手?浏览器哪个好用哪款浏览器好用等额本息等额本金哪个好房贷是等额本金划算还是等额本息划算炒股软件哪个好用玩股票哪个软件好?手机管家哪个好手机管家 用什么最好?看书软件哪个好推荐几个好用的手机看书软件网络机顶盒哪个好什么牌子的网络机顶盒好用?
php网站空间 长沙域名注册公司 linode日本 highfrequency 香港机房 私人服务器 腾讯云数据库 koss 宕机监控 哈喽图床 windows2003iso web服务器架设软件 qq数据库 dd444 日本bb瘦 秒杀预告 赞助 中国电信测网速 天翼云盘 上海联通宽带测速 更多