editions59ddd.com

R&DConnectionsNovember2004TestingandTimeLimitsBrentBridgeman,AmandaMcBride,&WilliamMonaghanTestingandtimelimits.
It'sanalmostinevitableunion—andforgoodreason,manywouldargue.
Imposingtimelimitsontestscanservearangeofimportantfunctions.
Timelimitsareessential,forexample,ifspeedofperformanceisanintegralcomponentofwhatisbeingmeasured,aswouldbethecasewhentestingsuchskillsashowquicklysomeonecantype.
Limitingtestingtimealsohelpscontainexpensesassociatedwithtestadministrations,suchaspayinghourlyfeesforproctorsinapaper-basedadministrationorforseattimeatcomputertestingcenters.
Butlimitingtestingtimetoodrasticallycanthreatenatest'svalidity,ortheabilityofthetesttoaccuratelyreflectwhatthetestwasdesignedtomeasure.
Thisisparticularlytrueifthetestisnotintendedtomeasurehowquicklythetesttakercananswerquestionsorifthetestingtimeissolimitedthatalargenumberofexamineestakingthetestcannotcompleteit;thatis,ifthetestis"speeded.
"Speedednessintestingreferstotheeffectthattimelimitshaveontesttakers'scores.
Whenatest'stimelimitsareconstrainedtothepointthatmosttesttakersdonothaveenoughtimetoconsiderandanswereachquestion,thetestissaidtobe"speeded.
"Atestisspeededtotheextentthatthosetakingitscorelowerthantheywouldhaveiftheyhadbeengivenanunlimitedamountoftimetocompleteit.
FortestssuchastheGREandCollegeBoard'sSAT,whichareintendedtomeasureskillsrelatedtoacademicabilityratherthantherateatwhichexamineescanwork,thespeedatwhichtesttakersanswerthequestionsshouldplayaminorrole,atmost,indeterminingtestscores(Briel,O'Neill,&Scheuneman,1993;Donlon,1984).
Consequently,timelimitsforsuchtestsshouldgivemosttesttakersenoughtimetofinishthetest,andamodesttimeextensionshouldhavearelativelysmalleffectonoveralltestscores(Bridgeman,Cline,&Hessinger,2003).
Whileit'spossiblethattimelimitscanaffectthescoresofalltesttakers,somehavesuggestedthatsuchlimitsmaydifferentiallyaffectfemaleandminoritytesttakers.
Someclaimthatthe"fast-paced,orspeedednature"oftheSATputsfemaletesttakersatadisadvantageoncertaintestsectionsbecausetheyapproachproblem-solvingdifferentlythantheirmalecounterparts—femaletesttakers,theysay,aremorelikelytoworkproblemsoutcompletely,toconsidermorethanonepossibleanswer,andtochecktheirwork(Becker,1990;Linn,1992).
Othershavenotedwhatseemstobeacommonbeliefamongtesttakersandtheirfamilies(andevenamongsomeschoolcounselors)thatgivingexamineesmoretimetocompleteatestcouldsubstantiallyimprovetheirscores.
Thishasraisedconcernsoverthepossibilitythatnondisabledstudentsmayattempttoobtainextended-timeaccommodations(whichETSprovidestoexamineeswithdocumenteddisabilitiesthatrequireadditionaltestingtime,suchaslearningdisabilities,Attention-Deficit/HyperactivityDisorder,orsightproblems),andthusgainaperceivedadvantageonstandardizedtests(Bridgeman,Trapani,&Curley,2003;Mandinach,Cahalan,&Camara,2002).
Butifevidencesuggeststhatextratimedoesnotimprovetesttakerperformance,studentswouldhavelittleornomotivationtomanipulatethesystemtoreceiveextratest-takingtimethatthey'renotentitledto.
AndtherewouldbelessListening.
Learning.
Leading.
reasontoflag1thescoresofstudentswhoweregrantedextendedtime,apracticethathasengenderedfiercedebatesinceitsimplementationdecadesago.
EffectofExtraTimeonSATTestScoresWithallthisinmind,theobviousquestionsseemtobe,whathappenswhentesttakersaregivenmoretimetocompleteastandardizedtestDotesttakers'scoresimprovewhentheyaregivenmoretimeAndifso,byhowmuchTobegintoanswerthesequestions,Bridgeman,Trapani,andCurley(2003)placedSATReasoningTestsectionswithafewernumberofquestionsintothestandard30-minutevariablesectionoftwonationaltestadministrations.
Thissectiondoesnotcounttowardthefinalscoresoftesttakers,butisusedtotryoutnewquestionsandtoensurethatscoresonneweditionsofthetestarecomparabletothoseonearliereditions.
Theresearcherscreatedthereducednumbersectionsbydeletingquestionsfromaverbalsectionthatcontained35questions,toproducetwosetsofforms,onewith27questionsandanotherwith23.
Thescoresonthe23questionscouldthenbecomparedtothescoresonthesame1"Flagging"referstothepracticebywhichadministratorsofstandardizedtestsplaceasterisksorothersimilarnotationsonthescorereportsofpeoplewithdisabilitieswhotakeexamsundercertainnonstandardconditions.
Theseconditionsusuallyinvolveanaccommodationonoramodificationtothetestandmayincludeprovidingpeopletoreadthetestinstructionsandquestionsaloud,large-printandBrailleformsofthetest,individualizedadministration,orextendedtime.
Accommodationsareintendedtoeliminateirrelevantsourcesofdifficultythatarerelatedtothedisabilitybutnottotheconstructbeingassessed.
It'sworthnotingthatthenumberofstudentsrequestingextratimehasgrownbyabout26percentoverthepastfiveyears(Camara,Copeland,&Rothschild,1998).
It'salsoimportanttonotethat,asofOct.
1,2001,ETSnolongerflagsscoresofteststhatwereadministeredunderanaccommodationofextendedtime.
23questionsinthesectionscontainingthe27or35questions.
Thiswasdoneforboththemathandtheverbalsectionsofthetest.
AscanbeseeninFigures1,2,3,and4,theresearchersfoundthatallowingmoretimeperquestion(theequivalentoftime-and-a-half)hadminimalimpactonverbalscores,producinggainsoflessthan10pointsonthe200-800SATscale.
Infact,inthefirststudy,scoresforthelowerabilitygroup(thosewhoscoredbelow400)actuallydecreasedwithextratime.
TheseresultssuggestthattheSATverbalsectionisonlyslightlyspeeded.
Themathsectionappearstobemorespeededthantheverbalsection,butnothighlyspeeded:Theequivalentoftime-and-a-halfraisedscoresabout20points,althoughtheincreasewassomewhatgreater(17-26points)forhigherabilitystudents(abilitylevel>600).
Forbothsections,increasingthetimetendedtobenefithigh-scoringstudentsmorethanlower-scoringstudents,withextratimecreatingnoincreaseinscoresforstudentswithSATscoresof400orlower(abilitylevel600>60030-Items30-Items25-Items25-Items35040045050055060065070060025-Items20-Items17-ItemsAbilityGroupsStudy135040045050055060065070060025-Items20-Items17-ItemsFigure3.
Meanscoreson17M1itemswithstandardtiming(embeddedina25-itemsection),andwithtwolessspeededconditions(embeddedina20-itemsectionandasacomplete17-itemsection).
AbilityGroupsStudy235040045050055060065070060025-Items22-Items35040045050055060065070060025-Items22-ItemsFigure4.
Meanscoreson22M2itemswithstandardtiming(embeddedina25-itemsection),andwithalessspeededcondition(acomplete22-itemsection).
Source:Bridgeman,Trapani,&Curley,2003.
AbilityGroupsStudy1AbilityGroupsStudy235040045050055060065070070060035-Items27-Items23-ItemsAbilityGroupsStudy1Figure1.
Meanscoreson23V1itemswithstandardtiming(embeddedina35-itemsection),andwithtwolessspeededconditions(embeddedina27-itemsectionandasacomplete23-itemsection).
35040045050055060065035-Items27-Items23-Items410-600>60060030-Items25-ItemsAbilityGroupsStudy1650600550500450400350AbilityGroupsStudy2Page3of6600Figure2.
Meanscoreson25M1itemswithstandardtiming(embeddedina30-itemsection),andwithalessspeededconditions(acomplete25-itemsection).
25-Items30-ItemsEffectofExtraTimeonQuantitativeandVerbalGREScoresAswiththeSAT,timelimitsfortheGREGeneralTestareintendedtobesetsothatmosttesttakerscancompletethetest.
Amodesttimeextension,then,shouldhavearelativelysmalleffectontestscores.
TheresultsfromtheSATstudy,however,cannotbeappliedtothecurrentcomputer-adaptiveGREGeneralTestbecauseofthecontentandtimingdifferencesofthetwotests,andbecauseofthedifferencesbetweencomputer-adaptivetesting(CAT)andpaper-basedadministration.
InaCAT,unlikepaper-basedtests,differentexamineesreceivedifferentsetsofquestions.
2UnlikemanyCATs,theGRECAThasafixednumberofquestionsandstricttimelimitsforeachsection,althoughitisnotintendedtobeaspeededtest.
ToinvestigatespeedednessandtheGRECAT,Bridgeman,Cline,andHessinger(2003)performedastudyinwhicharesearchsectionwasaddedtotheendofregularadministrationsoftheCATGRE.
VolunteerstookeitheraverbaloraquantitativeGREsectionwitheitherstandardtimingorone-and-a-halftimesthestandardtimelimit.
Toencouragemotivatedperformance,participantswereeligibleforacashpaymentiftheydidaswellontheexperimentalsectionastheydidontheoperationalsections.
2Incomputer-adaptivetesting,thecomputerselectstherangeofquestionsthatisappropriatetoeachtesttaker'sabilitylevel.
Testtakersreceiveasetofquestionsthatmeettestdesignspecificationsandgenerallyareappropriateforeachtesttaker'sperformancelevel.
Questionsarechosenfromalargepoolofpossiblequestionscategorizedbycontentanddifficulty.
(Thecontentandtypesofquestionsaresimilartothatfoundincomparablepaper-basedtests.
)Thecomputer-adaptiveteststartswithquestionsofmoderatedifficulty.
Asthecandidateanswerseachquestion,thecomputerscoresthequestionandusesthatinformation,aswellasthecandidate'sresponsestopreviousquestions,todeterminewhichquestionispresentednext.
Aslongasthetesttakerrespondscorrectly,thecomputertypicallyselectsanextquestionofgreaterdifficulty.
Incontrast,ifthetesttakeranswersaquestionincorrectly,thecomputertypicallyselectsanextquestionoflesserdifficulty.
Subsequentquestionsarepresentedbasedinpartonthetesttaker'sperformanceonpreviousquestionsandinpartonthetestdesign.
Inotherwords,thecomputerisprogrammedtofulfillthetestdesignasitcontinuouslyadjuststofindquestionsofappropriatedifficultyfortesttakersofallperformancelevels.
AsTables1and2show,resultsfromthisstudyindicatethatextratimehadaminimaleffectonoverallscores,addingonlyabout7pointstoverbalscoresand7pointstoquantitativescoresonthe200-800scorescale.
And,aswasthecaseintheSATstudy,scoresunderthedifferentconditionswerecomparableacrossgenderandethnicgroups,althoughquantitativescoreswereslightlyhigherforlowerabilityexamineeswhohadmoretime.
Note,however,thattherearesomeimportantdifferencesbetweentheSATandGRE.
TheSATsubtractsafractionofapointforeveryquestionthatisansweredincorrectly,sothatitisbettertoleaveaquestionunansweredthantogiveanincorrectanswer.
TheGRE,ontheotherhand,hasapenaltyforleavingquestionsunansweredattheend.
QuestionsontheSATarearrangedforthemostparttobecomesuccessivelymoredifficult.
Lowerabilitytesttakersaremorelikelytoguessandgiveincorrectanswerstothelattersetofquestions,resultinginanegativeeffectontheirscores.
However,thisisnottrueforsectionswithreadingpassages,whichmakeupthemajorityoftheverbaltest.
Orderofthoseitemsisdependentuponwherethetopicstheindividualitemsrefertoappearinthepassage.
OntheGRECAT,lowerabilitytesttakerswouldreceivequestionsatorclosetotheirabilityleveltowardtheendofthetest,lesseningtheirneedtoguess.
ImpactofTimeLimitsonComputer-AdaptiveTestsAsmentionedearlier,theGRECATisnotintendedtobeaspeededtest,buthasafixednumberofquestionsandsectiontimelimits.
Sowhathappenswhentimelimitsareimposedonteststhatgivedifferentquestionstodifferentexaminees,particularlyifquestionsthataresupposedtobeequallydifficulttendtohavesubstantialdifferencesinthetimeittakestoanswerthemBridgemanandCline(2000)foundthatsomeofthequestionsintheGRE'sanalyticalandPage4of6quantitativesectionscouldbeansweredmuchmorequicklythanothers.
Theresearchersalsonotedthatwhilesomeofthisvariationinresponsetimewasrelatedtothedifficultyofthequestions—moredifficultquestionstendedtotakelongertoanswerthanlessdifficultones—therealsowassubstantialvariationinthetimerequiredtoanswerquestionsofroughlythesamedifficultylevelandmeetingthesamecontentspecifications.
Table1SampleSizes,Means,andStandardDeviationsforResearchGREQuantitativeScoresTimingconditionStatisticStandard(45min.
)Extended(68min.
)Differencen3,9043,749M6646717SD125121Table2SampleSizes,Means,andStandardDeviationsforResearchGREVerbalScoresTimingconditionStatisticStandard(30min.
)Extended(45min.
)Differencen4,1974,098M4544617SD122120Source:Bridgeman,Cline,&Hessinger,2003.
Giventhesefindings,itseemedconceivablethatexamineesreceivingtime-consumingtests(i.
e.
,thosewhogetadisproportionatenumberofitemsthattakealonger-than-averagetimetoanswer)couldbedisadvantagedand,asaresult,receivelowerscorescomparedtotesttakerswhogetalesstime-consumingtest.
Yet,uponfurtherinvestigation,BridgemanandCline(2000)couldfindnoevidenceofimpactontotaltestscores.
Inarelatedstudy,however,BridgemanandCline(2004)didfindevidencethattesttakersontheanalyticalsectionoftheGREwereindeedaffectedbythiscombinationofconditions,whichresultedintesttakershavingtoguessonthefinalquestionsinordertofinishthetestbeforerunningoutoftime.
Testtakersatthehigherabilitylevelstendedtoguessmorethanthoseatthelowerabilitylevelsbecausethequestionsadministeredtohigherabilityexamineesweretypicallymoretime-consuming.
Sinceguessingincreasesthechancesofansweringitemsincorrectly(whichwouldloweratesttaker'sscore),thesefindingsindicatethatexamineeswhoareadministeredtestswithadisproportionatenumberoftime-consumingitemsarelikelytogetlowerscoresthanthoseofcomparableabilitywhoreceivetestscontainingitemsthatcanbeansweredmorequickly.
It'sworthnotingthattheGRE'sanalyticalsectionhasbeenreplacedbytwoessaypromptsthatassessanalyticalwritingskills.
Althoughthepotentialproblemnotedabovecontributedtothisdecision,itwasnottheonlyconsideration(Bridgeman&Cline,2004).
ImplicationsThisresearchindicatesthatindividualstakingeithertheSATortheverbalandmathsectionsoftheGRECAThavesufficienttimetoanswerthequestions.
Thesetestsarenotspeededtoanysignificantdegree,andgivingtesttakersmoretimetocompletetheseitemsdoesnotresultinsignificantscoregains.
Thescoregainsthatwereachieved(lessthan10pointsfortheverbalsectionandlessthan30pointsforthemathsection,ona200-800scale)wereextremelyminorandwouldcertainlynotmakeorbreakastudent'seducationalaspirations.
Moreover,scoregainswerenotconsistentacrossabilitylevels:Fortheseassessments,high-scoringtesttakerstendedtobenefitmorethanlower-scoringstudents,withextratimecreatingnoincreaseinscoresforstudentswithSATscoresof400orlower.
Furthermore,racial/ethnicandgenderdifferenceswereneitherincreasednorreducedwithextratime,challengingargumentsthattheso-called"speeded"natureoftheSATdisadvantagesminorityandfemaletesttakers.
TheseresultsshouldhelptoreducethemotivationforstudentswhoarenotdisabledtoPage5of6manipulatethesysteminanattempttoobtainunwarrantedextended-timeaccommodations.
Atthesametime,testusersshouldnotbeoverlyconcernedthatsomestudentsmightbegaininganunfairadvantageinthismanner,sinceanysuchadvantagewouldlikelybequitesmall.
StudieswereconflictingregardingwhetherornottheAnalyticsectionoftheGRECATwasspeeded.
Althoughthemostrecentstudy(Bridgeman&Cline,2004)makeastrongargumentthatthetestwasindeedspeeded,itisnowamootpointsinceETSnolongeradministersthissection.
However,theinformationobtainedinthisstudyshouldproveusefultodevelopingfutureCATswithstricttimelimits.
ReferencesBecker,B.
J.
(1990).
ItemcharacteristicsandgenderdifferencesontheSAT-Mformathematicallyableyouths.
AmericanEducationalResearchJournal,27,65-87.
Bridgeman,B.
(2004,April).
Speedednessasathreattoconstructvalidity.
PaperpresentedattheannualmeetingoftheNationalCouncilonMeasurementinEducation,SanDiego,CA.
RetrievedOct.
19,2004,fromtheETSWebsite:http://www.
ets.
org/research/dload/NCME_2004-Bridgeman.
pdfBridgeman,B.
&Cline,F.
(2004).
Effectsofdifferentiallytime-consumingtestsoncomputer-adaptivetestscores.
JournalofEducationalMeasurement,41,137-148.
Bridgeman,B.
,&Cline,F.
(2000).
Variationsinmeanresponsetimesforquestionsonthecomputer-adaptiveGREGeneralTest:Implicationsforfairassessment(ETSRR-00-7).
RetrievedOct.
19,2004,fromtheETSWebsite:http://ftp.
ets.
org/pub/res/researcher/RR-00-07-Bridgeman.
pdfBridgeman,B.
,Cline,F.
,&Hessinger,J.
(2003).
EffectofextratimeonGREQuantitativeandVerbalscores(ETSRR-03-13).
RetrievedOct.
19,2004,fromtheETSWebsite:http://ftp.
ets.
org/pub/res/researcher/RR-03-13-Bridgeman.
pdfBridgeman,B.
,Trapani,C.
,&Curley,E.
(2003).
EffectoffewerquestionspersectiononSATIscores(CollegeBoardReportNo.
2003-2).
RetrievedOct.
19,2004,fromtheCollegeBoardWebsite:http://www.
collegeboard.
com/research/pdf/rdcbreport20032web_23502.
pdfBriel,J.
B.
,O'Neill,K.
A.
,&Scheuneman,J.
D.
(1993).
GREtechnicalmanual.
Princeton,NJ:ETS.
Camara,W.
,Copeland,T.
,&Rothschild,B.
(1998).
EffectsofextendedtimeontheSAT:Reasoningtestscoregrowthforstudentswithlearningdisabilities(CollegeBoardReportNo.
98-7).
RetrievedOct.
19,2004,fromtheCollegeBoardWebsite:http://www.
collegeboard.
com/research/pdf/rr9807_3912.
pdfDonlon,T.
F.
(Ed.
).
(1984).
TheCollegeBoardtechnicalhandbookfortheScholasticAptitudeTestandAchievementTests.
NewYork:CollegeEntranceExaminationBoard.
Linn,M.
C.
(1992).
Genderdifferencesineducationalachievement.
InSexequityeducationalopportunity,achievement,andtesting:Proceedingsofthe1991ETSInvitationalConference(pp.
11–50).
Princeton,NJ:ETS.
Mandinach,E.
,Cahalan,C.
,&Camara,W.
(2002).
Theimpactofflaggingontheadmissionprocess:Policies,practices,andimplications(CollegeBoardReportNo.
2002-2).
RetrievedOct.
19,2004,fromtheCollegeBoardWebsite:http://www.
collegeboard.
com/research/pdf/02595020604txtcvr_11433.
pdfR&DConnectionsispublishedbyETSResearch&DevelopmentEducationalTestingServiceRosedaleRoad,19-TPrinceton,NJ08541-0001SendcommentsaboutthispublicationtotheaboveaddressorviatheWebat:http://www.
ets.
org/research/contact.
htmlCopyright2004byEducationalTestingService.
Allrightsreserved.
EducationalTestingServiceisanAffirmativeAction/EqualOpportunityEmployer.
EducationalTestingService,ETS,andtheETSlogoGraduateRecordExaminations,andGREareregisteredtrademarksofEducationalTestingService.
CollegeBoardandSATareregisteredtrademarksoftheCollegeEntranceExaminationBoard.
SATReasoningTestisatrademarkoftheCollegeEntranceExaminationBoard.
Listening.
Learning.
Leading.
Page6of6