Consensordelphi教程

TheMillenniumProjectFuturesResearchMethodology—V3.
0THEDELPHIMETHODbyTheodoreJ.
GordonI.
HistoryoftheMethodII.
DescriptionoftheMethodIII.
HowtoDoItIV.
StrengthsandWeaknessesoftheMethodV.
FrontiersoftheMethodVI.
SamplesofApplicationsBibliographyTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethodiiAcknowledgmentsSomecontentsofthisreporthavebeentaken,insomecasesverbatim,frominternalpapersofTheFuturesGroupwiththeirpermission.
ThesepaperswerewrittenbyJohnG.
Stover,TheodoreJ.
Gordon,andothersdescribingtheDelphimethodanditsapplications.
Themanagingeditoralsogratefullyacknowledgesthecontributionsofreviewersofthedraftofthispaper:Dr.
IanMilesofTheProgrammeofPolicyResearchinEngineeringScienceandTechnology,intheUnitedKingdom;Dr.
BrianFreeoftheFuturesEnvironmentCouncilofAlberta,Canada;Dr.
MikaMannermaaoftheFuturesResearchCentreatTurkuSchoolofEconomics,Turku,Finland;Dr.
HaroldA.
LinstoneofPortlandStateUniversity,UnitedStates;andDr.
PeterBishopoftheUniversityofHouston,intheUnitedStates.
Andfinally,specialthankstoElizabethFlorescuandNedaZawahriforprojectsupport,BarryBluesteinforresearchandcomputeroperationsandSheilaHartyforediting.
TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod1I.
HISTORYOFTHEMETHODThemodernrenaissanceoffuturesresearchbeganwiththeDelphitechniqueatRAND,theSantaMonica,California,"thinktank"intheearly1960s.
ThequestionsofRANDthinkers,atthetime,primarilydealtwiththemilitarypotentialoffuturetechnologyandpotentialpoliticalissuesandtheirresolution.
Theforecastingapproachesthatcouldbeusedinsuchapplicationswerequitelimitedandincludedsimulationgaming(individualsactingoutthepartsofnationsorpoliticalfactions)andgeniusforecasting(asingleexpertorexpertpaneladdressingtheissuesofconcern).
Quantitativesimulationmodelingwasquiteprimitive,andcomputersthatwouldultimatelymakesuchquantitativetechniquespractical,werenotyetcapableenough.
TheRANDresearchersexploredtheuseofexpertpanelstoaddressforecastingissues.
Theirreasoningwentsomethinglikethis:experts,particularlywhentheyagree,aremorelikelythannon-expertstobecorrectaboutquestionsintheirfield.
However,theyfoundthatbringingexpertstogetherinaconferenceroomintroducesfactorsthatmayhavelittletodowiththeissueathand.
Forexample,theloudestvoiceratherthanthesoundestargumentmaycarrytheday;or,apersonmaybereluctanttoabandonapreviouslystatedopinioninfrontofhispeers.
Aswithnormalthinkers,thegive-and-takeofsuchface-to-faceconfrontationsoftengetsinthewayofatruedebate.
Oneofthelittleknownin-houseresearchprojectsundertakenbyRANDatthetimeinvolvedcombiningopinionsofhorse-racinghandicappers.
Thesepeople,afterall,aresupposedlyexpertsintheirfield.
Furthermore,theiropinionsaboutthefuture(theoutcomeofhorseraces)arepublisheddailyandcanbecheckedagainstrealitywithin24hours.
Soaprojectwasimplementedtodeterminejusthowtocombinehorse-raceforecastsbydifferentexpertstoimprovethelikelihoodthatthecompositeopinionwasbetterthananysingleexpert.
TheworkontheDelphimethodfollowed.
OlafHelmer,NicholasRescher,NormanDalkey,andothersatRANDdevelopedtheDelphimethod,whichwasdesignedtoremoveconferenceroomimpedimentstoatrueexpertconsensus.
Thename,ofcourse,wasdrawn(humorously,theythought)fromthesiteoftheGreekoracleatDelphiwherenecromancersforetoldthefutureusinghallucinogenicvaporsandanimalentrails.
Theybeganfromaphilosophicalbaseandaskedinitially,"justhowmuchcouldbeknownaboutthefuture"(HelmerandRescher,1959)TheDelphimethodwasdesignedtoencourageatruedebate,independentofpersonalities.
Anonymitywasrequiredinthesensethatnooneknewwhoelsewasparticipating.
Further,toeliminatetheforceoforatoryandpedagogy,thereasonsgivenforextremeopinionsweresynthesizedbytheresearcherstogivethemallequal"weight"andthenfedbacktothegroupasawholeforfurtheranalysis.
Theseaspects:anonymityandfeedback,representthetwoirreducibleelementsoftheDelphimethod.
ThisgeneralapproachhasbeenusedthousandsoftimessincethefirstpublishedDelphistudy,ReportonaLong-RangeForecastbyGordonandHelmer.
ThisRAND"bestseller,"publishedin1964,containedforecastsofscientificandtechnologicalbreakthroughsthrough2000andbeyond;the82panelistswhocontributedincludedIsaacAsimov,ArthurClarke,BertranddeTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod2Jouvenel,IthieldeSolaPool,DennisGabor,PeterGoldmark,HaroldGuetzkow,andWilliamPickering,tonameafew.
(GordonandHelmer,1964)Thetemptationtoreviewalloftheforecastswiththeadvantageofhindsightisgreat(onesuchreviewwasdonebyAmantin1970),butacursoryreviewshowsmanyforecaststhatwereontarget,suchas:economicallyusefuldesalinationofseawateroralcontraceptivesadventofultralightmaterialsautomatedlanguagetranslationtransplantingorgansmorereliableweatherforecastscentralizeddatabanksartificialorgansx-raylaserspsychotropicdrugsself-replicatingmoleculessyntheticproteinsfeasibilityofcontroloverhereditarydefectsTherewerebigmisses,too,including:controlledthermonuclearpowerbiochemicalgeneralimmunizationlimitedweathercontrolworldpopulationby2000lessthan6billionmannedlandingonMarsNevertheless,thestudyand,moregenerally,RAND'sinterestindevelopingsystematicmethodsforforecastingapparentlylegitimizedthefield.
SincethisfirstDelphi,literallythousandshavebeenperformedontopicsaswiderangingasthefutureofreligionandthefamily,tospaceexploration.
Themethodcontinuestobeusedextensively.
InSeptemberof2008,areviewwasmadeoftheScopusdatabase(whichincludesarticlesfrom15,000peer-reviewedprofessionaljournalsfrom4,000publishers,proceedingspapers,andtradepublications)and105publicationswereidentifiedinresponsetothesearchterms"Delphistudy.
"Byfar,thelargestfieldofstudywhichemployedthemethodwashealthsciences.
AnincompleteGooglesearchon"DelphiStudies2008"producedresearchreportsandstudydescriptionsthatincludedthefollowingtopics,allpublishedinthefirstninemonthsof2008,listedheretoindicatetheextremelybroadscopeofrecentapplications:TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod3Asthmaindicators(refiningthelist)BackpaindefinitionsChestx-raysChronicanorexianervosaClinicalinformationsystemsElectronicrecordsmanagementGerontologyGISscienceHealtheducationHealthpromotionHIPAAcomplianceontheBattlefieldMedicalEvacuationprocess.
LibraryscienceLoweringCOemissionsfromdieselsMathematicalproblemsolving(inprimaryschool)Noxiousstimuli(neonatal)NursingeducationPalliativemedicineParentingandchildhealthPhysiotherapyPovertyalleviationPowersector-KeralaState,IndiaPsychiatryPublicrelationsresearchprioritiesSevereAcuteRespiratorySyndromeTechnologymanagementTwostudiesfromthissetaredescribedhereasrecentexamplesoftheuseofthemethod:Pare,G,C.
Sicotte,M.
Jaana,andD.
Girouard,"Prioritizingclinicalinformationsystemprojectriskfactors:ADelphistudy,"reportingonrisksassociatedwithimplementingaclinicalinformationsystem.
TheresearchinvolvedaDelphiamong21expertsinvolvedinCanadianCISprojectstoidentifyriskfactorsofclinicalinformationsystems.
Dionne,C.
E.
,etal.
,"Aconsensusapproachtowardthestandardizationofbackpaindefinitionsforuseinprevalencestudies,"inwhichaDelphistudywasperformedwith28expertsinbackpainresearchfrom12countries.
Thisworkwasdesignedto:"identifystandardizeddefinitionsoflowbackpainthatcouldbeconsistentlyusedbyinvestigatorsinprevalencestudiestoprovidecomparabledata….
[and]thestudyproduceddefinitions[thatcould]providestandardsthatmayimprovefuturecomparisonsoflowbackpainprevalencefiguresbyperson,placeandtimecharacteristics,andofferopportunitiesforstatisticalsummaries.
"TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod4II.
DESCRIPTIONOFTHEMETHODThisdescriptionwillbeillustratedbyahypotheticalexample:supposewewanttoestablishaforecastforthedatebywhichamannedMarslandingwouldoccur.
WiththeDelphimethod,expertsfromtherequireddisciplinesarefirstidentifiedandaskedtoparticipateintheinquiry.
Forthisexample,theexpertsmightincluderocketscientists,geologists,andbioscientistswhoareexpertsontheplanet,plannersfromspaceagenciesthatmightbeinvolvedinsuchaproject,andotherswhoexpressedanopinionthatmannedlandingsmightbeabadidea.
Duringtheinitialcontact,thenominatedpersonsaretoldabouttheDelphiandinvitedtoparticipate.
Theyareassuredofanonymityinthesensethatnoneoftheirstatementswillbeattributedtothembyname.
Thequestionsarerefinedbytheresearchersandpursuedthroughanumberofsequentialquestionnaires.
Inthefirstquestionnaire,participantsmightbeaskedtoprovidetheirjudgmentonadateatwhichamannedlandingmighttakeplace.
Theanalysiswouldidentifytherangeofopinionsaboutthedate.
Inasecondquestionnaire,therangewouldbepresentedtothegroup,andpersonsholdingopinionsattheextremesoftherangewouldbeaskedtoreassesstheiropinioninviewofthegroup'srangeandprovidereasonsfortheirpositions.
Forexample,areasonforalatemannedlandingmightbethatrobotlanderscandoallthatmencando,solittlereasonexiststospendthemoneyforhumanon-siteinvolvement.
Thesereasonswouldbesynthesizedbytheresearchersattheendofroundtwo;thesynthesizedreasonswouldformthebasisforthethirdquestionnaire.
Inthisthirdquestionnaire,thenewgroupjudgmentonadatewouldbepresentedtotheparticipants,alongwithreasonsfortheextremeopinions.
Eachmemberofthegroupwouldbeaskedtoreassesshisorherpositioninviewofthereasonspresented.
Theymightalsobeaskedtorefute,ifappropriate,theextremereasonswithanyfactsattheirdisposal.
Forexample,someonemightattempttorefutetherobotargumentbysayingthathumanpresencewillberequiredtoperformacertainclassofengineeringactivityortoconstructhabitatsforlatercolonization.
Inafourthandfinalround,theseargumentswouldbepresented,alongwiththeevolvinggroupconsensus,andareassessmentrequested.
Inasense,theDelphimethodisacontrolleddebate.
Thereasonsforextremeopinionsaremadeexplicit,fedbackcoollyandwithoutangerorrancor.
Moreoftenthannot,groupsofexpertsmovetowardconsensus;butevenwhenthisdoesnotoccur,thereasonsfordisparatepositionsbecomecrystalclear.
Plannersreviewingthismaterialcanmakejudgmentsbasedonthesereasonsandtheirownknowledgeandgoals.
Becausethenumberofrespondentsisusuallysmall,Delphisdonot(andarenotintendedto)producestatisticallysignificantresults;inotherwords,theresultsprovidedbyanypaneldonotpredicttheresponseofalargerpopulationorevenadifferentDelphipanel.
Theyrepresentthesynthesisofopinionoftheparticulargroup,nomore,noless.
ThevalueoftheDelphimethodrestswiththeideasitgenerates,boththosethatevokeconsensusTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod5andthosethatdonot.
Theargumentsfortheextremepositionsalsorepresentausefulproduct.
QuestionsincludedinaDelphimaybeofanysortsthatinvolvejudgment,including,forexample,thesizeofafuturemarket,whetherornottheCEOshouldreceivearaise,ortheproperpolicytoachieveagoal.
Inplanningapplications,thequestionsgenerallyareofthreetypes.
forecastsontheoccurrenceoffuturedevelopments.
Forecastsoffuturedevelopmentscallforanswersaboutwhenaneventisexpectedtooccuroraboutthefuturevalueofsomeparameter;desirabilityofsomefuturestate.
Questionsdealingwithdesirabilityaskforjudgmentsaboutwhetheraneventoughttooccur,andthebasisfortherecommendation;andthemeansforachievingoravoidingafuturestate.
Questionsdealingwithpolicyinvolvethetraditionalreporter'squestionsaboutimplementation:who,what,when,where,andhowmuchButtothissetwemustadd:towhatend.
Inotherwords,questionsaboutpolicyoughttobelinkedcloselytotheobjectivessoughtandthelikelihoodthatanypolicywill,infact,accomplishitsintendedgoals.
Thesethreetypesofquestionsmayrequiredifferentkindsofexperts.
Thelikelihoodquestionsmayinvolvehands-onexperienceandintimateknowledgeofthefrontiersofresearch.
Thedesirabilityquestionsmayinvolveamoral,political,orsocialdimensionquitedistinctfromthedisciplinaryexpertiseinvolvedinjudginglikelihood.
Thepolicyquestionmayinvolveknowledgeoftheartofthepossible.
InsomemodernapplicationsofDelphi:1.
Thequestionsrelatetothevalueofindependentvariablesthatareusedinquantitativesimulationmodels.
Inthisapplication,aconsensusisnotrequired;rather,ifdisagreementexistsaboutthevalueofanyvariable,theextremescanbetestedinquantitativemodelstodeterminewhetherornotthedifferencehasanyimportantsignificance.
2.
In-depthinterviewswithexpertshavebeenusedwithgreatsuccessasanalternativetoquestionnaires.
Inthisapproach,thesamekindsofexpertsarefirstidentified,invitedtoparticipate,assuredoftheiranonymity,and,inmostinstances,promisedareportbasedontheinterviewsequence.
Appointmentsaremadeattheconvenienceoftheinterviewees.
Interviewprotocolsarepreparedandtestedtoelicitjudgments.
High-levelstaffmembers,familiarwiththestudy'sobjectives,actasinterviewers.
Feedbackcanbeintroducediftworoundsofinterviewsareemployed;however,single-roundstudiesareusedmorefrequently.
Inthese,"feed-forward"isoftenemployed,presentingtorespondentsinformationaboutemergingconsensusderivedfromthepriorinterviews.
True,thisprocessintroducesdifferencesamongthevariousinterviews,butrememberthattheexerciseisnotdesignedtobestatisticallysignificantbutrathertoelicitideasthatcanbeimportanttosubsequentanalyses.
Expertin-depthinterviewsareanexcellentmeansofobtainingsuchideas.
Anadvantageofone-on-oneinterviewsisthattheyprovideflexibility,whichisabsentinTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod6questionnaires.
Forexample,aninterviewprovidestheopportunitytoprobethereasonsbehindtheforecasts,tosearchforbiasesintheforecasts,andtofollowuponunexpectedhintsdroppedbytheinterviewees.
3.
Forsomeapplications,groupmeetingsamongexpertshavenowbecomepractical.
Delphihaditsbirthinconcernaboutspuriousfactorsthatintrudeinface-to-facemeetingsamongexperts.
Newtechnologycanminimizethesefactors.
SomeDelphi-likestudieshavebeenperformedon-line(Shota,1993)andexperimentsdatebacktothe1970'swhenMurrayTuroffexperimentedwithearlycomputer-basedcommunicationstolinkexpertstogetherinnetworks.
(Turoff,1972).
MostrecentlyheandhiscolleagueshavedescribedaSocialDecisionSupportSysteminwhichlargegroupsofpeople(thousands)interactandvotedynamically(canchangevotesasinDelphi)onsocialissues.
(Turoff,Hiltz,Cho,Li,andWang,2002)Conferenceroomvotingmachineshaveproveduseful.
TheConsensor(AppliedFuture,Westport,CT)builtandrentsonesuchmachine.
OthersincludeOptionFinder(Minneapolis)andthePCVoter(TheFuturesGroup).
Typically,thesemachinesprovideeachparticipantwithasmallterminal,whichisconnectedthroughaserialcircuittoapersonalcomputer.
InthePCVoter,eachsmallterminalhastwoknobs.
Thefirstknoballowstheusertoprovidequantitativejudgmentsaboutaquestionposedbythemeeting'smoderatorforexample:"Whatistheprobabilitythatlimitedweathercontrolwillbeavailableby2020"Usingthesecondknob,participantscanprovideapercentageassessmentofconfidenceintheiranswer.
Thecomputer'ssoftwareintegratestheanswersoftheexpertsatthemeeting,discountingthosewhohavelowconfidenceintheiranswers,andprovidesonadisplayscreenahistogramshowingthedistributionofthegroup'sopinions.
Anonymityispreservedbecausetheinputsareprivateandunseenbyothers;thedisplayprovidesfeedback.
Itistruethatanonymityislostinanydiscussionoftheresults,butdiscussionisanoptionoftheparticipants.
Thisapproachhasbeenfoundquiteusefulwhenquickresultsareneededandtheissueis,"hot";thatis,thetopicofthestudyislikelytoevokestrongemotionalresponses.
Anexampleofa"hot"issueisthefutureofexecutivecompensationamongtheexecutivesandcompensationcommitteemembersofacorporation.
4.
Severalrecentstudieshaveplacedtheirquestionnaireson-linetoreducethecommunicationsdelaysandburdens.
ThesestudieshavesubstitutedthecommunicationstoolsavailableviaInternetfortheoldpaperformsthatweretransmittedbyfax,postalmail,ormorerecentlye-mail,Nevertheless,thestudiesstillemployedsequentialquestionnairesandpreservedtheDelphiprinciplesofanonymityandfeedback,5.
Recently,theMillenniumProjectoftheWorldFederationofUnitedNationsAssociationshasdeveloped"RealTimeDelphi"amethoddescribedindetailelsewhereonthisCDROM.
Briefly,inthisapproachaDelphi-likequestionnaireisaccessedbyrespondentsfromanon-lineInternetwebsite.
Theanswersaretypedinandonsubmittalarefedtoadatabaseontheserver.
Theaverageresponseiscomputedandfedbacktotheparticipantswhoarealsoaskedforthereasonsbehindtheiranswers.
Eachrespondentcanseethereasonsgivenbyallrespondentsandmayatanytimechangetheiranswers.
Thereisnoexplicitsecondround.
Whentherespondentcomesbacktothestudyinaminuteoraday,theoriginalinputformwiththeirprioranswersispresentedtohimorher.
Ofcourse,bythenothersmayhavecontributedjudgments,theaveragesTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod7ormediansmayhavechangedandgiventhisnewinformation,therespondentmaywishtorevisetheirprioranswers.
InthiswaytheDelphirequirementsofanonymityandfeedbackaremetandtheprocess,onceunderway,yieldsthedistributionofthegroup'sresponsesandreasonsfortheextremepositions.
Theprocesscanbesynchronousorasynchronous,andifimplementedonanInternetsitecaneasilyinvolveworldwidepanels.
.
III.
HOWTODOITThekeytoasuccessfulDelphistudyliesintheselectionofparticipants.
SincetheresultsofaDelphidependontheknowledgeandcooperationofthepanelists,itisessentialtoincludepersonswhoarelikelytocontributevaluableideas.
Inastatisticallybasedstudy,suchasapublicopinionpoll,participantsareassumedtoberepresentativeofalargerpopulation;inDelphi,non-representative,knowledgeablepersonsareneeded.
Sothefirstproblemishowtoselectpotentialparticipants.
Knowledgeablepersonsareusuallyidentifiedthroughliteraturesearchesforwhohaspublishedonthesubjectunderstudy,recommendationsfrominstitutions(e.
g.
,theWorldFutureSociety)andotherexpertsinaprocessknownas"daisychaining.
"Literaturesearchesnecessarilyresultinlistsofpeoplewhohavepublishedonthetopicofinterest;thisapproachmissespeoplewhomayhavesomethingtocontributebuthavenotpublished.
Recommendationsfrominstitutionssufferfromthesameshortcoming:therecommendationsarelimitedtoonlythosewhoareknowntotheinstitutions.
"Daisychaining"hasthepotentialdisadvantageofidentifyingcliques.
Onepossibilitythathelpsensurethattherequiredskillsarerepresentedistoformamatrixinwhichtherequiredskillsarelisted.
InthecaseoftheMarslandingquestion,forexample,thecolumnsmightbe:solarsystemplanetarygeology,rocketsystemdesign,robotics,datacollection,telemetry,etc.
Therowsofthematrixconsistofthenamesoftheprospectiveparticipants.
Thecellsarecheckedtoindicatethe"coverage"thatthenomineesareexpectedtoprovide.
Asfor"unknown"peoplewhoareoutsideofthenormallinesofcommunicationbutwhomaybeabletocontributenewandinnovativeideas,herearesomesuggestions:usebulletinboardssothatcontributorswhohavesomethingtosayinthisinformalenvironmentmayidentifythemselves;getrecommendationsfromuniversityprofessorsaboutbrightstudents;advertiseforparticipantsandqualifiedapplicantsthroughpreliminaryDelphi's.
Moststudiesusepanelsof15to35people,althoughinsomeapplicationshundreds,eventhousandsofrespondentshavebeeninvolved.
.
Thelengthoftheinvitationlistshouldanticipateanacceptancerateofbetween35and75percent.
Oncethelistofnomineesisformed,eachpersonshouldbecontactedindividually.
Formlettersshouldnotbeused.
Theinitialcontactmaybebytelephone,butlettersshouldconfirmtheTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod8invitation.
Theselettersshouldcontainadescriptionoftheproject,itsobjectives,thenumberofroundstobeincluded(orthetimecommitmentanticipated),thepromiseofanonymity,and,ifappropriate,aconfirmationofthepanelist'sacceptance.
Thenextstepistoformulatethequestions.
Thequestionsmustbesharpandanswerable.
Asmallpanelmightbeusedtohelpformulatethequestions.
Forexample,sayourquestionisapolicyDelphionnuclearproliferationandcentersonthequestion:WhateffectiveandpracticalmeansexisttopreventproliferationofnuclearweaponsThequestionwouldbesenttoasmallexpertpanel(10personsorso)workingwiththeresearchteam.
Essay-typeanswerswouldbepermittedatthisstage.
Theanswersmightinclude:Militaryintervention,greatlyenhancedcustomsoperations,newnonproliferationtreaties,spacesurveillance,etc.
Theresearchstaffwouldcollatetheanswersandformanon-essayquestionnaire.
Thequestionmightbe:Thefollowingmeanshavebeensuggestedforpreventingnuclearproliferation.
Pleaserateeachoftheseonascaleof1to10,with1beingtheleasteffectiveandpracticaland10beingthemosteffectiveandpractical.
Addothersuggestionstothelistifyouhaveanideathatmightbeaseffectiveandpracticalasthoselisted.
Ifyouthinkanyofthesemeansareextremelygoodor,attheotherextreme,counterproductive,pleaseprovideyourreasons.
Thisnon-essayquestionwouldserveasthebasisforthefirstquestionnaire.
Thequestionnairefirstwouldbetested,perhapsusingthesmalladvisorypanel.
Thetestingwouldincludeactuallyfillinginthequestionnaire.
Thistestisdesignedtofindflawsinthewaythequestionsareaskedandtofindanylurkingpossibilitiesformisinterpretation.
Oncequalifiedinthisway,thequestionnairewouldbesenttotheparticipants.
Thecoverletterwouldremindtheparticipantsabouttheobjectivesofthestudy,establishtheschedulefortheresponse,andincludethereturnaddress.
Themediathatarepracticalfortransmittingthequestionnairesandresponsesarepostalmail,fax,ande-mail.
Ourexperienceindicatesthataresponseratefrom40to75percentoftheparticipantscanbeanticipated.
ExceptforRealTimeDelphi,theturnaroundtimeisontheorderofweeks,nomatterwhatthemediumofcommunicationsis.
Inthisexample,respondentswouldprovidetwonumbersforeachoption,thefirstdepictingtheirjudgmentsabouteffectivenessandthesecond,practicalityoftheactionssuggested.
Theywouldbeaskedtoprovidereasonsfortheirjudgmentsandtosuggestmeansforimprovingeffectivenessandpracticalityaswellastostateotheralternativestheymighthaveinmind.
.
Theresearchstaffwouldcollatetheresults.
AfeedbackroundwouldbeusedtopresenttheTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod9results,thereasonsfortheextremepositions,andacallforreassessment.
Thereasons,inDelphistyle,wouldbeplacedinfrontoftheparticipantswhoansweredthefirstround.
Therespondentswouldbeaskedtoreconsidertheirformeranswersinviewofthereasonsfortheextremeopinions.
Questionsthathavequantitativeanswersor"checkthebox"responsesarealwayseasiertouse.
Whenthequestionsareinthisform,theycanbecollatedusingspreadsheetsoftware.
Infact,inoneapplication,thequestionnaireitselfwasonaspreadsheettransmittedbye-mail,andresponseswereentereddirectlyonthespreadsheetwhichwasreturnedtotheresearchteamasae-mailattachment.
However,itisnotalwayspossibletousethisapproachordesirabletoaskquestionsinthisform.
Forexample,wemaywantsuggestionsfromthepanelaboutpoliciesfordiminishingpopulationgrowthrate:here,theessenceisinthedetailthatthepanelmightprovide.
Inaddition,evenwhenquantitativeratherthannarrativeanswersarecalledfor,notesfromthepanelistsmaycontainthemostvaluableinformation:referencestootherpeople,pastexperiences,uncertainty,pointingtodata,etc.
Forthisreason,allquestionnairesmustbereadanddigestedbyseniormembersofthestudyteam.
Phrasingofquestionsisimportant.
Acommonmistakeistoincludetwoeventsinthesamequestion.
Example:"Whenwillbicyclesbemademostlyofplasticsandusedinurbantransportation"Differencesinthewaypeopleanswerthisquestionmaystemnotonlyfromtheirperceptionsaboutfutureusesofplasticinbicycles,butalsofromdifferencesintheirbeliefsaboutwhenbicycleswilllikelybeusedforurbantransportation.
Evensubtlechangesinwordingmayaffectanswers.
Forthisreason,itisdesirabletoconductatestoftheinstrument,notonlybyobtaininganswersfromatestgroup,butalsobydiscussingwiththatgrouptheirinterpretationsofthequestions.
KahnemanandTverskypointoutthattheframingofaquestionoftencontainsseedsoftheanswersitwillprovoke.
Presentedwiththechoicebetweentwotherapiesforafataldisease,simplystatingthecurerateintermsofsurvivalratherthanmortalitybiasesanswersinthatdirection.
Toavoidframingbiasonemighttestthesamequestionaskedindifferentwaysandcompareresults.
ThedatafromaDelphicanbedisplayedinseveralways.
Thegroupjudgmentshouldbebasedonthemedianratherthanthemean,sincesingleextremeanswerscan"pull"themeanunrealistically.
Furthermore,itisincumbentontheanalysttoshowthespreadofopinion,whichcanbedonegraphicallyornumericallybyshowingarangeofresponses,oftentheinterquartilerange(therangethatcontainstheanswersofthemiddle50percentoftherespondents).
Anexampleisshownbelow:TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod10Specialcareinpresentingresultsmustbetakenifthedistributionisunusual,e.
g.
bimodal.
OnequantitativefeedbackoptionofferedinRealTimeDelphiisadisplayofthepercentageornumberofresponsessortedintoquintilegroupsstretchingfromtheminimumtothemaximumresponse,asfollows:Avg1stGroupofAnswers2ndGroupofAnswers3rdGroupofAnswers4thGroupofAnswers5thGroupofAnswersTotalResponsesStandardDeviation46.
581410.
0to28.
01828.
0to46.
01346.
0to64.
01364.
0to82.
0282.
0to100.
06021.
5IVSTRENGTHSANDWEAKNESSESOFTHEMETHODDelphistudiesaredifficulttoperformwell.
Agreatdealofattentionmustbegiventothechoiceofparticipants;thequestionnairesmustbemeticulouslypreparedandtestedtoavoidambiguity.
Multi-roundstudiesrequireagreatdealoftime;inevitably,someparticipantswilldropoutduringtheprocess.
Therefore,othertechniquesforcollectingexpertjudgmentarewelcome.
ManyofthesetechniquesaredescribedelsewhereonthisCDROM.
TheprimarystrengthofDelphiisitsabilitytoexplore,coollyandobjectively,issuesthatrequirejudgment;aweaknessofDelphiistheeasewithwhichquestionscanbeaskedforwhichbettertechniquesexist.
Atoneextremearequestionsaboutthefutureforwhichfactualanswersexistandthusrequireminimaljudgment.
Forexample:"Givencontinuedtrendsinimmigration,birth,anddeathrates,howmanypeopleovertheageof70willbeintheUnitedStatesin2050"Thisquestionhasanumericalanswerthatcanandshouldbecomputed.
FactualquestionsshouldTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod11notbecandidatesforDelphi.
However,thequestionofwhetherornotcurrentpopulationtrendswillcontinueisamatterofjudgmentand,therefore,appropriateforDelphi.
Inshort,Delphiisapowerfultechniquewhenusedtoseekanswerstoappropriatequestions.
Suppose,forexample,thatwewanttoforecastthefuturesizeofthemarketforagivenproduct.
ADelphistudymightinvolvesalesandmarketingpersonnel,retailers,andexpertsinconsumerpreferences.
Thesepeoplemightbeaskedforadirectestimate;todosowouldrequirethattheysomehowintegrateallfactorsaffectingthemarket,suchaspricing,changingfashion,competition,consumerspending,etc.
Participantsmightfocusondifferentfactorsandaccountforthemdifferently;withoutadirectquestionaboutthemarketdeterminants,thesize-determiningfactorswouldremainhidden.
Therefore,abettertechniquewouldbetoaskfortherespondentstoidentifythefactorsimportanttofuturemarketsizeandtoestimateprobabilitiesandconsequenteffectonsales,weretheytooccur.
Then,usingthisinformationandaquantitativetechnique,suchasTrendImpactAnalysis,thesizeofthefuturemarketcouldbeestimated,notonlyinviewofthepastmarkettrendbutalsoinconsiderationoftrend-changingfactorsuncoveredbyDelphiparticipants.
Finally,aweaknessoftheDelphimethodisthetimethatittakes.
Asingleroundcaneasilyrequirethreeweeks;athree-roundDelphiisatleastathree-tofour-monthaffair,includingpreparationandanalysistime.
Delphihasitscritics.
FredWoudenberg,forexample,reportsthat"themainclaimofDelphi––toremovethenegativeeffectsofunstructured,directinteraction-cannotbesubstantiated.
"HefoundthatDelphidoesnotproducemoreaccurateanswersthanothermethods,andthatconsensusoccursasaresultofpressurebroughtonparticipantswhohaveextremeopinions.
Thesequestionsarewellworthexploringinmoredepth.
(Woudenberg,1991;Kastenetal.
,1993;andRoweG.
etal.
,1991).
GeneRoweandGeorgeWright(RoweandWright,2001)reviewedtheliteraturecomparingDelphitootherstructuredgroupproceduressuchasgroupsthathavebeeninstructedtoargueboththepositiveandnegativesideofaquestion,andgroupsthatusedastructuredformofinformationexchange.
Theyfoundthat"Giventheequivocalnatureofthesestudies,…thereappearstobenoclearcutrationaleforadoptinganyofthesetechniquesinpreferencetoDelphi.
"True,theDelphimethodmakesparticipantswithextremeopinionsworkharderthanothers.
Ifopinionsarenotstronglyheld,participantsmayswitchpositionsratherthanwritereasonsfortheirestimates.
Ontheotherhand,thosewithstrongopinionsstatewhy.
Theoriginalimpetusbehindthemethodwastoseekconsensus,becauseexpertconsensuswasbelievedmorelikelytobeaccuratethananindividualforecast.
Dalkey(1970)testedthispropositionusingobscurequestionswithknownanswersandfoundweaksupportfortheconcept.
Today,consensusislessimportantformanyinvestigatorsthanpreviously;nowausefulproductoftheDelphimethodiscrystallizationofreasonsfordisagreement.
Furthermore,Delphiisnowseenasnomoreorlessthanasystematicmeansofsynthesizingthejudgmentsofexperts––theaggregatejudgmentrepresentingakindofcompositeexpertcomposed,inthedomainofinterest,oftheexpertiseofallparticipants.
DoesthemethodproduceanaccurateviewofthefutureItisnomoreaccurate,probably,thanTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod12anyexpert,singleorcomposite.
Butsupposewewantedtoformascenariobasedonexpertviewsofwhatmightbepossible.
OrsupposeweneededajudgmentaboutwhetherornotwecouldmountamannedMarsmissionandifso,how.
Orsupposewewantedtoexploretherangeoffutureeventsthatcouldaffectpopulationgrowthorweaponryorwar.
NobetterwayexiststocollectandsynthesizeopinionsthanDelphi.
V.
FRONTIERSOFTHEMETHODThecentralquestionontheDelphifrontieris"Howcanquestionsbeaddressedtothepersonsmostlikelytoanswerthemwell"HerearesomethoughtsthatweredevelopedinthePhase1MillenniumFeasibilityStudy(GordonandGlenn,1993).
Self-RatedExpertise.
Ifthenumberofquestionsandparticipantsissmall,thenonemightsay"let'shaveeverybodyanswerallquestions.
"ThisapproachisessentiallytakeninmostDelphis.
Iftheanswersofallparticipantsareaccordedequalweight,theanalysisiseasy:forquestionsthatcanbeansweredinnumericalform,theanswersareoftensimplydisplayedasanaverage(orinstatisticalterms,suchasmedianandstandarddeviation).
Sincesamplesizesaresmall,thiskindofanalysiscangivetheappearanceofnumericalrespectabilitywhennonereallyexists.
Fornonnumericquestions,theparticipants'answerscanbedisplayedasaspreadofopinions.
Someresearchers(e.
g.
,DalkeyandHelmer)attemptedtointroducearefinement:self-ratedexpertise.
Thereasoningwentsomethinglikethis:anexpertoughttobebelievedmorethananon-expert;therefore,weoughttogiveanexpert'sopinionmoreweightthananon-expert.
Infact,weakevidenceexiststhatforobscurequestionswithknowableanswers(e.
g.
,seeDalkey,1970)experts,withoutuseofreferencebooksordatabases,areabletogivemoreaccurateanswersthennon-experts.
ButhowcanexpertsbeidentifiedaprioriDalkeyandothershaveusedvarioussystemsofself-rating;forexample,1.
Areyouanexpertinthisfield,workinginitdaily2.
Doyouworkinthisfieldoccasionally3.
Areyouknowledgeableaboutthisfieldthroughoccasionalprofessionalreading4.
Wouldyouclassifyyourselfasaninformedlayman5.
AreyouuninformedaboutthisfieldByassigning"points"foreachlevelofself-ratedexpertiseintheanalysisofthegroup'sresponse,answerscanbediscountedforlackofexpertisetoarriveatagroupopinion.
IfthisapproachweretobeusedinaDelphistudy,thenallquestionswouldbesenttoallparticipants.
Eachquestionwouldhaveappendedaself-ratingsection.
ThisapproachwouldTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod13provideaweighingyardstickbuthasseveraldisadvantages.
Expertsformanyofthequestionsposedmaynotexist;Evenifexpertsexist,shouldtheiranswerscarrymoreweightthannonexperts(Afterall,ifexpertscouldanswerthequestionsassociatedwithpoverty,forexample,whyhaven'ttheyalreadydoneso)Peoplewithgoodandusefulsuggestionscouldbedissuadedfromcontributingsincetheymighthavetoadmitaself-depreciatinglevelofignorance.
Lock-and-KeyApproach.
Inthisapproach,administratorswouldattempttomatchthecapabilitiesofparticipantswiththerequirementsofthequestions.
Eachquestionandeachrespondentis"profiled,"andquestionsareaddressedonlytothoserespondentswhoseprofilesmatch.
Respondentprofilescancomefromaquestionnairethatiscompletedbytherespondentatthetimeofjoiningthepanelandcanbeupdatedperiodically.
Theprofilewouldconsistofbest(orseveral)choicealternativesofthefollowingsort:1.
WhattermbestdescribesyouroccupationScientist______Politician_Physician_____etc.
Businessperson____Cleric_____Serviceworker________Engineer_Artist_Manufacturer________Teacher_Publisher_______Retailer_2.
Ifscientist,whatisyourdiscipline(Parallelquestionsforengineers,businesspersonsteachers,etc.
)ParticlephysicsGeneticsBiomedicineetc.
OrganicchemistryNuclearphysicsAstronomyMaterialsPsychologyEconomicsSocialSciencePoliticalscienceAgriculture3.
Wouldyouconsideryourselfa:GeneralistSpecialist4.
Areyouinterestedin(ordoyouhaveexperiencein):TechnologicalforecastingPoliticalprocessesUtopiasetc.
Value-relatedissuesPolicyanalysisSciencePolicyWorldaffairsInotherwords,therespondentswouldbeaskedtoidentifythemselveswithrespecttothesamedescriptorsthatwillbeusedtoprofilethequestions.
Beforeanyquestionnaireissentout,theTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod14profilewouldbecategorizedaccordingtothesamesetofdescriptorsthattherespondentsusedtodescribethemselves.
Then,throughamatchingprocessofthesortdescribedbelow,thedegreeofmatchbetweenaquestionandeachrespondentcouldbescored:DesiredQuestionAttributesWeightRespNo.
1RespNo.
2RespNo.
3RespNo.
4RespNo.
5Scientist501100Materials1001010Generalist301110SciencePolicy210100TechnFcsting110100SCORE31811130Inthissimulatedexample,supposethattheadministratorsfeltthatthequestioncouldbestbeansweredbyamaterialsscientist,whoconsideredhimselfageneralist,withexperienceinsciencepolicyandtechnologicalforecasting.
Noteachofthesedescriptorswasviewedasbeingequallyimportant,however.
Beingascientistwasweighedat5;beinganexpertinmaterialswasweighedat10;etc.
Supposefurtherthatdataexistforeachpotentialrespondentwithrespecttoeachattribute(fromthequestionnairecompletedatthetimetheyjoinedthepanel),soasimplescoringmatrixcouldbeconstructedthatassessedeachrespondentwithrespecttothedesiredattributes.
Bytakingweightedsums,eachrespondentcouldbeassignedascore.
Inthisexample,respondent1scored3,andrespondent2scored18.
Aftercompletingsuchanexercise,respondentscouldberankorderedbyscoreandthetopsetselectedasasubpaneltowhichthequestionwouldbeaddressed.
Thenumberofpeopleinasubpanelmightbekeptcommonacrossallquestions,determinedbybudgetconsiderations,orlimitedtosay90percentofthetoppossiblescore.
Ofcourse,ifthenumberofrespondentswaslarge,thisprocesswouldhavetobeautomated.
Theprogramwouldhaveaccesstoadatabaseoftherespondents'characteristics.
Itwouldcallforquestionattributesandweights,automaticallyscanthedatabase,scoreeachrespondentandpresentadifferentrank-orderedlistofrespondentsforeachquestion.
Theselectednumberofthesecouldthenbeaddressedautomatically.
FreeTextSearch.
Thisapproachissimilartothelock-and-keyapproachexceptthatitenvisionsamoreopenendedandfullyautomatedmeansofmatchingquestionrequirementstorespondentexpertise.
Supposethateachrespondentwasaskedtosubmitafreetextresumetotheprojectwhentheyjoined.
Theresumesmightbeconstrainedtoagivenlengthandfollowapredeterminedformatrequiring,forexample,thattherespondentsdescribetheircurrentandpastactivities,interests,publications,etc.
Adatabasewouldbeformedoftheseresumes.
Now,wheneachquestionisformulated,asetofkeywordswouldbeassociatedwiththequestion,orthesignificantwordsinthequestionitselfwouldbeviewedasakeywordset.
Asbefore,eachkeywordwouldbeassignedaweight.
Thedatabaseofresumeswouldbesearchedforthesekeywordsand,basedon"hits,"eachpotentialrespondentwouldreceiveascore.
TheyTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod15couldberankorderedasbefore.
NarrowingtheUniverse.
Inthisapproach,ratherthanaskingabouttherespondents'expertiseandinterest,weletthemselectquestionsofinterestandusetheirpastselectionstoguidetheroutingofquestionsinfutureinquires.
Suppose,afterarespondentjoinedthenetworktheyreceived,foratime,allquestions.
Theywould,however,beinstructedtoansweronlyquestionsofinteresttothemandwhichtheyfeltcompetenttoaddress.
Then,overtime,theadministratorscouldlearnwhichcategoriesofquestionseachrespondentincluded.
Whensufficientdatawereavailable,onlythetypesofquestionsofpastinterestwouldbesenttothem.
Thedanger,ofcourse,isthatthisapproachislimiting.
Ifthescanningprocessiscutofftooearly,thenrespondent/questionmatchesthatcouldhavebeenproductivemightbemissed.
Tominimizethispotentialdetriment,respondentsmightbesentbriefdescriptionsofotherquestionsnotincludedintheirquestionnairesandaskedifanyofthesemightbeofinteresttothem.
PastPerformanceScoring.
Herewedefineanexpertasapersonwhocanprovidecorrectanswerstodifficultquestions.
Wewantexpertstoanswerquestionsaboutthefutureonthenetwork;therefore,wewillfirstposequalifyingquestionsthatexpertsshouldgetright.
Answeringthesequalifyingquestionscorrectlyisthe"ticketofadmission"totherealinquiry.
Forexample,iftheadministratorswantedtomakeaforecastofthefuturevalueoftheGermanmark,thequalifyingquestionmightbe"Onyesterday'sdate,whatwasthevalueoftheGermanmarkvis-à-vistheU.
S.
dollar"Onlythosepeoplewhoansweredclosetocorrectlywouldbepresentedthefuture-orientedquestion.
Thisapproachhasobviousproblems.
Thetestquestionswouldhavetobedesignedwithmeticulouscaretobeappropriate.
Moreimportant,manyrespondentsmightfeelthatsuchatestoftheircompetenceisinappropriate.
Avariationonthisthemeistokeeparecordofeachrespondent's"battingaverage"bysubject;thatis,theratioofcorrecttototalforecastsmadeinthepast.
Importantquestionswouldbesenttothosewhohavedonewellinthepast.
Whileinitiallyattractive,thisapproachhasanumberofshortcomingsalso.
First,itwouldhavetobebuiltupovertime.
Differentrespondentswouldhaveadifferentnumberof"timesatbat.
"Finally,whetheraforecasthasindeedoccurredisoftendifficulttotell,eveninretrospect.
TheTreeApproach.
Inthisapproach,asequentialseriesofquestionsisaskedoftherespondents,eachrequiringamoredetailedknowledgeofthefield.
Atsomepointalongtheline,therespondentsself-disqualify.
Here'sanexample.
SupposeourquestionhastodowiththefutureoftheelectoralcollegesystemofelectionsintheUnitedStates:WilltheelectoralcollegesystemofelectionintheUnitedStateschange;ifso,towhatformandwhenTheadministratorsmightask:DomainQuestion:HaveyoueverthoughtmuchabouthowpresidentsareelectedintheUnitedStates(Yes,No)If"no,"nofurtherquestions.
TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod16QualifyingQuestion:Canyounamethreeotherwaysinwhichthismightbedone(Yes,No)If"no,"nofurtherquestions.
ActualQuestion:DoyouthinktheelectoralsystemintheUnitedStateswillchangewithinthenext20years(Yes,No)If"no,"thenask:WhynotIf"yes,"thenask:WhyTowhatformWhenThedisadvantageofthisapproachisitscomplexityanditsneedforformulatingprecisequestions.
Ifthiscouldbetolerated,itwouldcertainlyleadtoappropriatematchingofquestiontorespondent.
ThegeneralprinciplesofaDelphistudy,anonymity,andfeedbackofpriorroundinformationtothecurrentround,havebeenusedinseveralnovelwaysinrecentworkbytheMillenniumProject.
Somehighlightsoftheseapplicationsarelistedbelowtoillustratethewiderangeofpossibilitiesforthismethod.
Thequestionofrewards.
OneofthekeyprinciplesofDelphihasbeenanonymityofrespondents,atleastwithrespectto"whosaidwhat.
"Thesuccessesofpredictionmarkets(seethechapteronthismethodelsewhereonthisCDROM)raisetheissue:wouldabuilt-inrewardsystemimproveaccuracyAtonelevelthiscouldmeanallowingtherespondentstospecifythatatleastsomeoftheirspecifiedcontributionscouldbeattributedtothem.
Or,asinthecaseofpredictionmarkets,respondentsmightbegivenprizesorrewardsifitislaterfoundthattheyhaveproducedaccurateorotherwiseusefulforecasts.
Inpredictionmarketstherewardscanbepsychicwhenthecurrencyofthemarketis"play"moneyorrealwhenthecontractsareboughtandsoldwithrealmoneyorwhen"winners"aregivenprizesordonationstofavoritecharitiesaremadeintheirname.
AttheveryleastthepredictioncontractsofferameanstoencouragecontinuedparticipationthroughthesequentialDelphirounds.
AppliedtoRealTimeDelphi,arewardsystemcouldencouragefrequentrevisitstotheonlinequestionnaire.
Delphihasbeenusedinscenarioconstruction.
Delphiwasusedinscenarioconstructioninastudywhichdealtwithanti-terrorism.
Thestudybeganwitharequestforanti-terroristscenariosdistributedtolistservsoftheMillenniumProjectoftheAmericanCouncilfortheUnitedNationsUniversityandtheWorldFuturesStudiesFederation.
Thescenariossubmittedinresponsetothisrequestwereanalyzedtoidentifyandratepoliciesandactionsthatmightbeusefulincounterterrorismstrategies.
Thisworkwaspostedon-linewithafurtherrequestforcomments,modificationstoexistingscenarios,andaddedscenarios.
Thesubmittedscenariosandothersfromoutsidethiseffortwerereviewedtoidentifyactionsandpoliciesthatmightbeusefulincounterterrorismstrategies.
Thefifty-nineactions/policiesidentifiedwerethensubmittedtoaninternationalpanelforjudgmentsabouttheireffectiveness,plausibility,andunexpectedpotentialdownsiderisks.
(GlennandGordon,2002)AnotherexampleoftheuseofthetechniquewasmadebytheMillenniumProjectinformingTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod17normativescenarios.
MillenniumProjectparticipantsidentifiedandratednormsthatweretobeusedasanormativescenarioframework.
Inthefirstround,participantsselectedthefollowingtopfournormsaroundwhichtoformthescenario:environmentalsustainability,plenty,globalethics(theidentifiedandaccepted),andpeace.
Theothersinorderofpreferencewerehealth,freedom,universaleducationaccess,equity,preservationofthehumanspecies,enlightenment,excitingandmeaningfullife,self-actualization,longevity,everyonehaseverythingtheywant,andsecurity.
Anormativescenariowasdevisedaroundthesenorms;thescenariofocusedonactionstoaddresstheGlobalChallengesbeingtrackedbytheProject.
Theseactionsconnectedthepresentworldtothenormativefutureof2050.
Forthesecondround,ascenarioreviewpaneloflong-termnormative-orientedparticipantswasformed;theywereaskedtoreviewandimprovethedraftofthescenariotoillustrateoptimisticpossibilitiesforthefuture(GlennandGordon,2002)TheDelphiapproachhasbeenusedinconstructionofaStateoftheFutureIndex(SOFI).
In1999–2000,theGlobalLookoutPaneloftheMillenniumProjectwasaskedtoidentifyindictorsbywhichthestatusof15globalchallengescouldbemeasured.
Thesenominatedindicatorsweresubsequentlyevaluatedbythepanelintermsoftheiravailabilityandusefulness.
Theresults,plusareviewofotherindexstudies,weresubmittedtotheGlobalLookoutPanelin2001tocollectjudgmentsaboutpotentialindicatorsfortheSOFI.
Therespondentsprovidedjudgmentsaboutwhatthebest(norm)andworst(dystopian)statuswasfortheindicatorin2011.
Theyalsoratedtheimportanceofreachingthenormanddystopianstates.
Thecriteriaforassigningahighweighttoavariablewere:thenumberofpeopleaffected;thesignificanceoftheeffect;whethersomegroupsseemtobeaffecteddifferentially;thetimeoverwhichtheeffectwouldbefelt;andwhethertheeffectisreversible.
MillenniumProjectstaffworkedwiththevariablesidentifiedinthisquestionnaire,obtaining20years(wherepossible)ofhistoricaldatafromthemostauthoritativesources,andforecastingeachvariableusingatimeseriesapproach(andlater,usingTrendImpactAnalysis)toformaStateoftheFutureforecast.
(GlennandGordon,2002Delphiprincipleshavebeenusedinpolicystudies.
TuroffwroteaboutthisuseofDelphiasearlyas1970(Turoff,1970).
Asarecentexample,in2002-2003,theMillenniumProjectperformedastudyofroutestopeaceintheMiddleEast.
Inthisapplication,thefirstquestionnairelistedpre-conditionstopeaceandactionsthatcouldbetakentosatisfythepreconditions.
Thepanelwasaskedtoaddtothelistofactionsandtoevaluatethesetintermsofimportance,likelihoodofimplementation,andtheirpotentialtoevokeundesiredorunintendeddownstreamconsequences.
Extensivewrittencommentarywasgeneratedbyboththerespondentsandthosewhoelectednottoparticipate.
Thedatawereanalyzednotonlyfromthepanelasawholebutalsofromthestandpointoftheantagonists,usingsub-samplesofresponsesfromArabcountriesandIsrael.
Fromthisbodyofdata,theProjectformedasetofnormativescenariosthatbeganfrompointsofagreementandillustratedpathstopeace.
(GlennandGordon,2003)TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod18VI.
SAMPLESOFAPPLICATIONSTheDelphimethodcanbeusedinalmostanyforecastingapplication.
.
TheMillenniumFeasibilityProject,performedbytheUnitedNationsUniversityfortheU.
S.
EnvironmentalProtectionAgencyinearly1993,producedanapplicationofpotentialinterest.
ThisstudyincludedaDelphithatexaminedimportantfutureeventsandpoliciesthatcouldaffectworldpopulationgrowthandtheenvironmentinthenext25yearsandwillbeusedasanexamplehere.
TherespondentsforthisDelphihadself-identifiedthemselvesinapriorstudyasexpertorgreatlyinterestedinpopulationorintheenvironment;thislistwasaugmentedbyrecommendationsbythestudy'ssponsor(EPA),aliteraturescan,andrecommendationsofpeoplewhohadalreadyacceptedourinvitationtoparticipate.
Inthefirstround,76questionnairesweresentoutviaairmail,fax,ande-mail;42responseswerereceived.
Thesecondroundwenttothesamelist.
Ofthosewhocompletedthefirstround,25respondedtothesecondround.
Thequestionnaireswerefirstsenttoasmallplanningcommittee,largelyinopen-endedform;theresponsesfromthesmallergrouphelpeddefineamuchmoreextensiveformat.
Thequestionsdealtwithpasthistory(e.
g.
,"Whatdevelopmentsandtrendshavebeenimportantinshapingtoday'spopulationgrowthpatterns")andfuturedevelopments(e.
g.
,"DoyouthinkthesetrendswillcontinueWhatothernoveldevelopmentswilldeflectthetrendsinthefutureWhatwillbetheirconsequences").
Thedesignofthequestionswasstraightforward.
Thesubjectsofpopulationandenvironmentweredealtwithinseparatesectionsofthequestionnaire,buttheflowforeachwaslargelythesame.
Inthefirstround,questionsfocusedonpastforcesforchange,bothfromthestandpointofhistoricalimportanceandfutureinfluence.
Theresearchersprovidedthegroupwithaninitiallistandaskedthemtoextendthelistandprovidejudgmentsabouthistoricalandfutureimportance.
Theythenaskedaboutnewforcesforchangeandpotentialfutureunprecedentedevents;again,theresearchersprovidedalist,askedforotherideasthatmightextendthelist,andrequestedjudgmentsaboutthelikelihoodandimpactofthefuturedevelopments.
Inthecaseofpopulation,thequestionnairealsocalledfordirectestimatesaboutthefuturesizeofthepopulationandpopulationgrowthratesinseveralcountriesandregions.
Italsoaskedaboutfactorsthatmightbedifferentbetweendevelopedanddevelopingcountries.
Inthecaseoftheenvironment,thequestionnaireincludedanadditionalsectiondealingwiththeinteractionbetweenpopulationandtheenvironment.
Thesecondroundfedbackthenewlysuggesteditemstothepanelandrequestedjudgmentssimilartothosecalledforinthefirstround.
Italsoaskedforjudgmentsaboutpoliciesthatmightbeimplementedtoimprovethefuturesituation.
Thesepolicyquestionsfocusedontheoneortwofutureeventsthatwerejudgedtohavelowprobabilitybuthighfavorableimpact,iftheyweretooccur.
Thisquestionasked,ineffect,whatmightbedonetoimprovetheprobabilitiesoftheseevents.
Toillustratethenatureofthequestionsasked,thepopulationsectionofthesecondquestionnaireTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod19isreproducedbelow.
Thefullquestionnairesandtheresultsofthestudyareincludedinreference(GordonandGlenn,1993).
UNUMILLENNIUMPROJECTFEASIBILITYSTUDYDELPHIONENVIRONMENTANDPOPULATION—ROUND2QUESTIONNAIRE1.
POPULATIONSECTIONThefirstroundofthisDelphionPopulationandEnvironmentaskedthepaneltoestimatepopulationandgrowthratesforseveralregionstotheyear2018.
Table1belowliststhepanel'smedianresponses.
Theinter-quartilerangesforthepopulationestimatesareshowninparenthesesnexttothepopulationestimates.
(Theinter-quartilerangeisfrom25%abovethelowestto25%belowthehighestestimates;hence,50%oftheestimatesfallwithinthisrange.
)Forpurposesofcomparison,1992year-endpopulationandgrowthratesarelistedfirst(citiesare1990data).
Table1.
1992and25yearforecastsofpopulationfromPopulation&EnvironmentDelphiIRegion1992PopulationRatesMedianForecast(&Ranges)RatesWorld5.
4billion1.
7%8.
0billion(7.
0-8.
5)1.
5%DevelopedCountries1.
2billion0.
5%1.
4billion(1.
3-1.
4)0.
35%DevelopingCountries4.
2billion2.
0%6.
5billion(5.
7-7.
0)1.
8%Africa654million3.
0%1.
2billion(1.
0-1.
4)2.
5%China1.
2billion1.
3%1.
5billion(1.
4-1.
6)1.
0%India883million2.
0%1.
3billion(1.
2-1.
4)1.
5%UnitedStates256million0.
8%300million(300-320)0.
65%Brazil151million1.
9%230million(210-250)1.
6%Japan124million0.
3%130million(130-137)0.
2%Iran60million3.
3%%105million(100-120)3.
0%%%%%%%Mexico88million2.
3%%140million(128-140)2.
0%Tokyo18.
1million0.
9%21million(20-22)0.
5%MexicoCity20.
2million2.
9%32.
5million(30-35)2.
0%Inthefirstround,thepanelistswereaskedtoratesomeforcesthatledtothereductionoftheTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod20worldpopulationgrowthratefrom2.
06%inthelate1960stothe1.
7%thencurrentandtoassesshowtheseforcesmightchangeoverthenext25years.
Table2belowshowstheaverageofthepanel'sjudgments.
Inmakingtheseassessments,theyusedScaleAshownbelow.
ScaleAHistoricInfluenceFutureInfluence1=Veryimportant1=Greatlyincreasinginimportance2=Important2=Increasinginimportance3=Marginallyimportant3=Remainingthesameinimportance4=Unimportant4=Decreasinginimportance5=Counterimpact5=Nolongerafactor,ormixedTable2-Theimportanceofsomehistoricfactors(firstnumber)onglobalpopulationgrowthandpossiblefuturechangesinimportance(secondnumber)2.
02.
2Availabilityofinexpensive,simpleeffectivecontraceptives;2.
02.
1Familyplanningandpublichealthprograms2.
02.
8China'spopulationpolicy2.
12.
1Governmentpoliciesindevelopingcountriesthatencouragesmallerfamilies2.
12.
0Increasingnumberofyearsthatwomenattendschool2.
12.
3Risingincomesandthespreadofmiddleclassvalues2.
52.
4Riseof"woman'spower.
"QUESTION1.
1Weaskedthepaneltosuggestadditionalforcesthatmightberesponsibleforthehistoricchangesinpopulationgrowth.
Manynewsuggestionswerereceived.
RespondentswereaskedtoreviewthelistpresentedbelowandtoprovidetheirjudgmentsaboutthehistoricalandpossiblefutureimportanceoftheseforcesusingScaleA,bothhistoricallyandoverthenext25years.
.
1.
1.
1______Moveawayfromagriculturalsociety/primarysector1.
1.
2______DemonstrationbytheNorththatfewerchildrencanmeanmorewealth1.
1.
3______Legitimizationofcontraception1.
1.
4______Availabilityofmalecontraceptives1.
1.
5______DecreasingCatholicChurch'ssocialinfluence1.
1.
6______Increasingfuturisticorientation1.
1.
7______War,famine,disease,andpestilence1.
1.
8______Environmentaldeterioration1.
1.
9______Spreadofnewcommunicationsmedia(television,etc)TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod211.
1.
10_____Educationaboutrelationofenvironmentandpopulation1.
1.
11_____Decreasesininfantandchildmortality1.
1.
12______Improvedliteracybyimprovedchildren'sschoolingPopulationgrowthrateshaveremainedhighindevelopingcountries.
UsingtheScaleAabove,theywereaskedtojudgetheimportanceofhistoricalinfluencesandhowtheymightchangeinimportanceoverthenext25years.
Table3presentsthepanel'sestimates.
Historicalassessmentisthefirstnumber;futureassessmentisthesecondnumber.
Table3-Somereasonsforhighpopulationgrowthindevelopingcountries1.
63.
0Needofchildrenforsocialsecurity,tosupportparentsintheiroldage.
1.
82.
4Poverty2.
02.
7Lowlevelsofliteracy2.
22.
8Lackofinformationandaccesstocontraceptives2.
23.
1Highinfantmortality2.
83.
4GovernmentpoliciessupportinglargefamiliesQUESTION1.
2Inthefirstround,thepanelistswerealsoaskedtosuggestotherforcesthatmightaccountforpopulationgrowthratesremaininghighinmanydevelopingcountries.
Manynewsuggestionswerereceived.
TheywereaskedtoreviewthedevelopmentsandtoprovidejudgmentsabouttheirhistoricalandpossiblefutureimportanceusingScaleA.
1.
2.
1______Masculinityassociatedwithincreasingnumbersofchildren1.
2.
2______Closeadherencetoreligioustenetsthatleadtoavoidanceofcontraceptives1.
1.
3______LowlevelsofliteracyandlackofunderstandingecologicalviewofplanetEarth1.
2.
4______Discriminationagainstwomen(littleautonomyoreducation,andlackofsocialpower)1.
2.
5______Beliefbygovernmentsthatlargerpopulationsmeangreaterpoliticalstrength1.
2.
6______Ruralareasreceivelessattentionfrompopulationprogramsthanurbanareas1.
2.
7______Family-based,labor-intensiveeconomiesneedingchildren'sinput1.
2.
8______LiberalimmigrationpoliciesinrichercountriesInthefirstround,thepanelistswereaskedtoassessnewforcesandunprecedentedeventsthatmightinfluencepopulationgrowthinthefuture.
Theywereaskedforjudgmentsaboutthelikelihoodofoccurrenceandimpactsoverthenext25years.
Table4showstheaverageoftheTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod22panel'sresponsesabouttheeventsincludedinthefirstround.
Inmakingtheseassessments,theyusedScaleBshownbelow.
ScaleBLikelihoodEventualimpactonpopulationwithinofoccurrencethenext25yearsofgrowth1=almostcertain1=reducesgrowthrateby30%ormore2=likely2=reducesgrowthrateby5-30%.
3=evenor50/50chance3=noimpact.
4=unlikely4=increasesgrowthrateby5-30%.
5=almostimpossible5=increasesgrowthrateby30%ormoreTable4Likelihoodofoccurrenceofnewforcesorunprecedentedevents(firstnumber)thatmightinfluencepopulationgrowthandeventualimpactoverthenext25years(secondnumber)1.
82.
7Simpletestforidentifyingthesexofunbornchildren1.
93.
4Increasingsurvivalinmiddleageandearlyoldageduetocuringorimprovedtherapyforheartdisease,cancerandstroke2.
12.
3Simple,safe,effectivemalebirthcontrolpill2.
32.
4ChangesindeathratesduetospreadofAIDSindevelopingcountries2.
42.
6MassivestarvationofthescaleofSomaliaatleastonceeverythreeyears2.
52.
0Longterm(atleastoneyear)contraceptivesaswidelyusedandacceptedasbirthcontrolpillsaretoday2.
62.
4Newdeadlyviruses,includingAIDSmutations2.
82.
6Simplemethodforselectingsexatconception2.
92.
1Risingincomesinmostdevelopingcountries3.
32.
3ChangeintheVatican'sposition:useofcontraceptivesbecomesavailablewithoutlimit3.
42.
1TwochildrenperfamilybecomesthesocialnorminthemajorityofthedevelopingworldQUESTION1.
3Inthefirstround,thepanelistswereaskedtosuggestothernewforcesorunprecedentedeventsthatcouldinfluencepopulationgrowth.
Manynewsuggestionswerereceived.
Theywereaskedtoreviewthelistbelowandprovidetheirjudgmentsabouttheirlikelihoodandimpacts,iftheyweretooccuroverthenext25yearsusingScaleBinthefirstspaceinfrontofeachitemforthelikelihoodofoccurrenceandinthesecondspacefortheitem'sfutureimpactoverthenext25years.
1.
3.
1______Widespreaduseofrelativelycheapandeasywaystoaffecttheagingprocess,resultingindiminishedmortalityandextensionofthelifespanbyabout5years1.
3.
2______Increasingsterility,worldwide,by10%duetoenvironmentaldegradation,higherlevelofstress,andotherfactors1.
3.
3______Increasingimpotencyby10%duetoenvironmentaldegradation,higherlevelofstress,andotherfactorsTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod231.
3.
4______Publichealthprogramsdecreasemortalityofinfantandyoungadultsby5-10%1.
3.
5______3%ofbirthsvianewmethodsofimpregnationandprenataldevelopment("Outsidewomb"fertility,artificialinsemination,surrogatemotherhood,othersuchtechniques)1.
3.
6______Importantnegativechangesintheenvironment(e.
g.
,accumulationoftoxicwastes,failuresofmono-agriculturecrops,contaminationofdrinkingwater)resultinginincreasedmortality1.
3.
7______Doubling,worldwide,oftoday'slevelofmaleandfemalehomosexuality1.
3.
8______Basiceconomicneedsmetfor90%ofglobalpopulation(minimumacceptablehealthcare,food,andshelter)1.
3.
9______Successfulnewprototypehabitatsinoceans,coldregions,orinearthorbitstimulatespopularfrontierspiritandalternativestopreviousurbanizationpatterns1.
3.
10______20%increaseofpeopleactiveinreligionsthatencouragehigherfertilityQUESTION1.
4WiththeinformationprovidedbythepanelinroundI,weidentifiedtwopossiblefutureeventsthatwerejudgedtobeunlikely,butneverthelesspotentiallyeffectiveinreducingpopulationgrowthrates,iftheyweretooccur.
Bothhavebeenconsideredbeforeandoneisthefocusoflargeinternationalprograms.
Withoutrepeatingwhathasbeensaidandtriedmanytimes,weaskbelowforyoursuggestionsaboutnovelpolicyapproachesthatyouthinkmightbepractical,and,ifimplemented,improvetheprobabilityofthesedevelopments.
Pleasewriteyoursuggestionsinthespaceprovidedbelow.
Changethestatementsifyouwish.
Addotherpolicydomainsifyouwish.
Pleasebeasspecificaspossibleanduseseparatesheetsofpaperorelectronicspaceasyouneed.
1.
4.
1Novelpoliciesthatcouldleadtoasocialnormoftwochildrenperfamilythroughoutthemajorityofthedevelopingworld:1.
4.
2NovelpoliciesthatcouldleadtotheVatican'sacceptanceofcontraceptionwithoutlimit:1.
4.
3.
Otherpolicyareasofyourselectionandhowyouwouldaddressit:TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod24Toillustratethequalityoftheresponsesreceived,someoftheresultsofthisquestionnairearesummarizedbelow;acompleteanalysis-includingsummariesofanswerstoallquestions-appearsinGordonandGlenn,1993.
Usingthescaleforhistoricalinfluence,thepanelfoundthefollowingtendevelopmentsmostsignificantintheevolutionofworldpopulationgrowth.
TheTenMostImportantHistoricalFactors2.
0Availabilityofinexpensive,simpleeffectivecontraceptives;2.
0Familyplanningandpublichealthprograms2.
0China'spopulationpolicy2.
0Legitimizationofcontraception2.
0Decreasesininfantandchildmortality2.
1Governmentpoliciesindevelopingcountriesthatencouragesmallerfamilies2.
1Increasingnumberofyearsthatwomenattendschool2.
1Risingincomesandthespreadofmiddleclassvalues2.
2Moveawayfromagriculturalsociety/primarysector2.
3Spreadofnewcommunicationsmedia(television,etc.
)Further,thepaneljudged,usingthescaleforfutureinfluence,thatamongforcesalreadyinplay,thefollowingwouldbethetenwiththemostgreatlyincreasingimportanceoverthenext25yearsTheTenHistoricalForcesThatPromisetobeofIncreasingImportance1.
8Spreadofnewcommunicationsmedia(Television,etc.
)2.
0Increasingnumberofyearsthatwomenattendschool2.
0Environmentaldeterioration2.
1Governmentpoliciesindevelopingcountriesthatencouragesmallerfamilies2.
1Familyplanningandpublichealthprograms2.
1Improvedliteracybyimprovedchildren'sschooling2.
2Availabilityofinexpensive,simpleeffectivecontraceptives2.
2Legitimizationofcontraception2.
2Educationaboutrelationofenvironmentandpopulation2.
2DecreasesininfantandchildmortalityThequestionnaireincludedsimilarquestionsaboutdevelopingcountries:reasonsforhighpopulationgrowthinthesecountriesandthepotentialimportanceoftheseforcesoverthenext25years.
Hereisasummaryofthetopitems:TheTenMostImportantReasonsforHighPopulationGrowthinDevelopingCountries1.
6Needofchildrenforsocialsecurity,tosupportparentsintheiroldage.
1.
7Discriminationagainstwomen(littleautonomy,education,andlackofsocialpower)TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod251.
8Poverty1.
8LowlevelsofliteracyandlackofunderstandinganecologicalviewofplanetEarth2.
0Family-based,labor-intensiveeconomiesneedingchildren'sinput2.
1Closeadherencetoreligioustenetsthatleadtoavoidanceofcontraceptives2.
1Ruralareasreceivelessattentionfrompopulationprogramsthanurbanareas2.
2Lackofinformationandaccesstocontraceptives2.
2Highinfantmortality2.
3MasculinityassociatedwithincreasingnumbersofchildrenTheTenHistoricalForcesThatPromisetobeofIncreasingImportanceinDevelopingCountriesovertheNext25Years2.
2LowlevelsofliteracyandlackofunderstandingecologicalviewofplanetEarth2.
4Poverty2.
6Ruralareasreceivelessattentionfrompopulationprogramsthanurbanareas2.
7Lowlevelsofliteracy2.
8Discriminationagainstwomen(littleautonomyoreducation,andlackofsocialpower)2.
9Lackofinformationandaccesstocontraceptives3.
0Needofchildrenforsocialsecurity,tosupportparentsintheiroldage.
3.
1Family-based,labor-intensiveeconomiesneedingchildren'sinput3.
1Highinfantmortality3.
2BeliefbygovernmentsthatlargerpopulationsmeangreaterpoliticalstrengthWiththeinformationprovidedbythepanelinroundI,wealsoidentifiedtwopossiblefutureeventsthatwerejudgedtobeunlikely,butneverthelesspotentiallyeffectiveinreducingpopulationgrowthrates,iftheyweretooccur.
Thesewere;ImpactProbabilityDevelopment2.
13.
4Twochildrenperfamilybecomesthesocialnorminthemajorityofthedevelopingworld.
2.
33.
3ChangeintheVatican'sposition;useofcontraceptivesbecomesavailablewithoutlimit.
Theseitemswereofparticularinteresttoussincetheyrepresentedpotentialpolicyopportunities.
Bothhavebeenconsideredbeforeandoneisthefocusoflargeinternationalprograms.
Weaskedforsuggestionsaboutnovelpolicyapproachesthatmightbepractical,andifimplemented,improvetheprobabilityofthesedevelopments.
Wealsoaskedourrespondentstonotrepeat,ifpossible,previouslymadesuggestions.
Someofthepanel'sideasfollow:Novelpoliciesthatcouldleadtoasocialnormoftwochildrenperfamilythroughoutthemajorityofthedevelopingworld:TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod26-Family-sizetaxesthatincreasesubstantiallyforeverychildovertwo.
-Prohibitionofchildlabor.
-Policyfocusonwomen'saccesstoworkandeducation.
-Freecirculationofcontraceptivesinpublichealthprogramssupportedbychurchesandotherreligiousorganizations.
-Internationalinformationutilitiesineducation,health,andtrainingininformationsociety.
-Withenougheffortandresources,thepresentpolicies(national,bilateral,andmulti-lateral)thattrytomakefamilyplanningservicesuniversallyavailableandtopromoteinformation,educationandcommunicationaboutfamilyplanning,shouldmakethetwo-childnormvirtuallyuniversalwithin25years.
-Withoutsignificantreductionsinpovertyandinfantmortality,andincreaseinwomen'seducationandempowerment,thesepolicieswouldneedtobecoerciveinnatureoratleastprovidestrongeconomicdisincentivestohavingmanychildren.
SomeNovelPoliciesThatCouldLeadtotheVatican'sAcceptanceofContraceptionWithoutLimit:-RemoveHolySeefromtheUnitedNationsongroundsthatitisnotreallyacountryandgiveitthesamestatusastheWorldCouncilofChurches.
-Promotionofcontraceptivesaccompaniedwithstrongdisseminationofmoralvalues.
-TheologicaldoctrinedevelopedbyUS.
CatholicBishopsinsupportofsustainabledevelopment(atUNCED1992).
-Policiesthatfocusonresponsibilitiesofwomentomakechoices.
-Onlydeclineoforganizedreligionislikelytoaffectthesituation.
-Allowpriestsandnunstomarryandpayfortheraisingofchildren.
-ReconsiderationofthetheologyofSt.
ThomasAquinas-especiallythediscardingoftheThomasticviewof"naturallaw"asitappliedtohumansexuality.
Christsaidnothingwhatsoeverabouthumansexuality.
Separatethenotionofprocreationastheonly"naturalend"fromthatofenjoyment.
Wealsoaskedrespondentstosuggestotherareasthatwereripeforpolicyintervention.
Herearesometheysuggested:-Researchforlong-termimplantableovulationsuppressiondevice.
-Globalteleviseddebateonpopulationpolicy,environmentalprotection,andsocialethics.
-Sexeducationinschools-Allnationswillhavetoeventuallyadoptpoliciesthatclearlystatethefreedomofindividualchoice.
-Policiesthatemphasizespeopletotakechargeoftheirlivesandreducedependenceongovernments.
TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod27BIBLIOGRAPHYAmant,R.
,"ComparisonofDelphiForecastingStudiesin1964and1969,"Futures,Vol2,No.
1,March1970.
ThisarticleisanearlyattempttodeterminejusthowaccurateearlierforecastsmadebytheDelphimethodhaveproventobe.
Itshowedthatscientificandtechnologicalforecastswereindeedlikelytobemoreaccuratethanforecastsinsofterareas,suchasfashion,politics,orsocialbehavior.
Italsoshowedthatsometimes,whetheranevent,evenonesuccinctlystated,hadindeedoccurredwasquitedifficulttotell.
Azani,HosseinandKhorramshahgol,Reza,"AnalyticDelphiMethod,"EngineeringCostsandProductionEconomics,July1990.
Biji,Rob,"DelphiinaFutureScenarioStudyonMentalHealthandMentalHealthCare,"Futures,April1992.
Dalkey,Norman,B.
BrownandS.
W.
Cochran,"TheDelphiMethodIII:UseofSelfRatingstoImproveGroupEstimates,"TechnologicalForecastingandSocialChange,1,1970Dionne,C.
E.
,etal.
,"Aconsensusapproachtowardthestandardizationofbackpaindefinitionsforuseinprevalencestudies,Spine,Volume33,Issue1,January2008,Pages95-103Gibson,LayJamesandMiller,Mark,"ADelphiModelforPlanning'Preemptive'RegionalEconomicDiversification,"EconomicDevelopmentReview,Spring1990.
Ginsburg,Alan,"IntegratingEvaluationIntoDecisionmaking,"PublicManager,Winter1992-93.
Goldfisher,Ken,"ModifiedDelphi:AConceptforProductForecasting,"JournalofBusinessForecasting,Winter1992-93.
Glenn,JeromeandTheodoreGordon,StateoftheFuture2002,MillenniumProject,AmericanCouncilfortheUnitedNationsUniversity,Washington,DC.
2002.
Glenn,JeromeandTheodoreGordon,StateoftheFuture2003,MillenniumProject,AmericanCouncilfortheUnitedNationsUniversity,WashingtonDC.
2003.
Gordon,T.
J.
andGlenn,J.
C.
,IssuesinCreatingtheMillenniumProject:InitialReportfromtheMillenniumProjectFeasibilityStudy,UnitedNationsUniversity,August1993.
Thisworkwas,inpart,anexplorationofthefeasibilityofconductingworldwideDelphisonissuesofglobalimportance,usingscholars,futurists,andotherinterestedandinformedparticipantsinpanelsthatcommunicatedbye-mail,fax,andairmail.
Gordon,T.
J.
andHelmer,Olaf,ReportonaLongRangeForecastingStudy,RANDCorp.
,R-2982,1964.
Thisreportisstillinterestingtoread.
ItcontainsmanyinnovationsthatareusedintheanalysisandpresentationofDelphiresults.
Forexample,thereportspresentsargumentsforTheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod28usingmedianratherthatmeansofthegroup'sresponses,showshowrangesofopinionscanbepresentedgraphically,distinguishesbetweenlikelihoodandprobability,etc.
Helmer,OlafandRescher,Nicholas,"OntheEpistemologyoftheInexactSciences,"ManagementSciences,Vol.
6,No.
1(1959).
InthislandmarkworkHelmerandReschersetoutthephilosophicalbackdropforDelphiandsetlimitsofexpectationaboutwhatcanandcannotbeknownwhenthequestionsbeingaddressedfallintothecategoryof"inexactscience.
"Theyincludeallofsocialscienceandpoliticalscienceas"inexact.
"Kastien,M.
R.
,etal.
,"Delphi,TheIssueofReliability,"TechnologicalForecastingandSocialChange,Nov.
1993.
Lindstone,H.
,andTuroff,M.
(ed.
),TheDelphiMethod,AddisonWesleyPublishingCo.
,1975.
OneofthebestdetaileddiscussionsoftheDelphimethod.
McCarthy,Kevin,"Commentonthe'AnalyticDelphiMethod',"InternationalJournalofProductionEconomics,May1992.
Mitchell,Vincent-Wayne,"UsingDelphitoForecastNewTechnologyIndustries,"MarketingIntelligenceandPlanning,Vol10,Issue2,1992.
Nimgade,AshokandSonk,Joseph,"OrdinalQuantificationoftheConceptofTechnologyThroughPooledConsensus,"R&DManagement,Jan.
1991.
Olshfski,Dorothy,andJoseph,Alma,"AssessingTrainingNeedsofExecutivesUsingtheDelphiTechnique,"PublicProductivityandManagementReview,Spring1991.
Pare,G,C.
Sicotte,M.
Jaana,andD.
Girouard,"Prioritizingclinicalinformationsystemprojectriskfactors:ADelphistudy",ProceedingsoftheAnnualHawaiiInternationalConferenceonSystemSciences,2008.
Rohde,William,"PastPredictionsTakeShape,"HealthcareExecutive,May/June1991.
Rowe,Gene,GeorgeWright,"ExpertOpinionsinForecasting:TheRoleoftheDelphiTechnique,"achapterinPrinciplesofForecasting,J.
ScottArmstrong(ed)Springer(2001).
Rowe,Gene,GeorgeWright,andF.
Bolger,"Delphi:ARevaluationofResearchandTheory,"TechnologicalForecastingandSocialChange,Vol.
39,May1991.
Russo,DavidM.
,andMcLaughlin,"TheYear2000:AFoodIndustryForecast,"Agribusiness,November1992.
Shota,Ushio,"TheFutureofHighTech:ForecastsfortheNextDecade,"TokyoBusinessToday,April1993.
TheMillenniumProjectFuturesResearchMethodology—V3.
0TheDelphiMethod29TechnologicalForecastingandSocialChange,SpecialIssueonDelphi,1975,no.
2.
Discussescritiquesofthemethod.
Turoff,Murray"TheDesignofaPolicyDelphi,"TechnologicalForecastingandSocialChange2,No.
2(1970).
Turoff,Murray,DelphiConferencing:Computer-BasedConferencingwithAnonymity,TechnologicalForecastingandSocialChange3,159-204,1972.
Turoff,Murray,StarrRoxanneHiltz,Hee-KyungCho,ZhengLi,andYuanqiongWang,SocialDecisionSupportSystems(SDSS),Proceedingsofthe35thHawaiiInternationalConferenceonSystemSciences,2002Tversky,A.
andD.
Kahneman,"TheFramingofDecisionsandthePsychologyofChoice,"Science,211,453-458,January30,1981.
http://psych.
hanover.
edu/classes/Cognition/papers/tversky81.
pdfVickers,Brent,"UsingGDSStoExaminetheFutureEuropeanAutomobileMarket,"Futures,October1992.
Woudenberg,Fred,"AnEvaluationofDelphi,"TechnologicalForecastingandSocialChange,Sept.
1991.
ContainsabroadreviewoftheliteratureonDelphiandmanyreferencestothemethodandpaststudies.