Fertilisationadalovelace

January2021TheuseofpublicengagementfortechnologicalinnovationLiteraturereviewandcasestudiesBEISResearchPaperNumber2021/003PrefaceTheaimofthisreportcommissionedbytheDepartmentforBusiness,EnergyandIndustrialStrategy(BEIS)istobringtogetherevidenceonthecurrentuseofpublicengagementinpolicydevelopmentandregulationoftechnologicalinnovation.
ThestudyprovidesanevidencebasetosupporttheworkoftheBetterRegulationExecutive(BRE)atBEISthatleadstheregulatoryreformagendaacrosstheUKgovernment.
Inparticular,theresultsfromthisstudyareintendedtohelptheBREinitsactivitiesandengagementwithgovernmentdepartmentsandregulatorsonhowtobestconductpublicengagementwithaviewtoinformingandcreatinganarrativearoundnewandemergingtechnologies.
Thefindingsfromthisstudyarethusmainlytargetedatpolicymakersandregulators.
Theanalysisandresults,however,arealsolikelytobeofrelevancetootherstakeholdersinterestedandinvolvedintechnologicalinnovationsuchasfundersofresearchandinnovation,academia,industry,andthegeneralpublic.
Thisstudyfocusesonthreemainresearchquestionsinthespecificcontextoftechnologicalinnovation:WhatexistingexamplesarethereofpublicengagementtechniquesandhowandwhenhavetheybeenappliedWhatevidenceofimpactexistsinrelationtodifferenttypesofpublicengagementontechnologicalinnovation(e.
g.
intermsofinformingthedesignofregulatoryframeworks,newbusinessmodels,marketadoption,andpublictrust)Hastheeffectivenessofthepublicengagementtechniquesaroundtechnologicalinnovationbeenformallyevaluated,andwhat,ifany,werethelearningsThisreportwaswrittenbythefollowingresearchersfromRANDEurope:Camillad'Angelo,AdvaitDeshpande,EmilyRyenGloinson,JoeFrancombe,CaglaStevenson,MannVirdee,andSalilGunashekar.
RANDEuropeisanot-for-profitresearchorganisationthataimstoimprovepolicyanddecisionmakinginthepublicinterest,throughresearchandanalysis.
RANDEurope'sclientsincludeEuropeangovernments,institutions,non-governmentalorganisationsandfirmswithaneedforrigorous,independent,multidisciplinaryanalysis.
FormoreinformationaboutRANDEuropeorthisdocument,pleasecontact:DrSalilGunashekar(ResearchLeaderinScienceandEmergingTechnology)RANDEurope,WestbrookCentre,MiltonRoad,CambridgeCB41YG,UnitedKingdomTel.
+44(1223)3532562Email:sgunashe@randeurope.
orgAcknowledgementsWewouldliketothanktheprojectteamattheBetterRegulationExecutivefortheiradvice,guidanceandsupport,andforengagingconstructivelyandcollaborativelywithusoverthecourseoftheproject.
Inparticular,wewouldliketothankKalavatiPatel,DorotaDenning,KimWagerandStuartRoddam.
Weareverygratefultothemanystakeholderswhokindlyagreedtobeinterviewedaspartofthisstudy.
WewouldalsoliketothanktoourqualityassurancereviewersatRANDEurope,DrSusanGuthrieandDrHarrietTeare,fortheircriticalreviewandvaluablecommentsonthisreport.
Crowncopyright2021ThispublicationislicensedunderthetermsoftheOpenGovernmentLicencev3.
0exceptwhereotherwisestated.
Toviewthislicence,visitnationalarchives.
gov.
uk/doc/open-government-licence/version/3orwritetotheInformationPolicyTeam,TheNationalArchives,Kew,LondonTW94DU,oremail:psi@nationalarchives.
gsi.
gov.
uk.
Wherewehaveidentifiedanythird-partycopyrightinformationyouwillneedtoobtainpermissionfromthecopyrightholdersconcerned.
Anyenquiriesregardingthispublicationshouldbesenttousat:enquiries@beis.
gov.
uk5ContentsPreface_2Acknowledgements3Executivesummary9Background9Studyobjectives9Methodology_10Casestudiesillustratingtheuseofpublicengagementtechniquesfortechnologicalinnovation11Keyfindings15Adiverserangeofpublicengagementtechniquescouldbeusedinthecontextoftechnologicalinnovation15Differentpublicengagementtechniqueshave,tovaryingdegrees,hadanimpactonselectedoutcomes15Theusefulnessofpublicengagementtechniquesiscontextual16Concludingremarks17Chapter1:Introduction_18Backgroundandcontext_18Studyobjectives21Summaryofthemethodology_21Structureofthisreport22Chapter2:Examplesofpublicengagementtechniquesappliedinthecontextoftechnologicalinnovation_24ResearchQuestion1:Whatexistingexamplesarethereofpublicengagementtechniquesandhowandwhenhavetheybeenapplied24Arangeoftraditionalandatypicalpublicengagementtechniqueshavebeenusedinthecontextoftechnologicalinnovation25Therangeofpublicengagementtechniquesappliedtotechnologicalinnovationcoverboth'one-way'methodsand'two-way'methods_68Publicengagementtechniqueshavebeenappliedtoadiverserangeoftechnologies_69Thepublicengagementtechniquesappliedtotechnologicalinnovationcoverarangeofstakeholdertypes706Publicengagementtechniqueshavebeenappliedtovaryingdegreesacrossthetechnologydevelopmentandadoptionpathway71Chapter3:Existingevidenceofimpactinrelationtodifferenttypesofpublicengagementtechniquesontechnologicalinnovation80ResearchQuestion2:Whatevidenceofimpactexistsinrelationtodifferenttypesofpublicengagementontechnologicalinnovation80Thereisevidencethatdifferentpublicengagementtechniqueshavehadanimpactondifferentoutcomesassociatedwithatechnologicalinnovation_81Atypicalpublicengagementtechniquesandmarketoutcomeslackevidence84Chapter4:Evidenceofwhethertheeffectivenessofpublicengagementtechniquesaroundtechnologicalinnovationhasbeenformallyassessed,andassociatedlearnings86ResearchQuestion3:Hastheeffectivenessofthepublicengagementtechniquesaroundtechnologicalinnovationbeenformallyevaluated,andwhat,ifany,werethelearnings_86Fewpublicengagementprocesseshavebeenformallyevaluated87Existingevaluationsandassessmentsprovidesomekeylearningsrelatedtospecifictechniques_87Chapter5:Discussionandconcludingremarks92Whatdoestheevidencesayaboutpublicengagementinthecontextoftechnologicalinnovation93Adiversespectrumofpublicengagementtechniquescouldbeusedinthecontextofatechnologicalinnovation93Differentpublicengagementtechniqueshave,tovaryingdegrees,hadanimpactonselectedoutcomes93Theusefulnessofpublicengagementtechniquesiscontextual94Takingalltheevidencetogether,severalcross-cuttinglearningsemerge95Theuseofmultipletechniquesoverthecourseofthepublicengagementprocesscanhelptoengagedifferent'publics'appropriately_95Spreadingpublicengagementovertimeallowsforreflectionandembeddingofconcepts96Havinganimpactontrustinatechnologicalinnovationrequirestimeandconsidereddebatetoincreaseaccountabilityandmoresystematicpublicengagement_96Amulti-stakeholder,collaborativeapproachtopublicengagementhelpstodevelopinformedandconsideredjudgements97Usingonlineanddigital-enabledpublicengagementtechniquescanpotentiallyincreasethespeed,scale,inclusivityandgeographicalcoverageofengagement97Usingsomeatypicaltechniquescanpotentiallyrenderpublicengagementmoretangibleanduser-friendlyandcouldalsoincreasethediversityofparticipation987Havinganimpactonoutcomessuchasregulation,policyandmarketadoptionofatechnologicalinnovationtypicallyrequiresbuy-inandengagementwiththerightstakeholders98Itisimportanttobuildevaluationintopublicengagementprocessestotrackimpactsandoutcomesovertime_99Finalreflections99AnnexA:Casestudiesillustratingtheuseofpublicengagementtechniquesinthecontextoftechnologicalinnovation_101A.
1.
Casestudy1(CS01):Seriousgametocrowdsourcethepublic'sviewsonmoraldecisionsfacedbyautonomousvehicles103A.
2.
Casestudy2(CS02):Exploringpublicperceptionsonautonomousvehiclesusinglivepublictrials,workshops,sentimentmapping,andobservationalstudies107A.
3.
Casestudy3(CS03):Citizens'jurytounderstandpublicattitudestowardsEthicalAI112A.
4.
Casestudy4(CS04):Foresightgamingformulti-stakeholderdialoguestoexplorenanotechnologyandResponsibleResearchandInnovationpractices_117A.
5.
Casestudy5(CS05):Determiningpublicperceptionontheuseofvirtualrealityinhealthcarethroughsociallistening122A.
6.
Casestudy6(CS06):UsingthevTaiwanplatformtocarryoutapublicdebateontheregulationofUberinTaiwan_126A.
7.
Casestudy7(CS07):EngagingexpertandcitizenperspectivesonAIusingaworkshopandonlineplatform130A.
8.
Casestudy8(CS08):Engagingthepubliconfacialanalysisandautomateddecision-makingthroughtheuseofBioMetricMirror–aninteractiveapplication_135A.
9.
Casestudy9(CS09):CitizenandMulti-ActorConsultationonHorizon2020(CIMULACT)toformulatescienceandtechnologypolicyresearchagendaintheEuropeanUnion139A.
10.
Casestudy10(CS10):RapidonlinedeliberationtoexplorepublicattitudestotheuseofCOVID-19-relatedtechnologies144AnnexB:Overviewofgeneralpublicengagementtechniques149AnnexC:Descriptionofmethodologicalapproach_179C.
1.
Phase1:Deskresearch180C.
1.
1.
Focussedreviewoftheacademicandgreyliterature_180C.
1.
2.
Scopingconsultations_186C.
2.
Phase2:Casestudies186C.
2.
1.
Casestudyselection186C.
2.
2.
Interviewswithcasestudystakeholders_1878C.
2.
3.
Reviewofselecteddocumentationandarticlesrelatedtocasestudies_______188C.
3.
Phase3:Analysis189C.
3.
1.
SynthesisworkshoptotriangulateevidencefromPhases1and2_189C.
3.
2.
Finalreportwithkeyfindings189C.
4.
Caveatsoftheanalysis190AnnexD:Long-listofcasestudiesillustratingtheuseofpublicengagementtechniquesinthecontextoftechnologicalinnovation191References_2029ExecutivesummaryBackgroundWiththepromisetoimprovelives,andtheofferofabundantopportunities,technologicalinnovationisregardedasacrucialenablerfortheadvancementofsocietiesandeconomiesatlarge.
Technologiesandtheirapplications,however,alsopresentanarrayofsocial,economicandregulatorychallenges.
Howtoharnessthebenefitsoftechnologicalinnovationwhileaddressingtherisksassociatedwiththesedevelopmentsisthesubjectofmuchdiscussionbygovernments,regulators,industry,academiaandthegeneralpublic.
Akeyaspectassociatedwiththesediscussionsanddebatesispublicengagement,whichisincreasinglybeingrecognisedbystakeholdersasacriticalinstrumenttoencouragetransparencyandopenness,increaserepresentativeness,andbuildtrustindecisionmakingandthetechnologiesthemselves.
Publicengagementisabroadtermthatisusedinavarietyofsectors(e.
g.
inresearch,healthcareandpolicymaking).
Itencompassesdiversewaysinwhichmembersofthegeneralpubliccanbebroughttogethertoengagewithissuesthatareofpublicimportance.
Inthecontextoftechnologicalinnovation,publicengagementisusedtodescribetheinvolvementofadiversegroupofpeople(thegeneralpublic,butalsootherkeygroupssuchaslobbyists,civilsocietyorganisationsandsocialinfluencers)indiscussionsanddebatesaboutpotentialapplicationsofnewandemergingtechnologies,theirgovernance,regulationandthewiderissuesthatcouldarisefromthewaythattheyaredevelopedandadopted.
StudyobjectivesUnderstandingwhatworks,inwhatcontextsandforwhatpurposeinrelationtopublicengagementandtechnologicalinnovationisimportanttoinformpolicyandregulation.
TheoverarchingaimofthisstudycommissionedbyBEISistobringtogetherevidenceonthecurrentuseofpublicengagementfortechnologicalinnovation.
Theworkreportedherefocusesonidentifyingandunderstandingthetypesofengagementtechniquesusedintherealworld,howthesehavebeenappliedtotechnologicalinnovationacrossdifferentsectorsandanypotentialoutcomestheymayhave,aswellastheeffectivenessoftheseapproaches.
Thestudysynthesisesevidenceonthecurrentuseofpublicengagementfortechnologicalinnovation.
Thestudy,whichistargetedatregulatorsandpolicymakers,providesanevidencebasetosupporttheworkoftheBREatBEISthat10leadstheregulatoryreformagendaacrosstheUKgovernment.
Theworkfocusesonthreemainquestionsinthecontextoftechnologicalinnovation:1.
ResearchQuestion1:Whatexistingexamplesarethereofpublicengagementtechniquesandhowandwhenhavetheybeenapplied2.
ResearchQuestion2:Whatevidenceofimpactexistsinrelationtodifferenttypesofpublicengagementontechnologicalinnovation(e.
g.
intermsofinformingthedesignofregulatoryframeworks,newbusinessmodels,marketadoption,andpublictrust)3.
ResearchQuestion3:Hastheeffectivenessofthepublicengagementtechniquesaroundtechnologicalinnovationbeenformallyevaluated,andwhat,ifany,werethelearningsMethodologyWeadoptedamixed-methodsapproachtoaddresstheobjectivesofthestudy,whichwasconductedovera12-weekperiod.
Theapproachenabledustorapidlyidentify,analyseandsynthesiseevidencerelatedtotheuseofpublicengagementtechniquesinthecontextoftechnologicalinnovation.
Theapproachbroadlyconsistedofthreedistinctbutoverlappingphases:(i)Phase1:Deskresearch;(ii)Phase2:Casestudies;and(iii)Phase3:Analysis.
Inthefirstphaseofthestudy,weconductedarapidreviewoftheacademicliteraturecoveringpublicengagementtechniquesappliedinthecontextoftechnologicalinnovation.
Thereviewincludedacademicarticlesfrom2016to2020.
Thiswassupplementedwithtargetedsearchesofthegreyliteratureandscopingconsultations(bytelephoneande-mail)withexpertsintheareaofpublicengagement.
Inthesecondphaseofthestudy,wedevelopedtencasestudiestoprovideconcreteexamplesofhowpublicengagementhasbeenappliedinthecontextoftechnologicalinnovation,whetherthetechniqueshavebeeneffective,andifsohow.
Foreachcasestudy,theinformationwasgatheredthroughinterviewswithstakeholdersandbyreviewingaselectionofarticlesconnectedtothecasestudy.
Inthefinalphaseofthestudy,wesynthesisedtheevidencefromthedeskresearchandcasestudiesbydevelopingasetofkeythemesregardingapproachestopublicengagementfortechnologicalinnovationandtheireffectiveness.
Thecross-analysisoftheevidencewascarriedoutinaninternalworkshopattendedbycoremembersofthestudyteam.
11CasestudiesillustratingtheuseofpublicengagementtechniquesfortechnologicalinnovationBelowwepresentshortsummariesofthetencasestudiesdevelopedtodemonstrateadeeperunderstandingofavarietyofpublicengagementtechniquesappliedindifferentcontexts.
Thecasestudiesspandifferenttechnologyareas,sectors,organisations,countrycontexts,andtimeperiods.
Casestudy1(CS01):Seriousgametocrowdsourcethepublic'sviewsonmoraldecisionsfacedbyautonomousvehiclesAkeychallengearoundthedevelopmentofautonomousvehiclesliesinthemoraldilemmasthattheyarelikelytoface(e.
g.
decidingwhoshouldliveandwhoshoulddiewhenfacedwithapotentialcollisionwithapedestrian).
Tothisend,researchersdevelopedawebsite,MoralMachine,thatuseda'seriousgame'withscenariostocrowdsourcethepublic'sviewsonmoraldecisionsfacedbyautonomousvehicles.
Theaimwastogenerateabetterunderstandingofthepublic'sviewsabouthowautonomousvehiclesshouldsolvemoraldilemmas,aswellastohelpraiseawarenessaboutthistopicamongstthepublic.
Theplatformwasaneffectivelarge-scaledatagatheringexercise,collecting40milliondecisionsintenlanguagesfrompeoplein233countriesandterritories.
Ithelpedtoshowthataseriousgameisaneffectivemethodtocrowdsourcethepublic'sviewsonacontroversialtopic.
Theexerciseidentifiedstronguniversalmoralpreferences.
Althoughthefindingshavenotledtoanyformaloutcomes,inprincipletheycouldbeusedtocontributetodevelopingglobal,sociallyacceptableprinciplesformachineethics,whichcouldinformconsiderationsbycarmanufacturersandpolicymakers.
Casestudy2(CS02):Exploringpublicperceptionsonautonomousvehiclesusinglivepublictrials,workshops,sentimentmapping,andobservationalstudiesLedbyTransportLaboratoryLondon(TRL),theGATEway(GreenwichAutomatedTransportEnvironment)projectemployedarangeofpublicengagementmethodstoexplorepublicperceptionstowardsconnectedandautonomousvehicles(CAVs)inordertounderstandsocietalfactorsaffectingtheadoptionofCAVs.
TheGATEwayProjectdemonstratesthepotentialoflivepublictrialsasamethodforpublicengagement,whilealsohighlightingtheroleofdiverseengagementtechniquesinengagingdifferentpublicaudiences.
InsightsfromtheGATEwayhaveshapedtheworkofcertainactors,e.
g.
thevehicledesigncoursesoftheRoyalCollegeofArt.
Engagementtechniquesusedbytheprojecthavealsobeenincorporatedintomarket-orientedtest-bedenvironments,asexemplifiedbytheLondonSmartMobilityLivingLab(SMLL).
Casestudy3(CS03):Citizens'jurytounderstandpublicattitudestowardsEthicalAI12Theethicaluseofartificialintelligence(AI)hasbecomeincreasinglyimportantinautomateddecisionsystemsthatuseAItoinformormakedecisionsonwhichactionstopursue.
TheRoyalSocietyfortheencouragementofArts,ManufacturesandCommerce(RSA)andDeepMindpartneredonadeliberativeengagementexerciseusingacitizenjuryforpublicengagementontheimpactsofethicalAI.
Theprojectwasnotformallyassessed,butanecdotalevidencefrominterviewsanddeskresearchindicatesthatparticipatingcitizensfeltbetterinformedandhadabetterunderstandingofautomateddecisionsystemsfollowingtheproject.
Intervieweesindicatedthatthepublicengagementformatalsoillustratedtheimportanceofusingpublicengagementtechniquesforotherorganisations,althoughthisimpactwasnotcitedinrelevantdocuments.
Casestudy4(CS04):Foresightgamingformulti-stakeholderdialoguestoexplorenanotechnologyandResponsibleResearchandInnovationpracticesResponsibleResearchandInnovation(RRI)isimportanttoincorporatesocietalneedsandvalues,andusesdeliberativeandtransparentapproachestodevelopethicallyacceptableandsociallydesirableproducts.
TheprojectusedtheJointResearchCentre(JRC)game,theSES,innationalmulti-stakeholderdialoguestoexplorenanotechnologyandRRIpractices.
Theprojectensuredthatparticipantscouldcomfortablyexpresstheiropinionandthetechniqueresultedingreateruptakeofpublicengagementtechniquesatthesciencecentresthatrantheexercise.
Intheparticipatoryprocess,recommendationssuggestedthattoopenupthenanotechnologysystem,decision-makersshouldpromotegeneratinginterestandmotivationforRRIthatisgenuine,ratherthanfocussingontop-downregulation.
ItalsoprovidedrecommendationsanddirectionsfortheEuropeanCommissionandotheractorsontheimportanceofpublicengagementinnanotechnologyprojects,althoughnoevidencewasfoundoftheuptakeoftheserecommendationsinpractice.
Casestudy5(CS05):DeterminingpublicperceptionontheuseofvirtualrealityinhealthcarethroughsociallisteningVirtualreality(VR)providesanimmersiveenvironmentthatenablesuserstohaveanalteredexperienceofreality.
Itisincreasinglyusedacrossthehealthcaresectortosupportpatientsduringtreatment,howeverthepublicperceptiononitsuseremainsunder-studied.
Here,theresearchteamusedamethodcalled'sociallistening'inordertocollectandanalyseFacebookcommentsinresponsetoavideoontheuseofVRinhealthcare.
AnalysisofthesecommentsshowedthatthepublicwasgenerallyexcitedabouttheuseofVRwithinahealthcaresetting,butalsoidentifiedseveralpotentialconcernsofitsuse.
Theseconcernshighlightpotentialfuturebarrierstothiskindofresearchandprovideinsightintohowthisareacouldprogress.
13Casestudy6(CS06):UsingthevTaiwanplatformtocarryoutapublicdebateontheregulationofUberinTaiwanvTaiwanisadeliberativedigitalplatform.
Itfacilitatesconstructivedebateandhelpsidentifyareasofconsensusonspecificissuesamongstcitizens,stakeholders,andgovernment–withtheintentionofincreasingengagement,scrutinyandtransparencyindecision-making.
Inthisexample,vTaiwanwasusedintheregulationoftheUberXservicefromtheride-hailingappUber.
vTaiwanusesadedicatedAI-facilitatedsocialmediatool,Pol.
is,whichallowsuserstodraftwaysinwhichaproblemmaybeaddressed–aswellasrespondtootherusers'solutionsbyagreeingordisagreeingwiththem.
TheapplicationofthevTaiwanprocesstotheregulationofUberXdemonstratedimpactsthroughitsabilitytodiffuseapotentialrowbetweenUberdriversandtraditionaltaxidriversaroundwhetherUberwasanapportaxiserviceinTaiwan.
ByengagingcitizensandstakeholdersitreachedconstructiveoutputsregardinglegalrequirementsforbothtraditionaltaxisandUbercars,andtheseoutputssubsequentlybecamelaw.
TheeffectivenessofthevTaiwanprocessdiscussedinthiscasestudyhasbeenhighlightedinanumberofpublishedarticles.
Casestudy7(CS07):EngagingexpertandcitizenperspectivesonAIusingaworkshopandonlineplatformThiscasestudyexploresatwo-stagepublicengagementprocessonAI,conductedundertheauspicesoftheHumanBrainProject(HBP)andoverseenbytheDanishBoardofTechnology(DBT).
Inthefirststageofthepublicengagement,anexpertworkshopwasorganised.
Theworkshopbroughttogethercross-disciplinaryexpertstoconsidertheethical,economic,legalpoliticalandsocialimpactsofAIusinga'360'approach.
Inthesecondstage,findingsfromtheexpertworkshopwereusedtoinformanEU-wide'citizenconsultation'onAIusinganonlineplatform.
Thecasestudyhighlightsthepotentialforlinkedapproachestopublicengagementdrawingonbothexpertstakeholdersandthebroaderpublic,aswellastheutilityofonlineplatformsasmechanismsforengagingpublicperspectives.
Theextent,however,towhichtheserecommendationswillbetakenintoaccountbytheEuropeanCommissionisnotclearasofDecember2020.
Casestudy8(CS08):Engagingthepubliconfacialanalysisandautomateddecision-makingthroughtheuseofBioMetricMirror-aninteractiveapplicationFacialanalysisapplicationsareincreasinglybeingusedtoinformdecision-makingprocesses,andconcernshavebeenraisedregardingthetransparencyandethicsofthesetechnologies.
Here,theresearchteamdevelopedaninteractiveapplication'BioMetricMirror'whichperformedfacialanalysisonthepublic,andsubsequentlyprovidedinferencesabouttheparticipants'demographicandpsychometriccharacteristics.
Throughthisapplication,thepublicwereabletoengagewiththis14technologyinanexperientialway,whichprompteddiscussionandreflectiononnotonlythepotentialforthistypeoftechnologybuttheethicalchallengesthatmustbeconsideredifthisweretobecomemoreprevalentinsociety.
Casestudy9(CS09):CitizenandMulti-ActorConsultationonHorizon2020(CIMULACT)toformulatescienceandtechnologypolicyresearchagendaintheEuropeanUnionTheCIMULACTprojectfocussedonadvancesinRRIintermsofenhancedcooperationwithscienceandsocietytopromotescientificexcellence,andsocialresponsibilityandawareness.
CIMULACTcarriedoutcitizenandmulti-actorconsultationstocontributetoresearchagendaformulationforscienceandtechnologypolicyintheEuropeanUnion.
Theprojectwaswell-receivedbyparticipants.
IthadanimpactontheresearchagendainHorizon2020,astheprojectresultswereusedintheformulationofH2020WorkProgrammes2018-2020.
TheprojectwasalsoacknowledgedintheInterimEvaluationofHorizon2020andwaschosenasagoodpracticecaseforcitizenengagementinagendasettinginopenscienceattheOECD.
Casestudy10(CS10):RapidonlinedeliberationtoexplorepublicattitudestotheuseofCOVID-19-relatedtechnologiesThereappearstobealackofpublictrustinsomedigitaltechnologiesdevelopedtoaddressdifferentaspectsofCOVID-19,includingcontacttracingapps.
Toensurepublictrustandbuy-inofthesetechnologies,itisimportanttodetermineunderwhatcircumstancesthepublicconsiderstechnologicalsolutionssuchastheCOVID-19contacttracingapptobeappropriate.
Tothisend,theAdaLovelaceInstitute,togetherwithcollaborators,conductedarapidonlinedeliberationtoexploretheattitudesofmembersoftheUKpublictotheuseofdigitalCOVID-19technologies,includingtheNHScontacttracingapp.
Theprocesswaseffectiveinenablinganinformeddialoguewiththepublicinarapidlychanginganduncertainenvironment.
Theprocessdemonstratedthatitispossibletogatherpublicinputinashortspaceoftimeandcouldpotentiallybeappliedtootherfutureareasinwhichtimeislimited.
Althoughitistoosoontoindicatewhethertheengagementprocesshasimpactedonpolicyorregulation,theengagementexerciseidentifiedfourrequirementsthatwouldhelptoensurepublictrustandbuy-inregardingfutureCOVID-19technologies.
Thesewereto:(1)provideatransparentevidencebase;(2)offerindependentreviewofthetechnology;(3)clarifydatause,rightsandresponsibilities;and(4)addresstherisksandneedsofvulnerablegroups.
Theserequirementsfedintoachecklistaimedatgovernment,policymakersandtechnologydeveloperstohelpthemwiththefuturedevelopment,designanduseofCOVID-19technologies.
15KeyfindingsBelowwesummarisethekeyfindingsfromtheresearch.
AdiverserangeofpublicengagementtechniquescouldbeusedinthecontextoftechnologicalinnovationSpecifically,weobservethat:Traditionalapproacheshavebeenusedmostfrequently.
Theseincludewell-establishedmethodssuchassurveysandpublicdialogues,forwhichconsiderableguidancealreadyexists;andAtypicalapproacheshavebeenusedinsomeinstances.
Thesegenerallytendtobeinnovative,'digital-enabled'techniques,usingonlinetoolsorimmersiveVRtechnologiesandsimulations.
Thereisevidencetosuggestthesecouldholdpotentialtobeusedmoreinthefuture.
Accordingly,thereportandcasestudiesalsocoverthesetechniques.
Publicengagementtechniqueshavebeenappliedtoadiverserangeoftechnologies(e.
g.
cognitivetechnologies(includingAIandmachinelearning),data-driventechnologies,medical/biotechnologies,greentechnologies,nanotechnologies).
Publicengagementtechniquesappliedtotechnologicalinnovationcoverarangeofstakeholdertypes(e.
g.
thegeneralpublic,potentialusersorconsumersofthetechnology,policymakers,regulators,industryandexpertsfromvariousdisciplines).
Differentpublicengagementtechniqueshave,tovaryingdegrees,hadanimpactonselectedoutcomesOverall,theevidenceonimpactsisstrongestfortraditionalapproaches(i.
e.
surveys,publicdialogues),withlessevidencearoundatypicaltechniques,whichtendtoconsistofproof-of-conceptstodemonstratethefeasibilityofaparticularapproach.
Themostreportedoutcomeistoraiseawarenessandunderstandingoftechnologicalinnovation.
Thereareseveralinstancesinwhichpublicengagementhashadanimpactonpolicy,regulation,orethics.
Thereareseveralinstancesinwhichpublicengagementhashadanimpactonaspectsoftrust.
Therearelimitedinstancesinwhichpublicengagementhashelpedinformmarketadoptionoftechnologicalinnovationorcontributedtochangesinbusinessmodels.
16TheusefulnessofpublicengagementtechniquesiscontextualTheeffectivenessofpublicengagementtechniqueswithregardtotechnologicalinnovationisoftennotformallyassessed,andevidenceaboutmanyapproachesislimited.
Theevidencesuggeststhat:Consultativeapproachessuchassurveys,interviewsandfocusgroupsareawell-establishedmethodtogatherinformationontechnologicalinnovationfromaselectedsampleofthepublic.
Deliberativemethodsareparticularlyusefultoexplorecomplextopicsindepth,andwhenthereispotentialuncertaintyorcontroversy.
Therearemultipleinnovativemethods(e.
g.
art-basedandexperientialtechniques)thatcouldhelptorenderpublicengagementmoremeaningful(i.
e.
byhelpingthepublictounderstandhowatechnologicalinnovationmightfitintotheirwidereverydaylives),andwidenparticipation.
Theuseofonlineanddigital-enabledapproaches(acrossthespectrumofengagement)representsanovelapproachthatcanspeeduptheprocessofengagement,capturetheviewsofthepublicatscaleandenhancetheexperienceofengagement(e.
g.
throughtheuseofimmersiveexperiencesthatenablethepublictoexperiencereal-worldscenarios).
Theevidencealsohighlightsthatdifferenttechniqueshavebeenused'upstream'(i.
e.
atanearlierstage)and'downstream'(i.
e.
atalaterstage):Techniquesacrosstheengagementspectrumhavebeenused'upstream'togathertheviewsofthepublicordiscussissuesin-depthtohelpinformaspectsofregulation,ethicsandpolicy.
Thereisevidencethatdeliberativetechniquesmaybeparticularlysuitedforthispurpose.
Techniquesacrosstheengagementspectrumhavealsobeenused'downstream'tohelpinformacceptanceandpotentialmarketadoptionoftechnologies.
Thereisevidencethatexperientialtechniquesmaybeparticularlysuitedforthispurpose.
Toaidtheunderstandingofthedifferentpublicengagementtechniquesidentifiedinthisstudyinthecontextoftechnologicalinnovation,wehaveprovidedthefollowinginformationastablesinaseparateExcelfilepublishedalongsidethisreport(thesetablesshouldbereadandinterpretedinconjunctionwiththenarrativepresentedinthisreport):Overviewofthedifferentpublicengagementtechniquesusedinthecontextoftechnologicalinnovationidentifiedintheliteratureandcasestudies(Table1;thisispresentedinChapter2andthefirstsheetintheaccompanyingExcelfile)17Overviewoftheevidencepresentedinthisreportoneffectivenessofpublicengagementtechniques(inthecontextoftechnologicalinnovation)inrelationtodifferentoutcomes(Table2;thisispresentedinthesecondsheetintheaccompanyingExcelfile)Overviewoftheevidencepresentedinthisreportonthepotentialimplicationsofapplyingthedifferentpublicengagementtechniquesinthecontextoftechnologicalinnovation(Table3;thisispresentedinthethirdsheetintheaccompanyingExcelfile)ConcludingremarksOnthebasisoftheanalysis,wealsooffersomecross-cuttinglessons:Theuseofmultipletechniquesoverthecourseofthepublicengagementprocesscanhelptoengagedifferent'publics'appropriately.
Spreadingpublicengagementovertimeallowsforreflectionandembeddingofconcepts.
Havinganimpactontrustintechnologiesandtechnologicalinnovationrequirestimeandconsidereddebatetoincreaseaccountabilityandmoresystematicpublicengagement.
Amulti-stakeholder,collaborativeapproachtopublicengagementhelpstodevelopinformedandconsideredjudgements.
Usingonlineanddigital-enabledpublicengagementtechniquescanpotentiallyincreasethespeed,scale,inclusivity,andgeographicalcoverageofengagement.
Usingsomeatypicaltechniquescanpotentiallyrenderpublicengagementmoretangibleanduser-friendlyandcouldalsoincreasethediversityofparticipation.
Havinganimpactonoutcomessuchasregulation,policyandmarketadoptionoftechnologicalinnovationtypicallyrequiresbuy-inandengagementwiththerightstakeholders.
Itisimportanttobuildevaluationintopublicengagementprocessestotrackimpactsandoutcomesovertime.
18Chapter1:IntroductionBackgroundandcontextPublicengagementisabroadtermthatisusedinavarietyofsectors(e.
g.
inresearch,healthcareandpolicymaking).
Itencompassesnumerousanddiversewaysinwhichmembersofthegeneralpubliccanbebroughttogethertoengagewithissuesthatareofpublicimportance.
Itisaprocessthatinvolvesinteractionandlisteningtogeneratebenefitandincludesanaspirationtobetterconnectsocietieswithresearchandinnovation.
Benefitsincludeincreasedaccountability,alignmentofvalueswiththeneedsofcitizens,buildingtrustandensuringrelevanceandresponsiveness(NCCPE2020b).
Inthecontextofpolicymaking,publicengagementcanhelptoincreasethelegitimacy,justiceandeffectivenessofdecision-making(Fung2006).
Challengeswithpublicengagementinclude:ensuringthatthemostappropriatemethodsofengagementareusedwithdiversestakeholdergroups;theavailabilityoffunding;havingtherequiredtimetoplansuccessfulprojects;ensuringthatpolicymakersandresearchersrecogniseitsvalue;communication;andhavingtherequiredcapacityandskills(UniversityofReading2019).
AsillustratedinTable4inAnnexB,publicengagementencompassesavarietyofmethodsortechniques,andcanbeappliedinavarietyofdifferentcontexts.
Inthecontextoftechnology1andtechnologicalinnovation,2publicengagementisusedtodescribetheinvolvementofadiversegroupofpeoplethatarenotlimitedtothegeneralpublic,butalsoincludeotherkeygroupssuchaslobbyists,civilsocietyorganisations,andsocialinfluencersinkeydiscussionsanddebates.
Thesediscussionsareconcernedwiththepotentialapplicationsofnewandemergingtechnologies,3theirgovernance,regulationandthewiderissuesthatcouldarisefromthewaythattheyaredesignedandimplemented(Saunders2018).
Publicengagementisavaluablemeanstogetsocietalperspectivesonthepotentialimpactsofscienceandtechnologyonpolicy,legal,ethical,andotherissues(PytlikZillig&Tomkins2011)aswellasmorebroadlycontributetotheeffectiveoversightofscienceandtechnology(Gunashekaretal.
2019;Sutcliffeetal.
2020).
Itcanofferincreasedlegitimacyandaccountability,empowercommunitiesandincreasesocialcohesion,anddeveloppublicservicesthataremoreefficientand1Technologyinvolvestheapplicationofknowledge,methodsandprocessesforpracticalpurposes(OECD2001).
2Inthisreport,weregardtechnologicalinnovationtobetheapplicationsoftechnologieswithinoneormoresectors,e.
g.
theapplicationsofmachinelearningorblockchaintechnologyinhealthcareandfinancialservices.
3Newandemergingscienceandtechnologylacksanaccepteddefinitionbuttypicallyischaracterisedastechnologiesthathavethefollowingattributes:novelty;relativelyfastgrowth;impactonsocietyandtheeconomy;anduncertainty(Rotoloetal.
2015).
19effective,andcreatepersonalbenefitstocitizens(Brodieetal.
2009;Selinetal.
2017).
Thesebenefitsareofparticularimportancetotechnologicalinnovationwherescientificuncertaintiesoftendependondiverseandvalue-basedperspectives,wherethevaluesofexpertsaswellascitizensshouldbeconsidered(Zhaoetal.
2015).
Publicengagementinscienceandtechnologydevelopmentisincreasinglybeingrecognisedbypolicymakersasanimportantmechanismtobringopennessandregulatorytransparencyindecisionmakingandtohelpaddressissueslikepublicmistrustinscience(Eatonetal.
2014;Rempeletal.
2018).
Lackofpublicengagement,ordecisionmakingwithoutpublicsupport,tendstoresultinalackoflegitimacy,whichtakestheformofpublicdistrust(Clift2019;Zhaoetal.
2015).
Technologicalinnovationisincreasinglyregardedas'political'duetoitspotentialimpactonsociety,whichisassociatedwithrisksandbenefits,andethicaldisagreementaboutwhatconstitutesthecommongoodandresponsibleresearch(Zhaoetal.
2015).
Forinstance,lackofengagementhascontributedtosomehigh-profilepubliccontroversiesintheUK,withanegativeimpactonthepublicacceptanceoftechnologiesandtechnologicalinnovationsuchasnuclearpower,geneticallymodifiedorganisms,andnanotechnologies,amongstothers(Clift2019).
Globally,citizenshavetendedtotrustscientists,althoughtherehasbeenaperceptionamongstthepublicinEuropeancountriesthatsciencedoesnot'speaktruthtopower',andthereisdistrustofsometechnologicalinnovationandscientificfields(Funketal.
2020;Hendriksetal.
2016;Hennen2013;Kabat2017).
Thishasestablishedtheimportanceofpublicengagementtobuildtrust,managerisksandethicalproblems,andtounderstandthelegalandsocioeconomicchallengesrelatedtonewtechnologiesandtheirapplications(Gunashekaretal.
2019).
Traditionally,publicengagementhasfocussedonthemobilisationoffixedpublicopinionswherethepublicisperceivedtooperatewitha'blankslate'todevelopopinions(Lezaun&Soneryd2007;Selinetal.
2017;Welpetal.
2006).
Thepublicareperceivedtohaveadeficitorlackofknowledgeonscienceandtechnologywhichinfluencestheirrejectionoftechnologyandscientificideals(Selinetal.
2017;Stilgoeetal.
2014).
Thefocusonpublicengagementinthesemodelshavebeentobuildtrusttosupportpublicacceptanceofscienceandtechnology(Stilgoeetal.
2014).
Intheearly2000s,therewasashifttowardsaviewofengagementashappeningwithmultiplesubgroupsofthepublics.
Thesedifferentsubgroups,however,werestillperceivedas'anti-scientific','concerned'orasproviding'barriers'toinnovation(Selinetal.
2017).
Publicengagementhassinceseenashiftfromthe'one-way'communicationofinformationtoimprovethepublicunderstandingofscienceandtechnologytoa'two-way'communicationprocess.
Moreparticipatorymethodsaimtoactivelygatherinputs(i.
e.
opinions,expertiseandvalues)fromthepublicandstimulatedebate(Wilsdon&Willis2004).
Therehasalsobeenanincreasingfocusonpublic20engagementatdifferentphasesofbothpolicydevelopmentandimplementation(PytlikZillig&Tomkins2011).
Forinstance,theshifttowardsafocusonresponsibleresearchandinnovationinpublicengagementhasbuiltonideasof"anticipatorygovernance,real-timetechnologyassessment,constructivetechnologyassessment,value-sensitivedesignandopeninnovation"thatincorporatesthepublicanduserengagement(Stilgoeetal.
2014).
Newerandatypicalformsofpublicengagementthatfocusoncreatingnewexperiencesforthepublicmayhelptostrengthenthisshifttowardsresponsibleresearchandinnovation.
Asnotedabove,themethodsandapproachestoengagethepublicandthetopicscoveredcanvarywidely.
Examplesofmoretraditionalmethodsincludesurveys,publicconsultations,workshops,focusgroups,townhallmeetings,noticeandcommentopportunitiesandcitizenjuries(Involve2018;PytlikZillig&Tomkins2011;Saunders2018;Ward2009).
Morerecent,innovativemethodsincludetheuseofcreativearts-basedapproaches,seriousgamesanddigitaltechnologies/onlineplatforms(Involve2018;Saunders2018;Ward2009).
Innovativemethodsoftenhavealessdirectlinktoscientificprocesses,buthavebeenfoundtocreateseveralbenefits(Rasketal.
2012).
Inareviewofseveralinnovativepublicengagementcases,innovativemethodshavebeenfoundtoalsoinvolveanumberofdiverseactorswithrepresentationfromthethirdsector(i.
e.
asocialsectorthatconsistsofstakeholderswhocomefromneithertheprivatenorthepublicsectors(NationalAuditOffice,n.
d.
))andthefourthsector(anemergingfieldofhybridgroupsofpeoplewithestablishedinterestsandwhocanbringaboutsocialcooperationthroughhybridnetworking)(Rasketal.
2012).
Innovativemethodsareoftenalsoorientedtowardsaddressingsocietalchallengesandareversatileandpracticalinnature(Rasketal.
2012).
Inthisevolvingcontextofincreasingrecognitionoftheimportanceofpublicengagementinrelationtotechnologicalinnovation,theBREatBEISisleadingtheregulatoryreformagendaacrossUKgovernment.
TheWhitePaperonRegulationfortheFourthIndustrialRevolutionsetsoutplanstotransformtheUK'sregulatorysystemtosupportinnovationwhileprotectingcitizensandtheenvironment(BEIS2019).
Aspartofthisagenda,BREaimstobetterunderstandwhichpublicengagementtechniques,particularlynovelandcreativetechniques,cansupportappropriateregulationoftechnologicalinnovation,andprovideadvicetoregulatorsandpolicymakers.
ThisworkisintendedtoidentifyrelevantevidencetosupporttheBRE'scommitmentto"providesupport,adviceandsharebestpracticewithpolicymakersandregulatorsonpublicengagementtechniquestosupportappropriateregulationoftechnologicalinnovation"(BEIS2019).
21StudyobjectivesUnderstandingwhatworks,inwhatcontextsandforwhatpurposeinrelationtopublicengagementandtechnologicalinnovationisimportanttoinformpolicyandregulation.
Theoverarchingaimofthisstudyistobringtogetherevidenceonthecurrentuseofpublicengagementfortechnologicalinnovation(e.
g.
tosupportpolicyandregulationdevelopment,andmarketadoptionoftechnologicalinnovation).
Theworkfocusesonidentifyingandunderstandingthetypesofengagementapproaches/techniquesusedintherealworld,howthesehavebeenappliedtotechnologicalinnovationacrossdifferentsectorsandanypotentialoutcomes,aswellastheeffectivenessoftheseapproaches(e.
g.
intermsofinfluencingregulation).
Thestudysynthesisesevidenceonthecurrentuseofpublicengagementfortechnologicalinnovation.
Thestudy,whichistargetedatregulatorsandpolicymakerswillprovideanevidencebasetosupporttheworkoftheBRE.
Theworkfocusesonthreemainquestionsinthecontextoftechnologicalinnovation:1.
ResearchQuestion1:Whatexistingexamplesarethereofpublicengagementtechniques,andhowandwhenhavetheybeenapplied2.
ResearchQuestion2:Whatevidenceofimpactexistsinrelationtodifferenttypesofpublicengagementontechnologicalinnovation(e.
g.
intermsofinformingthedesignofregulatoryframeworks,newbusinessmodels,marketadoption,andpublictrust)3.
ResearchQuestion3:Hastheeffectivenessofthepublicengagementtechniquesaroundtechnologicalinnovationbeenformallyevaluated,andwhat,ifany,werethelearningsWeexplorethewholespectrumofpublicengagementmethodsortechniquesthatcouldbeusedinthecontextoftechnologicalinnovation–fromthemoretraditionalapproachestoinnovative,creativeones.
Wealsopresentanunderstandingoftherangeandnatureofstakeholdersinvolvedintheengagementprocess.
Furthermore,weprovideinsightsintotheeffectivenessofthetechniquesthatcouldsupportthedecision-makingofregulatorsandpolicymakers.
SummaryofthemethodologyWeadoptedamixed-methodsapproachtoaddresstheobjectivesofthestudywhichwasconductedovera12-weekperiod.
Theapproachenabledustorapidlyidentify,analyseandsynthesiseevidencerelatedtotheuseofpublicengagementtechniquesinthecontextoftechnologicalinnovation.
Theapproachbroadlyconsistedofthreedistinctbutoverlappingphases:(i)Phase1:Deskresearch;(ii)Phase2:Casestudies;and(iii)Phase3:Analysis.
22Inthefirstphaseofthestudy,weconductedarapidreviewoftheacademicliteraturecoveringpublicengagementtechniquesappliedinthecontextoftechnologicalinnovation.
Thereviewincludedacademicarticlesfrom2016to2020.
Thiswassupplementedwithtargetedsearchesofthegreyliteratureandscopingconsultations(bytelephoneande-mail)withexpertsintheareaofpublicengagement.
4Theprimaryaimofthescopingconsultationswastoidentifypotentiallyrelevantliteratureandexamplesofpotentialcasestudies.
Inthesecondphaseofthestudy,wedevelopedtencasestudiestoprovideconcreteandrecentillustrationsofhowpublicengagementhasbeenappliedinthecontextoftechnologicalinnovationandwhetherandhowthetechniqueshavebeeneffective.
Asthecasestudiesspanneddifferentpublicengagementtechniques,technologyareas,countries,sectors,organisationsandtimeperiods,theyenabledustopaintarichanddiverse'ontheground'pictureassociatedwithemergingdevelopmentsinpublicengagementpractices.
Foreachcasestudy,theinformationwasgatheredthroughinterviewswithstakeholdersandbyreviewingaselectionofarticlesconnectedtothecasestudy.
5DetaileddescriptionsofthecasestudiesareprovidedinAnnexA.
Inthefinalphaseofthestudy,wesynthesisedtheevidencefromthedeskresearchandcasestudiesbydevelopingasetofkeythemesregardingapproachestopublicengagementfortechnologicalinnovationandtheireffectiveness.
Thecross-analysisoftheevidencewascarriedoutinaninternalworkshopattendedbycoremembersofthestudyteam.
ThemethodsforthisstudyaredescribedindetailinAnnexCalongwiththelimitationsoftheanalysis.
StructureofthisreportTherestofthereportisstructuredasfollows:Chapter2focusesonResearchQuestion1,andprovidesadetailedoverviewofexamplesofpublicengagementtechniquesandhowandwhentheyhavebeenappliedinthecontextoftechnologicalinnovation.
Chapter3addressesResearchQuestion2,andfocussesonexistingevidenceofimpactinrelationtodifferenttypesofpublicengagementontechnologicalinnovation.
4Throughoutthereport,scopingconsultationintervieweeinputsarecitedusingtheidentifierINTXXwhereXXisanumberbetween01and07.
5Throughoutthereport,casestudyintervieweeinputsarecitedusingtheidentifierCSXX-YYwhereXXisanumberbetween01and10,andYYisanumberbetween01and03.
23Chapter4addressesResearchQuestion3,andconsiderskeyinsightsfromtheevidenceastheyrelatetowhethertheeffectivenessofpublicengagementtechniquesaroundtechnologicalinnovationhavebeenformallyevaluated,andanypotentiallearningsthatmayhavebeenmade.
InChapter5,weprovidesomeconcludingremarks,andreflectontheimplicationsofthefindingsforregulatorsandpolicymakers.
InAnnexA,wepresentthetencasestudieswedevelopedtohighlightadeeperunderstandingofavarietyofpublicengagementtechniquesappliedtotechnologicalinnovationindifferentcontexts.
InAnnexB,weprovideanoverviewofaselectionofgeneralpublicengagementtechniques(includingthosethathavebeenusedincontextsoutsideoftechnologicalinnovation).
InAnnexC,weprovideadetaileddescriptionofthemethodologicalapproachtoimplementthestudyalongwithkeycaveatsoftheanalysis.
InAnnexD,wepresentthelong-listofcasestudiesfromwhichthefinaltencasestudieswereshort-listed.
24Chapter2:ExamplesofpublicengagementtechniquesappliedinthecontextoftechnologicalinnovationInthissection,wepresentthekeyfindingsassociatedwithResearchQuestion1.
ResearchQuestion1:WhatexistingexamplesarethereofpublicengagementtechniquesandhowandwhenhavetheybeenappliedBox1:SummaryofevidenceonexamplesofpublicengagementtechniquesfromtheliteratureSummaryofevidenceonexistingexamplesofpublicengagementArangeoftraditionalandatypicalpublicengagementtechniqueshavebeenusedinthecontextoftechnologicalinnovation.
Publicengagementtechniquesappliedtotechnologicalinnovationcoverboth'one-way'methods(i.
e.
theflowofinformationisunidirectional)and'two-way'methods(i.
e.
theflowofinformationisbidirectional).
Publicengagementtechniqueshavebeenappliedtoadiverserangeoftechnologies(e.
g.
cognitivetechnologies(includingAIandmachinelearning),data-driventechnologies,medical/biotechnologies,greentechnologies,nanotechnologies).
Publicengagementtechniquesappliedtotechnologicalinnovationcoverarangeofstakeholdertypes(e.
g.
thegeneralpublic,potentialusersorconsumersofthetechnology,policymakers,regulators,industryandexpertsfromvariousdisciplines).
Publicengagementtechniqueshavebeenappliedtovaryingdegreesacrossthetechnologydevelopmentandadoptionpathway:-Differenttechniqueshavebeenused'upstream'toexploreaspectsofpolicy,ethics,regulationandtrust.
25-Differenttechniqueshavebeenused'downstream'toexplorepotentialacceptabilityandmarketadoption,i.
e.
bringingexistingtechnologyintothemarketplace.
-Sometechniquescouldbeusedtogatherlong-termperspectivesovertimeacrossthedevelopmentandadoptionpathway.
ArangeoftraditionalandatypicalpublicengagementtechniqueshavebeenusedinthecontextoftechnologicalinnovationMostoftheexamplescoveredintheliteratureconsistofso-called'traditional'orclassicalpublicengagementtechniques.
6Theseincludewell-establishedmethodstocommunicatewiththepublicusingeducationalcontent(e.
g.
videos)(Middleton2017)orthrougheventssuchassciencefestivals(PintofScience2020).
Therearealsomethodstoconsultandlistentothepublic'sviewsandattitudes,suchassurveys(bothonlineandface-to-face),interviewsandpublicconsultations(BEIS2020).
Therearealsomanyexamplesofface-to-faceandonlineparticipatorymethods(e.
g.
publicdialogues,futureworkshopsandcitizens'juries).
Inadditiontothemoretraditionalexamples,theliteraturecontainsseveralexamplesofcreative,experimentalandatypicalpublicengagementtechniques.
Thesegenerallytendtobeinnovative,'digital-enabled'techniquesandincludeapproachestoconsultandlistensuchassociallistening(e.
g.
usingtextminingandcontentanalysisofcommentsonsocialmedia)(Kelleretal.
2017)andparticipatoryapproachesusingvariousexperimentaltools(e.
g.
algorithms)oronlinediscussionplatforms(vTaiwan,n.
d.
).
Newermethodsincludegamification(e.
g.
facetofacegamesandonlinegameswithscenarios)(Bontouxetal.
2016),participatoryfuturesapproaches(i.
e.
techniquesincorporatingfuturesmethods)(Ramosetal.
2019),andexperimentalapproachesor'livinglabs'involvingexperimentswithpeopleusingorexperiencingatechnology(e.
g.
livetrialsofprototypes,lab-basedtestingofprototypesorVRsimulations)(Dupontetal.
2016).
6Inthisreport,publicengagementtechniqueshavebeengroupedintothreebroadoverarchingcategories:(1)Communicate,whicharecharacterisedbythedeliveryofinformationtothepublictoinformoreducate.
Theseincludetechniquessuchaseducationaltools(e.
g.
videos,presentations,blogs)andfestivals;(2)Consult,whicharecharacterisedbytheactiveorpassivecollectionofinformationfromthepublictolisten,gainknowledgeandunderstanding.
Theseincludetechniquessuchassurveys,interviews,crowdsourcing,sociallistening,focusgroups,publicconsultation;and(3)Participate,whicharecharacterisedbydiscussion,collaborationandco-creationwiththepublic.
Theseincludetechniquessuchasdeliberativetechniques,experientialtechniques,arts-basedtechniques.
ThethreecategoriesbroadlymapontotheInternationalAssociationofPublicParticipation(IAP2)spectrumofpublicparticipation(IAP22018)thatiscomprisedofthefollowingcategories:inform,consult,involve,collaborate,andempower.
'Communicate'and'consult'roughlycorrespondtotheIAP2categories'inform'and'consult',respectively.
Thecategory'participate'broadlyincludesthethreeIAP2spectrumcategories'involve','collaborate'and'empower'.
26Table1providesanoverviewofthedifferentpublicengagementtechniquesidentifiedintheliteratureandcasestudiesinrelationtotechnologicalinnovation(thistableisalsopresentedinthefirstsheetintheaccompanyingExcelfile).
Asnotedpreviously(seenote6),inthisreport,publicengagementtechniqueshavebeenbroadlygroupedintothreeoverarchingcategories:communicate,consult,andparticipate.
Thetechniquesarealsogroupedaccordingtowhethertheyconsistoftypicaloratypicaltechniques.
Examplesofthetypesoftechnologiestowhichthetechniqueshavebeenappliedarealsoindicated.
27Table1:OverviewofthedifferentpublicengagementtechniquesusedinthecontextoftechnologicalinnovationidentifiedintheliteratureandcasestudiesHighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesCommunicateArts-basedImagesofconnectedandautonomousvehiclesTechniquestoengagethepublicthatspan:thevisualarts,performingarts,games,artinstallationsandothertechniques.
InthiscasethetechniqueconsistedofcreatingimagesofconnectedandautonomousvehiclestobeTechnologicalinnovation:ConnectedandautonomousvehiclesSector:TransportCountry:UKCasestudy02;AnnexB.
28HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesdisplayedinLondon'sTransportMuseum.
Theimagesweredesignedtoreflecttheviewsofthepubliconconnectedandautonomousvehicles(whichhadbeencollectedaspartofthevehicledesigncoursesoftheRoyalCollegeofArt).
29HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesCommunicateFestivalPintofScienceForumtoengagethepublicandmakeuseofdiverseformatsforpublicengagement,includingtalks,discussions,workshops,hands-onactivitiesandperformance.
PintofScienceisaninternationalsciencefestivalinwhichresearcherscommunicatetheirTechnologicalinnovation:MultiplescienceandtechnologytopicsSector:MultipleCountry:Global(PintofScience2020)30HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesresearchtothepublicinapubsetting.
CommunicateVideoYourDNAYourSayVideodesignedtoeducateandinformthepubliconaparticulartopic.
Providesinformationonaparticulartopicinamannerthatisengagingandinteresting(i.
e.
usingmetaphorsandlaylanguagetoTechnologicalinnovation:Genomics;CRISPR-mediatedsomaticgenomeeditingSector:HealthCountry:Global(Middleton2017)31HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesdemystifygenomicdatasharingandachildasthemaincharactertogivethevideosanon-threateningandpositivefeel).
ThevideowastranslatedintoPolish,French,German,Russian,Arabic,Swedish,Icelandic,Portuguese,Italian,Japanese,Mandarin,SpanishandUrdu,andcan32HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesthereforebeusedtoreachaudiencesthatarenotconversantwithEnglish.
ConsultConversationcaféSyntheticbiologysciencecaféInformaldialoguewherepeopleareinvitedtodiscussanissueinaninformalsetting(whichcouldhelptoensurethattheprocessisasopenandaccessibleaspossibletoanybodywhoisTechnologicalinnovation:SyntheticbiologySector:HealthCountry:Canada(Navid&Einsiedel2012)33HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesinterestedinthetopic).
ConsultCrowdsourcingMoralMachineProcesstogatherideas,functions,services,orcontactsfromalargeandundefinednetworkofpeople.
Theprincipleofopennessunderpinstheexerciseandusesbottom-upprocesstofindgoalsthatareoftentop-Technologicalinnovation:AutonomousvehiclesSector:TransportCountry:Global(Awadetal.
2018)34HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesdown.
Tendstohappenonline.
MoralMachineisacrowdsourcingwebsitethatalsocontainsseriousgamingelements(seebelow)ConsultCrowdsourcingvTaiwanProcesstogatherideas,functions,services,orcontactsfromalargeandundefinednetworkofpeople.
TheTechnologicalinnovation:Ride-hailing/car-poolingapps;FintechCasestudy06;AnnexB35HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesprincipleofopennessunderpinstheexerciseandusesbottom-upprocesstofindgoalsthatareoftentop-down.
Tendstohappenonline.
vTaiwanisacrowdsourcingplatformthatalsocontainsdeliberativeelements(seebelow).
Sector:Transport;financialservicesCountry:Taiwan36HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesConsultFocusgroupFocusgroupGuideddiscussionofasmallgroupofthepublictogatherinformationonaparticulartopic.
Canbeconductedfacetofaceoronline.
Technologicalinnovation:CRISPR-mediatedsomaticgenomeeditingSector:HealthCountry:US(Persaudetal.
2019)ConsultInterviewInterviewStructuredconversationtogatherinformationfromthepubliconaparticulartopic.
Exploresviews,Technologicalinnovation:Facialanalysistechnology;Smartcities(Woutersetal.
2019)37HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesnormativepositions,beliefs,experiences,andmotivationsofanindividualparticipant.
Thetechniqueprovidesanin-depthunderstandingofacertaintopic.
Canbeconductedfacetofaceoronline.
Sector:surveillance;urbanplanningCountry:France;US38HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesConsultPublicconsultationCitizenconsultationTechniqueusedtoaskgroupsofpeopletodiscusstheiropiniononissues.
Unlimitednumbersofparticipantscanbesentinformationaboutthesubject,downloaditonline,andrespondviaemailorcommentonthewebsite.
FormalconsultationscanTechnologicalinnovation:AISector:MultipleCountry:EU-wideCasestudy07;AnnexB.
39HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesalsoconsistofregulatorytoolsthatareoftenundertakenaspartofthelegislativeprocess(CabinetOffice2018).
ConsultSentimentmappingSentimentmappingTechniquetogatherandanalysesentimentsofpublicperceptionsdata.
ThetechniqueusesnaturallanguageprocessingtoidentifytheemotionsTechnologicalinnovation:ConnectedandautonomousvehiclesSector:TransportCasestudy02;AnnexB.
40HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesexpressedintext(e.
g.
throughtheanalysisofcommentsonsocialmediaplatforms)(Hauthaletal.
2020).
Country:UKConsultSociallisteningTextminingandcontentanalysisofsocialmedia(e.
g.
Twitter,Facebook)Techniquetogatherthe'natural'unfilteredopinionsofthepubliconaparticulartopic.
ViewsaregatheredviatextminingorcontentanalysisofTechnologicalinnovation:VR;CRISPR-mediatedgenomeeditingSector:Health(Calabreseetal.
2019;Kelleretal.
2017)41HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcescommentspostedbythepublic(typicallyonsocialmediae.
g.
Facebook,Twitter).
Country:USCRISPR-mediatedgenomeeditingConsultSurveyTraditionalsurveyToolusedtogatherquantitativeorqualitativeinformationfromagivenpopulation.
Canbeconductedfacetofaceoronline.
Technologicalinnovation:MultiplescienceandtechnologytopicsSector:MultipleCountry:Multiple(BEIS2020)42HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesConsultSurveyAttrakDiffquestionnaire(withaspecificUsereXperience(UX)method)Onlinesurveythatevaluatesapotentialproduct(e.
g.
softwareoraconsumerproduct)bygauginghowtheattractivenessoftheproduct(e.
g.
intermsofitsusabilityorappearance)isexperiencedbypotentialusers.
Inthisexample,thesurveywasusedtounderstandhowdifferentgroupsTechnologicalinnovation:ElectriccarsSector:TransportCountry:France(Dupontetal.
2016)43HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesperceiveelectriccars,withaviewtoinformingacceptabilityfordifferentgroupsandinturninformdissemination.
ConsultSurveyStatedpreferencesurveyOnlinesurveyincorporatingabest-worstchoiceexperimenttoexploretheparticipants'mostandleastpreferredattributesofTechnologicalinnovation:AutonomousvehiclesSector:TransportCountry:US(Shabanpouretal.
2018)44HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesautonomousvehicles.
ParticipateArts-basedParticipatorytheatree.
g.
EuropeanTheatreLabTechniquesthatusecreativeapproachestobringcomplexissuesalivewithaudienceswhowouldnottakepartinmoretraditionalprocesses.
ItencouragesaudienceinteractionandexploresdifferentapproachesTechnologicalinnovation:AI;SmartcitiesSector:UrbanplanningCountry:EU-wide(Engasser&Saunders2018)45HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcestodealwithanissue.
Onenewformofparticipatoryprocessisimmersivetheatre,whichencouragespublicparticipationandco-creation.
Forexample,theStageYourCityProjectledbytheEuropeanTheatreLabusesimmersiveandparticipatorytheatre46HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcestoengagethepublicinthefutureofcitiesandtechnologysuchasAI(EngasserandSaunders2018).
ParticipateArts-basedUrbanwalkandphotographyTechniquestoengagethepublicthatspan:thevisualarts,performingarts,games,artinstallationsandothertechniques.
ThisapproachusesphotographytohelpTechnologicalinnovation:SmartcitiesSector:UrbanplanningCountry:US(Altamirano-Allende&Selin2016)47HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesparticipantscaptureandreflectonaparticulartopic.
Inthisexampleparticipantstookpartinanurbanwalkandwereinvitedtotakephotographsoftheirsurroundings.
Creativemethodthatcanfacilitatedeeperreflectionsandamoremeaningfulengagementand48HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesconversationaroundtechnologicalinnovationandthefuture.
ParticipateDeliberativeDeliberativeworkshops/PublicDialogueWorkshops/DeliberativePolicyWorkshopsDialogueeventsthatfocusonin-depthandinformeddiscussionsofissuesthatareeithercomplexorcontroversial.
Canbeconductedfacetofaceoronline.
DeliberativetechniquesoftenTechnologicalinnovation:Connectedandautonomousvehicles;DigitaltechnologiestocombatCOVID-19;Drones;Genomics;Machinelearning;Mitochondrialreplacementtherapy;(AdaLovelaceInstitute2020;Cameron&Maguire2017;CDEI2020;Drew2016;McCool2019;Miletal.
2019)49HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesinvolvea'mini-public'(i.
e.
arelativelysmallsampleofthepublicthatisdemographicallyrepresentativeofthelargerpopulation)butcanalsoextendtolargersamples(e.
g.
inthecaseofconsensusconferencesorcitizens'summits).
Neuralinterfaces;OnlinetargetingSector:Transport;publichealth;datascience;healthCountry:EU;Taiwan;UK50HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesParticipateDeliberativeCitizens'/communityjuryTechniqueinwhichthepublicdiscussanddeliberateoveratopic(typicallyoneclearlyframedquestion)indepth.
Theaimistoreachadecisionfollowingdeliberationontheissue,eitherbyconsensusorvoting.
Typicallyconductedfacetoface.
Technologicalinnovation:Automateddecisionsystems;WholeGenomeSequencing;BigDataAnalyticsSector:Health;DatascienceCountry:Australia;UK(Degelingetal.
2020;Newsonetal.
2019;RSA2019)51HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesParticipateDeliberativeDeliberativemappingTechniquethatcombinesvariedapproachesandinvolvesbothexperts(around20participants)andthepublic(around40participants)toratediversepolicyoptionsagainstasetofcriteria.
Theobjectiveistocreateaprocessthatallowsformoredemocratic,robustandaccountableTechnologicalinnovation:Geoengineering;Xenotransplantation;Energy-relatedtechnologiesSector:Energy;HealthCountry:UK(Bellamyetal.
2016)52HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesdecisionmakingthatreflectspublicvalues.
Thecitizensandexpertsdiscussissuesseparatelyandthentogetherataworkshop.
ParticipateDeliberativeDistributeddialogueDecentralisedapproachthataimstodevelopdiscussionsthataredispersed,ongoingandembeddedonatopic.
SomeoftheengagementisTechnologicalinnovation:BioenergySector:EnergyCountry:UK(BBSRC2013)53HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesorganisedbyparticipantsorgroupsthemselves.
Itengagesresearchcommunities,stakeholdersandthepublicforstrategyandpolicydevelopment.
Itofteninvolvesseveraldialogueeventsorganisedbyresearchersandinterestedparties,indifferentgeographicalareas54HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesandusingdiversemediums,toengageawiderangeofcommunitiesonagivenissue.
ParticipateDeliberativevTaiwanDedicatedAI-facilitatedsocialmediatool,Pol.
is,whichcrowdsourcesproposalsfromthepublic,andalsoallowsthepublictovoteonotherpeople'sproposalsbyagreeingorTechnologicalinnovation:Ride-hailing/car-poolingapps;FintechSector:Transport;financialservicesCountry:TaiwanCasestudy06;AnnexB.
55HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesdisagreeingwiththem.
Thistechniquealsoincorporatesaspectsofcrowdsourcing(seeabove).
ParticipateExperientialBioMetricMirrorTechniquesthatenablethepublictoexperienceatechnology(eitheratestprototypeintherealworld,interactiveapplications,orTechnologicalinnovation:FacialanalysistechnologySector:Multiple(surveillance)Casestudy08;AnnexB.
56HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesusingVRsimulations).
BioMetricMirrorconsistsofaninteractiveapplicationthatsimulatesfacialanalysistechnology.
Country:USParticipateExperientialVRsimulationTechniquesthatenablethepublictoexperienceatechnology(eitheratestprototypeintheTechnologicalinnovation:AutonomousboatsSector:Transport(Venverlooetal.
2020)57HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesrealworldorusingVRsimulations).
InthiscaseitwasaVRsimulationofautonomousboats.
Country:US58HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesParticipateExperientialLivinglabe.
g.
FabLivingLabTechniquesthatenablethepublictoexperienceatechnology(eitheratestprototypeintherealworldorusingVRsimulations).
FabLivingLabisaplatformthatcombinestheconceptofa'livinglab'witha'FabLab'(FabricationLaboratory–i.
e.
anopenprototypingTechnologicalinnovation:SmartcitiesSector:UrbangovernanceCountry:France(Dupontetal.
2016)59HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesspace,similartotheconceptofadigitalmanufacturingworkshopinitiatedbytheMassachusettsInstituteofTechnology).
Theplatformisusedtoprospectivelyevaluatethepotentialusesandacceptabilityofinnovation,i.
e.
concepts,technologiesor60HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesprojects,andenablesmultiplestakeholderstoparticipatetogethertoco-create,prototypeandtestfutureproductsandservices.
ParticipateExperientialLivinglabe.
g.
GATEwayTechniquesthatenablethepublictoexperienceatechnology(eitheratestprototypeintheTechnologicalinnovation:ConnectedandautonomousvehiclesCasestudy02;AnnexB.
61HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesrealworldorusingVRsimulations).
GATEwayconsistedoflivepublictrialsofconnectedandautonomousvehicles.
Sector:TransportCountry:UKParticipateFutureworkshopCard-basedworkshopIMAGINETechniquetoplanandformavisionofthefuture.
Futureworkshopsareusuallyfocussedonlocalissuesorchallenges,oronTechnologicalinnovation:NanotechnologySector:MultipleCountry:Austria(Feltetal.
2014)62HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcestheplanningofalocalactionconcerningaparticulardevelopment.
Theaimistoinvolveparticipantswhoaredirectlyaffectedbyaproblemandcouldoffersolutions.
Thisspecifictechniqueusessetsofcardsassupportmaterialtoallowthepublictodevelop63HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesvisionsofaparticulartopic.
ParticipateFutureworkshopCIMULACTTechniquetoplanandformavisionofthefuture,andtosetoutandprioritisethestepsrequiredtoachievethevision.
Technologicalinnovation:R&IpolicySector:MultipleCountry:EU-wideCasestudy09;AnnexB.
ParticipateSeriousgamingMoralMachineTechniquesthatserveastoolstogaincomplexknowledgeandstimulatereal-lifeTechnologicalinnovation:AutonomousvehiclesSector:Transport(Awadetal.
2018)64HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourceseventsand/orprocesses,andtoprovidetheparticipantwithaproblem-solvingenvironmentfortraining.
MoralMachineisawebsitethatfeaturesa'seriousgame'tocrowdsourcethepublic'sviewsonhowtheywouldwantautonomousCountry:Global65HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesvehiclestosolvemoraldilemmasinthecontextofunavoidableaccidents.
66HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)KeysourcesParticipateSeriousgamingJointResearchCentreScenarioExplorationSystemTechniquesthatserveastoolstogaincomplexknowledgeandstimulatereal-lifeeventsand/orprocesses,andtoprovidetheparticipantwithaproblem-solvingenvironmentfortraining.
Role-playinggamedevelopedfortheapplicationofTechnologicalinnovation:NanotechnologySector:MultipleCountry:EU-wideCasestudy04;AnnexB.
67HighlevelpublicengagementcategoryPublicengagementtechniqueSpecificexampleEstablishedAtypicalDescriptionanddistinctivefeaturesExemplarcontextwithinwhichthetechniquehasbeenapplied(technologicalinnovation;sector;country)Keysourcesfuturesthinkingtopolicy-making.
Actorsfromthreestakeholdergroups(businesses,policymakersandcivilsocietyorganisations)trytoachievelong-termobjectiveswhilstbeingobservedbythepublic.
Source:RANDEuropeanalysis68Therangeofpublicengagementtechniquesappliedtotechnologicalinnovationcoverboth'one-way'methodsand'two-way'methods'One-way'methodsconsistoftechniquesthathaveunidirectionalinformationflowbetweenthepublic/usersandthedesigners/expertsoftheengagementexercise.
Theseincludetechniquesforcommunicatinginformationtothepublic(e.
g.
throughsciencefestivalsoreducationalmaterialssuchasvideos),andtechniquesthatcangatherortracktheviewsofthepublic(e.
g.
throughsurveys,interviews,focusgroupsandsociallistening).
Certaintypesofone-waypublicengagementtechniquesthatensureclear-cutanswers,suchassurveys,7canbehelpfulwhenaimingtounderstandpublicacceptancerelatedtoasetofspecificornarrowquestionsabouttechnologicalinnovation(Dupontetal.
2016,Shabanpouretal.
2018).
Forexample,surveyshavebeenusedtoconsultthepubliconthemostandleastattractivefeaturesofautonomousvehicles(Shabanpouretal.
2018)andtheperceptionsthepublichasofelectriccarstoinformtheiradoption(Dupontetal.
2019).
Techniquesforcommunicatingtothepublic,suchassciencefestivals,talksorvideos,canrepresentafunandeffectivemechanismtoconveycomplextopicstoalargenumberofpeople(NCCPE2020a).
Forexample,theGenomeEditingPublicEngagementSynergy(GEPES)project,apartnershipbetweentheWellcomeGenomeCampusandtheNationalCoordinatingCentreforPublicEngagement,hascollatedaseriesofeducationalresources(e.
g.
talks,seminars,films,websites,onlinecourses,publicationsetc.
)toengagethepublicwithgenomeediting(NCCPE2018).
Innovativeandcreativeone-waymethodsalsoexist.
Thesemethodsinclude:seriousgames,suchastheMoralMachineexperiment,tocrowdsourcethepublic'sviewsonmoraldecisionsrelatedtoAI(Awadetal.
2018);andscientificorconversationcafés,whichaimtodiscusstopicswiththepublicinaninformalsetting,suchasacaféorbar(Involve2018).
'Two-way'publicengagementtechniquesconsistofbidirectionalinformationflowandfacilitateanexchangebetweentheuser/publicandthetool/administratorinordertocapturenuancesandthediversityofviewsandopinionsofparticipants.
Well-establishedmethodsinthiscategoryincludecertaintypesofconsultativemethods,suchaspublicconsultationsandcitizens'panels,andface-to-faceparticipatorymethods,8suchasdeliberativeworkshops,publicdialoguesandcitizens'juries(Involve2018).
Differenttypesofparticipatorytechniquesfacilitatein-depth,informeddiscussionwiththepublicthatcapturesdeeperunderlyingvaluesandtrade-offs.
Thesetypesoftechniquesareparticularlyusefultoexplorecomplexandcontroversialquestionsaroundtechnologyandtechnologicalinnovation,whereknowledgeiscontestedandtheremightbeimportantethicalandsocial7Thisreporthasincludedawideandinclusivedefinitionofpublicengagement.
Itshould,however,benotedthatinmanycontexts,techniquessuchassurveysareconsideredtobearesearchtool,ratherthanapublicengagementtechnique,particularlyinanacademicsettingormarketresearch(whencompaniesaregatheringinformationfromconsumers).
8Techniquesthatallowcollaborationandco-creationwiththepublic(bothonlineandoffline).
69considerations(Involve2018).
Newerorinnovativeparticipatorytechniquesincludethetestingofreal-worldprototypesthroughlivinglabs(Dupontetal.
2016),experientialtechniquesusingVRandimmersiveexperiences(Venverlooetal.
2020),experimentalinteractiveapplications(Woutersetal.
2019)andgamifiedapproaches.
Thesetypesofexperientialtechniqueshavebeenusedtoenablethepublictoexperiencethetechnologicalinnovation,eitherinarealworldoralabsetting,andfacilitateanexchangebetweenparticipantsandthoseresponsibleforthepublicengagementprocess.
Forexample,livetrialsofautonomousvehiclesinLondonthroughtheGATEwayProjectgavethepublictheopportunitytoexperience(andprovidefeedbackon)thetechnologicalinnovation.
Inanotherexample,theFabLivingLabplatform,whichusesimmersiveVRcombinedwithalivinglabapproach,hasbeenusedtoengagethepublicandotherrelevantstakeholders(e.
g.
urbanplanners)inthecityofNantesinFrancetoco-design,prototypeandtestfutureproductsandservicestodevelopsmartcities(Dupontetal.
2016).
PublicengagementtechniqueshavebeenappliedtoadiverserangeoftechnologiesPublicengagementtechniqueshavebeenusedacrossdifferentareasofemergingscienceandtechnology.
Cognitivetechnologies(includingAIandmachinelearning),data-driventechnologies,andmedical/biotechnologiesarethemostcommontypesoftechnologiescoveredinthereviewedliterature.
Othertypesoftechnologiesincludenanotechnologies,neurotechnologiesandgreentechnologies.
70ThepublicengagementtechniquesappliedtotechnologicalinnovationcoverarangeofstakeholdertypesInsomeofthemoretraditionalexamplesofpublicengagement,stepsareoftentakentoensurethatthepeopleinvolvedrepresentadiversecross-sectionofsociety(thiscanbeamini-public).
9Inthisconnection,deliberativetechniquessuchasface-to-facepublicdialogueshavetypicallyaimedtoengagewithadiverseandrepresentativegroupofthegeneralpublicwhoarenotnecessarilyfamiliarwiththescienceortechnology.
Inseveraloftheseexamples,theengagementwascarefultoincludegroupsthatareeitherfrequentorinfrequentusersofatechnologicalinnovationtoensurethatviewsarebalanced.
Forexample,apublicdialogueaboutgenomicsincludedgroupsexpectedtobebothless(blackandminorityethnicgroups)andmoreenthusiastic(heavyusersoftheNHS)aboutgenomicsanddatasharing(IpsosMORI2019).
Similarly,surveys,suchasthewell-establishedBritishSocialAttitudessurvey(NatCen2021)andPublicAttitudestoSciencesurvey(BEIS2020),aswellasothers(AdaLovelaceInstitute2019),typicallyaimtoensurethatthesamplesarerepresentative.
Insomecases,aspecificsubsetofthepopulationisengagedtobetterunderstandspecificconcerns.
Forinstance,anumberofdeliberativepublicdialogueshavesoughttoincludespecificgroupsofinterest(e.
g.
minoritiesorvulnerablegroups)inordertobetterunderstandspecificconcernsamongstthesegroups.
Inoneexample,apublicdialogueononlinetargetingincludedfourspecificinterestgroups:youngpeople,ethnicminorities,thosewithexperienceofmentalhealthissuesandthosewithfinancialvulnerability(CDEI2020).
Insomecases,wherethetechnologicalinnovationismoreclearlydefined,techniqueshavebeenusedtoengagepotentialusersofthetechnology.
Forexample,focusgroupswerecarriedouttogaugetheviewsandconcernsofsicklecelldiseasepatientsonparticipationinfutureCRISPR-mediatedsomaticgenomeeditingclinicaltrials(Persaudetal.
2019).
Inanotherexample,civilservantswereinvolvedas'thepublic'inapublicengagementprocesstodetermineprinciplesforanethicalframeworkarounddatascienceingovernment(Drew2016).
Atveryearlystagesofdevelopmentwhenthetechnologicalinnovationandpotentialpublicsarepoorlydefined,publicengagementhasalsobeendirectedatexpertswhoareveryfamiliarwiththetechnologytohelpdetermineemergentpublicsandpotentialethicalconcernsthatcouldarise(Buckleyetal.
2017).
Expertsarealsousedaspartofdeliberativeprocesses(e.
g.
aspartofanexpertpanel)toprovideinformationtothepubliconaparticulartopic(Involve2018).
9Asampleofthepublicthatisdemographicallyrepresentativeofthelargerpopulationthatisrandomlyselectedtodiscussanissueofpublicconcern(SmithandSetl2018).
71PublicengagementtechniqueshavebeenappliedtovaryingdegreesacrossthetechnologydevelopmentandadoptionpathwayBox2:DefiningupstreamanddownstreampublicengagementtechniquesUpstreamanddownstreampublicengagementtechniquesInthefollowingdiscussion:'Upstream'publicengagementreferstoengagementwiththepublicaroundpotentiallycontroversialscienceandtechnology,atanearlierstageofdevelopmentbeforeeventualapplicationsareclearlydetermined'Downstream'publicengagementtakesplaceatastagewhereatechnologyismoremature,forexampleinprototypeform,andmaybewaitingtobeexploited.
Source:(WilsdonandWillis2004)Anumberoftechniqueshavebeenused'upstream'togathertheviewsofthepublicordiscussissuesindepthtohelpinformaspectsofregulation,ethicsandpolicySurveysareawell-establishedandrelativelystraightforwardmethodtogatherthepublic'sviewsandconcernsrelatedtodifferenttypesoftechnologicalinnovationSurveysareawell-establishedtechniquetocollectinputfromasampleofthepublic,includingforexampletheperceptionoftrustrelatedtoaparticulartechnologicalinnovation,concernandacceptabilityofatechnologicalinnovation,opinionsontheroleofthetechnologicalinnovationinsociety,andtheintroductionofnewtechnologiestocertainsectors.
Severallongstandingsurveys,includingtheBritishSocialAttitudessurvey(NatCen2021),andthePublicAttitudestoScienceSurvey(BEIS2020),havebeenusedtogatherpublicperceptionsofarangeofissues,includingemergingscienceandtechnology(e.
g.
nanotechnology,GMfoodsandattitudestoscienceregulation).
Otherexampleshaveincludedsurveystounderstandthepublic'slevelofconcernforthepotentialapplicationsofbrain-computerinterfaces(Sampleetal.
2020)andpublicawarenessandacceptabilityofhydrogenenergytechnologies(Hienukietal.
2019).
Surveyscanbeusedtodeterminebroad,generalinformationonalargenumberofpeople,oranswerstomorespecific,issue-basedquestionsamongstasmaller,targetedgroupofindividuals.
Althoughtheycanbeusedasastandalonemethod,theycanalsoformacomponentofabroaderengagementprocess(e.
g.
asafollow-uptoothertypesofengagementmethods).
Ingeneral,surveysarearelativelylow-costtechniquethatcanbeappliedatscale,particularlywhenconductedonline,andcanbeusedtogatherbothqualitativeandquantitativeinformation.
72Deliberativemethodshavebeenusedtoprovidein-depthinsightintotheviewsofthepublictoinformaspectsofethics,policyandregulationDeliberativemethodsconsistofdifferenttypesofmethodsthatenablethepublictoconsideratopicindepth,typicallythroughfacilitateddiscussion,withaviewtoarrivingataninformedopinionordecision(Involve2018).
Deliberativemethodsfacilitateanexchangebetweentheuser/publicandthemethodandtool,inordertocapturenuancesandthediversityofviewsandopinionsofparticipants,andsoareparticularlyusefulinexploringcomplexorcontroversialtopicsrelatedtotechnologicalinnovation,includingearlyonindevelopment(Involve2018).
Well-establishedexamplesofmethodsthatinvolvedeliberationincludepublicdialogues,citizens'juries,anddeliberativemapping.
Thesetechniquesfallinthe'involve''collaborate'and'empower'categoriesoftheIAP2spectrum.
Severalintervieweeshighlightedthatpublicdialogueshavefrequentlybeenusedtoexplorethepublic'sviewsandconcernsaroundtechnologicalinnovation(INT01;INT06).
Deliberativepublicdialogueshavealsobeenusedtoinformaspectsofpolicy,ethics,regulationandalternativestoregulationconcerningscienceandtechnologyinanumberofsectors,includinghealth,transportanddatascience.
Oneintervieweenotedthattheyhavebeenusedtypicallytoinformaspecificdecisionorinformprinciplesbywhichadecisioncanbetaken,ratherthantoinformregulation(INT01).
IntheUK,thereareseveralrecentinstancesofupstreampublicdialogueexercises(suchasworkshops)beingusedtoinformethicsconcerningAIanddata-driventechnologies(CDEI2020;Drew2016).
TheRoyalSocietyconductedapublicdialoguein2019tounderstandpublicopiniononthefieldofneuralinterfaces,howitmaydevelopinthefutureandtoco-createrealisticvisionsforthefutureofthisemergingtechnology(Miletal.
2019).
IntheUS,researchersconductedaworkshopwithexpertsinnanobiosensorstocarryoutanearlyethicsassessmentofnanobiosensorsappliedtothemanagementoflivestockdiseaserisksintheearlystagesoftheirdevelopment.
Theaimwastogathertheviewsofexpertsonanyethicalconcernsoffuturepublics(e.
g.
futureusersofatechnologicalinnovation)toinformaspectsofanticipatorygovernance(Buckleyetal.
2017).
IntheUK,theHumanFertilisationandEmbryologyAuthority(HFEA)usedabreadthofengagementtoolsincludingworkshops,apublicsurvey,openmeetingsandfocusgroupstodeterminepublicacceptabilityofmitochondrialreplacementtherapywithaviewtoinformingtheHFEA'sadvicetotheUKgovernmentonproposedregulations(Sheikh2013).
Citizens'jurieshavealsobeenusedtoexplorecomplexethicalquestionsrelatedtotechnologicalinnovation.
Forexample,in2018to2019,acommunityjuryofmembersofauniversityinAustraliawasusedtoconsidertheacceptabilityandlegitimacyofusingpathogenwholegenomesequencingandbigdataanalyticstoenhancepublichealthresearchandcommunicablediseasesurveillance(Degelingetal.
2020).
Inanotherexample,theRSAconvenedacitizens'juryin2018to73deliberateontheethicaluseofAIanditsusetohelpmakedecisionsforDeepMindonitsservices(RSA2019)(seeCasestudy03;AnnexA).
Citizens'juriesaretypicallyusedtoexploreparticularlycontroversialorpolarisingtopicswithethicalconsiderations(Involve2018).
Theaimoftheseexercisesistoreachaconsensusonaclearlyframedquestion(Involve2018).
Thistechniquecanbeconsideredtofallinthecategories'involve'and'empower'asittypicallyaimstoimplementthedecisionsofthepublic.
Moreinnovativeexamplesofdeliberativemethodsincludedigital-enableddeliberativeapproaches.
In2015,thegovernmentofTaiwanusedanonlineplatform('vTaiwan')toengagethepublicinalarge-scaledeliberationexerciseontheregulationofUberinTaiwan(CentreforPublicImpact2019)(seeCasestudy06;AnnexA).
TheplatformusesanAI-facilitatedsocialmediatool,Pol.
is,whichallowsuserstopropose,voteandreachconsensusonsolutions.
Thisapproachrepresentsarelativelynovelapproachtodeliberationinthatitisbothlarge-scale(e.
g.
inoneexercise31,115votesweremadeondifferentproposals)andonline.
Unlikesomeofthemoreestablisheddeliberativetechniques,thistechniquecanbeconsideredtofallinthecategory'empower'asitaimstoimplementthedecisionsofthepublic.
Inanotherexample,theAdaLovelaceInstituteincollaborationwithothersranarapidonlinedeliberationwith28membersofthepublictoexploretheuseofdigitaltechnologiestocombatCOVID-19(AdaLovelaceInstituteetal.
2020)(seeCasestudy10;AnnexA).
Theprocesswasconductedonlineusingtwoonlineplatforms.
Oneintervieweenotedthatthismayrepresentaparticularlyusefulapproachtoconductdeliberationrapidly(i.
e.
overaperiodof10weeksratherthanmonths),suchasincrisissituationsorarapidlyevolvingtopic(CS10-02).
Deliberativeapproachesprovideinformedandconsideredpublicopiniondata,andtypicallyinvolvethedevelopmentofviewsovertime,togetbeyondinitial,surfacereactions(AdaLovelaceInstituteetal.
2020)(INT01;INT06).
Theyareparticularlyusefultoexploretopicsin-depth(e.
g.
challenges,concerns,opportunities,ethicalquestions,futureuses),considernuances,andproposesolutions(Involve2018).
Astheseprocessestypicallytakeplaceoverarelativelylongperiodoftime(i.
e.
severalweekstomonths)andrequireconsiderableresourcetosetupandfacilitate,theyaretypicallycostly(Involve2018).
Neweronlineapproaches,however,mayrepresentamorecost-effectiveapproach(AdaLovelaceInstitute2020).
DeliberativemappingisatechniquetoincreasethediversityofframingsDeliberativemappingtypicallytakesplaceduringmeetingsandworkshopsandinvolvesexpertsandmembersofthepublicusingeconomic,social,ethicalandscientificcriteriatoratedifferentpolicyoptionsinsituationswherethereisnoobviouswayforward(Involve2018).
Deliberativemappingrepresentsamethodthathasbeenusedinthecontextofenergyandmedicaltechnologies.
Thisisan'upstream'methodthataimsto'openup'thediscussiontoabroaderdiversityof74framings(i.
e.
decisionsonhowatopicispresented)andfuturepathways(Bellamyetal.
2016).
In2012,researchersconductedadeliberativemappingexercisetoengagebothexpertsandthepublicongeoengineering(Bellamyetal.
2016).
Thetechniquehasalsobeenappliedtootheremergingtechnologiessuchasxenotransplantation(Daviesetal.
2003)andenergy-relatedtechnologies(Burgessetal.
2004).
Thistechniquehelpstobroadenperspectivesratherthanfocusonsinglesolutionsandisparticularlyusefulwhenconsideringacomplexpolicyissuewherethereisnoobviouswayforward(Involve2018).
Thisapproachcanberelativelycostlysinceitinvolvesconsiderabletimeandresourcestoorganiseseveralmeetingsandexpertinput(Involve2018).
Deliberativemappingcanbeconsideredtofallinthecategory'collaborate'asittypicallyaimstoco-createsolutionswiththepublic.
SociallisteningcanbeusefulforexploringunfilteredviewsonlineandatscaleSociallisteningrepresentsarelativelynovelunstructuredmethodof'listening'thatcanenablethecollectionofunfilteredand'natural'views(Calabreseetal.
2019;Kelleretal.
2017)(seeCasestudy05;AnnexA).
Socialmediarepresentsonesuchtoolthatcanbeusedtounderstandandanalysepublicdiscussionaroundtechnologicalinnovation(Calabreseetal.
2019;Kelleretal.
2017).
Themethodtypicallyinvolvesanalysisofasetofcommentsfromthepublicononlineforums(e.
g.
socialmediasuchasFacebookorTwitter)eitherviatextminingorcontentanalysis.
OnestudyintheUSusedsociallisteningtoanalyseFacebookcommentstoexaminepublicopinionabouttheuseofVRtechnologyinhealthcare(suchastohelppatientscontrolpain,treatanxietydisorders,supportphysicalrehabilitation,andtodistractpatientsduringtreatment)(Kelleretal.
2017).
AnotherstudyconductedcontentanalysisonTwittertounderstandthepublic'sperceptionssurroundingCRISPR-mediatedgenomeediting(Calabreseetal.
2019).
Thistechniquerepresentsalow-costmethodtogatherlarge-scaleviews.
Thistechniquecouldbeusedtogivepolicymakersarapid'snapshot'ofdataonpublicviewsatagiventimeonaparticulartopic,ortoinformfurtherpublicengagement.
Indeed,onecasestudyintervieweenotedthatitwouldbeusefultocombinesociallisteningwithmorefocuseddiscussion(e.
g.
usingparticipatorymethods)withparticipantstoprobesomeoftheissuesandopinionsraisedtoultimatelyexploretheunderlyingviewsandconcerns(CS05-01;seeCasestudy05;AnnexA).
ExperimentaltechniqueshavebeenusedtoenablethepublictoexperienceanewtechnologyoraprototypeNewerexperiential-basedtechniquesincludeexperimentalmethodsandlivinglabs.
ExperimentalmethodsincludetheuseofVRsimulationsandimmersiveexperiences(Venverlooetal.
2020),andexperimentalinteractiveapplications(Woutersetal.
2019)(seeCasestudy08;AnnexA)thatenablethepublictoexperienceaprototypeofatechnologicalinnovationtypicallyinalab-basedsetting.
Forexample,researchersinAustraliaconductedpublicengagementaroundfacialanalysistechnologyusinganinteractiveapplication(BiometricMirror),ademonstratorthat75enabledpeopletoexperiencethepotentialimpactsoffacialanalysistechnology(Woutersetal.
2019)(seeCasestudy08;AnnexA).
Inanotherexample,researchersconductedanimmersiveVRexperimenttoassessthepublic'sperceptionsofautonomousboatsthatareintendedtobedeployedintheNetherlands(Venverlooetal.
2020).
IntheUK,theGATEwayProjectinvitedmembersofthepublictoparticipateinlive'publictrials'ofconnectedandautonomousvehicles(CAVs)(seeCasestudy02;AnnexA).
TheprojectenabledlaypeopletoexperiencedifferenttypesofCAVsinareal-worldsetting,using,forexample,driverlesspodshuttleservices,automatedvaletparking,andautomateddeliveryservices.
Livinglabsaimtocollaboratewiththepublictoco-designsolutions.
Thesetypesofexperientialtechniquesoffercreativewaysthatallowpeopletoplacethemselvesinafutureworldandexperimentwithnewbehavioursorvalues(Ramosetal.
2019).
Byprovidingsimulatedreal-worldexperiences,someofthesetechniquesprovideemotionalandevocativeexperiencesforthepublic,andcanbettersupportthemtoreflectonhowtechnologicalinnovationwillimpacttheireverydaylives(INT07).
Thesetypesofexperientialtechniquesdeploypublicengagement'inthefield'toenableparticipantstoengagewithrealisticapplicationscenarios(Woutersetal.
2019).
Theyareusefultoexploretechnologicalinnovationsthathavehardwareassociatedwiththemandcanbeexperienced.
Thesetechniquesarelikelytorequireconsiderabletimeandresourcestosetupandrun.
Itcanbeusefultosupplementthesetypesofmethodswithotherengagementapproaches(e.
g.
survey,interviews,consultationmeetingordeliberativemethods)tofurtherprobeandunderstandthepublic'sperceptionsandconcernsaroundtheirexperience(seeCasestudies02,08;AnnexA).
Arts-basedtechniquesrepresentanalternative,creativeapproachtoengageawiderspectrumofthepublicArts-basedmethodsspanmultipleapproachesincludingthevisualarts,performingarts,games,immersiveartinstallationsandotherapproaches(e.
g.
poetryandbooks).
Together,thesetechniquesrepresentauser-friendlyandcreativeapproachtoengageawiderspectrumofthepubliconcomplexandcontroversialtopics(Altamirano-Allende&Selin2016;Engasser&Saunders2018).
Creativemethodscanelicitreflectionfromthepubliconcomplextechnologicalinnovationsinawaythatisfun(Altamirano-Allende&Selin2016,Engasser&Saunders2018).
Forexample,intheUS,alarge-scalepublicengagementexerciseaimedtoexplorethedevelopmentoftechnologyintheurbanenvironmentandinvitedparticipantstotakephotographsduringacitytourandexploretheirfindingsduringfacilitatedsessions(Altamirano-Allende&Selin2016).
Otherapproachesinvolvetheuseofparticipatoryandimmersivetheatrewhichusesdigitalelementssuchasaugmentedrealityandsmartphoneappstoengageandencouragefeedbackfromtheaudienceinreal-time.
Forexample,theStageYourCityproject,ledbytheEuropeanTheatreLab76(https://www.
europeantheatrelab.
eu/),usedparticipatoryandimmersivetheatretoengagethepublicinthefutureofcities(Engasser&Saunders2018).
Thesetypesoftechniquesaimtoinvolveandcollaboratewiththepublictoco-createsolutions.
Gamesrepresentafunandengagingwaytoraiseawarenessofatopicortoinvolvethepublicinfuturescenarios.
Theyhavealsobeenusedtoconsultthepublic–forexample,researchersintheUSdevelopedMoralMachine,anonlineplatformandgameforgatheringthepublic'sviewsonmoraldecisionsmadebymachineintelligence,suchasself-drivingcars(Awadetal.
2018)(seeCasestudy01;AnnexA).
Theaimwastocrowdsourcethepublic'sviewsonhowtheywouldwantautonomousvehiclestosolvemoraldilemmasinthecontextofunavoidableaccidents.
Thiswasalarge-scaleandglobalexercisethatgatheredtheviewsofnearly40millionpeople.
Inanotherexample,theJointResearchCentredevelopedarole-playinggame,theScenarioExplorationSystem,aforesightgamingsystemdevelopedfortheapplicationoffuturesthinkingtopolicymaking(Bontouxetal.
2016;Bontouxetal.
2020)(seeCasestudy04;AnnexA).
Therole-playinggamewasusedaspartoftheNANO2ALLprojecttoengagethepublicandotherstakeholders(e.
g.
policymakers)toreflectin-depthaboutfutureapplicationsofnanotechnologies,andtoensurethatresearchandinnovationismoreresponsibleinthisarea.
Throughplayingthegame,usersgainanemotionalconnectiontotheirassignedroleinthegame.
Thisconnectionmaythusfacilitateconstructiveengagementwithstakeholdergroupswhomightnototherwiseengagewithcomplexissues.
Thesetypesoftechniquesrangeacrossdifferentcategories,frominformingandconsultingthepublictoinvolvingthemandcollaboratingwiththemtodevelopsolutions.
Participatoryfuturestechniquesrefertoarangeofapproachesthatcanbeconsideredacrossoverbetweenpublicengagementandthefieldoffuturesstudies(Ramosetal.
2019).
10Theytypicallycombineconventionalfuturesmethods(e.
g.
scenario-basedmethods)withgames,immersiveexperiences,artandwriting(Ramosetal.
2019).
Participatoryfuturesalsoincludedeliberativemethodsbycombiningworkshopsanddiscussionwithcreativeordigital-enabledapproaches.
Theyhavebeenusedwiththepublictoexploreorshapepotentialfutures(Ramosetal.
2019).
Ashighlightedbyoneinterviewee,theaimofparticipatoryfuturesistoenablethepublictoexperiencethefutureinatangibleandmeaningfulmannerbyengagingpeople'semotionsinanon-analyticalmanner(INT07).
Bymovingawayfromanarrowviewofthetechnologicalinnovationinisolation,theycanfacilitatediscussionsaroundsocialandculturesissuesandassumptionsthatsurroundatechnologicalinnovation.
Thesetypesofapproachesaimtoinvolveandcollaboratetogetherwiththepublictodevelopsolutions.
10Thefieldoffuturesstudiesusesmultiplemethodstosystematicallyexploreemergingtechnological,social,economicandpoliticaltrends;visionsandscenariosofthefuture;andpotentialpolicyoptions.
77Todate,arts-basedapproacheshavelargelybeenusedinthecontextofcommunicatinginformationtoenhancepublicunderstandingofscienceandtechnology.
Theseapproacheshavealsobeenusedlessinthecontextoftechnologicalinnovation.
Goingforwards,theyhavethepotentialtobeusedtoagreaterextentinaparticipatorymannertohaveinformedandinclusivediscussions(Engasser&Saunders2018).
Forexample,newformsoftheatresuchasparticipatoryandimmersivetheatrerepresentspotentialforparticipationandco-creation,whichcouldhelptoimprovetheinclusivenessofpolicymaking(Engasser&Saunders2018).
Arts-basedapproachesarerelativelycostlytorunastheyrequireconsiderableskilltosetup(asinthecaseoftheatre)orthehiringofprofessionalstaff(e.
g.
agamedesigner).
Anumberoftechniqueshavealsobeenused'downstream'tohelpinformacceptanceandpotentialmarketadoptionoftechnologicalinnovationSurveys(somewithexperimentalcomponents)havebeenusedtoanalysefactorsthatcouldhelpinformdecisionsregardingmarketadoptionoftechnologiesSurveysmaybeusefulforinformingmarketadoptionofanewtechnologicalinnovationastheypermitthegatheringofviewsatscaleforthesegmentationofconsumers.
Thiscanhelptoidentifydifferentsubgroupsofthepublicandthefactorsthatarelikelytoinfluenceacceptabilityofatechnologicalinnovation.
Thereareseveralexamplesofsuchuseofsurveysintheliterature.
AnonlinesurveyinGermanywasusedtoanalysefactorsthatmightinfluencethelikelihoodofadoptionofanewproduct,inthiscaseacarbonderivedfoammattress(usingcarboncaptureandutilisationtechnology)(Arningetal.
2018).
Numerousstudieshavealsousedsurveystodetermineacceptabilityofnewtransporttechnology,suchasdriverlessandelectriccars.
IntheUS,anonlinesurveyincorporatingabest-worstchoiceexperimentwasusedtoinvestigateparticipants'preferencesaboutthemostandleastattractivefeaturesofdriverlesscars(Shabanpouretal.
2018).
InSwitzerland,asurveyincorporatingamethodcalled'phenomenology'aimedtoidentifyusers'perceptionsofrisksrelatedtodriverlesscars,whichcanaffecttheefficiencyandtheperceptionoftheservice(Pfeifferetal.
2020).
AstudyinFranceusedanovelsurveytool,theAttrakDiff-2questionnaire,aspecific'UsereXperience(UX)method',11toidentifyfactorsthatdeterminepotentialacceptabilityofelectriccarsenablingtheiruptakebyconsumerswithaviewtosupportingsmartcitypolicies(Dupontetal.
2019).
Thesetypesoftechniqueaimtoconsultthepublictogatherinformationorviewsondifferentoptionstoinformdecision-making.
11Onlinesurveythatevaluatesapotentialproduct(e.
g.
software,consumerproducts)bygauginghowtheattractivenessoftheproduct(e.
g.
intermsofusability,appearance)isexperiencedbypotentialusers.
78LivinglabshavebeenusedtotestprototypeswithaviewtoinformingmarketadoptionLivinglabs12aredefinedasinteractionspacesinwhichmultiplestakeholdersfromacademia,industry,thepublicsectorandthepubliccometogethertotestprototypesofatechnologicalinnovationina'real-world'environmentpotentiallytohelpinformthescalingupandroll-outofthattechnologicalinnovation.
Theyhavebeenusedinthecontextofautonomousvehiclesandsmartcities(Venverlooetal.
2020).
Forexample,theGATEwayProject,apartnershipbetweenindustryandthepublicsector,conductedlivetrialsofconnectedandautonomousvehiclesoverathree-yearperiodinLondon(SmartMobilityLivingLab2020)(seeCasestudy02;AnnexA).
Thesetrialsprovidedthepublicwiththeopportunitytoexperience(andprovidefeedbackon)differenttypesofconnectedandautonomousvehiclesinalive,real-worldsetting.
Alongsidepublictrials,theGATEwayProjectalsoemployedotherengagementmethods(seeCasestudy02;AnnexA).
IncollaborationwiththeRoyalCollegeofArt,theprojectranaseriesofpublicworkshops.
tounderstandpeople'sperceptionstowardsCAVsandtoexplorehowthedesignofCAVsmightinfluencepublicattitudestowardsthem.
Inanotherexample,researchersdevelopedthe'FabLivingLab'platformtoenablemultiplestakeholderstocollaboratetoco-design,prototypeandtestfutureproductsandservices(Dupontetal.
2016).
Theplatformhasbeenusedinaparticipatoryprocesstorebuildaneco-neighbourhoodinthecityofNancyinFrance.
Livinglabsgivethepublicanopportunitytoexperienceanewtechnologicalinnovationoraprototypeandinformitsdevelopmentinasafeandacceptablemanner.
Thesemethodsinvolvecontrolledexperimentsinalivesetting,andsorequireconsiderabletimeandresources,andarethereforecostly.
Theyareparticularlyusefulinthelaterstagesofdevelopmentofatechnologicalinnovationtohelpinformacceptability,withaviewtoscalinguptheinnovation.
Itisusefultosupplementthelivetrialapproachwithotherpublicengagementapproaches(e.
g.
survey,interviews,consultationordeliberativemethods)tofurtherunderstandandprobethepublic'sviews,perceptionsandconcernsaroundtheirexperience(seeCasestudy02;AnnexA).
Thesetypesofapproachesaimtoinvolveandcollaboratewiththepublictodevelopsolutions.
DistributeddialoguehasbeenusedtoconductpublicengagementwithscienceandtechnologyovertimeDistributeddialogueconsistsofadecentralisedapproachtopublicengagementthataimstodevelopongoingdiscussionsaroundatopic,withmultipleengagementeventsorprocesseshappeningindifferentgeographicalareas(Involve2018).
TheBiotechnologyandBiologicalSciencesResearchCouncil(BBSRC)organiseda12Livinglabsdonothaveauniversallyaccepteddefinition.
Theyhavebeendefinedasenvironmentsandinteractionspacesinwhichdifferentstakeholders(e.
g.
thepublic,industry,governmentandacademia)cancometogethertotestandscaleupnewproductsorservices(ofteninareal-worldsetting).
79'BioenergyDistributedDialogue'todevelopanongoingdiscussionaroundbioenergyresearchwithitsresearchcommunityandmembersofthepublic(BBSRC2013).
Thistechniquefallsintothecategory'involve'asitaimstounderstandandtakeintoaccounttheviewsandconcernsofthepublic.
Thisapproachrecognisesthatpublicengagementdoesnotnecessarilyconsistofaone-offdiscreteeventbutcanhappenonanongoingbasis.
Itacknowledgesthatthepublic'sviewsonemergingtechnologyisnotfixed,buttypicallyevolvesovertime.
Therefore,theseapproachescanbeusefulforbothdevelopingongoinginformeddiscussionsandtrackingpublicopinionovertime.
Thereareexamplesofinnovativeapproachesthataimtocollectandmappublicengagementovertimeincontextsoutsideoftechnologicalinnovation(INT02).
Box3providesanexampleofalongitudinalapproachtopublicengagementinthecontextofenergyandthenet-zerotransition(INT02).
Along-termapproach,however,maynotbeappropriateinthecontextoftechnologicalinnovationduetothe'pacingproblem'.
13Itisusuallyacknowledgedthatengagingthepublicearlyoninthetechnologicalinnovationpathwaycanhelptomitigatepotentialnegativeunforeseenconsequences.
Box3:Exampleofalongitudinalapproachtopublicengagementinenergyandthenetzero-transitionMappingpublicengagementovertime:energyandnet-zerotransitionMappingapproachesaimtoanalysediverseformsofpublicengagementonanongoingbasistoformamorecomprehensiveevidencebaseofpublicviewsandconcernsaroundemergingscienceandtechnology.
IntheUK,theObservatoryforSocietalEngagementwithEnergyaimstomapdiverseformsofsocietalengagementinenergyandnet-zerotransitions(UKERC2021).
Thisapproachaimstodevelopdigital,systematicreviewandcrowdsourcingmethodstomapsocietalengagementwithenergyandnet-zerotransitionsacrossthewholeUKenergysystemandaroundkeyemergingissues.
Whilenotexplicitlyusedinthecontextoftechnologicalinnovation,thisrepresentsa'whole-systemsapproach'togenerateevidenceaboutdifferentformsofpublicparticipationonanongoingbasis(UKERC2021.
13Thepacingproblemreferstothegapbetweenthedevelopmentofatechnologyandsocial,political,ethicalandregulatorydevelopments.
80Chapter3:ExistingevidenceofimpactinrelationtodifferenttypesofpublicengagementtechniquesontechnologicalinnovationInthissection,wepresentthekeyfindingsassociatedwithResearchQuestion2.
ResearchQuestion2:WhatevidenceofimpactexistsinrelationtodifferenttypesofpublicengagementontechnologicalinnovationBox4:SummaryofevidenceonimpactsfrompublicengagementtechniquesappliedtotechnologicalinnovationEvidenceonimpactsfrompublicengagementtechniquesMostoftheevidenceonimpactsofpublicengagementtechniquesisfoundinthegreyliterature,andrelatestowell-establishedmethods,inparticular,deliberativetechniques(e.
g.
publicdialogues).
-Variousparticipatorymethods,inparticulardeliberativetechniques(e.
g.
publicdialoguesandcitizens'juries)haveinformedaspectsofregulation,ethics,policyandtrustinrelationtotechnologicalinnovation.
Experientialtechniqueshavealsobeenusedtoexploretrustandconfidence.
-Consultative(i.
e.
surveys)andparticipatory(i.
e.
livinglabs)techniqueshavebeenusedtodetermineacceptanceoftechnologicalinnovationandtestprototypesrespectively,andtheresultscouldbeusedtoinformmarketadoption(i.
e.
bringingexistingtechnologicalinnovationintothemarketplace),buttheevidenceislacking.
Thereiscurrentlylimitedevidenceontheimpactsofsomeofthemoreinnovativeatypicalapproaches(e.
g.
games,experientialapproachesandarts-basedapproaches).
Mostoftheevidenceontheimpactsofpublicengagementtechniquesisfoundinthegreyliterature,andrelatestowell-establishedmethods,inparticulardeliberativemethodssuchaspublicdialogues.
Forexample,Sciencewisehasconductedformal81evaluationsofanumberoftheirengagementprocesses,inlinewiththeirstatedaimofsupportingdialogueinareaswherethereisaclearcommitmenttotheprojectfindingsinformingpolicydevelopment(Sciencewise2021e).
Thisevidencesuggeststhatface-to-facedeliberativemethodshavehadanimpactonaspectsofpolicy,ethics,andregulation(Arningetal.
2018;CDEI2020;RSA2019;Sciencewise2021e;Shabanpouretal.
2018).
Table2(presentedinthesecondsheetintheaccompanyingExcelfile)presentsanoverviewoftheevidencepresentedinthisreportontheimpactsinrelationtodifferentoutcomesforthedifferentpublicengagementtechniquescoveredintheliteratureandcasestudies.
ThereisevidencethatdifferentpublicengagementtechniqueshavehadanimpactondifferentoutcomesassociatedwithatechnologicalinnovationDeliberativemethodshaveinformedaspectsofregulation,alternativestoregulation,ethicsandpolicyinrelationtoatechnologicalinnovationDeliberativemethods,suchaspublicdialogues,futureworkshopsandcitizens'juries,havehadanimpactoninformingaspectsofpolicy,ethics,regulationandalternativestoregulation14concerningscienceandtechnologyinanumberofsectors,includinghealth,transport,datascienceandgeneralresearchandinnovationpolicy(CentreforPublicImpact2019;Drew2016;Hebákováetal.
2018;RSA2019;Sciencewise2021e).
Overall,asnotedbyaninterviewee,deliberativemethodshavebeeneffectiveatprovidingrobustperspectivesfromabroadsampleofthepublic(INT01).
Theyhavetypicallybeenusedtoinformaspecificdecisionorinformprinciplesbywhichadecisioncanbetaken(INT01).
PublicdialogueshavebeenusedtoinformalternativestoregulationsuchasethicalguidelinesrelatedtothedevelopmentoftechnologiesIntheUK,thereareseveralrecentinstancesofpublicdialoguesbeingusedtoinformethicsconcerningAIanddata-driventechnologies.
Forexample,in2019,theCentreforDataEthicsandInnovation,togetherwithSciencewise,ranaseriesofworkshopsacrosstheUKinordertounderstandopinionsandperspectivesheldbymembersofthepublicconcerningonlinetargeting(CDEI2020).
ThisinformedrecommendationstotheUKgovernment,focusingonthegovernanceand14Alternativestoregulationconsistoftwobroadcategories:(1)alternativestoregulation;and(2)alternativemodelsofregulation.
Alternativestoregulationinclude:nonewintervention/donothing;informationandeducation;andincentive/market-basedstructures.
Alternativemodelsofregulationinclude:self-regulation;goals-basedregulation;andco-regulation(NationalAuditOffice2014;OECD2021).
Examplesofalternativemodelsofregulationinclude:independentrecommendationschemes;ratingsystems;andlabelling(NationalAuditOffice2014).
AnexampleofinformationandeducationincludestheFoodStandardsAgency'FoodHygieneRatingScheme'translatestheresultsoffoodsafetyinspectionsintoafoodhygienescorethatestablishmentscandisplay(NationalAuditOffice2014).
Anexampleofaself-regulatorysystemcurrentlyoperatingwithinindustryistheLionQualityMarkEggAssurancescheme,whichensureseggshavebeenproducedtohighstandardsoffoodsafety(NationalAuditOffice2014).
82implementationofonlinetargetingpractices(CDEI2020).
In2019,theRoyalSocietyforArts,ManufacturesandCommerce(RSA)andDeepMindconvenedacitizens'jurytodeliberateontheethicaluseofAI,whichfedintoatoolkitonconductingpublicengagementaroundethics(RSA2019)(seeCasestudy03;AnnexA).
TheGovernmentDataSciencePartnershipranapublicdialogueondatascienceethics,includingdeliberativeworkshops,whichinformedtheprinciplesfortheDataScienceEthicalFrameworkfortheUKGovernment,whichguidesresponsibledatauseingovernmentandthewiderpublicsector(Drew2016).
Deliberativemethodshavebeenusedtoinformthedesignofconsultations,policyandstrategyAshighlightedbyinterviewees,intheUKandinternationally,deliberativetechniques(e.
g.
publicdialoguesandworkshops)havebeenusedtoinformanddeveloppolicyandstrategyaroundtechnologicalinnovation,andresearchandinnovationmoregenerally(INT06).
Forexample,in2018,GenomicsEnglandranapublicdialoguewithsupportfromSciencewisetoexplorepublicattitudestomainstreaminggenomicmedicineintotheNHS.
Theresultsoftheexercisewereprovidedtothe100,000GenomesProjectparticipants'panel,theAssociationofBritishInsurersandtheGenomicsEnglandseniorleadershipteam(Sciencewise2021c).
ThisisexpectedtohaveanumberoffutureimpactsonpolicyaroundtheNHSGenomicMedicineService,includinginformingrevisionsoftheNHSConstitution,theNationalGenomicsHealthcareStrategy2019,andtheScienceandTechnologyCommitteeInquiryintoCommercialGenomicTesting(Sciencewise2021c).
Inanotherexample,aSciencewisepublicdialogueondronesinformedtheDepartmentforTransport'sconsultationondrones(Collier2016).
IntheEUcontext,'visioning'(afuturesmethodtoimaginethefuture)workshopswereheldthroughout2015and2016with1000participantsin30differentEuropeancountriesaspartoftheCIMULACT(CitizenandMulti-ActorConsultationonHorizon2020)project(Hebákováetal.
2018)(seeCasestudy09;AnnexA).
TheresultsinformedthedesignandimplementationofHorizon2020workprogrammes2018-2020andFrameworkProgramme9(Hebákováetal.
2018).
Anumberofstakeholders,includingthepublic,developedtheirvisionsofsustainableanddesirablefuturesandtranslatedtheseinto23suggestionsforHorizon2020topics,alongwithpolicyrecommendations.
DeliberativemethodshavebeenusedtoinformregulatoryframeworksDeliberativemethods(includingpublicdialogueandonlinedeliberativemethods)intheUKandinternationallyhaveinformedthedevelopmentofregulation.
IntheUK,theHumanFertilisationandEmbryologyAuthorityusedabreadthofengagementtoolsincludingworkshops,apublicsurvey,openmeetingsandfocusgroupstodeterminepublicacceptabilityofmitochondrialreplacementtherapySheikh2013).
TheoutputsofthedialogueinformedtheHFEA'sadvicetotheUKgovernmentonitsregulatoryframework,withthegovernmentincludingthesafeguardsraisedbythedialogueintotheresultingregulations(Sciencewise2021d).
In2015,thegovernment83ofTaiwanusedanonlineplatform(calledvTaiwan)toengagethepublicinlarge-scaledeliberationontheregulationofUberinTaiwan,whichledtoconcreterecommendationstothegovernmentonitsregulatoryframework(CentreforPublicImpact2019)(seeCasestudy06;AnnexA).
Theprocessresultedinthegovernmentadoptingnewregulationsinlinewiththerecommendations(CentreforPublicImpact2019).
VariousparticipatorymethodshavebeenusedtoexplorepublictrustandconfidenceThereissomeevidenceintheliteraturethatsuggeststhatvariousparticipatoryapproaches(bothtraditionalface-to-facedeliberativemethodsaswellasmoreatypicalapproaches)havebeenusedtoexplorepublictrust.
Forexample,publicdialogueshavebeenusedtoexploretrustandconfidenceinusingaparticulartechnologicalinnovation,andhavealsohelpedtoincreasepublictrustandconfidenceinatechnologicalinnovation(Degelingetal.
2020;FoodStandardsAgency2020;McCool2019;Miletal.
2019).
TheFoodStandardsAgencyconductedapublicdialogueonemergingfoodtechnologies(GMfoods,nanotechnologyinfoods,foodfromclonedanimalsandculturedmeat)andaspartoftheprocessinvestigatedbeforeandafterviewstodetermineifthesehadchanged(i.
e.
weretheynegative,positive,neutral)asaresultoftheengagement(FoodStandardsAgency2020).
Inanotherexample,lookingatpublicviewsofneuralinterfaces,publictrustofthetechnologicalinnovationsignificantlyincreasedoverthecourseoftheworkshops(Miletal.
2019).
Recently,theAdaLovelaceInstituterananonlinedeliberativeprocess,whichidentifiedfourrequirementsthatwouldhelptoensurepublictrustandbuy-inregardingfutureCOVID-19technologies(i.
e.
provideatransparentevidencebase;offerindependentreviewofthetechnology;clarifydatause,rightsandresponsibilities;andaddresstherisksandneedsofvulnerablegroups)(AdaLovelaceInstitute2020)(seeCasestudy10;AnnexA).
Thesefindingsfedintothedevelopmentofachecklist(basedaroundthefourrequirementsoutlinedabove)aimedatgovernment,policymakersandtechnologydeveloperstohelpthemwiththefuturedevelopment,designanduseofCOVID-19technologies,includingcontacttracingapps,toensurepublictrustandbuy-in(AdaLovelaceInstitute2020).
Similarly,citizens'jurieshavealsohelpedtoincreasedpublictrustintechnologicalinnovation.
Forexample,communityjurieswereconductedtoconsidertheacceptabilityandlegitimacyofusingnewtechnologiestoenhancepublichealthresearchandcommunicablediseasesurveillance,andtheprocessofdeliberationincreasedsupportforusingbigdataanalyticstoenhancecommunicablediseasesurveillance,becauseparticipantswereabletoarticulatetheirconcernsandthinkthroughdifferentconditionsthatwouldneedtobemetforthemtoacceptitsimplementation(Degelingetal.
2020).
Inanotherexample,theRSAranacitizens'juryonautomateddecision-makingsystems.
Asaresultofthisengagement,intervieweeswhofacilitatedtheexercisefeltthatthetrustinautomateddecision-84makingsystemsincreased(INT02,INT03)(seeCasestudy03;AnnexA).
Onestudyalsosuggestedthatcommunicatingtothepublicthroughtrustedchannelsandprocessesthatareinclusive(e.
g.
newschannels,socialmedia,talkshows,andotherfrequentlyusedinformationdistributionplatforms)couldincreasepublictrustintechnologicalinnovation(Sutcliffeetal.
2020).
Thereisoneexampleofanatypicaltechniquebeingusedtoexploretrust.
TheexerciseusedGoogleAutodraw(anAI-enabledsmartphoneapp)toexamineusers'perceptionoftrust(Bitkinaetal.
2020).
ThestudyaimedtoexamineandevaluatetheacceptanceofAItechnology(i.
e.
theGoogleAutodrawapp)byencouraginguserstoengagewiththeAI-basedtool,andlookedspecificallyatperceivedcharacteristicssuchastrust,usabilityandutilityrelatedtothistechnology.
TheinsightsfromthestudycouldbeusedbyAIdeveloperstoimprovetheirproductattractivenessamongstusersandincreasethetrustintheirtechnologies.
AtypicalpublicengagementtechniquesandmarketoutcomeslackevidenceOnlinesurveysandlivinglabshavebeenusedtodetermineacceptanceoftechnologicalinnovationandcouldbeusedtoinformmarketadoptionThereissomeevidencefromtheacademicliteratureonhowcertaintechniques,inparticularsurveysaswellassociallistening,couldhypotheticallybeusedtoinformaspectsofmarketadoption(i.
e.
bringingexistingtechnologicalinnovationintothemarketplace)(Arningetal.
2018;Shabanpouretal.
2018).
Itremains,however,tobedeterminedwhethertheseformsofengagementhavehadanimpactonoutcomes.
Surveyscanhelptogatherclear-cutanswersandcanbehelpfultounderstandpublicacceptancerelatedtoasetofclearlydefinedquestionsassociatedwithtechnologiesoraspecifictechnologyapplication.
Theycanalsobeadministeredatscaleandrelativelyquickly.
Forexample,anonlinesurveyinGermanywasusedtoanalysefactorsthatmightinfluencethelikelihoodofadoptionofacarbonderivedfoammattress(usingcarboncaptureandutilisationtechnology)(Arningetal.
2018).
IntheUS,anonlinesurveyof1,253participantsincorporatingabest-worstchoiceexperiment,wasusedtoinvestigatepeople'spreferencesaboutthemostandleastattractivefeaturesofdriverlesscars(Shabanpouretal.
2018).
ResearchersinFranceusedanovelsurveytool(theAttrakDiff-2questionnaire,aspecificUXmethod)toidentifyfactorsthatdeterminehowdifferentgroupsperceiveelectriccars(Dupontetal.
2019).
Theaimwastoinformfactorsinfluencingacceptabilitywithaviewtosupportingsmartcitypolicies.
Sociallisteninghasbeenusedtogatherviewsfromalargesampleofthepubliconacceptanceoftechnologicalinnovation.
Forexample,analysisofcommentsbyusersonsocialmediaplatformssuchasFacebookandTwitterhavebeenusedto85understandpublicdiscussionsaroundnewtechnologicalinnovationsuchastheuseofVRinhealthcare(Kelleretal.
2017)andCRISPR-mediatedgenomeediting(Calabreseetal.
2019).
Thesetechniquescouldpotentiallyofferusefulinsightsforbringingemergingapplicationsoftechnologiesintothemarketplaceatscale.
This,however,remainshypotheticalatpresent,withexamplescitedintheliteraturehavingbeenusedas'proof-of-concepts'inresearchcontexts.
ThereiscurrentlylimitedevidenceontheimpactsofsomeofthemoreatypicalapproachesThereiscurrentlylimitedevidencefromtheliteratureontheimpactsofarts-basedandexperientialtechniquesonspecificoutcomes,whichtendtorepresent'proof-of-concept'approaches.
TheMoralMachineexperiment,whichusedagametocrowdsourcethepublic'sviews,resultedinglobalinsightsonmoralpreferencesaroundAI.
Theexperimentalsohelpedtoincreaseawarenessofethicalconsiderationssurroundingautonomousvehiclesandmachinesingeneral(CS01-01;seeCasestudy01;AnnexA).
Thefindingscouldinfuturebeusedtoinformglobalprinciplesaroundmachineethicsandinformconsiderationsbycarmanufacturersandpolicymakers,butthishasnothappenedtodate(Awadetal.
2018).
Biometricmirror(thefacialanalysisapplication)succeededinraisingawarenessofthistechnologicalinnovationamongstthepublicinanengagingandthought-provokingmanner(Woutersetal.
2019)(seeCasestudy08;AnnexA).
TheBioMetricMirrorapplicationalsocontinuestobeexhibitedinbothartandscienceexhibitionsglobally,continuingtostimulateconversationaroundtheuseofthistechnologicalinnovationandthepotentialethicalconcerns.
Thereis,however,noevidencethatthisapproachhasledtoconcreteoutcomesasyet.
TheGATEwayProject,inwhichthepublicwasinvitedtotakepartinlivetrialsofconnectedandautonomousvehiclesproducedwide-ranginginsightsonpublicperspectivestowardsconnectedandautonomousvehicles(seeCasestudy02;AnnexA).
Thereis,however,limitedevidencethattheGATEwayProjecthashadasignificantimpactonactorswithintheCAVecosystemoronregulatoryframeworksrelatingtoCAVs,thoughthiscouldchangeastheCAVmarketmatures.
86Chapter4:Evidenceofwhethertheeffectivenessofpublicengagementtechniquesaroundtechnologicalinnovationhasbeenformallyassessed,andassociatedlearningsInthissection,wepresentthekeyfindingsassociatedwithResearchQuestion3.
ResearchQuestion3:Hastheeffectivenessofthepublicengagementtechniquesaroundtechnologicalinnovationbeenformallyevaluated,andwhat,ifany,werethelearningsBox5:SummaryofevidenceandkeylearningsontheeffectivenessofpublicengagementtechniquesappliedtotechnologicalinnovationUpstreamanddownstreampublicengagementtechniquesFormalevaluationsofdeliberativetechniques(e.
g.
publicdialogues,futuresworkshops)offersomelearningsontheeffectivenessofthesemethods,namely:-Deliberativetechniquesareeffectiveatexploringin-depthquestionsrelatedtotechnologicalinnovation.
-Publicdialoguesareeffectiveatproducingrobustperspectivesfromarepresentativesampleofthepublic.
-Havingaclearpurposeisimportanttothesuccessofpublicdialogues.
-Exploringcomplexquestionsaroundtechnologyrequiressufficienttimeandresources.
Anecdotalevidencefromtheacademicliteraturereviewedofferssomeinformalassessmentsofothertypesofpublicengagementtechniques.
Keylearningsfromthesemethodsinclude:87-Surveysareawell-establishedmethodtogatherperspectivesontechnologicalinnovationfromasampleofthepublic.
-Experientialtechniques(e.
g.
immersivesimulationsusingVR,experimentalapplicationsandlivinglabs)givethepublicanopportunitytoexperienceatechnologicalinnovationortestaprototypein'real-world'scenarios.
-Arts-basedandgamifiedtechniquesmayprovidefunanduser-friendlyexperiencestoeffectivelyengageamorediversepublic.
-Distributeddialoguemaybeeffectiveatgatheringlong-termperspectivesovertime.
FewpublicengagementprocesseshavebeenformallyevaluatedBasedontheevidencecollectedinthisstudy,theeffectivenessofpublicengagementtechniquesisrarelyassessedformally,andevidenceaboutmanyapproachesislimited.
Mostoftheevidenceoftheeffectivenessofpublicengagementtechniquesisfoundinthegreyliterature(see,e.
g.
Involve2018;Sciencewise2021e)fromanumberofkeyorganisationsthatareactiveinthisarea(AdaLovelaceInstitute,Involve,TheRoyalSocietyandSciencewise).
Theseorganisationshaveconductedsomeformalevaluationsofsomeofthedeliberativemethods,whichoffersomelearningsontheeffectivenessofthesemethods.
Inadditiontoformalevaluations,anecdotalevidencefromtheacademicliteraturereviewedofferssomeinformalassessmentsofothertypesofpublicengagementtechniques.
Thesecanoffersomelearningsconcerningfactorsthatcancontributetoeffectivenessandsuccess.
Table3(presentedinthethirdsheetintheaccompanyingExcelfile)presentsanarrayofcriteria(i.
e.
scale,resourcesoftimeandcost,inclusivity,userfriendlinessandflexibility)basedontheevidenceintheliteratureandcasestudiestohighlighttheimplicationsofapplyingthedifferentpublicengagementtechniquescoveredintheliteratureandcasestudies.
ExistingevaluationsandassessmentsprovidesomekeylearningsrelatedtospecifictechniquesDeliberativemethodsareeffectiveatexploringin-depthquestionsrelatedtotechnologicalinnovationTheevidencefromformalevaluationsoftheSciencewisepublicdialogues(e.
g.
genomicmedicines,mitochondrialreplacementtherapyanddrones)(Collier2016;IpsosMORI2019;Watermeyer&Rowe2013),15andfromtheliteraturesuggeststhatdeliberativemethodsareaneffectivemethodofpublicengagement.
Byprovidingextensiveinformationfromarangeofexpertsthatengagewiththepublic15SeetheSciencewiselibraryforafulllistofevaluations(Sciencewise2021e)88aspartofthisprocessandensuringconditionsforreasonableandextendeddebate,deliberativemethodselicitmoreconsideredjudgementsthanothermethods(Degelingetal.
2020;Involve2018).
Inaddition,oneintervieweementionedthatbyemployingaformalrecruitmentprocess,theytypicallycontainarepresentativesampleofthewiderpublic(INT01).
Aswasalsonotedbyseveralinterviewees(INT01,INT02,INT05),deliberativemethodsaretypicallymorecostlythanothermethodsduetotheamountoftimeandresourcesthatareneededtodesignandimplementthem(Involve2018).
Belowaresomekeylearningsonfactorsthatcontributedtoeffectiveness.
ExploringcomplexquestionsaroundatechnologicalinnovationrequiressufficienttimeandresourcesSuccessfulexamplesofpublicdialoguefacilitatedextensive,open-endeddiscussionsbetweenthepublicandexperts,providingthepublicwithinformationandtimetovoiceanddiscusstheirconcerns(Watermeyer&Rowe2013).
EvaluationsoftheSciencewisepublicdialogueonmitochondrialreplacementtherapy(Watermeyer&Rowe2013)andthepublicdialogueorganisedbytheRoyalSocietyonneuralinterfaces(Miletal.
2019)concludedthatparticipantswerebroadlygivenenoughtime.
Incontrast,anevaluationofaSciencewisepublicdialogueondronesfoundthattight(externallyimposed)programmetimescalesprobablyimpactedonthequalityofcitizens'proposals(Collier2016).
Thisisimportantwhendiscussingcomplexethicalandsocialaspectsofnewtechnologiesandtechnologicalinnovation(Sutcliffeetal.
2020).
ResearcherswhoconductedacommunityjuryinAustraliaontheuseofwholegenomesequencingandBigDataanalyticsinpublichealthconcludedthatallowingtimeforextendedandopen-endeddialogueprovidedthepublicwithsufficientinformationandtimetovoiceanddiscusstheirconcerns,whichhelpedtoincreaseunderstandingandacceptability(Degelingetal.
2020).
Inparticular,thedesignofanengagementprocessthattakesplaceoverseveralroundsallowsparticipantstimetothinkbeyondtheirinitialresponses(Miletal.
2019).
Thedesignofatwo-roundprocessaspartoftheRoyalSocietydialogueonneuralinterfaces,withatwoweekgapbetweeneachround,allowedparticipantstimetothinkbeyondtheirinitialresponses,andtypicallythisresultedinincreasedtrust,acceptabilityandexcitementaroundthepotentialbenefitsofthetechnologicalinnovation(Miletal.
2019)(INT04).
Inanotherexample,theDepartmentforTransportandSciencewiseconductedapublicengagementprocessaboutconnectedandautonomousvehiclesthatincorporatedamulti-roundprocessandparticipantswhowereinterviewedafterthefinalworkshopreflectedthattheirperspectiveschangedoverthecourseofthedialogue(McCool2019).
InarapidonlinedeliberationconductedbytheAdaLovelaceInstituteandothers,havingtheengagementtakeplaceovermultiplesessionsenabledparticipantstoreflectandembedlearningbetweenmeetings(seeCasestudy10;AnnexA).
89Oneintervieweealsonotedthatrobustpublicdialoguerequiresenoughtimetogathertheexpertsneededtoadequatelyframethedialogue,includingtodevelopmaterialsthatarehigh-qualityandunderstandable(INT01).
Forthisreason,anumberofintervieweeshighlightedthatthistypeofengagementprocesstendstoberelativelycostlyandrequiressignificantinvestmentofresources(INT01,INT02,INT05).
HavingaclearpurposeisimportanttothesuccessofpublicdialoguesSuccessfulexamplesofpublicdialoguehadclarityontheoutcomethatwasbeinginfluenced(e.
g.
policy,regulation),thevaluethatthepubliccouldbring,andtheparametersthatcouldbeinfluenced(Allan2016).
Thepublicmaybescepticalaboutpublicengagementiftheylackclarityabouthowtheengagementwillinfluencetheultimatedecision-makingprocess(Salisbury&Nicholas2005).
Thefailureofpublicengagementcanbeduetolackofclarityaboutwhytheengagementisbeingheld,whichcanleadtoalackoftrustonthepartofthepublic(Salisbury&Nicholas2005).
Forexample,anevaluationoftheGMNationdebateonGMcropsin2003foundthatthepublicengagementprocesssufferedfrompoorlydraftedobjectivesandlackofclarityonhowthefindingswereusedbygovernment(Horlick-Jonesetal.
2006).
Surveysareawell-establishedmethodtogatherinformationonatechnologicalinnovationfromasampleofthepublicSurveysareusefulforgatheringinformationonatechnologicalinnovationfromdifferentsamplesofthepublic.
Surveysrepresentamethodthatiscost-effectiveandpotentiallystatisticallyrepresentativeofapopulation(Sampleetal.
2020).
Surveysarealsoversatileinthattheycanbeusedinastandalonemanner,butarealsousefulwhenusedaspartofalargerengagementprocess,eitherattheirinceptiontoinformsubsequentengagementprocessesusingalternativemethods(e.
g.
deliberativemethods)orasafollow-uptoprobespecificthemesmoredeeply(Participedia2021a).
Forexample,havingconductedasurveyontheuseoffacialrecognitiontechnology,theAdaLovelaceInstituteisorganisingaCitizenBiometrics'Counciltoprovidepublicperspectivesandvaluesonbiometricsandtoco-producepracticalrecommendationsincollaborationwithpolicymakers,regulatorsandotherstakeholders(Participedia2021b).
Surveyscanalsopermitthegatheringofviewsatscale(withnolimitonthenumberofquestionnaireparticipants),whichmaybeusefultosegmentconsumersintosubgroupsandfactorsthatinfluencethem,toinformpolicyortheadoptionofanewtechnology.
Here,anumberofstudieshaveusedsurveyswithexperimentalcomponentsthatassessusers'experiencesofatechnologicalinnovationtoexamineandidentifydifferentgroupsofusers,intermsofacceptabilityandpreferencesregardingelectriccars(Dupontetal.
2019)anddriverlesscars(Shabanpouretal.
2018).
90ExperientialtechniquesgivethepublicanopportunitytoexperienceatechnologicalinnovationorprototypeandcanhelptoelicitinsightfulresponsesaboutpotentialethicalimplicationsByprovidingrealorsimulatedreal-worldexperiences,experientialmethods(bothexperimentalandlivinglab-basedapproaches)enablepublicengagementtobeconducted'inthefield',allowingparticipantstoengagewithrealisticapplicationscenariosorrealprototypes(seeCasestudies02and08;AnnexA).
Inthecaseofexperimentalapproaches(thatuselab-basedprototypesofatechnologicalinnovationorVRsimulations),the'provocative'natureoftheengagementcanprovideathought-provokingandentertainingapplicationwhichenableengagementfromthepubliconcomplexethicalissues.
Forexample,inthecaseoftheBiometricMirrorinteractiveapplication,theresearcherswereabletodemonstratetothepublicthepotentialforfacialanalysistechnology,enablingthepublictoformareasonedopiniononthistypeoftechnologicalinnovationbeforeitreachesfullmaturity(seeCasestudy08;AnnexA).
Theinteractivenatureoftheseapproachescanenablethepublictobetterrecognisethepotentialimpactsandchallengesposedbyaparticulartechnologicalinnovation.
Livinglabsgoastepfurtherbyenablingthepublictotestatechnologicalinnovationorprototypeinareal-worldsetting.
Thismethodisparticularlyusefultoobtainstakeholderinputonatechnologicalinnovation.
Itis,however,usefultosupplementthesemethodswithotherapproaches(e.
g.
surveys,interviews,consultationordeliberativemethods)tofurtherprobeandunderstandthepublic'sperceptionsandconcerns.
Arts-basedandgamifiedtechniquesmayprovidefunanduser-friendlyexperiencesArts-basedandgamifiedtechniquesrepresentuser-friendlyandcreativeapproachestoengageawidersectionofthepubliconcomplexandcontroversialtopics(Altamirano-Allende&Selin2016;Engasser&Saunders2018)(seecasestudies01,04;AnnexA).
Forexample,participatorytheatreisatechniquethatcanhelptopromoteaudienceengagementandparticipation,particularlygroupsthatmaynotparticipateinformaltechniques,andcouldpotentiallymakepolicymakingaroundtechnologicalinnovationmoreinclusive(Engasser&Saunders2018).
Creativemethodscanelicitreflectionfromthepubliconcomplextechnologyinawaythatisfun(Altamirano-Allende&Selin2016;Engasser&Saunders2018).
Gamesrepresentafunandengagingwaytoraiseawarenessofatopicortoinvolvethepublicinfuturescenarios.
Similarly,participatoryfutures,whichsometimesincludegamifiedapproaches,allowpeopletoplacethemselvesinafutureworldandexperimentwithnewbehavioursorvalues(Ramosetal.
2019).
Therefore,togetherthesetechniquescanbettersupportthepublictoreflectonhowtechnologicalinnovationcouldimpacttheireverydaylives(Woutersetal.
2019).
91Distributeddialoguemaybeeffectiveatgatheringlong-termperspectivesonatechnologicalinnovationovertimeDistributeddialoguecanbeusedtoengagewiththepublicovertimeratherthanthroughaone-offengagementeventorexercise.
Thelengthofthisvariesbutapproachescouldtakeplaceoverseveralmonthstoyears.
Similarly,distributeddialoguerepresentsadeliberativetechniquethathasbeenusedtogatherongoingviewsofthepubliconscienceandtechnology,althoughcostsforthistechniquemaybeslightlyhigher(BBSRC2013).
Ithasbeenrecognisedthatpublicengagementdoesnothavetobeadiscreteone-offprocess,andpublicperspectivesareconstantlychangingovertime(INT01).
Therefore,asnotedbytwointerviewees,longitudinalapproachescouldprovideamorecomprehensiveevidencebaseandcontributetodevelopingasustainedconversationwithmultiplepublics(INT01,INT02).
Itis,however,unclearhowthefindingsfromtheseexampleshavebeenused,andsoitremainstobedeterminedhowlongitudinalapproachescanhelptoinformscienceandtechnologypolicyinpractice.
92Chapter5:DiscussionandconcludingremarksTechnologiesareproliferatingtheworldoverwithanarrayofapplicationsthatspanmanysectorsandservices.
Withthepromisetoimprovelives,andtheofferofabundantopportunities,technologicalinnovationisregardedasacrucialenablerfortheadvancementofsocietiesandeconomiesatlarge.
Atthesametime,however,astheworldbecomesprogressivelymoredependentontechnologiesandtheirapplications,suchinnovationsalsopresentanarrayofsocial,economicandregulatorychallenges.
Howtoharnessthebenefitsoftechnologicalinnovationwhileaddressingtherisksassociatedwiththesedevelopments,isthesubjectofmuchdiscussionbygovernments,regulators,industry,academiaandthegeneralpublic.
Akeyaspectofthesediscussionsanddebatesispublicengagementinthecontextoftechnologiesandtechnologicalinnovation.
Aswehaveseeninthisreport,suchengagementisincreasinglybeingrecognisedbystakeholdersasacriticalinstrumenttoencouragetransparencyandopenness,increaserepresentativeness,andbuildtrustindecisionmakingandthetechnologiesthemselves.
Integratingpublicengagementintoregulatoryprocessescouldalsoreinforceconfidenceinregulatorybodies.
Againstthisbackdrop,thisstudyhascollatedandanalysedevidenceontheuseandapplicationofarangeofpublicengagementtechniquesfortechnologicalinnovation.
Throughafocussedreviewoftherecentliteratureandtendetailedcasestudies,thestudyhighlightsvariouselementsassociatedwiththespectrumofpublicengagementtechniquesthatarebeingusedaroundtechnologicalinnovation(seeTable1inChapter2foranoverviewofthisevidence).
Theliteraturereviewprovidesanoverviewofthebreadthofevidenceassociatedwiththistopic,focussingonthedifferenttypesofpublicengagementtechniquesthathavebeenappliedtotechnologicalinnovation,theireffectivenessandanylessonsthatmightbelearntfromapplyingthetechniques.
ThecasestudiesofferadeeperandmoreconcreteunderstandingofhowaselectionofpublicengagementtechniqueshavebeenusedinspecificcontextsintheUKandglobally.
Whilethefindingsfromthisstudyareprimarilytargetedatpolicymakersandregulatorsinvolvedintechnologicalinnovation,wenotethattheymayalsobeofinteresttootherstakeholders,suchasfundersofresearchandinnovation,academia,industryandthegeneralpublic.
Inthesectionsbelow,wesummarisethekeyfindingsfromtheresearchandoffersomeconcludingreflections.
93WhatdoestheevidencesayaboutpublicengagementinthecontextoftechnologicalinnovationAdiversespectrumofpublicengagementtechniquescouldbeusedinthecontextofatechnologicalinnovationSpecifically,weobservethat:Traditionalapproacheshavebeenusedmostfrequently.
Theseincludewell-establishedmethodssuchassurveysandpublicdialogues;andAtypicalapproacheshavebeenusedinsomeinstances.
Thesetendtobeinnovative,'digital-enabled'techniques,usingonlinetoolsorimmersiveVRtechnologiesandsimulations.
Thereisevidencetosuggesttheseapproachescouldholdpotentialtobeusedmoreinthefuture.
Publicengagementtechniqueshavebeenappliedtoadiverserangeoftechnologies(e.
g.
cognitivetechnologies(includingAIandmachinelearning),data-driventechnologies,medical/biotechnologies,greentechnologies,nanotechnologies).
Publicengagementtechniquesappliedtotechnologicalinnovationcoverarangeofstakeholdertypes(e.
g.
thegeneralpublic,potentialusersorconsumersofthetechnology,policymakers,regulators,industryandexpertsfromvariousdisciplines).
Differentpublicengagementtechniqueshave,tovaryingdegrees,hadanimpactonselectedoutcomesOverall,theevidenceisstrongestfortraditionalapproaches(i.
e.
surveys,publicdialogues),withlessevidencearoundatypicaltechniques,whichtendtoconsistofproof-of-conceptstodemonstratethefeasibilityofaparticularapproach.
Themostreportedoutcomeistoraiseawarenessandunderstandingoftechnologicalinnovation.
Publicengagementtechniquesinthecategories'communicate'and'participate'werefoundtohelpinraisingawarenessandunderstandingofaparticulartechnologicalinnovationamongstthepublic.
Techniquesinthecategory'consult'werelesslikelytohaveanimpactonawareness,sincethesetechniquestypicallyaimtogatherinformationfromthepublicratherthaninform.
Thereareseveralinstancesinwhichpublicengagementhashadanimpactonpolicy,regulationorethics.
Participatorymethodsthatincludedeliberation,inparticularpublicdialogues,butalsofutureworkshopsandcitizens'juries,havehadthemostimpactoninformingaspectsofpolicy(both94researchbutalsoaroundtheuseoftechnologicalinnovation),regulationandinformingethicalframeworks.
Thereareseveralinstancesinwhichpublicengagementhashadanimpactonaspectsoftrust.
Participatorymethodsthatincludedeliberation,andinparticular,publicdialoguesandcitizens'juries,havebeenusedtomeasureaspectsoftrustinaparticulartechnologicalinnovation,buthavealsodirectlyledtoincreasedtrustasaresultoftheprocess.
Theseapproacheshavebeenusedtoconsidertrust,bothasstandalonemethodsbutalsoaspartofwiderengagementprocessesusingothertechniques,includingsomeofthemoreatypicalexperientialapproachesusingsimulationsandimmersivemethods.
Thissuggeststhatdeliberativemethodsareparticularlyhelpfultonurturepublictrustintechnologicalinnovation.
Therearelimitedinstancesinwhichpublicengagementhashelpedinformmarketadoptionofatechnologicalinnovationorcontributedtochangeinbusinessmodelsandthisistheoutcomewhichlacksthemostevidenceintheliteraturereviewed.
Experientialtechniqueslikelivinglabs,holdpotentialtoinformthescaleupofaspecifictechnologicalinnovation,butcurrentlylackevidenceofimpact.
TheusefulnessofpublicengagementtechniquesiscontextualTheeffectivenessofpublicengagementtechniqueswithregardtotechnologicalinnovationisoftennotformallyassessed,andevidenceaboutmanyapproachesislimited.
Theevidencesuggeststhat:Consultativeapproachessuchassurveys,interviewsandfocusgroupsareawell-establishedmethodtogatherinformationfromaselectedsampleofthepublic.
Theseapproachesrepresentversatiletoolsthatcanbeusedasastandalonemethodoraspartofawiderengagementprocess.
Forexample,surveyscanbeusedtoinformadeliberativeprocessorasafollow-uptoexperientialtechniquestofurtherprobethepublic'sviewsfollowingtheexperience.
Deliberativemethodsareparticularlyusefultoexplorecomplextopicsin-depth,andwhenthereispotentialuncertaintyorcontroversy.
Thesetechniquesprovidethetimeandspaceforthepublictovoiceanddiscusstheirconcernsandconsideranissuefrommultipleangles.
Therefore,thesetechniqueshavebeenparticularlyusefulatearlierstagesoftechnologydevelopment,toinformaspectsofpolicy,regulationorethicalprinciples.
Inaddition,byallowingtimefordebateandreflection,thesetechniquesareparticularlyusefultoexplorequestionsoftrustaroundaparticulartechnologyortechnologicalinnovation.
95Therearemultipleinnovativemethodsthatcouldhelptorenderpublicengagementmoremeaningful,andwidenparticipation.
Inparticular,arts-basedapproachesandexperientialtechniquesrepresentpotentiallyeffectiveandcreativeapproachestoengageawiderspectrumofthepublicwithcomplexandcontroversialissues.
Bymakingthetechnologicalinnovationmoretangibleorstimulatingcreativethinking,theseapproachescanhelpthepublictoconsiderhowatechnologicalinnovationcouldimpacttheirlives.
Theuseofonlineanddigital-enabledapproaches(acrossthespectrumofengagement,involvingcommunicate,consultandparticipate)representsanovelapproachthatcanspeeduptheprocessofengagement,capturetheviewsofthepublicatscale,andenhancetheexperienceofengagement.
Consultativemethodssuchassociallisteningandcrowdsourcingplatformsareeffectivemethodsofcapturingviewsatscale.
Combiningtraditionaldeliberativeapproacheswithdigitalelementscanhelptospeeduptheprocessofengagementincaseswherethepublic'sopinionmayneedtobesoughtmorequickly.
TheuseofVRtocreatesimulationsandimmersiveenvironmentscanenablethepublictoexperienceatechnologicalinnovationandfacilitateengagement.
Takingalltheevidencetogether,severalcross-cuttinglearningsemergeTheuseofmultipletechniquesoverthecourseofthepublicengagementprocesscanhelptoengagedifferent'publics'appropriatelyEngagementprocessesthatemployedmultiplepublicengagementtechniquestendedtobeeffective.
Theseapproachesincludedthecombinationofdeliberativetechniquesorexperientialtechniques(suchaslivinglabs)withconsultativetechniques(suchassurveysorinterviews).
Combiningmultiplemethods(e.
g.
formaltechniqueswithinnovativemethods)makestheengagementmorerelevanttomultiplegroupsandcanthereforeengagearangeofdifferentaudiencesatonce,andpotentiallyincreasestherobustnessandrelevanceofthedatathatarecollected.
Forexample,theGATEwayProjectusedmultiplemethodstoengagearangeofdifferentaudiences,fromthosewithastatedinterestinconnectedandautonomousvehiclestothosewithlessawareness,whoengagedspontaneouslyuponencounteringtheproject(seeCasestudy02;AnnexA).
TheGATEwayProjectshowcasesthewayinwhichpublictrialsmaybecombinedwithotherengagementmethods,includingcreativeworkshops,publicexhibitions,sentimentmappingandobservationalstudies,andameansbywhichmulti-dimensionalengagementapproachesmaybeusedtoincorporatediverseperspectivesandvariedstakeholdergroups.
Similarly,afterusershadexperiencedtheBioMetricMirrorapplication,96interviewswerecarriedoutwithasubsetoftheuserstoenablediscussionaroundhowaccurateusersfeltthetechnologywasaswellasethicalimplicationsofthetechnologicalinnovation(seeCasestudy08;AnnexA).
SpreadingpublicengagementovertimeallowsforreflectionandembeddingofconceptsEngagementprocessesthattookplaceoveralongerperiodoftimetendedtobeeffective.
CIMULACT(seeCasestudy09;AnnexA)andAI360(seeCasestudy07;AnnexA)areexamplesofextendedpublicengagementprocessesthattookplaceoverseveralmonthsandemployedmultiplemethodsandstages.
Similarly,anumberofpublicdialoguesintheUKemployedanextendedapproachovermultiplesessions,whichgavethepublicsufficienttimetolearnandembedinformation,andhelpedtoincreasepublicunderstandingandtrustaroundaparticulartechnologicalinnovation(Cameron&Maguire2017;FoodStandardsAgency2020;Miletal.
2019).
Inaddition,conductingpublicengagementoveraperiodoftime,asinthecaseofpublicdialogues,ensuresthatconsideredpublicopinionissought,andpotentiallyhelpstoincreasetrust.
HavinganimpactontrustinatechnologicalinnovationrequirestimeandconsidereddebatetoincreaseaccountabilityandmoresystematicpublicengagementThereareseveralinstancesinwhichdeliberativetechniques,inparticularpublicdialogues,havehadanimpactontrust.
Forexample,participantsreportedincreasedtrustasaresultoftheRoyalSocietydialoguesonneuralinterfaces(Miletal.
2019)andmachinelearning(Cameron&Maguire2017),andtheFSAengagementonfoodtechnologies(FoodStandardsAgency2020).
TheRSAcitizens'juryalsoledtoincreasedtrustinautomateddecision-makingsystems(seeCasestudy05;AnnexA).
Akeyfeatureoftheseprocessesthathelpedtoincreasetrustisthemulti-stepprocess,whichgavetheparticipantstimetoreflectontheirviewsandincorporateandembedlearningbetweensessions.
IntheexampleofAI360,thereisalsosomeevidencethattheprocessmayhavehelpedtoincreasetrustinAIsystems(seeCasestudy07;AnnexA).
Althoughthereissomeevidencethatcertaintypesofpublicengagementtechniqueshavehelpedtoincreasetrust,itshouldbenotedthattrustisoftennotformallyassessedaspartofthesemethodsbutratheranecdotallycapturedaspartoffeedbackfromparticipants.
Itisalsoimportanttoconsiderthatthereisevidenceofshort-termincreasesintrust,butitremainstobeseenwhetherthatincreaseismaintainedovertime.
Therefore,inordertobuildpublictrustatasocietallevel,thereisaneedformoresystematicpublicengagementprocessesinvolvingactorsandorganisationsacrosstheecosystem,withthepowerandinfluencetoshapethecourseofthetechnology'sfuturedevelopment(CS07-01).
97Amulti-stakeholder,collaborativeapproachtopublicengagementhelpstodevelopinformedandconsideredjudgementsExamplesofengagementthatinvolvedthepublicandotherstakeholders,suchasexperts,tendedtobeeffective.
Forexample,theAI360workshopandcitizenconsultationdemonstratethepossibilityofalinked,two-stepapproachtopublicengagement,incorporatingbothexpertopinionandbroaderpublicviews(seeCasestudy07;AnnexA).
Similarly,adistinctivefeatureofdeliberativemappingapproachesisthepublic-specialistinteractionthroughajointworkshop(Bellamyetal.
2016).
Themultipleexamplesofdeliberativetechniquestypicallyinvolvedexpertswhopresentedinformationtothepublic,includingpublicdialogues(AdaLovelaceInstitute2020)andtheRSAcitizens'jury(RSA2019).
TheJRC'sScenarioExplorationSystemrole-playingscenariogameinvolvedpolicymakersandindustryrepresentativesplayingalongsidemembersofthepublic,withparticipantsreportingthattheyappreciatedthe'realism'oftheconversations(seeCasestudy04;AnnexA).
Intheseexamples,thepresenceofexpertshelpedtogroundthediscussioninwhatisfeasibleratherthan'fantasy',whichcanhelptodevelopinformedandconsideredjudgements.
Usingonlineanddigital-enabledpublicengagementtechniquescanpotentiallyincreasethespeed,scale,inclusivityandgeographicalcoverageofengagementOnlineanddigitaltechniquescanhelptoengagethepublicatscaleandmuchfasterthanface-to-faceormoretraditionalapproaches.
Forexample,onlineapproaches(includingsurveys,crowdsourcingplatformsandgames)cancollecttheviewsofanunlimitednumberofparticipants(sometimesuptomillionsofparticipants)atmuchlowercostthantraditionalface-to-facesurveysorinterviews.
Forexample,theMoralMachinewebsite,whichcombinescrowdsourcingwithgamificationwasabletocollectcloseto40milliondatapointsfrom233countries(seeCasestudy01;AnnexA).
Itisalsopossibletocombinedigitaltoolswithtraditionalapproaches,asinthecaseoftherapidonlinedeliberationrunbytheAdaLovelaceInstitute,whichranadeliberativeprocessin10weekscomparedtoseveralmonths(seeCasestudy10;AnnexA).
TheonlineplatformvTaiwancombinescrowdsourcingandonlinedeliberationandcansupportupto100,000commentsfromparticipants(seeCasestudy06;AnnexA).
Digital-enabledpublicengagementtechniquesthatemployuser-friendlyandengagingplatformscouldalsohelptowidenparticipation.
Forexample,sociallisteninghasusedpopularsocialmediaplatformssuchasFacebookandTwitter(seeCasestudy05;AnnexA),MoralMachinefeaturedanengaginggame(seeCasestudy01;AnnexA),andthevTaiwanplatformuseselementsofgamification(suchasallowinguserstovote)(seeCasestudy06;AnnexA).
Inthisway,thesetypesofatypicaltechniquescouldhelptowidenparticipationtomoregroupsof98peoplethanmoretraditional,formalapproaches,whichmanypeoplemaynotchoosetoengagewith.
Acaveattodigitalapproachesisthatalthoughtheycanservetowidenparticipation,theycanalsolimitparticipationtothosemembersofthepublicthathaveaccesstotheinternet,orthosethatarealreadycomfortableworkingonline(orthosethathaveasocialmediaaccount).
Therefore,carefulconsiderationneedstobemadeduringrecruitmentorsampling.
Usingsomeatypicaltechniquescanpotentiallyrenderpublicengagementmoretangibleanduser-friendlyandcouldalsoincreasethediversityofparticipationMultipletypesofatypicaltechniquesrepresentnovelandeffectivemethodstoengagethepublicinamoretangibleanduser-friendlymanner.
Akeyaspectofatypicaltechniques,suchasexperientialandarts-basedtechniques,istheirabilitytoenhancetherelevanceofthetechnologicalinnovationtopeople'severydaylives,andthusrenderpublicengagementmoretangibleandmeaningfultopeople.
Arts-basedapproachesoftenusecreativemeanstohelpthepublicconsiderthefutureofatechnologicalinnovationandcangenerateemotionalandcognitiveengagement,allowingparticipantstorelatetoaparticularstoryandtopicthroughpersonalexperience.
Indoingso,thesetechniquescanhelptoreachgroupsofpeoplewhomaynotbeinterestedinatopicorparticipateinmoreformalengagementtechniques.
Forexample,participatorytheatreisatechniquethatcanhelptopromoteaudienceengagementandparticipationoncomplextopics(e.
g.
AI).
Thistypeoftechniquecouldpotentiallymakepolicymakingaroundtechnologicalinnovationmoreinclusive.
Similarly,experientialtechniques,suchasBioMetricMirror(seeCasestudy08;AnnexA),orVRsimulations(Venverlooetal.
2020),thatenablethepublictoexperiencearealisticversionofatechnologicalinnovationcanmakeengagementmorethought-provokingandelicitemotionalresponses.
Nonetheless,itcanbeusefultosupplementexperientialorarts-basedmethodswithmoreformalmethods(suchasasurvey,interviewordialogue)tocapturethepublic'sviewsfollowinganexperience.
Havinganimpactonoutcomessuchasregulation,policyandmarketadoptionofatechnologicalinnovationtypicallyrequiresbuy-inandengagementwiththerightstakeholdersForpublicengagementtohaveanimpactonaparticularoutcome,itisimportanttoengagewithandhavebuy-infromappropriatestakeholders.
Forexample,whilethereissomeevidencethattheGATEwayProjecthasshapedthinkingamongthoseorganizationsdirectlyinvolvedintheproject),theevidenceofwiderimpactontheCAVecosystemislimited(seeCasestudy02;AnnexA).
Inpart,thismaybeattributedtothefactthatCAVsremainatanearlystageofdevelopment.
Atthesametime,accordingtooneinterviewee,theprojectcouldalsohavedonemoretoengagekeyaudiences(suchaspolicymakingaudiences)withitsfindings(CS02-01;99seeCasestudy02;AnnexA).
Inanotherexample,therapidonlinedeliberationaboutonlinetechnologiestocombatCOVID-19alsoshowedthatpolicymakersupportandbuy-inisimportant(seeCasestudy10;AnnexA).
Thetighttimeframesmeanttherewaslesstimeforpolicymakerstobeinvolved.
Itwasnoted,however,thatgoingforwardsitwouldbeimportanttohavepolicymakerbuy-inandengagement,andclarityonhowtheworkwillfeedintothedecision-makingprocess(CS10-01;CS10-02;seeCasestudy10;AnnexA).
ItisimportanttobuildevaluationintopublicengagementprocessestotrackimpactsandoutcomesovertimeItisnotablethatthereisrelativelylimitedgood-qualityevidencethatevaluatestheeffectivenessofpublicengagementprocessesavailableintheliterature.
Inparticular,mostevaluationsofpublicengagementprocessestypicallyonlyincludeshort-termfollow-upandmanyoftenlookatintermediateratherthanfinaloutcomes(e.
g.
theproductionofapolicybrieforrecommendationsratherthanwhetherandhowthesehavebeentakenup).
Forexample,certainassessmentsdofollowuptoconsiderimpactsontrust,butthisisonlycarriedoutintheshort-term.
Whetherornotincreasedtrustpersistsoverthelongtermisnotclear.
Itisalsoapparentfromtheliteraturethatfewcategoriesofpublicengagementtechniqueshavebeenevaluated.
Mostformalevaluationsarefocussedondeliberativemethods.
Therearesomeinformalassessmentsthroughparticipantfeedbackorresearcherobservationsofsomeofthemoreatypicaltechniques.
Itisimportanttoincorporateappropriateevaluationintopublicengagementprocessestobeabletotracktheimpactonparticipantsengagedintheprocess,aswellasthelonger-termoutcomesoftheprocess.
Improvingtheevidencebasethroughlonger-termfollow-upwouldnotjusthelpustounderstandwhetherexistingpublicengagementisworking,itwouldalsoenableustobetterdesignpublicengagementthatiseffectiveatimprovingtherangeofdifferentoutcomes.
Thereisalsolikelytobescopeforlearningacrosscontextsandpurposes,aswellasacrossdifferentevaluationsthemselves.
FinalreflectionsTable2andTable3(providedastwosheetsintheaccompanyingExcelfile)provideanoverviewoftheevidencepresentedinthereportandsummarisedabove.
Table2providesasummaryofevidenceontheeffectivenessofthedifferentpublicengagementtechniquesinrelationtodifferentoutcomes.
Table3providesasummaryofevidenceontheimplicationsofapplyingthedifferentpublicengagementtechniquesdiscussedinthereport.
Theinformationinthetablesisintendedtohelppolicymakersandregulatorsinidentifyingappropriatepublicengagement100techniquestouseinspecificcontexts.
Tothiseffect,theyprovideamenuofdifferentoptionstoconsider.
Eachtechniquehasitsownstrengthsandweaknessesthatneedtobefactoredinwhenselectingoneormorepublicengagementtechniquestoimplement.
Themostsuitabletechniquetoadoptisverymuchdependentonthespecificcontextualfactorsthatrelatetothetechnologyinquestionandthegeographicalcontext,sectorandintendedaudience(s)involved.
Itisalsoimportanttorecognisethatalltechnologiesarenotthesameandthereforebringdisparatechallengesforengagement.
Ultimatelythereisno'one-size-fitsall'publicengagementtechniquethatcanbeusedinallcircumstances.
Indeed,ashighlightedbytheevidencepresentedinthisreport,publicengagementprocessesthatemployacombinationofdifferenttechniqueshavetendedtobemoreeffective.
Itshouldbenotedthatthetablesarebasedonevidencecitedintheliteraturereviewedandthecasestudies.
Theliteraturereviewedandtechnologicalinnovationfeaturedinthereviewweredefinedbythescopeofthesearchstrategy.
Assuch,thetablesarenotintendedasadefinitiveguideandcannotbeusedtodrawpredictableconclusionsaboutwhichpublicengagementtechniquetouseinwhichcontext.
Table2:Overviewoftheevidencepresentedinthisreportoneffectivenessofpublicengagementtechniques(inthecontextoftechnologicalinnovation)inrelationtodifferentoutcomesThistableispresentedinthesecondsheetintheaccompanyingExcelfile.
Table3:OverviewoftheevidencepresentedinthisreportonthepotentialimplicationsofapplyingthedifferentpublicengagementtechniquesinthecontextoftechnologicalinnovationThistableispresentedinthethirdsheetintheaccompanyingExcelfile.
101AnnexA:CasestudiesillustratingtheuseofpublicengagementtechniquesinthecontextoftechnologicalinnovationInthissection,wepresentthetencasestudieswedevelopedtodemonstrateadeeperandmoreconcreteunderstandingofavarietyofpublicengagementtechniquesappliedindifferentcontexts.
Thetencasestudiesspandifferenttechnologicalinnovations,sectors,organisations,geographicalcontexts,andtimeperiods.
Thecasestudiesthuspaintaricherandmorediverse'ontheground'pictureassociatedwithpublicengagementtechniquesfortechnologicalinnovation.
AsnotedinAnnexC,thefinalselectionoftencasestudieswasagreedinaniterativemannerincollaborationwithBREbyworkingthroughalong-listofpotentialcasestudies.
Thelong-listwasidentifiedusingthefocussedliteraturereviewandsuggestionswereceivedfrompublicengagementexpertsweconsulted.
Foreachcasestudy,wecollatedandanalysedthefollowinginformationbasedoninterviewswithstakeholdersandafocussedreviewofaselectionofarticlesconnectedtoeachcasestudy:WhatisthetechnologicalinnovationThiscoversabriefdescriptionofthe(new)technologicalinnovationinthecontextofthisexample(i.
e.
inthiscountry/region,sectorandtimeperiod).
WhatwasthepurposeofthepublicengagementWhywasthepublicengagementexercisecarriedoutForexample,whatweretheissueswiththetechnologicalinnovationinthecontextofthisexample(i.
e.
insightsintoregulation,cultural,sectorandtimeperiodinfluences)HowwasthepublicengagementcarriedoutWhatpublicengagementmethodorapproachwasused(i.
e.
howareengagementtechniquesbeingapplied)WhichbodiesandstakeholderswereinvolvedintheengagementprocessHoweffectivewasthepublicengagementWhateffectdidtheengagementapproachhaveon(forexample):(a)designandimplementationofregulatoryframeworksforthetechnologicalinnovation;(b)newbusinessmodelsinrelationtotechnologicalinnovation;and(c)public102trustandconfidenceintechnologicalinnovationandrelevantregulatoryframeworksHaseffectivenessofthepublicengagementtechniquebeenformallyassessedWhat'lessons'canbelearntfromthisexampleWhywastheengagementtechniqueeffectiveornoteffectiveWhat,ifany,werethelearningsfromtheassessmentBelowwepresentalistofthetencasestudies.
Eachoftheseispresentedinturnintherestofthisannex.
Casestudy1(CS01):Seriousgametocrowdsourcethepublic'sviewsonmoraldecisionsfacedbyautonomousvehiclesCasestudy2(CS02):Exploringpublicperceptionsonautonomousvehiclesusinglivepublictrials,workshops,sentimentmapping,andobservationalstudiesCasestudy3(CS03):Citizens'jurytounderstandpublicattitudestowardsEthicalAICasestudy4(CS04):Foresightgamingformulti-stakeholderdialoguestoexplorenanotechnologyandResponsibleResearchandInnovationpracticesCasestudy5(CS05):DeterminingpublicperceptionontheuseofvirtualrealityinhealthcarethroughsociallisteningCasestudy6(CS06):UsingthevTaiwanplatformtocarryoutapublicdebateontheregulationofUberinTaiwanCasestudy7(CS07):EngagingexpertandcitizenperspectivesonAIusingaworkshopandonlineplatformCasestudy8(CS08):Engagingthepubliconfacialanalysisandautomateddecision-makingthroughtheuseofBioMetricMirror–aninteractiveapplicationCasestudy9(CS09):CitizenandMulti-ActorConsultationonHorizon2020(CIMULACT)toformulatescienceandtechnologypolicyresearchagendaintheEuropeanUnionCasestudy10(CS10):RapidonlinedeliberationtoexplorepublicattitudestotheuseofCOVID-19-relatedtechnologies103A.
1.
Casestudy1(CS01):Seriousgametocrowdsourcethepublic'sviewsonmoraldecisionsfacedbyautonomousvehiclesCasestudytitleSeriousgametocrowdsourcethepublic'sviewsonmoraldecisionsfacedbyautonomousvehiclesConcisesummaryofthecasestudyAkeychallengearoundthedevelopmentofautonomousvehiclesliesinthemoraldilemmasthattheyarelikelytoface(e.
g.
decidingwhoshouldliveandwhoshoulddiewhenfacedwithapotentialcollisionwithapedestrian).
Tothisend,researchersdevelopedawebsite,MoralMachine,thatuseda'seriousgame'withscenariostocrowdsourcethepublic'sviewsonmoraldecisionsfacedbyautonomousvehicles.
Theaimwastogenerateabetterunderstandingofthepublic'sviewsabouthowautonomousvehiclesshouldsolvemoraldilemmas,aswellastohelpraiseawarenessaboutthistopicamongstthepublic.
Theplatformwasaneffectivelarge-scaledatagatheringexercisethatcollected40milliondecisionsintenlanguagesfrompeoplein233countriesandterritories.
Ithelpedtoshowthataseriousgameisaneffectivemethodtocrowdsourcethepublic'sviewsonacontroversialtopic.
Theexerciseidentifiedstronguniversalmoralpreferences.
Althoughthefindingshavenotledtoanyformaloutcomes,theycouldinprinciplebeusedtocontributetodevelopingglobal,sociallyacceptableprinciplesformachineethicsthatcouldfeedintoconsiderationsbycarmanufacturersandpolicymakers.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleAutonomousvehiclesTransportSeriousgameGlobal(233countries)WhatisthetechnologicalinnovationThetechnologicalinnovationinthisexampleconsistedofautonomousvehicles.
Autonomousvehiclesaredevelopingrapidlyandarenearingmarketadoption(Shabanpouretal.
2018).
Theyareexpectedtogreatlytransformmobilityandthewaytransportationsystemsoperate(Shabanpouretal.
2018).
Thereare,however,anumberofchallengesassociatedwithdevelopingandadoptingautonomous104vehicles,includingaroundliability,privacy,andsecurityofthistechnology(Shabanpouretal.
2018).
WhatwasthepurposeofthepublicengagementAkeychallengearoundthedevelopmentofautonomousvehiclesliesinthemoraldilemmasthattheyarelikelytoface–e.
g.
,decidingwhoshouldliveandwhoshoulddiewhenfacedwithapotentialcollisionwithapedestrian(Awadetal.
2018).
Carmanufacturersandpolicymakersarecurrentlystrugglingwiththesemoraldilemmas,whichrequirethedevelopmentofethicalguidelines.
Ithasbeenrecognisedthatpublicacceptanceofautonomousvehicleswillrequiretheirunderstandingandacceptanceoftheethicalprinciplesprogrammedintoautonomousvehicles(Awadetal.
2018).
This,inturn,requiresagoodunderstandingofthepublic'sviewsabouthowautonomousvehiclesshouldsolvemoraldilemmas.
Therefore,thegoalofthepublicengagementexercisewastocrowdsourcethepublic'sviewsonhowtheywouldwantautonomousvehiclestosolvemoraldilemmasinthecontextofunavoidableaccidents(Awad,Dsouzaetal.
2018).
Theaimwastogatherviews,raiseawarenessandpromotediscussionaboutthistopicamongstthepublic(CS01-01).
HowwasthepublicengagementcarriedoutAteamofresearchersdesignedawebsite,MoralMachine,tocollectdataonthemoralacceptabilityofdecisionsmadebyautonomousvehiclesinsituationsofunavoidableaccidents,inwhichtheymustdecidewhoissparedandwhoissacrificed(Awadetal.
2018).
Thewebsitefeatureda'seriousgame'thatusedimagestoshowusersrandomlygeneratedunavoidableaccidentscenarios(basedonthe'trolleyproblem')16withtwopossibleoutcomes,dependingonwhethertheautonomousvehicleswervesorstaysoncourse(CS01-01).
Usersclickedontheoutcomethattheyfoundpreferable.
Aftercompletingasessionof13accidentscenarios,participantscouldthencompleteasurveythatcollected,amongstothervariables,demographicinformationsuchasgender,age,income,andeducation,aswellasreligiousandpoliticalattitudes.
OriginallywritteninEnglish,thewebsitewasprogressivelytranslatedintoninelanguages(Arabic,Chinese,French,German,Japanese,Korean,Portuguese,RussianandSpanish)betweenNovember2016andMarch2017(Awadetal.
2018)(CS01-01).
Thecountryfromwhichusersaccessedthewebsitewasgeo-localizedthroughtheIPaddressoftheircomputerormobiledevicetoconductageographicalanalysisofmoralpreferences.
16Thetrolleyproblemisathoughtexperimentinethicsinvolvingethicaldilemmasofwhethertosacrificeonepersontosavealargernumber.
105DatawerecollectedbetweenJune2016andMarch2017(Awadetal.
2018).
Theresearchersusedvarioussocialmediachannelstopromotethewebsiteonamonthlybasis,andthisalsosubsequentlygotpickedupbyotheronlinechannels(e.
g.
YouTube,Reddit),enablingthewebsitetogoviral(CS10-01).
Theplatformaimedtocollecttheviewsofusersallovertheworldwithnoaccountorlog-indetailsneededtoparticipate(CS01-01).
HoweffectivewasthepublicengagementThepublicengagementprocesshasnotbeenassessedformallybuttheauthorsoffersomekeyobservations.
Oneintervieweefeltthatthepublicengagementprocesshelpedtoincreaseawarenessamongstthepublicofethicalconsiderationssurroundingautonomousvehiclesandmachinesingeneral(CS01-01).
Thewebsitewaseffectiveatcollectinglarge-scaleviews,gatheringapproximately40milliondecisionsintenlanguagesfrompeoplein233countriesandterritories(Awadetal.
2018)(CS01-01).
Basedonthesedata,theresearcherswereabletoidentifythreestronguniversalmoralpreferences:thepreferenceforsparinghumanlives,thepreferenceforsparingmorelives,andthepreferenceforsparingyounglives.
Theauthorsalsoidentifiedmoralpreferencesthatvariedacrosscountriessuchaspreferencesbasedongenderorsocialstatus(Awadetal.
2018).
Althoughthefindingshavenotledtoanyformaloutcomes,thedatacollectedcould,inprinciple,beusedtocontributetodevelopingglobal,sociallyacceptableprinciplesformachineethicsthatcouldfeedintoconsiderationsbycarmanufacturersandpolicymakers(CS01-01).
Oneintervieweereportedthatakeyfactorthatcontributedtothesuccessoftheengagementwasthegamifiedaspect,whichconsistedofanengaginggame(CS01-01).
Inaddition,theresearchershiredaprofessionaldesignertocreateanappealingwebsite(CS01-01).
Thegameitselfconsistedofrelativelysimpleexperimentalscenariosthatrequiredachoicebetweentwooptions(CS01-01).
Otherimportantfactorshighlightedbyanintervieweeascontributingtothesuccessoftheplatformincluded:thechoiceofaninterestingandcontroversialtopicthathelpedtogarnerinterestandattention;thechoiceofusingawebsitetomakethegamewidelyaccessibletogatherviewsatscale;andthetranslationintomultiplelanguages,whichwidenedoutreach(CS01-01).
Akeycaveatoftheapproachusedfortheengagementwasthatthesamplewasself-selected,andnotrepresentativeofthesocio-demographicsofeachcountry(Awadetal.
2018)(CS01-01).
Theauthorsarguethatthesamplewaslikelytoreflectapopulationthatisinterestedindriverlesscartechnology,andmorelikelyto106participateinearlyadoption,whichmayhaveimplicationsforthemoralviewscollected,andhencepotentialusersofthisstudysuchascarmanufacturersandpolicymakers(Awadetal.
2018).
What'lessons'canbelearntfromthisexampleAseriousgamecanbeusedtogatherlarge-scaleviews:Thiswasanexperimentalprocessthatdemonstratedthefeasibilityofcrowdsourcingthepublic'sviewsonacontroversialtopic(CS01-01).
Itisimportanttochooseaninterestingresearchquestionortopic:Theprojectwascostlyandtime-consuming,andrequiredthehiringofawebdesigner(CS01-01).
Theresearcherschoseacontroversialtopicthatgarneredconsiderableattentionfromthepublic,whichtheyconsideredmadetheuseofresourcesworthwhile(CS01-01).
Therefore,ifchoosingthistypeofengagementapproach(i.
e.
todesignagamefromscratch),itwouldbeworthconsideringthetypeoftopicandquestion(s)carefullytoensuremaximalinterest.
Itisimportanttobeclearonthepurposeoftheengagement:Thepurposeshouldbetohelptodeterminehowbesttodesigntheengagement.
Forexample,isitforscientificdatacollection,engagement,orbothIfthepurposeissolelyengagement,thenthewebsitecouldbemadesimpler.
Incontrast,ifdatacollectionismoreimportant,thenthismightnotbethebesttool(CS01-01).
Itisimportanttodecideonthereachoftheengagement:Thistoolwaseffectiveatgatheringlarge-scaleglobalviews.
Ifthegoaloftheengagementistounderstandmorelocalviews,thenotherchannelsandtechniquesofengagementmightbemoreappropriate(CS01-01).
Here,acaveattonoteisthatifparticipantsareself-selectingthenthiswilldrivethesample.
107A.
2.
Casestudy2(CS02):Exploringpublicperceptionsonautonomousvehiclesusinglivepublictrials,workshops,sentimentmapping,andobservationalstudiesCasestudytitleExploringpublicperceptionsonautonomousvehiclesusinglivepublictrialsConcisesummaryofthecasestudyLedbyTransportLaboratoryLondon(TRL),theGATEway(GreenwichAutomatedTransportEnvironment)projectemployedarangeofpublicengagementmethodstoexplorepublicperceptionstowardsconnectedandautonomousvehicles(CAVs)inordertounderstandsocietalfactorsaffectingtheadoptionofCAVs.
TheGATEwayProjectdemonstratesthepotentialoflivepublictrialsasamethodforpublicengagement,whilealsohighlightingtheroleofdiverseengagementtechniquesinengagingdifferentpublicaudiences.
InsightsfromtheGATEwayhaveshapedtheworkofcertainactors,e.
g.
thevehicledesigncoursesoftheRoyalCollegeofArt.
Engagementtechniquesusedbytheprojecthavealsobeenincorporatedintomarket-orientedtest-bedenvironments,asexemplifiedbytheLondonSmartMobilityLivingLab(SMLL).
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleConnectedandautonomousvehicles(CAVs)TransportLivetrials,creativeworkshops,exhibitions,sentimentmapping,observationalstudiesUnitedKingdom,2015–2018WhatisthetechnologicalinnovationTheGATEwayProjectrespondedtotherapidtechnologicaldevelopmentofCAVsandthepotentialforthistobringsignificantchangestothefutureoftransportglobally(TRLpublish2018a).
Specifically,theGATEwayProjectfocussedonthreetypesofCAV:driverlesspods,self-parking(auto-valet)services,andautomateddeliveryservices(TRLpublish2018a).
Initsfinalreport,theGATEwayProjectidentifiedsixkeydriversofgrowthintheuseofCAVs:(1)thedigitalrevolution,includingsmartdevices,theInternetofThings(IoT),wirelessconnectivity,AIandbigdata;(2)societaltrends,includingincreasingmovestowardssharedownership;(3)airqualitychallenges;(4)safetyandwelfare;(5)capacitylimits,includingurban108populationgrowthandcongestion;and(6)growinginvestmentinCAVs(TRLpublish2018a).
WhatwasthepurposeofthepublicengagementBroadly,theGATEwayProjectaimedto'demonstratethesafeandefficientintegrationofsophisticatedautomatedtransportsystemswithincomplexreal-worldsmartcityenvironments'(TRLpublish2018a).
Aspartofthis,akeyaimoftheprojectwastoexplorepublicperceptionsofCAVsinordertounderstandsocietalfactorsaffectingadoptionofautomatedvehicles(TRLpublish2018a).
17TheinsightsgainedthroughtheprojectwereintendedtoinformthedecisionsmadebyactorsfromdifferentsectorsoftheCAVecosystem,includinginsurance,vehicledesign,connectivityandurbanplanning(TRLpublish2018a).
PartnersintheprojectincludedRoyalSunAlliance(RSA),O2Telefónica,theRoyalBoroughofGreenwichandDGCities(TRLpublish2018a).
HowwasthepublicengagementcarriedoutThekeymethodusedbytheprojectwasaseriesoflive'publictrials'ofCAVs.
Thesepublictrialsprovidedmembersofthegeneralpublictheopportunitytoexperience(andprovidefeedbackon)differenttypesofCAVinalive,real-worldsetting.
Thetrialsincluded:AdriverlesspodshuttleserviceoperatedontheThamesPathoftheGreenwichPeninsulainLondon,designedtoprovidealinkingservicebetweentransporthubs.
Membersofthepublicwereinvitedtousetheserviceeitheronapre-bookedor'walk-up'basisandprovidefeedbackthroughqualitativesurveys(TRLpublish2018a).
Around320membersofpublicparticipatedinthistrial,with118completingthesurvey(TRLpublish2018a).
18Anautomatedvaletparkingtrialinwhich35participants(includingparticipantswithadditionalmobilityneeds)gaineddirectexperienceoftheautomatedparkingfunctionalityofaCAV.
Thetrialincludedpre-trialinterviews,andpost-trial17Morebroadly,theGATEwayProjecthadsixkeyobjectives:(1)tounderstandthetechnical,cultural,societalandlegalchallengesandbarrierstotheimplementationandadoptionofautomatedvehicles;(2)toinspireover35,000individualsengagedwithautomatedtransporttechnology;(3)togeneratevaluable,exploitableknowledgeofthesystemsrequiredfortheeffectiveandsafevalidation,deploymentandintegrationofautomatedtransportinaSmartCity'real-world'environment;(4)tocreatethefirstfullyvalidatedtestbedintheheartofLondon,theSmartMobilityLivingLab:London;(5)topositionUKPLCattheforefrontoftheglobalCAVmarketplace;and(6)todemonstratethesafeandefficientintegrationofsophisticatedautomatedtransportsystemsintocomplexreal-worldSmartCityenvironments(TRLpublish2018a).
18Aspartofthistrial,33participantscompletedinterviewsbeforeandaftertheirexperienceofusingtheservice(FernándezMedina&Jenkins2017).
109workshopsandquestionnairestoexploretheimpactoftheexperienceonuserattitudes(Harrow2018;TRLpublish2018a).
Anautomateddeliveryservicetrial,undertakenincollaborationwithOcado,inwhich108residentswithintheRoyalBoroughofGreenwichreceivedgrocerydeliveriesviaCAV'pods'.
19Asurveycapturedparticipants'immediateperceptionsontheexperience(Tong2017;TRLpublish2018a).
Alongsidepublictrials,theGATEwayProjectalsoemployedotherengagementmethods.
IncollaborationwiththeRoyalCollegeofArt(RCA),theprojectranaseriesofpublicworkshopstounderstandpeople'sperceptionstowardsCAVsandtoexplorehowthedesignofCAVsmightinfluencepublicattitudestowardsthem(Phillips&Roberts2016;TRLpublish2018a).
DesignersfromtheRCAusedinsightsgainedthroughtheworkshopstocreateaseriesofimagesofCAVs,whichwerethendisplayedinLondon'sTransportMuseumforsixweeks,reachingover30,000visitors(TRLpublish2018a).
AnotherdimensionoftheGATEwayProjectwas'sentimentmapping'(TRLpublish2018a;TRLpublish2018b).
Here,focusingonthedriverlessshuttleservicetrial,theprojectgatheredandanalyseddataonperceptionstowardsCAVsbefore,duringandafterthetrial.
Sentimentmappingwasconductedbyencouragingpeopletovisitsentimentmappingwebsiteswheretheycouldprovidetheiropinionsandtheirlocation(TRLpublish2018b).
Capturinglocationdataenabledthemappingofwherepositive,negativeorneutralcommentshadbeenreceived.
Forthetrial-basedsentimentmapping,thisenabledanalysisofthewayinwhichcommentsdifferedatdifferentpointsonthetrialroute(TRLpublish2018b).
Duringthetrial,thewebsiteswereusedtocaptureviewsofbothusersandonlookersontheGATEwayCAVs(TRLpublish2018a).
20Intotal,therewereover21,279visitstothesentimentmappingwebsitesand746commentsleftbyparticipants(TRLpublish2018b).
Finally,theGATEwayProjectalsoconductedobservationalstudiesonhowpedestriansandcyclistsbehavedaroundtheautomatedvehicles,againfocussingonthedriverlessshuttleservice(TRLpublish2018a).
HoweffectivewasthepublicengagementTheprincipalaimofthepublicengagementundertakenbytheGATEwayProjectwastoproducenewsocialknowledgeregardingpublicattitudestowardsCAVs19Whilethepodsweredriverless,asafetydriverwaspresentandcouldtakecontrolatanypointifnecessary.
20AsentimentmappingwebsitewasalsodesignedtoaccompanytheRCAexhibitionattheTransportMuseumhttps://www.
rca.
ac.
uk/news-and-events/news/rca-explores-sentiment-mapping-travel-management/110(TRLpublish2018a)(CS02-01).
Inthisrespect,theprojectwasbotheffectiveandalsoexemplaryinitsuseofinnovativeengagementmethodstoexplorepublicviews.
Firstly,theGATEwayProjectdemonstratedthewayinwhichlivepublictrialscanbeusedasaneffectivetechniqueforpublicengagementwithnewtechnologies.
Theproject'slivetrialsnotonlyreachedawideaudience(comprisedbothofthosewhopre-registeredtoparticipateandthosewhoencounteredthetrialsspontaneously),butalsoprovidedopportunitiestoexploretheimpactoffirst-handuserexperienceonpublicattitudestowardsthetechnology(FernándezMedina&Jenkins2017)(CS02-01,CS02-02).
TheconductoflivepublictrialsalsoenabledtheprojecttoconsiderpublicattitudestowardsCAVsfromdifferentperspectives.
Whilethetrialsthemselvesfocusedonuserexperience,observationalstudiesconductedinparallelexaminedtheviewsofpedestriansandcycliststowardsCAVs(CS02-01,CS02-02).
Secondly,andrelatedly,theGATEwayProjectalsodemonstratedtheeffectivenessofacombined,mixed-methodapproach–includinglivepublictrials,workshops,exhibitions,surveysandsentimentmapping–inordertoreachawiderangeofpublicaudiences(CS02-01,CS02-02).
Whilefurtherhelpingtoensuretheparticipationofthemoregeneralpublic(aswellasproactiveCAVenthusiasts),therangeofengagementmethodsalsohelpedtoensurethecoverageofdifferentdemographicgroups,includingthosewithadditionalmobilityneeds(CS02-01).
Incertaincases,insightsproducedbytheGATEwayProjecthaveshapedtheworkoforganisationsinvolvedinthedevelopmentofCAVs.
TheRCA,forexample,hasincorporatedlearningsfromtheprojectintoitscoursesonautonomousvehicledesign(CS02-01).
Morebroadly,however,thereislimitedevidencethattheresultsoftheGATEwayhavehadasignificantimpactonactorswithintheCAVecosystemorontheregulatoryframeworkssurroundingCAVs,thoughthiscouldchangeastheCAVmarketmatures(CS02-01CS02-02).
Accordingtooneinterviewee,onechallengefacedbytheprojecthasbeentogetitsworkinfrontoftherightaudiences(bothintermsoftechnologydevelopmentandpolicymakingaudiences),somethingthatagreateremphasisonmarketingandcommunicatingmayhavehelpedtoaddress(CS02-01).
WhileevidencethattheGATEway'spublicengagementhasimpacteddirectlyupontheCAVecosystemisfairlylimited,theprojecthashadimpactinotherways.
Mostnotably,publicengagementtechniquesusedwithintheGATEwayProjecthavesubsequentlybeenincorporatedintotheLondonSmartMobilityLivingLab(SMLL).
BuildingontheGATEway,andbasedintheRoyalBoroughofGreenwich,theSMLLisacomprehensivetestbedenvironmentenablingmanufacturersandoperatorstodevelopnewCAVsystemsinacomplex,real-worldenvironment(TRLpublish2018a).
21CAVtestingperformedwithintheSMLL,includingtherecenttrialofthe21TheSMLLisbasedintheRoyalBoroughofGreenwichandQueenElizabethOlympicPark.
111UK'sfirstopen-architecturevehicle,hasfollowedtheGATEwayinusinglivepublictrialsandusersurveys(SmartMobilityLivingLab:London2020).
OtherSMLLtrials,whilenotusinglivepublictrials,haveaccompaniedvehicletestingwithdetailedanalysisofcustomerperspectivestowardsCAVsthroughworkshopsandsurveys(MERGEGreenwich2018).
WhileSMLLCAVtrialshavenotnecessarilyhadthesameexplicitfocusonpublicengagementastheGATEwayProject,noremployedthesamerangeofengagementtechniques,theuseoftheGATEway'smethodswithintheSMLLdemonstratesthewayinwhichtheGATEway'spublicengagementhasbeentakenupbyactorsattheleadingedgeoftheCAVdevelopmentprocess(CS02-02).
What'lessons'canbelearntfromthisexampleLivepublictrialsprovideaneffectivetechniqueforobtainingstakeholderperspectivesonanewtechnologicalinnovation:InthecontextoftheGATEway,livetrialshelpedtoengageawideaudience,includingthosewhoengageduponencounteringthetrialspontaneously.
Thelivetrialsalsoprovidedopportunitiesforexploringtheimpactoffirst-handexperienceofatechnologyandforexploringpublicattitudesfromdifferentperspectives(CS02-01,CS02-02).
Amulti-dimensionalapproachcanhelptofurtherensurethatpublicengagementcoversdifferentpublicaudiences:Thecombinationofpublictrials,creativeworkshops,publicexhibitionsandsentimentmappinghelpedtoensuretheparticipationofdiverseaudiences,includingfromacrossdifferentdemographicgroupsandamongthosewithadditionalmobilityneeds(CS02-01,CS02-02).
Tohelpensurethattheinsightsproducedbypublicengagementreachtherightaudiences,effectivemarketingandcommunicationsstrategiesareneeded:WhilefindingsoftheGATEwayhaveshapedtheworkofcertainactorswiththeCAVecosystem,accordingtooneinterviewee,theprojectcouldalsohavedonemoretoengagepolicymakingaudienceswithitsfindings(CS02-01).
MoreeffectivemarketingandcommunicationmayhaveincreasedtheopportunityfortheprojecttoinformpolicyandregulatoryframeworkssurroundingCAVs,forexamplethoserelatingtovehicledesign(CS02-01)Innovationtestbedsprovideonewayinwhichpublicengagementtechniquescanbeincorporatedintoatechnologydevelopmentprocess:PublicengagementtechniquesusedbytheGATEwayhave,insomecases,beenincorporatedintoCAVtrialsconductedwithintheSMLL(MERGEGreenwich2018;SmartMobilityLivingLab:London2020)(CS02-02).
112A.
3.
Casestudy3(CS03):Citizens'jurytounderstandpublicattitudestowardsEthicalAICasestudytitleCitizens'jurytounderstandpublicattitudestowardsEthicalAIConcisesummaryofthecasestudyTheethicaluseofAIhasbecomeincreasinglyimportantinautomateddecisionsystemsthatuseAItoinformormakedecisionsonwhichactionstopursue.
TheRoyalSocietyfortheencouragementofArts,ManufacturesandCommerce(RSA)andDeepMindpartneredonadeliberativeengagementexerciseusingacitizenjuryforpublicengagementontheimpactsofethicalAI.
Theprojectwasnotformallyassessed,butanecdotalevidencefrominterviewsanddeskresearchindicatesthatparticipatingcitizensfeltbetterinformedandhadabetterunderstandingofautomateddecisionsystemsfollowingtheproject.
Intervieweesindicatedthatthepublicengagementformatalsoillustratedtheimportanceofusingpublicengagementtechniquesforotherorganisations,althoughthisimpactwasnotcitedinrelevantdocuments.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleArtificialIntelligenceN/ACitizenjuryUnitedKingdom,2018WhatisthetechnologicalinnovationThiscasestudyaddressesEthicalAIinthecontextofautomateddecisionsystems.
Inthiscontext:EthicalAIisdefinedas:"AIthatisdesignedandimplementedbasedonthepublic'svalues,asarticulatedthroughadeliberativeandinclusivedialoguebetweenexpertsandcitizens"(Balarametal.
2018,9).
Automateddecisionsystems(ADS),orcomputersystems,informormakedecisionsonwhichactionstopursuerelatedtoanindividualorbusiness(Balarametal.
2018).
ADSdonothavetouseAI,butincreasinglydosotoimprovetheaccuracyofpredictions(RSA2019).
Forexample,ADShavebeenusedbytheprivatesectortograntloans,andthepublicsectorisexploringtheuseofADSforplanningandmanagingnewinfrastructure,toreducetaxfraud,andratingtheperformanceofschoolsandhospitals(RSA2019).
AIreferstothefieldofcomputersciencethatfocusseson113solvingcognitiveproblemsthatarecommonlyassociatedwithhumanintelligence(Balarametal.
2018).
WhatwasthepurposeofthepublicengagementTheRSAconvenedacitizen'sjuryin2018toenterintoadeliberativedialoguetounderstandtheethicaluseandethicalquestionsofAItohelpmakedecisionsforDeepMind,andhowothercompanies,organisationsandpublicinstitutionsshouldrespondtoethicalquestionsonAI.
22TheprojectexploredtheuseofAIsystemsbasedonthevaluesofthepublic,examininghowcitizensunderstandAIinitscontemporaryuses,andhowethicalreasoningisappliedtoitspotentialusesinthedeliveryofbothprivateandpublicservices(Balarametal.
2018)(CS03-02,CS03-03).
HowwasthepublicengagementcarriedoutThepublicengagementexercisewascarriedoutbytheRSAinthreestagesinApril,May,JuneandOctoberof2018.
Thesestageswere:asurvey,acitizenjury,andtwoworkshops.
Detailsofeachstageareasfollows:Survey:RSApartneredwithYouGovtocarryoutanonlinesurveyof2,000peopleinApril2018.
23Thesurveyquestionsfocussedonthepublic'sfamiliaritywith,concernandsupportfor,AIandADS(Balarametal.
2018).
Citizenjury:TheRSAbroughttogetheragroupof25-2924peoplefromacrossEnglandandWalestounderstandpeople'sviewsaboutconcreteusesofAIforautomateddecision-making(Balarametal.
2018;RSA,n.
d.
)(CS03-01).
25,26Thisnumberwasslightlyhigherthancitizenjuriesthatnormallycomprisebetween12-24people(CS03-01).
Thejuryconsistedofthreesessionsthattookplaceoverfourdaysintotal.
Session1tookplaceoveronelongweekend(FridayeveninguntilSunday);sessions2and3tookplaceovertwoSaturdays.
Thesameparticipantscametogetherondifferentoccasions.
Thefirsttwosessionswereheldonemonthapart,andthefinalsessiontookplacefourmonthslater.
Thiswasinlinewithbestpracticeforjuriesthatnormallylastbetweentwotosevendays(Balarametal.
2018):22DeepMindisanAI-basedcompanyandresearchlaboratoryintheUKthatwasfundedin2010andacquiredbyGooglein2014.
DeepMindprovidesresearchandbuildsartificialintelligencesystemsthataresafeandaimto'solveintelligenceandadvancescientificdiscoveryforall'(DeepMind2021a).
23Thesurveywascarriedoutonlineandtookplacebetween16and17April2018andwascarriedoutwith2,074adultsofdifferentages,genders,andethnicities(RSA2019)(CS-03-02).
24Therangereflectsthedifferentnumberofparticipantsattheeventsheldinfoursessions.
25Theparticipantsincludedarepresentativerangeofages,abilities,ethnicitiesandwereofdifferentsocio-economicbackgrounds.
Theparticipantswerealsosplitequallyintermsofattitudestowardstechnology.
2612and13May,2June,and13October2018.
114Onthefirstdayofthefirstsession,theRSAfocussedonbuildingtrustwithparticipantsandencouragedthemtoparticipateinaspeeddatingexercisewithexpertstoaskquestions(CS03-02).
TheseconddayofthefirstsessionfocussedoncasestudiesandthespecificapplicationofADSindifferentfields(CS03-02).
ThethirddayandsecondsessionfocussedonparticipantscreatingconditionsandrecommendationsforADSthatwaspresentedtoapanelconsistingofstakeholdersfromtheCitizenAdviceBureau,otherthinktanks,TechUKandacademia(CS03-02).
Thethirdandfinalsessionfocussedoncelebratingjurors'contributionsandnetworkingbetweenjurorsandexperts(CS03-02).
ThedeliberationsconsistedofcitizensspendingtimetolearnabouttheproblemofethicalAIanddiscussitfromdifferentperspectives,withinputfrom24expertwitnesseswhoexplainedkeyinformationandansweredquestionsfromthejurors(Balarametal.
2018).
27Oneintervieweenotedthatitwasinlinewithbestpracticeforcitizens'juriesasithadasinglequestion,expertswerepresent,andtherewasanopportunitytoexplorecitizens'views(CS03-01).
ThedeliberationsinthejuryweresummarisedbytheRSAaftereachsessionandfedbacktotheparticipatingcitizensforagreementbeforeanewsessionstarted(CS03-01).
Thefinalprojectreportwasbaseduponsummariesthatwereagreedbytheparticipatingcitizensandwerepresentedtoapanelofrelevantstakeholders,aswellasatafinalnetworkingevent(CSO3-01,CS03-02).
28ThejurywasledbyaprofessionalfacilitatorandsupportedbyRSAstaff(Balarametal.
2018)(CS03-02).
Workshops:TwoworkshopswereheldwithBlack,AsianandMinorityEthnicpeople,mainlymen,whowereconsideredtobemorelikelytobeaffectedbytheissuesunderdiscussion,suchasbeingdisproportionatelyaffectedbyfacialrecognitiontechnologyinpolicing(Balarametal.
2018).
HoweffectivewasthepublicengagementThepublicengagementtechniquewasnotformallyassessed.
Feedbackfromparticipants,however,andthosewhofacilitatedthejurysuggestedthattheprocesswaseffectivebecause(RSA2019):AnecdotalevidenceindicatesthatparticipatingcitizensfeltbetterinformedandhadabetterunderstandingofADSafterthejury(CS03-01).
Asaresultoftheengagement,intervieweeswhofacilitatedtheexercisefeltthatthe27ThisincludedapresentationofcasestudiesontheuseofADSintherecruitment,healthcareandcriminaljusticesectors.
28IncludingrepresentativesfromAdaLovelace,theCitizensAdviceBureau,DeepMind,theInformationCommissioner'sAuthorityandTechUK,andrepresentativesfromacademia(CS-03-02).
115trustinADSincreased(CS03-02,CS03-03).
Oneinterviewee,however,saidthattherewouldnotbeasignificantlevelofdistrustortrustforADSaftertheprocessduetoalackofawarenessaboutADSamongstparticipantspriortotheprocess(CS03-01).
Participatingcitizensvaluedthevarietyofinputsfromdifferentstakeholderswhoparticipatedintheprocess,includingcitizensandexperts(RSA2019).
Participatingcitizenssaidthattheyhaddevelopedtheirthoughtsandthattheiropinionshadconvergedfollowingtheprocess(RSA2019)Participantslearntquicklyfromtheprocess,andscaledupandalignedtheirresponsestobereflectiveofthosewhoareexpertsinADS,which,accordingtofacilitators,reflectedtheirabilitytounderstandtheissuesunderdiscussion(CS03-02,CS03-03)Aftertheengagement,theparticipatingcitizensdevisedalistofquestionstheywouldrecommendtheirpeerstoaskaboutADS.
TheRSAcreatedatoolkit29forinstitutionsandcitizensdetailingtheconditionsthatparticipantswouldliketoseebuilt-intotheautomateddecision-makingprocess,whichcapturedconditionsandconsiderationsatthedesignstage,creationstage,andtheapplicationstageofADS(RSA2019).
30Accordingtothefacilitators,thejuryillustratedtheimportanceofusingpublicengagementtootherorganisations,suchasTheAlanTuringInstitute,DeepMind,theInformationCommissioner'sOfficeandtheNationalHealthService(CS03-02,CS03-03).
Thisisreflectedin:(1)thesubsequentworkofDeepMind'sethicsandsocietyteam,whichispresentedonDeepMind'swebsite(DeepMind2021b);(2)theinspirationofaprojectbetweentheRSAandtheNationalHealthServiceonpatientAI(Singh2019);and(3)theadoptionofpublicengagementapproachesforexplainingAIprojectsbybothTheAlanTuringInstituteandtheInformationCommissioner'sOffice,soattendedthefinalevent(ICO&TheAlanTuringInstitute2021).
Itisnoted,however,thattheRSA'scitizenjuryhasnotyetbeencitedinrelevantdocuments.
What'lessons'canbelearntfromthisexampleTheRSAsummarisedseveralkeylessonsandbenefitsandchallengesrelatedtotheiruseofcitizenjuriesfortechnologicalinnovation(RSA2019):Citizenjuriesarenotnecessarilythebestwaytolearnfromminoritygroups:astheiruniqueexperiencescanbesilencedingroupdiscussions(RSA2019).
29Seehttps://www.
thersa.
org/reports/democratising-decisions-technology-toolkit.
30Theseincluded:(1)'WhatimpactwillADShaveonbroadersocialstructuresandinteractions';(2)'Isitsafeinwaymydetailsarebeingshared';(3)'WillIknowwhatADSisbeingused';(4)'How,orisitregulated';(5)'HowcanIcanchallengeit';and(6)'Howdoesitbenefitme'116Jurorsshouldreceiveinformationofahighqualitythatisunbiased:Awaytocircumventbiasistouseanindependentfacilitatorandadvisorygroupinthedesignandfacilitationoftheexercise,andtoprovideinformationtojurorsfromexpertwitnesses(RSA2019).
Considerthetechnologydesignstageatwhichapublicengagementexerciseismostuseful,thecomplexitiesofthequestionsbeingaskedandhowtheresultscanbestbeincorporatedintoimprovinganexistingsystem:Resultsshouldbeacknowledgedandactedupontoengenderpublictrust(RSA2019).
Ashortertimelinecanbeconducivetofacilitateon-goingengagement:ThefirstsessiontookplacefromFridaytoSundayinMay2018tomakejurorscomfortablewiththeprocess,buildtrustwiththedeliveryteamandotherjurors,provideconcentratedtimetohearfromexperts,andfacilitateaproperunderstandingofthecomplexityofADS.
Thelatersessions,however,tookplaceseveralmonthslater(inJuneandOctober),whichmighthaveinfluencedthelevelsofengagementofparticipants(RSA2019).
Re-considerthefacilitationteam'sroleinsynthesisingthediscussion:Inthefuture,participatingcitizensshouldbeempoweredtowritetheirownsummariestoensureownershipoftheoutcomesoftheprocess(RSA2019).
Deliberativemethodsareresourceintensiveandrequire'in-depth,long-termplanning,integrationandfacilitation'(p.
51):RSAarguesthatthedeliberativedialoguewashelpfulinelicitingrecommendationsandinsightsthatthejurorscameupwithcollectively,butthatthesemethodsshouldbecarefullyplannedandintegratedbeforecommencingtheprocess.
Decideonthepublicengagementtechniquebasedontheproblemandperspectivesthatshouldbeinvolved:Fullydevelopandunderstandthepublicengagementtechniquebeforeengagingwithittoavoidtheriskofalackofunderstandingofwhattheprocessfullyinvolves(CS03-01).
31Buildinanevaluationofthepublicengagementexercise:Oneintervieweesaiditisimportanttobuildinaproperevaluationtotracktheimpactofthecitizenjuryonachangeinpracticeforparticipantsandcommissioners(CS03-01).
Makethefocusonthepublicengagementexercisespecific:Twointervieweessaidacitizenjuryshouldfocusonspecificissues,asitisdifficulttoreachconsensuswhenlookingatseveralsectorsatonce(CS03-02,CS03-03).
31Forinstance,thereweremoreparticipantsinthecitizenjuryconvenedbytheRSAthanacitizenjurynormallywouldhave(INT01).
117A.
4.
Casestudy4(CS04):Foresightgamingformulti-stakeholderdialoguestoexplorenanotechnologyandResponsibleResearchandInnovationpracticesCasestudytitleTheEuropeanCommission(EC)JointResearchCentre's(JRC)ScenarioExplorationSystem(SES)inNANO2ALL'smulti-stakeholderdialoguesforResponsibleResearchandInnovation(RRI)practicesinnanotechnology.
ConcisesummaryofthecasestudyRRIisimportanttoincorporatesocietalneedsandvalues,andusesdeliberativeandtransparentapproachestodevelopethicallyacceptableandsociallydesirableproducts.
TheprojectusedtheJRCgame,theSES,innationalmulti-stakeholderdialoguestoexplorenanotechnologyandRRIpractices.
Theprojectensuredthatparticipantscouldcomfortablyexpresstheiropinionandresultedintheuptakeofpublicengagementtechniquesatthesciencecentresthatrantheexercise.
Intheparticipatoryprocess,recommendationssuggestedthattoopenupthenanotechnologyresearchandinnovationsystem(whichisimportanttoensureresearchconsiderssociety'sneedsandvalues),decision-makersshouldaimtocreategenuineinterestandmotivationforRRIratherthanenforcementbytop-downregulation,especiallyasregardstheprivatesector.
ItalsoprovidedrecommendationsanddirectionsfortheEuropeanCommissionandotheractorsontheimportanceofpublicengagementinnanotechnologyprojects,althoughnoevidencewasfoundoftheuptakeoftheserecommendationsinpractice.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleNanotechnologyMedicineSeriousgames:scenariosBelgium,France,Israel,Italy,Poland,Spain,Swedenin2017-2018WhatisthetechnologicalinnovationTheprojectfocussedonnanotechnologyandtheestablishmentofRRIpracticeswithinthisfield.
Thefocuswasontheapplicationofnanotechnologyin118nanomedicine,nanotextiles,andnano-enabledbraincomputerinterfaces(Kupper&Schuijer2018a;Kupper&Schuijer2018b).
RRIisbasedonthegeneralideathatresearchandinnovationshouldincorporatesocietalneedsandvalues,andrequiresdeliberativeandtransparentprocesseswithafocusonthecollectiveresponsibilityofresearch,innovatorsandsocietalactorsindeveloping(ethically)acceptableandsociallydesirableproducts(Kupper&Schuijer2016).
WhatwasthepurposeofthepublicengagementThepurposeoftheNANO2ALLprojectwastomakethepublicandotherstakeholdersreflectindepthaboutfutureapplicationsofnanotechnologiesandtoensurethatresearchandinnovationismoreresponsibleinthisarea(Bontouxetal.
2020).
ThefocusonRRIwasthroughthetransparentco-productionofknowledgeusinginclusiveandparticipatoryapproaches(Kupper&Schuijer2018a).
HowwasthepublicengagementcarriedoutTheprojectwasledbySociedadePortuguesadeInovaoandfundedbytheEC.
NANO2ALLconsistedofathree-phasedialogueapproach.
Thefirstdialoguephaseconsistedofasetofnationalcitizendialoguesinsixcountries(France,Israel,Italy,Poland,SpainandSweden).
Thesecondphaseconsistedofasetofnationalmultistakeholderdialoguesinthesamesixcountries.
ThethirdphaseconsistedoftheorganizationofafinalEuropeanstakeholderdialogueeventinBrussels.
Thisresultedinatotalof13dialogueevents(Kupper&Schuijer2018b).
32,33Theoutputofthecitizendialoguesfedintothemulti-stakeholderdialogues,andtheoutputofthemulti-stakeholderdialoguesfedintothefinalEuropeanstakeholderdialogue.
Partofthemulti-stakeholderdialogueconsistedofusingtheJRC'sSES,aforesightgamingsystemdevelopedfortheapplicationoffuturesthinkingtopolicy-making(Bontouxetal.
2020).
34TheSESusesagamewhereactorsfromthreestakeholdergroups(businesses,policymakersandcivilsocietyorganisations)trytoachievelong-termobjectiveswhilstbeingobservedbythepublicintwoscenarios(Bontouxetal.
2016).
Participantsreceivealimitedamountofresources,definetheirobjectivestoreachthehorizonofthescenario,takeactions32ThecitizendialogueswereconductedinApril-June,2017andmulti-stakeholderdialoguesinOctober2017-February2018.
33ThesetwodialoguesfedintotheNANO2ALLEuropeandialogueeventthatwasheldonApril9th,2018inBrusselswith29stakeholderswithnanoscientists,policy-makers,industry,civilsocietyorganisations,andintermediariessuchasRRIexperts,mediarepresentatives,socialscientists,ethicists,andstakeholdersfromearlierdialogues.
34Thedialogueparticipantswererecruitedbysciencecentresandincludedthefollowingtypesofactors:policy-makers,civilsocietyorganisations,businessandindustryrepresentatives,nanoscientists,citizendialoguerepresentatives,andactorswhodonothaveaformalstanceonnanotechnologies,suchasajournalistorartist(Kupper&Schuijer2018a).
119inturntoreachtheirdesiredobjective,play'reallifecards'toexplorefurtherinteractions,andgetjudgedandscoredbythe'publicvoice'(Bontouxetal.
2020).
Playersofthegame,whichincludedprivatestakeholdersandmembersofthepublic,tookontheroleofpolicy-maker,civilsocietyorganisation,researcher,andbusiness(Bontouxetal.
2020).
35TheSESteamcreatedagaminglogicintheNANO2ALLprojectaroundhowtechnophilicortechnophobicasocietyis,andhowcentralisedordecentralisedgovernanceis(Bontouxetal.
2020)(CS04-01).
Inthefirsthalfofthegame,variousfuturesofanano-applicationfieldwereexploredusingSESintwoparallelsessions.
Participantsworkedthroughascenariointhreestepsinscenariosthatunfoldedovera15-yeartimeperiod.
Inthesecondhalfofthegame,backcastingexercises36wereperformedinsubgroupsandpairstoidentifyactionsthatconnectdesirablefuturesidentifiedinthefirsthalftothepresent(Kupper&Schuijer2016).
HoweffectivewasthepublicengagementTheeffectivenessofthepublicengagementtechniquewasformallyassessedbyaskingparticipantstoprovidefeedbackandtocompleteashortsurvey(Kupper&Schuijer2018a;Kupper&Schuijer2018b).
Participantsfoundtheexercisetobe'fun'and'engaging'andreportedbothahighlevelofcomforttoexpresstheiropinion,andthattheygainedanawarenessoftheimpactofnanotechnology,whichchangedtheirperspectives(Kupper&Schuijer2018a)(CS04-02).
37Twofacilitatorssaidparticipantsappreciatedtherealismoftheconversations(CS04-01)andthattheygainedanexperienceofthecomplexdynamicsoftheRRIsystem(CS04-02).
Oneintervieweenotedthatthepublicengagementtechniqueopenedupthinkinginthesciencecentresonhowtousepublicengagement.
Forinstance,BialystokUniversityofTechnologyhasusedtheSESforteachingscienceandtechnologystudentsaroundhowtoengagewiththepublic(CS04-02).
TheJRCreportedthattheparticipantsweremoreinterestedinthementalexerciseandreflection,ratherthanaspecificdocumentedoutcomeofthesession(Bontouxetal.
2020).
Manyofthesuggestionsfromstakeholdersinthemulti-stakeholdereventreinforcedcurrentthinkingaboutthemesthatareimportanttothedemocratisationofRRI(Kupper&Schuijer2018b).
Thecitizenandmulti-stakeholderdialoguesfedintotheNANO2ALLEuropeandialoguethatprovideddirectionsandrecommendationsfortheECandotherimportantactorsinthenanotechnologyresearchandinnovationecosystem35TheNANO2ALLgameaskedparticipantstotakeondifferentrolesthantheirowntothinkaboutdifferentfamilyperspectives.
36Backcastingistheprocessofstartingfromadesirablefutureandlookingbacktotodaytoidentifythestrategicstepsthatarenecessarytoachieveaspecificfuture(Bibri2018).
37Participantsgavearatingof4.
6,4.
2,and4.
2respectivelyona5-pointscalefortheseaspects.
120(Kupper&Schuijer2018b)(CS04-01)38.
TherecommendationssuggestedthatcreatinggenuineinterestinRRIamongstresearchersandpolicymakers,ratherthanenforcementbytop-downregulation,wasamorepreferablewaytoopenupthenanotechnologyresearchandinnovationsystem.
TherecommendationsweresenttotheECinApril2018tobedraftedintoaninternalECpolicynoteonnanotechnologygovernance(Kupper&Schuijer2018b).
Thedialoguesalsocontributedtothedevelopmentofpublicroadmapsonthecommercialisationofnanotechnology(Garciaetal.
2019;NANO2ALL,2021b),39,40andtotheNANO2ALLethicspanelfindingsonpublicunderstanding,attitudesandfearsofnanotechresearch(NANO2ALL2019b).
Thedialoguesalsocontributedtoaseriesofflyersfordifferentstakeholders,41webinarsforapplicationfields(NANO2ALL,2021a),conferencesanddebates(NANO2ALL2017;NANO2ALL2018;NANO2ALL2018;NANO2ALL2019).
42Thismighthelpinformnanotechnologypolicyandpracticeinthefuture,althoughnoevidencehasbeenfoundontheuptakeofrecommendationsinpractice.
What'lessons'canbelearntfromthisexampleBasedontheJRC'sexperiencewithusingtheSESfortheNANO2ALLproject,severalkeylessonsemerge(Bontouxetal.
2020;Kupper&Schuijer2018a;Kupper&Schuijer2018b)(CS04-01):TheSEStoolisversatileandcreatesasafeanddynamicspace:Ithasbeenemployedindifferentcontexts,languages,andfordifferentstakeholders.
43Itfacilitatesconstructiveengagementwithstakeholdergroupswhomightnototherwiseengagewithcomplexissues(Bontouxetal.
2020)(CS04-01).
Thetoolhasbeenusedtoexplorediversetopicsandissues(CS04-01).
Theabilitytochangethescenariosinonesettingalsoexpandsthesepossibilities.
Thescaleofthequestionaskedwillalsodeterminethescaleofthediscussionofthequestioninthescenario(Bontouxetal.
2020).
Ensurethatdifferentstakeholderperspectivesarerepresentedinthegame:Participantstakeupunknownrolesandgainanemotionalconnectiontotherolein38Thisincludednanoscientists,policy-makers,industry,civilsocietyorganizations,mediarepresentatives,RRIexperts,socialscientistsandethicists.
39ThepublicdialoguesoutlineactionstobeundertakenbyEUandnationaldecision-makerstofosterRRI.
Italsofedintoworkinggroupsonnanotechnology(Garciaetal.
2019).
40Includingsafety,communication,industrialisation,networking,regulation,researchandtechnology,skillsandeducation,standardisation,technologytransferandinnovationfinancing,criticalrawmaterialsandsocietalaspects.
41Includingpolicy,industry,media,CSOs,generalpublicandacademia.
42SuchasthesuchastheInternationalCongress"NanotechnologyinEverydayLife"andtheESCITEannualconference,theB.
Debate,andakeynotepresentationonRRIduringanInfoSessionorganisedbyINCOBRAatPUCRSinPortoAlegre,Brazil.
43Thegamehasbeenusedwithpeoplefromdiversebackgroundsincludingage(12-67),gender,geographicalandprofessionalbackground(Bontouxetal.
2020).
121thegame.
Ithelpsparticipantsthinkstrategicallyaboutthelong-term.
Itis,however,importanttoensurethatdifferentstakeholdervoicesareconsideredinthegame,suchasindustryandthepublic(Kupper&Schuijer2018a;Kupper&Schuijer2018b)(CS04-02).
Itisalsoimportanttoconsiderhowtoincludethepublicvoiceinthegame,astherearemanypublicsratherthan'onepublic'thatcanrepresentdifferentstakeholderperspectives(CS04-02).
Theseroleshavetobesufficientlyindependenttofacilitateindependentdecision-making(Bontouxetal.
2020).
Thesuccessofthegamedependsonthemoderationqualityandcreativity(Bontoux,Sweeneyetal.
2020):althoughoneintervieweesaidthatitiseasytotrainpeopletobegamemasters(CS04-01).
Makethegameshorterandensurethatthereisareflectionsessionattheend:Thetimespanofagamecanbeaninvestmentfordialogueparticipantsandachallengeforfacilitatorswhenconveningthesession(Kupper&Schuijer2018a;Kupper&Schuijer2018b).
Thiscanhaveanimpactontheidentificationofactionstoenhancesocialinteractionsaroundparticulartechnologiesandcanalsolimitthetimespentonreflection.
Onefacilitatorfeltthatthesereflectionsareanimportantpartofthegamethatshouldbeincluded(CS04-02).
Somechallengescanarisewhenconductingthegameindifferentcontexts:Challengesincludetheneedtotrainmultiplemoderators,translatingthegamescenariosandculturaldifferences(Bontouxetal.
2020)(CS04-02).
122A.
5.
Casestudy5(CS05):DeterminingpublicperceptionontheuseofvirtualrealityinhealthcarethroughsociallisteningCasestudytitleDeterminingpublicperceptionontheuseofvirtualrealityinhealthcarethrough'sociallistening'ConcisesummaryofthecasestudyVRprovidesanimmersiveenvironmentthatenablesuserstohaveanalteredexperienceofreality.
Itisincreasinglyusedacrossthehealthcaresectortosupportpatientsduringtreatment.
Yet,thepublicperceptiononitsuseremainsunder-studied.
Here,theresearchteamusedamethodcalled'sociallistening'inordertocollectandanalyseFacebookcommentsinresponsetoavideoontheuseofVRinhealthcare.
AnalysisofthesecommentsshowedthatthepublicwasgenerallyexcitedabouttheuseofVRwithinahealthcaresetting,butalsoidentifiedseveralpotentialconcernsofitsuse.
Theseconcernshighlightpotentialfuturebarriersofthiskindofresearchandprovideinsightintohowthisareashouldprogress.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleVRHealthcareSociallisteningUnitedStates,2016WhatisthetechnologicalinnovationThispublicengagementexercisefocusedontheuseofVRtechnologyinhealthcare.
VRprovidesanimmersiveenvironmentthatenablesuserstohavealteredexperiencesofreality(Lietal.
2011;Malloy&Milling2010).
Itisbeingusedincreasinglyacrossthehealthcaresectortohelppatientscontrolpain,treatanxietydisorders,supportphysicalrehabilitationanddistractpatientsduringtreatment(Garrettetal.
2014;Hoffmanetal.
2000;Hoffmanetal.
2001;Lietal.
2011;Malloy&Milling2010;Morrisetal.
2010).
AreviewofrandomisedcontroltrialsdemonstratedthatVRisgenerallyeffectiveandwelltoleratedbypatientsacrossarangeofclinicalsettings(Dascaletal.
2017).
Whatwasthepurposeofthepublicengagement123ThepurposeofthispublicengagementtechniquewastobetterunderstandpublicopinionabouttheuseofVRinhealthcare(Kelleretal.
2017).
DespiteincreasingawarenessofVRanditspotentialbenefits,thereremainsalackofresearchonthefactorsdeterminingpublicacceptanceofVRinclinicalsettings.
Tobetterexplorethis,theresearchteamsetouttoexaminepublicperceptionofVRinhealthcareincludingtheunderstandingofVRtechnologyitself,concernsarounditsuseandpossibleareasforitsuseandfutureapplications(Kelleretal.
2017).
Ultimately,thisaimedtobuildapictureofthepotentialfacilitatorsandbarrierstoimplementationofVRinhealthcareandunderstandhowthepublicwouldperceivethisVRtechnologyiftheyweretofinditwithinahealthcaresetting(CS05-01).
HowwasthepublicengagementcarriedoutPublicengagementwascarriedoutusingamethodcalled'sociallistening'(CS05-01).
Thismethodallowsresearcherstoexploreunfilteredviewsontopics,suchasthosediscussedonsocialmediaandonlineforums(Stewart&Arnold2018).
Theresearchteam,basedattheCedars-SinaiMedicalCenter,used'sociallistening'togatherpublicperceptiononavideothathadbeenpostedontothesocialmediasiteFacebook(CS05-01).
Thevideo,producedbyNBCNews,depictedtheuseofVRinhealthcare,andwasrepackagedbyanewsaggregatorthatthenpostedthevideoontoFacebook(Kelleretal.
2017).
ThisvideosubsequentlywentviralandthereforeattractedalotofonlineattentionandpromptedcommentsfromFacebookusers(CS05-01).
TheresearchteamsawthisasanopportunitytogatherinvaluableinformationonhowthepublicfeltaboutVR(CS05-01).
Therefore,afterthevideowaspostedonline,theresearchteamdownloadedthecommentsfromthevideoanduploadedthemtotheircomputer-assistedsoftware,whichenabledsubsequenttext-miningandcontentanalysisofthecomments(CS05-01).
Postswerecategorisedintosentimentcategories(positive,negative,neutral),andintomajororminorthemes.
Majorthemesincludedhigh-levelaspects,suchasinterestinVRtechnology;desireforpersonaluse;aswellashealthcareusesincludingpainorlackofmobility.
Minorthemesofferedamoregranularcategorisationonthemajorthemes,includingdefiningpainintofurthercategoriessuchasdentistry,chronicpain,burns(Kelleretal.
2017).
Somecommentscouldnotbeanalysedasitwasimpossibletodeterminewhichhigh-ordercategorytheyfitinto,andthereforewereignored.
Inaddition,theresearchteamdeterminedtheself-identifiedgenderfromtheFacebookuserspageandperformedsentimentanalysisofthelanguagetoanalysewhetherperceptionofVRdifferedwithgender(Kelleretal.
2017).
HoweffectivewasthepublicengagementSociallisteningenabledthecollectionofa'snapshot'ofdataonpublicreactiontotheuseofVRinhealthcare(CS05-01).
Theresearchteamwereabletoanalyse1614commentsfromtheFacebookvideointotal,whichweredownloaded124approximatelytwoweeksaftertheinitialvideohadbeenposted(CS05-01).
Fromtheiranalysis,theresearchteamidentified1197(74.
16%)asexpressingapositiveperception,251(15.
55%)asexpressinganegativeperceptionorconcern,and560(34.
70%)expressingcommentscategorisedasneutral(Kelleretal.
2017).
TheengagementhighlightedspecificconcernsandpotentialbarrierstotheuseofVRinhealthcare,whichincludedthethreatthatVRmayposetopatienthealth(e.
g.
concernsaroundmotionsicknessorradiation),aswellasmoregeneralconcernsaroundthecostofsuchtechnologyaswellasincreasingsocietalrelianceontechnology(Kelleretal.
2017).
ThestudydemonstratedthereadinessofthepublictoengagewithVR(CS05-01);andthetechniqueofferedcrowd-sourcedavenuesforfuturestudyinthisarea(Kelleretal.
2017).
TheseincludedtheuseofVRtechnologyforpainreduction;tocombatalackofmobility,andtopositivelyinfluencementalhealthtreatment(Kelleretal.
2017).
ThestudyprovidedinsightintothepotentialbarriersregardingtheadoptionofVRwhichmayberelevantforfuturework(CS05-01).
Inthisconnection,participantsraisedconcernsaroundthecostofsuchtreatment,andthepossibilityofcross-infectionandcontaminationfromVRequipment(CS05-01).
ThisinformedhowtheresearchteamundertookfurtherworkonVRwithinaclinicalsetting.
Forexample,inastudyontheuseofVRforpatientswithchronicbackpain,theresearchteamusedthispriorknowledgeofpatientconcernstoalleviatepatientfearspriortotheirengagementwithVR(CS05-01).
Theengagementtechniqueenabledviewstobecapturedfromparticipantsacrossademographicandgeographicspectrum(Kelleretal.
2017).
Therewere,however,somelimitationstothisstudyincludingthatthevideowaspostedinEnglish,thuslimitingparticipationfromnon-Englishspeakers,aswellasthosethathadaFacebookaccount(Kelleretal.
2017).
What'lessons'canbelearntfromthisexampleSociallisteningcanbeapotentiallycost-effectivewaytoengagewithalargeparticipantsample:thistechniqueenabledtheresearchteamtoanalysealargenumberofsocialmediacommentsinresponsetothevideo(CS05-01).
Futurestudiesmaywishtousesociallisteningtorapidlycaptureabroad'snapshot'ofpublicopinion.
Sociallisteningcanprovideawaytocaptureparticipantviewsfromgroupsthatmaynotgenerallyengage:becauseviewswerecapturedfromexistingcomments,thisenabledunsolicitedviewstobecapturedthatwerenotinfluencedbythepresenceofaresearcher,aswellasenablingengagementwithgroupswho125maynothaveparticipatedinaformalisedengagementtechniquesuchasasurvey(Kelleretal.
2017).
Sociallisteningmayhavelimitedutilityiftherationalebehindparticipant'sviewsneedstobeunderstood:althoughtheengagementtechniqueprovidedvaluabledata,therewaslimitedparticipantinformationthatcouldbecapturedusingthismethod,reducingtheextenttowhichcontributingsocialfactorscouldbeanalysed(CS05-01).
Futurestudiesthatwishtousesociallisteningcouldcombinethistechniquewithfollow-upfocussedparticipantdiscussionwithasubsetoftheparticipants(CS05-01).
Thiswouldenabletherapidanalysisofalargesampleofcommentswhilstallowingforfollow-updiscussiononspecificparticipantviews.
126A.
6.
Casestudy6(CS06):UsingthevTaiwanplatformtocarryoutapublicdebateontheregulationofUberinTaiwanCasestudytitleUsingthevTaiwanplatformtocarryoutapublicdebateontheregulationofUberinTaiwanConcisesummaryofthecasestudyvTaiwanisadeliberativedigitalplatform.
Itfacilitatesconstructivedebateandhelpsidentifyareasofconsensusonspecificissuesamongstcitizens,stakeholders,andgovernment.
Theaimsaretoincreaseengagement,scrutinyandtransparencyindecision-making.
Inthisexample,vTaiwanwasusedintheregulationoftheUberXservicefromtheride-hailingappUber.
vTaiwanusesadedicatedAI-facilitatedsocialmediatool,Pol.
is,whichallowsuserstodraftwaysinwhichaproblemmaybeaddressed,aswellastorespondtootherusers'solutionsbyagreeingordisagreeingwiththem.
TheapplicationofthevTaiwanprocesstotheregulationofUberXdemonstratedimpactsthroughitsabilitytodiffuseapotentialdisputebetweenUberdriversandtraditionaltaxidriversaroundwhetherUberwasanapportaxiserviceinTaiwan.
ByengagingcitizensandstakeholdersitreachedconstructiveoutputsregardinglegalrequirementsforbothtraditionaltaxisandUbercars,andtheseoutputssubsequentlybecamelaw.
TheeffectivenessofthevTaiwanprocessinthiscasestudyhasbeenhighlightedinanumberofpublishedarticles.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleRide-hailing/car-poolingappTransportDiscussionanddeliberationviasocialmedia,andthenface-to-facemeetingstodiscussspecificproposals.
Taiwan,July-August2015WhatisthetechnologicalinnovationUberisaride-hailingcompany.
Uberprovidesamobileappthatcanbeusedtosubmitatriprequest,whichissenttoUberdriversinthevicinityoftheuserandalertsthemoftheuser'slocation.
Adriverwithcapacitywillthenacceptthisrequest,collecttheuser,anddrivethemtothedestinationrequestedthroughtheapp.
UberXisoneoftheservicesprovidedbyUberInc.
UberXcarscan127accommodatepartiesofupto4people,withonepersoninthefrontseatand3intheback(Uber2021).
WhatwasthepurposeofthepublicengagementWhenUberXstartedinTaiwanin2014,itwaspopularbutalsocausedproblemswithtraditionaltaxidrivers.
Uberregisteredasatechnologycompanyratherthanatransportcompanyonthebasisthatitisanappratherthanataxiservice.
ThisgaveUberseveralcompetitiveadvantagesoverexistingtaxidriversinareassuchasinsurance,faresandtaxes.
TheTaiwaneseMinistryofTransportandCommunications,however,stillconsideredUberatransportcompanyandsaidithadtoobeytaxilaws(Hsiaoetal.
2018).
WhenUberrefused,itfacedpenaltiesundertheHighwayAct(Hsiaoetal.
2018).
vTaiwanfacilitatedatransparentdebatebetweencitizensandstakeholders,suchastheAssociationofTaxiDriversinTaipei,TaiwanTaxiandUber(Tang2016)overaperiodofafewmonthsaboutwhatconstitutesfaircompetitionregulation(King2019).
HowwasthepublicengagementcarriedoutThevTaiwanengagementprocessiscentredaroundadedicatedAI-facilitatedsocialmediatool,Pol.
is,whichallowsuserstodraftwaysinwhichaproblemmaybeaddressed,aswellastorespondtootherpeople'ssolutionsbyagreeingordisagreeingwiththem(CS06-02).
Bydrawingamapthatclustersusersbasedontheiropinionsinrealtime,Pol.
isisableshowaxesofconsensusanddisagreement.
vTaiwan'skeytosuccessisinfocusingonareasofconsensusamongstcitizensandstakeholders,andgamifyingtheprocessoffindingcommonalities(TheAlternative2019).
Theengagementprocesswascarriedoutinfourstages:proposal,opinion,reflectionandlegislation(Simonetal.
2017;CS06-01).
ThedecisiontotransitionfromonestageinthevTaiwanprocesstothenextisnotpre-determined;itisdecidedthroughconsensusbythevTaiwancommunity.
vTaiwanarguethatthisfacilitatesmeaningfuldeliberationwhenallstakeholdersarepreparedtocollaborateanditerateonsolutions.
TheProposalStagetooktwoweeks.
TheMinistryofTransportationandCommunications,MinistryofEconomicAffairs,andtheMinistryofFinancewantedtoregulateUberX'soperationsinTaiwan.
WithconsensusfromthevTaiwancommunity,thistopicwasselectedforopenconsultation.
TheOpinionStagetookamonth,from15Julyto15August2015.
Itinvolvedgatheringrelevantfactsandresearch,simplifyingcomplexlegalesetodefinethescopeofthetopicinplainlanguage,draftingdefaultstatementstobegintheengagementprocess,andthenallowingvTaiwanuserstovoteonstatements128throughPol.
is.
Usershadtheoptionto'agree','disagree',or'pass'whengivenastatement,orhadtheoptiontocontributetheirownideawithalimitof140characters.
Topreventtrollingandtofocusonconsensus,userscouldnotdirectlyreplytootherusers'ideas(Hsiaoetal.
2018).
ThePol.
isURLlinkwasdistributedthroughFacebookadvertsandthroughstakeholdernetworkssothataffectedgroupssuchasUberdriversandtaxidriverscouldengageinthisprocess.
OneofthekeyinnovationsofthePol.
isplatformisthevisualexpressionofusers'viewsthroughreal-timemapsthatclusteropinionsintogroups.
Intheopinionstage,thePol.
isalgorithminitiallyidentifiedfourdistinctgroups:taxidrivers,UberXdrivers,taxi-passengersandUberXpassengers.
Theviewsofthesegroupscoalescedintoanti-Uberandpro-Ubergroups,clearlyseenintwostatementsthatgeneratedconsensus(CS06-02).
Athirdstatementthatgenerated95%consensusamongstallparticipantswas:'Thegovernmentshouldleveragethisopportunitytochallengethetaxiindustrytoimprovetheirmanagementandqualitycontrolsystem,sothatdriversandriderswouldenjoythesamequalityserviceasUberX's(Tang2016).
BytheendoftheOpinionStage,1,737participantshadtakenpart,196opinionshadbeenenteredintoPol.
isand47,539voteswerecast(Simonetal.
2017).
InorderforthePol.
isplatformtogivevisualexpressiontouser'sviews,theremustbeatleast20statementsonthespecificissuebeingdebated.
TheReflectionStagewasatwo-hourin-personconsultationwithacademicsandindustryexperts,citizenswhowereactiveusersfromthePol.
isplatformstageoftheconsultation,aswellasrepresentativesfromtheAssociationofTaxiDriversinTaipei,TaiwanTaxi,Uberandrelevantgovernmentministries.
Thisconsultationtookplaceon27August2015andwaslive-streamed.
Atotalof1,845peopleparticipatedthroughthelivecastoftheevent(Tang2016).
TheoutcomeofthereflectionstagewasanagreementthatUberneededtocomplywithlegalrequirementsofinsuranceanddrivertraining,whilsttaxiservicesagreedtobroadenservicesinresponsetothemarketdemandforride-hailingapps(Hsiaoetal.
2018).
TheLegislationStagetookplaceon23May2016,whentheconsensusreachedthroughthePol.
isplatformandReflectionStagewasenactedthroughanamendmentoftheregulationonautomobiletransportationmanagement.
HoweffectivewasthepublicengagementThevTaiwanengagementprocesstocarryouttheregulationofUberwaseffectiveinthreeways.
Firstly,thisprocessofengagementanddeliberationreducedtensionsbetweenUberdriversandtraditionaltaxidrivers(King2019);stakeholdersdisplayedastrongwillingnesstoengageintheprocess(Tang2016).
129Secondly,therewereseveralconstructiveproposalsthatemergedfromthisprocessrelatingtocompetitioninthetaxiindustry,whichsubsequentlybecamelaw.
Inparticular,taxisnolongerneededtobepaintedyellow;carpoolingtaxiswerefreetooperateonthebasisthattheydonotundercutthefaresofcurrenttaxis;carpoolingappsmustdisplaycaranddriveridentification,estimatedfareandcustomerrating;andper-ridetaxationmustbereportedtotheMinistryofFinance(King2019).
Thirdly,thevTaiwanprocessmeantthatUberchangeditsbusinessmodelinTaiwantobecomealegalride-hailingcompany(Hsiaoetal.
2018).
What'lessons'canbelearntfromthisexampleGamificationcanhelpengageusers:Bygettingusersto'agree','disagree'or'pass'onrandomlyselectedproposalsfromothervTaiwanusers,thePol.
isplatformintroducesanelementofgamification.
Theultimateaimistocomeupwiththeproposalthatcangeneratethemost'likes'fromadiversegroupofstakeholderswhohavedifferentopinionsonthesubjectbeingdebated(CS06-02).
Taiwan'sDigitalMinister,AudreyTang,saidthatthegoalofvTaiwanistogamifythesystemsothatthereisinstantgratificationandatangiblerewardbycontributinganopinion(Storey2018).
Focusingonconsensusisconstructive:vTaiwanfocusesonconstructiveengagementbyremovingpotentialobstaclestoreachingaconsensus.
Inordertopreventtrollingandtoencourageconstructivedialogue,userscannotdirectlyrespondtootherusers'proposals–theycanonlyvoteonthem(CS06-02).
Real-timedataanalysiscanbeapowerfultooltounderstandengagement:TheabilityofthePol.
isplatformtogivevisualexpressiontotheviewsbeingraisedthroughtheprocessinreal-timemeansthatareasofconsensusanddisagreementareeasiertounderstand.
Moreover,theuseofopen-sourcesoftware,inparticularthePol.
isplatform,enablesvTaiwantosupportparticipationonalargescale;theplatformcansupportupto100,000comments(CS06-01).
Amulti-stageprocesscanhelpensureabroadrangeofopinionsareconsidered:ThevTaiwanprocessensuresthatcitizens'voicesareheardintheOpinionStage,whilstalsobringingcitizens,stakeholders,andexpertstogethertofocusonworkingoutdetailsintheReflectionStage.
AlthoughtheLegislationStagedependsonthewillofthegovernmenttoimplementfindings,reachingaconsensusbetweencitizensandstakeholdersthroughthisprocessdemonstratesthattheoutputsarelikelytobedesirableandsuitabletoallpartiesinvolved(Hsiaoetal.
2018).
130A.
7.
Casestudy7(CS07):EngagingexpertandcitizenperspectivesonAIusingaworkshopandonlineplatformCasestudytitleEngagingexpertandcitizenperspectivesonAIConcisesummaryofthecasestudyThiscasestudyexploresatwo-stagepublicengagementprocessonAI,conductedundertheauspicesoftheHumanBrainProject(HBP)andoverseenbytheDanishBoardofTechnology(DBT).
Inthefirststageofthepublicengagement,anexpertworkshopwasorganised.
Theworkshopbroughttogethercross-disciplinaryexpertstoconsidertheethical,economic,legalpoliticalandsocialimpactsofAIusinga'360'approach.
Inthesecondstage,findingsfromtheexpertworkshopwereusedtoinformanEU-wide'citizenconsultation'onAIusinganonlineplatform.
Thecasestudyhighlightsthepotentialforlinkedapproachestopublicengagementdrawingonbothexpertstakeholdersandthebroaderpublic,aswellastheutilityofonlineplatformsasmechanismsforengagingpublicperspectives.
Theextent,however,towhichtheserecommendationswillbetakenintoaccountbytheECisnotclearasofDecember2020.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleArtificialIntelligenceDanishBoardofTechnologyExpertworkshopandcitizenconsultationEU-wide,March2019–September2020WhatisthetechnologicalinnovationTheAI360WorkshopandsubsequentcitizenengagementrespondedtotheincreasingproliferationofAIthroughoutEuropeansociety.
Inbothcases,AIwasconsideredinthebroadestpossibleterms,withnofixeddefinitionapplied.
Theadoptionofabroad,flexibledefinitionofAIwasintendedtofacilitateacomprehensive,'360-degreeevaluation'ofAI,ratherthanfocussingonmorespecificareassuchasthefutureofwork(CS07-01).
WhatwasthepurposeofthepublicengagementAgainstthebackdropoftheexpandingapplicationsofAI,theAI360WorkshopandcitizenconsultationsoughttoengageEuropeanstakeholdersinconversation131aboutthebenefits,opportunities,risksandchallengessuchdevelopmentsmightbring(Bitsch&Bang2020;Bitschetal.
2020).
AsactivitiesorganisedundertheauspicesoftheHBP,aflagshipEU-fundedresearchprojectexploringpossibilitiesforsimulatingthehumanbrain,bothengagementswereintendedtoprovidethebasisfor'societalinput'intothatproject,includingrecommendationsforhowtheHBPcouldbeproactiveinaddressingpublicperspectivesonAI(CS07-01).
InthecaseoftheAI360Workshop,thespecificpurposewastoconveneexpertdiscussioninordertoidentifythekeyopportunities,challengesandpotentialsolutionsrelatingtotheexpandingapplicationofAI(Bitsch&Bang2020;Bitschetal.
2020).
ThepurposeofthecitizenconsultationwastounderstandtheviewsofAIheldbythepublicatlarge(Bitsch&Bang2020;Bitschetal.
2020).
ByseekinggeneralpublicperspectivesonAI,thecitizenconsultationsoughtto'democraticallyqualify'debatesonAItypicallydominatedbyexpertopinion(Bitsch&Bang2020;Bitschetal.
2020).
Anotherstatedpurposeoftheconsultationwastoestablishgreaterpublicdialoguewithaviewtobuilding'publictrust'inAI-relatedtechnologies(Bitsch&Bang2020;Bitschetal.
2020).
44HowwasthepublicengagementcarriedoutInseekingtoidentifythekey'hotspots'relatingtoAI,theWorkshopemployeda'360approach'(Bitsch&Bang2020;Bitschetal.
2020).
Drawinginspirationfrommethodsusedbya2013EU-fundedprojectonDecisionSupportonSecurityInvestment(DESSI),this360approachinvolvedthesystematicandstructuredanalysisofAIacrossfive'dimensions',therebyfacilitatingaholisticviewofthetechnologyanditspotentialimpact(Bitsch&Bang2020;Bitschetal.
2020).
ThefivedimensionsusedfortheWorkshopwere'politicalimplications','rightsandethics','legalframework','economy'and'societalimplications'(Bitsch&Bang2020;Bitschetal.
2020).
45InthefirststageoftheWorkshop,expertsweredividedalongthefivedimensionsaccordingtotheirsubjectmatterexpertise,therebyenablinganassessmentofeachdimensionbyexpertswithstrongconnectionstothoseareas(Bitsch&Bang2020;Bitschetal.
2020).
Inthesecondstage,participantswerereorganisedtocreatemixed,multidisciplinarypanelsforeachdimension(Bitsch&Bang2020;Bitschetal.
2020).
46Inthethirdstage,mixedpanelsconsideredpotentialsolutionstotheissuesandchallengesidentifiedduring44Theabsenceofsuchadialogue,accordingtotheorganizers,wouldpotentiallycontributetoapublic'tech-lashagainstAI',therebycurtailingthepositivepotentialimpactsofthistechnologycouldhaveforsociety(Bitschetal.
2020).
45Foreachdimension,asetofmoredetailedcriteriaweredevelopedforparticipants'considerationBitschetal.
2020).
46Withinthesepanels,expertsratedthetechnologyagainstthedetaileddimensioncriteriausingascaleof1-5.
Ratingwasperformedbothbeforeandafterthepaneldiscussion,withtheresultspresentedinreal-timeoncentralscreen(Bitschetal.
2020).
132thefirsttwostages(Bitsch&Bang2020;Bitschetal.
2020).
47Atotalof28expertsparticipatedintheWorkshop.
Toperformthecitizenconsultation,theDBTusedanadaptedformofitsonlinedialoguetoolGlobalSay.
48Citizenswereinvitedtovolunteertohostmicrocitizensummitsofbetween5-8participantsfacilitatedbytheonlineplatformEngageSuite.
Theplatformguidedparticipantsthroughaseriesofdebates,includingpresentations,shortvideosandopportunitiesfordeliberation.
ThecontentoftheplatformdrewdirectlyonthechallengesandpotentialsolutionsidentifiedbytheAI360Workshop(Bitsch&Bang2020;Bitschetal.
2020).
Attheendofeachroundofdebate,participantswereaskedtoprovideanswerstoasetofquestions,withmultiplechoiceanswerstofacilitatequantitativeanalysis(Bitsch&Bang2020;Bitschetal.
2020).
Toengagecitizensintheconsultation,theDBTadoptedatwo-prongedapproach.
Firstly,in13Europeancountries,alocalimplementingpartneragreedtorecruit10localsummithosts(Bitsch&Bang2020;Bitschetal.
2020).
Secondly,bypublicisingtheconsultationaswidelyaspossibleonlineandthroughsocialmedia,theDBTattemptedtoencouragebroaderparticipationfromaself-selectingaudienceinterestedinthetopicofAIanditsfutureimpact(Bitsch&Bang2020;Bitschetal.
2020).
Overall,theconsultationsaw157summitstakeplaceacross13countries,reachingatotalof928participants(Bitsch&Bang2020;Bitschetal.
2020).
HoweffectivewasthepublicengagementAsnotedabove,thoughformingpartofalinked,two-stepprocess,theAI360Workshopandthecitizenconsultationhaddistinctaims.
InthecaseoftheWorkshop,theaimscentredaroundengagingexpertsindebatearoundkeychallengesandsolutionsrelatingtoAI;inthecaseofthecitizenconsultation,theaimsfocussedonqualifyingtheseexpertinsightsagainstbroaderpublicviews.
Whilebothformsofengagementwerebroadlyeffectiveinmeetingtheiraims,bothalsodemonstratedkeyshortcomings.
TheAI360Workshopdemonstratedthevalueofanall-round,multidisciplinaryapproachasamethodofexpertstakeholderengagement.
Accordingtointerviewees,however,italsolackedsufficientinvolvementfromtechnical/engineeringexperts(mostoftheparticipantsbeingfromlaw,ethics,philosophy,politicalscienceandeconomics)(CS07-02,CS07-03).
49ThelackofdeeptechnicalknowledgewithintheWorkshopmeantthatsomediscussionswere'surrealistic'ratherthanfocusedonconcretepossibilities(CS07-47Acrossthesethreestageswerenumerouspresentations,discussionsandfeedbackopportunities(Bitschetal.
2020).
48TheadaptedversionoftheGlobalSayplatformwasreferredtoasEuropeSay.
49Accordingtooneinterviewee,thiswaspossiblybecausetheorganisersfocussedmoreonengaging'socialscience'expertiseonAI,withtheresultthattechnicalexpertswerecomparativelyunder-represented(thoughnotwhollyabsent)attheworkshop(CS07-03).
13303).
Onthecitizenconsultationside,theconsultationdemonstratedtheeffectivenessofonlineplatformsasmechanismsforobtainingpublicviews(CS07-01).
Accordingtooneinterviewee,thetotalnumberofparticipantsintheconsultation(928)wasmorethantheanticipatednumberatthestartoftheprocess(CS07-01).
Atthesametime,theconsultationalsofaceddifficultiesgarneringmorewidespreadpublicengagementthrough'viral'dissemination,withmostoftheconsultation'sparticipantsengagedthroughin-countrypartners(CS07-01).
Together,theAI360Workshopandcitizenconsultationshaveledtotwokeyoutputs.
First,inlinewiththeinitialaimsoftheprocess,thetwoengagementshaveprovidedthebasisforan'Opinion'submittedtotheHBP(Aicardietal.
2020)(CS07-01).
TheOpinionseekstoconnectissuesraisedbytheengagementstotheworkoftheHBPandiscurrentlybeingconsideredaspartofinternaldiscussionswithintheproject(CS07-01).
Second,thetwoengagementshavealsobeenusedtoinformaDBTcommentaryonaEuropeanCommission(EC)WhitePaper,whichprovidesthebasisfornewEUguidelinesonAI(Bitsch2020).
TheDBTcommentaryincludedsevenkeyrecommendationsfortheEUguidelinesbaseddirectlyoninputsgainedbythe360Workshopandthecitizenconsultation(Bitsch2020).
50Accordingtooneinterviewee,however,itisnotcleartowhatextenttheserecommendationswillbetakenintoaccountbytheCommission(CS07-01).
WhiletheDBT'scitizenengagementmayhavecontributedtobuildingpublictrustinAIinasmallway,accordingtooneinterviewee,societal-levelpublictrustinAIwillrequirewidespreadproliferationofsuchengagementtechniques,ideallyinvolvingthose(suchastechnologydevelopersandpolicymakers)withthepowerandinfluencetoshapethecourseofthetechnology'sfuturedevelopment(CS07-01).
What'lessons'canbelearntfromthisexampleLinked,two-stepapproachescanbeusedtoengagebothexpertopinionandbroaderpublicviews:InthecontextoftheHBP,theDBTestablishedatwo-steppublicengagementprocessfocusingfirstonexpertconsultationandsubsequently50Thesevenrecommendationswereasfollows:(1)undertakeworktouncoverandaddressthefullrangeofpotentiallyabusiveusesofAI,includingsocietal,political,security,intelligence,andmilitarydomains;(2)makeobligatoryregulationforallAI;(3)reconsiderthedistinctionbetweenhighandlowriskAI;(4)encouragepublicauthoritiestocarryoutacomprehensiveinvestigationoftheirlocalchallengesbeforeimplementingAIsolutions;(5)buildtrustbysupplementingbalancedinformationwithinvolvingcitizensindeliberation,agendasetting,prioritizationanddecisionmaking;(6)encouragenationalandEuropeandeliberationandactionontheroleofdigitalspacesofinteraction,newsanddebatecentraltothefunctioningofourdemocracies,humanrightsandwell-beingofEuropeancitizens;and(7)humanoversightshouldbecomplimentedbyhumaninsightandexplainability(Bitsch&Bang2020).
134oncitizenconsultationsto'democraticallyqualify'expertviews(CS07-01,CS07-02,CS07-03).
Whenengagingexpertsandcitizensindiscussiononthepotentialfutureimpactoftechnologies,anall-round(360)approachcanbebeneficial:Boththe360WorkshopandthecitizenconsultationconsideredtheimplicationsofAIacrossmultipledifferentdimensions,therebypermittingamoreholisticunderstandingofopportunities,challengesandpotentialsolutionsthaniffocussingononearea(CS07-01).
Onlineplatformspresentaviabletoolforconductingcitizenconsultationstoengagepublicviews:Atthesametime,however,generatingtrulywidespreadengagementwithsuchplatformscanbeachallenge(Bitsch2020)(CS07-01).
Itisalsoimportanttorecognisethatthoseengaginginonlineplatformsmayrepresentafairlynarrowself-selectingsegmentofthepopulationatlarge.
Onepublicengagementprocessisnotsufficienttobuildpublictrustinatechnologyatasocietallevel:Whilesinglecitizenengagementexercisesmaycontributetobuildingpublictrustinatechnology,tobuildpublictrustatasocietallevel,thereisaneedforsystematicpublicengagementprocessesinvolvingactorsandorganisationsacrossatechnology'secosystem(CS07-01).
135A.
8.
Casestudy8(CS08):Engagingthepubliconfacialanalysisandautomateddecision-makingthroughtheuseofBioMetricMirror–aninteractiveapplicationCasestudytitleEngagingthepubliconfacialanalysisandautomateddecision-makingthroughtheuseofBioMetricMirror–aninteractiveapplication.
ConcisesummaryofthecasestudyFacialanalysisapplicationsareincreasinglybeingusedtoinformdecision-makingprocesses,andconcernshavebeenraisedregardingthetransparencyandethicsofthesetechnologies.
Here,theresearchteamdevelopedaninteractiveapplication'BioMirror'thatperformedfacialanalysisonthepublic,andsubsequentlyprovidedinferencesabouttheparticipants'demographicandpsychometriccharacteristics.
Throughthisapplication,thepublicwereabletoengagewiththistechnologyinanexperientialway,whichprompteddiscussionandreflectiononnotonlythepotentialforthistypeoftechnology,butalsoontheethicalchallengesthatmustbeconsideredifthisweretobecomemoreprevalentinsociety.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleFacialrecognitionandautomateddecision-makingMultipleSectors(Surveillance)InteractivefacialanalysisapplicationAustralia,2018-19WhatisthetechnologicalinnovationThispublicengagementexercisefocusedonfacialanalysisandautomateddecision-makingprocesses.
Facialanalysisapplicationsareincreasinglybeingusedtoinformdecision-makingprocesseswhichmayinfluenceanindividual'saccesstohealthcare,realestate,financialservicesandthejudicialsystem(Helbingetal.
2019).
Thistechnologyenablespersonalcharacteristics,suchasgender,age,raceoremotionalstate,tobeinferredfromphotographsorvideos(McStay2016;Mülleretal.
2009).
Whatwasthepurposeofthepublicengagement136Toexplorepublicopinionontheethicsoffacialanalysisandautomateddecision-making,researcherscreatedBioMetricMirror;aninteractiveapplicationwhichenabledmembersofthepublictohavetheirfacephotographed,andanalysedbyamachine-learninginstallationwhichprovidedinferencesontheirdemographicandpsychometriccharacteristics(Woutersetal.
2019).
Concernshavebeenraisedaroundtheuseoffacialanalysistechnologywithinautomateddecision-making,andalackoftransparencyontheapplicationofthesetechnologieshaslimitedtheopportunityforpublicdiscussionandcritiqueoftheseprocesses(Abduletal.
2018;Dattaetal.
2016;Ekstrandetal.
2018).
Thisapplicationaimedtoexplorepublicopinionofthistechnology,probingunderstandingaroundhowthesetechnologiesworkinpractice,aswellasethicalconcernsaroundtheiruse(Woutersetal.
2019).
Thetechnologywasbuiltusingpre-existingfacialanalysisinformationfromthe10kUSAdultFacesDatabase–adatabasecontainingthousandsofphotographsalongwithsubjectiveratingsonperceiveddemographic,psychologicalandsocialattributescrowd-sourcedfromthepublic(Bainbridgeetal.
2013).
Theresearchteamwantedtoenablethepublictoexperiencefacialanalysistechnologyinanexperientialway(CS08-01,CS08-02).
Furthermore,thestudyaimedtoprobeethicalquestionsaroundtheuseofthistypeoftechnology.
Thealgorithmsusedwithinthefacialanalysisinstallationwereinherentlysub-optimal,somethingwhichtheresearchteamusedtodemonstratethepotentiallimitationsofautomateddecision-makingwhenattemptingtoassigndemographicorpsychometriccharacteristics(CS08-01,CS08-02).
HowwasthepublicengagementcarriedoutBioMetricMirrorwasinstalledvialargepublicdisplayswithinapublicUniversityspaceinMelbourne,Australiafor50consecutivedays(Woutersetal.
2019).
Usersinteractedwiththisapplicationviagesturescapturedbysensors.
Onceusershadconsentedbyraisingtheirhand,theirphotographwastaken,andanalysedusingtheBioMetricMirrorfacialanalysisinstallation.
Theuserwasthenshownthepsychometricanalysisoftheirphotographdeterminedbytheinstallation,alongwithavalueandconfidenceinterval,whichindicatedthedegreeofcertaintyattachedtothisdecision(Woutersetal.
2019).
Followingthis,theuserwaspresentedwithpossiblespeculativescenariosusingtheresultsoftheiranalysis(Woutersetal.
2019).
Forexample,theapplicationquestionedhowtheuserwouldfeelabouttheirdatabeingsharedwithlawenforcementorarecruitmentagency,probingtheindividualtoconsiderhowtheywouldfeelaboutthesedecisions(seeFigure1forexamplesofuserinterface(Woutersetal.
2019).
Finally,semi-structuredinterviewswerecarriedoutwithasubsetoftheuserstoenablediscussionaroundhowaccuratetheythoughtthetechnologywasaswellasethicalimplicationsofthetechnologyitself(Woutersetal.
2019).
Howeffectivewasthepublicengagement137Theuseofaninteractiveapplicationsucceededinengagingthepublicontheissueoffacialanalysistechnologyandautomateddecision-making.
Intotal,798interactionstookplacewithBioMetricMirroracrossthe50daysofitsinstallation,and40individualswereinterviewed(Woutersetal.
2019).
BioMetricMirrorenabledreflectiononcomplexandpotentiallyharmfulethicalquestions,providingaspaceforfurtherdiscourseonfacialanalysistechnology(Woutersetal.
2019).
Theuseofspeculativescenarioswithintheapplicationprovidedparticipantswiththeopportunitytoreflectonthebroaderimplicationsofthesetechnologies(Woutersetal.
2019).
Follow-upinterviewswithparticipantsaftertheapplicationhighlightedtheirconcerns.
Oneuserdeclaredconcernthatanalysisdeterminingthemasaggressivemaybepassedontoanimmigrationofficer,whilstanotherconsideredthenegativeimpactsofthistypeofdatausewithinthejudicialsystem(Woutersetal.
2019).
Thestudyalsohighlightedareasofgeneralconcern.
Forexample,thattheprocessingandstoringoffacialanalysisdatamayresultinthestorageofaflawedreadoutofauserforanindeterminatetime(Woutersetal.
2019).
TheresearchteamreflectedthatBioMetricMirrorhadprovidedathought-provokingandentertainingapplicationwhichenabledengagementfromthepubliconcomplexethicalissuesaroundfacialanalysistechnology(CS08-01,CS08-02).
Theapplicationwasabletoprovideparticipantswithaspacetoexplorepsychometricanalysisandprofilinginafunandinformativeway(CS08-01,CS08-02).
Theresearchteamfoundthatusersgenerallyperceivedpsychometricanalysisasanenjoyableactivity(Woutersetal.
2019).
Participants,however.
Werealsoabletoconsiderthefallibilityofpsychometricprofilingduetotheinherenterrorswithintheanalyticalalgorithmsthattheapplicationused(Woutersetal.
2019).
BioMetricMirrorwaseffectiveatengagingthepublicbothnationallyandinternationally.
Theresearchteamcalculatedthat155newsarticlesonBioMetricMirrorwerepublishedthroughonlinemediaoutletsacross20countriesreachinganestimated204millionreaders(Woutersetal.
2019).
Thetechniqueresultedinfurtherreflectionandengagementonsocialmedia.
Atotalof2,588messageswerepublishedonsocialmediaeitherrespondingtothestudy,orfeaturingassociatedhashtags(Woutersetal.
2019).
Inaddition,theresearchteamobservedatwitterthreadthatspeculatedhowthewithdrawalgesturefromtheapplication(i.
e.
coveringtheparticipantseyes)couldbetranslatedintoreal-lifescenarios,andbecomeastandardisedwayforwithdrawingconsentfromfacialrecognitionandanalysis(Henry2018).
Thestudyprovidedinsightintofurtheravenuesforconsiderationinthisarea.
Forexample,thestudytypicallyfoundthatparticipantsunderestimatedtheextenttowhichfacialanalysistechnologiesarecurrentlyused,andmisunderstoodthe138objectivenessofalgorithmsthatrelyonsubjectivecrowd-sourceddata(Woutersetal.
2019).
What'lessons'canbelearntfromthisexampleTheuseofaninteractiveapplicationenabledtheresearchteamtodemonstratethepotentialforthistypeoftechnologyinthefield:Thisenabledparticipantstoengagewithrealisticapplicationscenarios(Woutersetal.
2019).
Inaddition,itgaveparticipantsthechancetoconsiderthistypeoftechnologybeforeitisfullyintegratedinsocietyandengagewithdiscussionanddebatewhilstthetechnologyisstillunderdevelopment(CS08-01,CS08-02).
Choosingaprovocativesubjectstimulatespublicengagement:Theprovocativenatureoftheinteractiveapplicationstimulatedpublicdiscussionontheethicsanduseofthistechnology(CS08-01,CS08-02).
Theresearchteamensuredthatthiswasbalancedwithadequateinformationtoenableparticipantwellbeing(Woutersetal.
2019).
Futurestudiesmaywishtoconsiderresearchquestionsthatcouldstimulateandencouragepublicdebate.
Thecreationofatoolthatcouldbeexhibitedindifferinglocationsenabledongoingengagementinthisarea:Althoughtheformalstudyiscomplete,BioMetricMirrorcontinuestobeexhibitedinbothartandscienceexhibitionsglobally,continuingtostimulateconversationaroundtheuseofthistechnologyandthepotentialethicalconcerns(CS08-01,CS08-02).
Futureengagementactivitiesmaywishtoconsideranapplication-liketoolthatcansupportengagementwiththepublicacrossdifferentlocationsandcontexts.
139A.
9.
Casestudy9(CS09):CitizenandMulti-ActorConsultationonHorizon2020(CIMULACT)toformulatescienceandtechnologypolicyresearchagendaintheEuropeanUnionCasestudytitleCitizenandMulti-ActorConsultationonHorizon2020(CIMULACT)toformulatescienceandtechnologypolicyresearchagendaintheEuropeanUnionConcisesummaryofthecasestudyTheCIMULACTprojectfocussedonadvancesinResponsibleResearchandInnovation(RRI)intermsofenhancedcooperationwithscienceandsocietytopromotescientificexcellence,andsocialresponsibilityandawareness.
CIMULACTcarriedoutcitizenandmulti-actorconsultationstocontributetoresearchagendaformulationforscienceandtechnologypolicyintheEuropeanUnion.
Theprojectwaswell-receivedbyparticipants.
IthadanimpactontheresearchagendainHorizon2020,astheprojectresultswereusedintheformulationofH2020WP2018-2020.
TheprojectwasalsoacknowledgedintheInterimEvaluationofHorizon2020andwaschosenasagoodpracticecaseforcitizenengagementinagendasettinginopenscienceattheOECD.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleScience,TechnologyandInnovationScience,TechnologyandInnovationWorkshops,consultations,andaconference30Europeancountries,2015–2018WhatisthetechnologicalinnovationTheCitizenandMulti-ActorConsultationonHorizon2020(CIMULACT)projectfocussedonScience,Technology,andInnovation(STI)asoneofthewaysfortheEUtocreatejobsandimprovethequalityoflifeinEurope(CIMULACT2021d;Hebakovaetal.
2018).
ItspecificallyfocussedonadvancesinRRI,relatedtoenhancedcooperationwithscienceandsocietytocombinescientificexcellenceandsocialawarenessandresponsibility(CIMULACT2021b;Hebakovaetal.
2018).
ThedefinitionofRRIinthisprojectfollowsvonSchomberg's(2011)definition,namelycreatingaspacewheredifferentactorsbecomemutually140responsivetoeachother,andaimedforagendasettingbasedonsocietalneeds(CIMULACT2021b).
WhatwasthepurposeofthepublicengagementTheaimofCIMULACTwastoimprovecitizenengagementintheresearchagendaformulationprocessinEuropebyprovidinginputstotheHorizon2020WorkProgramme2018-2020(H2020WP2018-2020)andthepreparationofFrameworkProgramme9(FP9)51(Hebakova,etal.
2018).
Themainobjectiveoftheprojectwastoengagecitizensandstakeholdersintheco-creationofEuropeanresearchagendas'basedonreal,validatedandsharedvisions,needsanddemands'(CIMULACT2021d)(CS09-02,CS09-01).
CIMULACTaimedtocontributetothedevelopmentofRRIinSTIbycreatingandimprovingthedialoguebetweencitizens,stakeholders,scientists,andpolicymakers(CS09-02).
Thegoalwastocreatescenariosofdesirableandsustainablefuturestocreaterecommendationsandsuggestionsforresearchandinnovationtopicsandpolicies(CIMULACT2021d;Hebakovaetal.
2018).
HowwasthepublicengagementcarriedoutCIMULACTwasathree-yearprojectcarriedoutby29consortiumpartnersandfundedbytheEuropeanCommission(EC)(Hebakovaetal.
2018).
52Itinvolvedworkshops,consultations,partnermeetings,andaconferencewithcitizensandexpertsfrom30Europeancountries.
TheCIMULACTprojecttranslatedinitialcitizens'visionsofthefutureintoresearchtopicsusingsixconsequentstepsconsistingofsevenactivities,focussingonbothverbalandnon-verbalcommunication(CIMULACT2021c;Hebakova,etal.
2018)(CS09-02):Nationalcitizenvisionworkshops(NCVs):FromNovember2015toJanuary2016NCVswereheldinthe30participatingcountrieswithmorethan1000citizenstoformulate179visionsforasustainableanddesirablefutureinEurope.
51Horizon2020wastheninthframeworkprogrammeoftheEUandbuildsonHorizon2020,whichcametoanendon31December2020(Karakas2019).
52ThisincludedTeknologirdet–DanishBoardofTechnology,FraunhoferGesellschaftzurForderungderAngewandtenForschungEv,OesterreichischeAkademiederWissenschaften,MissionsPubliques,StrategicDesignScenariosSprl,TechnologickeCentrumAkademievedCeskeRepubliky,AsociatiaInstitutuldeProspectiva,AppliedResearchandCommunicationsFund,GreendependentIntezetNonprofitKozhasznuKorlatoltFelelosseguTarsasag,PolitecnicodiMilano,TheAssociationforScienceandDiscoveryCentres,FundacioCatalanaperalaRecercailaInnovacio,AkademienderWissenschaftenSchweizVerein,HelsinginYliopisto,Teknologiradet–TheNorwegianBoardOfTechnology,InstytutTechnologiiEksploatacji-Panstwowy,InstytutBadawczy,AsociacijaZiniuEkonomixosForumas,SiaBaltijasKonsultacijas,UniversityCollegeCork,WageningenEconomicResearch,MediatedomainLda,UniversitaTaMalta,SlovenskaAkademiaVied,SlovenianBusiness&ResearchAssociation,RtdTalosLimited,4motionAsbl,Odraz-OdrziviRazvojZajednice,StatensGeotekniskaInstitutandAtlantisSymvouleftikiAnonymiEtaireiaAtlantisConsulting141Clusteringworkshop:InFebruary2016,the179visionswereclusteredaccordingtosocialneedsinaworkshopwiththeconsortiumpartners,11externalexpertsand"creativethinkers".
53Co-creationworkshop:InMilaninApril2016,100citizens,expertsandconsortiumpartnersco-created48researchprogrammescenarios,orsuggestionsforresearchprogrammesthatcanaddresstheunderlyingsocialneedsidentifiedinthepreviousstagesoftheproject(CIMULACT2016),basedonthesocialneedsfromtheclusteringworkshop.
NationalResearchandPolicyWorkshops(NPRS):977participantswereintroducedtoeightresearchprogrammescenariosbeforeorduringtheworkshopandenrichedatleastfourofthesescenarios.
54OnlineResearchandPolicyConsultations(ORPC):InparalleltotheNPRS,thescenarioswerediscussedinacross-Europeanopenconsultationorsurveywith3456participantstovalidateandprioritisethescenarios.
Corepartnermeeting:InNovember2016,thecorepartnerscompiledtheresultsandpreparedthemfortheEC.
55Pan-Europeanconference:InDecember2016,apan-EuropeanconferencewasheldtofinalisetheresearchtopicssuggestionswiththeCIMULACTpartners,experts,advisorsandprojectofficersattheEuropeanCommission.
56HoweffectivewasthepublicengagementTheprocessresultedinseveraldeliverablesaimedatEuropeanandnationalresearchpolicymaking.
Theseincluded179citizens'visions57and48researchprogrammescenarios,and23researchtopics(Hebakovaetal.
2018).
CIMULACTcarriedoutanimpactassessmentthatcomparedthe23citizen-basedreportswiththetopicsintheH2020WP2018-2020andhadinterviewsandanonlinesurvey53The26socialneedsthatwereidentifiedintheclusteringworkshopweremergedin12domainsofsocialneeds:(1)equality;(2)unityandcohesion;(3)citizenshipawarenessandparticipation;(4)holistichealth;(5)sustainablefood;(6)lifelongprocess;(7)strength-basededucationandexperientiallearning;(8)harmonywithnature;(9)personaldevelopments;(10)greenhabitats;(11)sustainableeconomy;and(12)sustainableenergy.
54Themethodsintheseworkshopsdifferedasprojectpartnerseitherdevelopedtheirownmethodologyorchoseonefromasetofthreesuggestedmethods.
55Theresultsofthemeetingconsistedofthechallenge,scopeandexpectedimpactofthe48suggestionsofresearchtopics.
5646CIMULACTpartners,11experts,2advisors,and16projectofficersfromtheECworkedtogethertorevisetheresearchtopicsbasedontheresultsfromtheNRPSandORPsheldfromAugustuntilOctober2016.
Thisresultedin23researchtopicsand40policyrecommendations.
Theticsrepresentedcitizens'desirablefutureandtheirexpectations,desiresandconcernsforEurope.
57TheCIMLUACTprojectfollowedtheCIVISTIproject'sdefinitionofvisionsas"apictureoranimaginationofadesirablefuture.
Avisioncanbebaseduponhopesanddreams—butalsouponconcernsandfearsinrelationtoproblemsorimaginedthreats,whichwedonotwanttobecomefuturereality.
Thetimespanofthevisionis30–40yearsfromnow"(Sotoudeh&Peissl2008).
142withresearchpolicyofficersatCommissionservices(DGs)ontheuseofCIMULACToutcomesinanimpactevaluation(Hebakovaetal.
2018).
58TheimpactevaluationfoundthatprojectresultswereusedintheformulationoftheH2020WP2018-2020,astherewasanoverlapbetween15ofthe23CIMULACTresearchtopicsand22topicsfromtheWP(Hebakovaetal.
2018).
SurveyandinterviewrespondentsalsoappreciatedtheapproachadoptedbyCIMULACTandfounditrelevant(Hebakovaetal.
2018).
Theyagreedthatcitizensarecompetentenoughtosuggestdirectionsforresearchandthattheirviewscanhelpdistinguishbetweenpoliticalpressureandrealcitizenneeds.
Intervieweesappreciatedthecoverageofissues,theirnovelty,andcompatibilitywithothersources(Hebakovaetal.
2018).
Twofacilitatorssaidthatcitizensappreciatedthattheycouldhaveanimpactonagendasettingandtotracetheirinputsateverystageoftheprocess(CS09-01,CS09-02),whichwasreflectedintheirwillingnesstoreturntoconsultations(CS09-01).
TheCIMULACTresearchtopicsalsoaddednewanduniqueperspectives,especiallyonsocietalaspects,inexpertorientedforesightstudies.
59PolicyofficersattheEuropeanCommissionwhoengagedwiththeprojectfoundthatCIMULACTwasahelpfulinstrumentforcitizenengagement,themethodologyproducedtrustworthyoutputsandagenuineconsultationprocess.
CIMULACTconsortiumpartnersnotedthattheygainednewexperiencesinco-creation,adoptedtheprojectmethods,improvedtheirmethodologicalknowledge,recruitmentapproaches,networkingandfacilitationskills(Hebakovaetal.
2018).
Moreover,theworkofCIMULACTwasacknowledgedintheInterimEvaluationofHorizon2020statingthattheCIMULACTprojectprovidedconcreteinputtotheEuroperesearchandinnovationagendaandimprovedtheengagementofcitizens(EuropeanCommission2017).
TheprojectalsofedintoH2020WP2018-2020andwasacknowledgedintheframeworkprogrammeintroduction(EuropeanCommission2020;Hebakovaetal.
2018).
TheCIMULACTprojectalsohadanimpactonotherwork,asOECDchoseCIMULACTasagoodpracticecaseforcitizenengagementinagendasettinginopenscience(OECD2017).
60What'lessons'canbelearntfromthisexampleCIMULACTsummarisedseveralkeylessonsfortheprocessintheirimpactevaluationreport(Hebakovaetal.
2018):58Theonlinesurveywasdistributedto215policyofficers.
Twointerviewswereconductedwithpolicyofficersbyphoneandthereweretwoface-to-facemeetingswiththreepolicyofficersinBrussels.
5910out16studiesincludedoneormorereferencestoresearchtopicsidentifiedinCIMULACT.
Here,foresightwasdefinedasa"structureddialogueonlong-termfutures"(CIMULACT2021a).
60Inaddition,theprojectproducedaninspirationcatalogueonmethodsforconsultingcitizensandstakeholders(Dagorne&Gudowsky2018),andadeliverableonthemeritsofcitizen-focussedconsultationscomparedtotraditionalforesightmethods,whichcanprovidemethodologicalinsightsforotheractors(Rosaetal.
2018).
143Focusondisseminationamong,andclosecollaborationwith,policymakersandrelevantECpolicyofficers:DisseminationandawarenessraisingproceduresshouldbesetupjointlywiththeECattheearlystagesoftheproject(Hebakovaetal.
2018).
Targetdisseminationefforts:Theprojectpaidparticularattentiontodisseminationandpromotiontoincreasetheimpactoftheproject.
61Amoredetailedoverviewofthetargetgroupsandmoreuser-orientedmaterialcouldhaveincreasedtheirimpact(Hebakovaetal.
2018).
Recognisethechallengeofconductingworkshopsacrossdifferentcountries:Thereweresomechallengeswithtimepressure,timelinesfortheonlinesurvey,andthenumberanddiversityofconsortiumpartnerswhoconductedtheworkshopsacrossEurope.
Oneintervieweealsosaiditisimportanttoconsiderthatthewillingnesstoshareandparticipateopinionswilldifferdependingontheinstitutionalhistoryofthecountry(CS09-01).
Ensurethatthepublicengagementprocessisflexible:Twointervieweesnotedthatflexiblepublicengagementprocessesallowcitizeninsightstoemergeorganicallyandtoadapttheprocesstoincludecitizen'sperspectivesatdifferentstagesintheprocess(CS09-01,CS09-02).
Establishstrongdatamanagementsystems:Oneintervieweenotedthatstrongdatamanagementsystemsarenecessarytocaptureandmanagecitizenperspectives(CS09-02).
Involvecitizensatallstagesofimplementationanddesign:Oneintervieweearguedthattofacilitateinnovationsthatarerelevanttocitizens,theyshouldbeinvolvedatallstagesofimplementationanddesign.
Itwouldalsobehelpfultoconsiderhowtoincludecitizenperspectivesinmorespecificsectorsandgeographicallocations(CS09-02).
61Atotalof902disseminationactivitieswerecarriedoutbyconsortiumpartnersand126presentationswereheldatworkshopsandconferences.
144A.
10.
Casestudy10(CS10):RapidonlinedeliberationtoexplorepublicattitudestotheuseofCOVID-19-relatedtechnologiesCasestudytitleRapidonlinedeliberationtoexplorepublicattitudestotheuseofCOVID-19-relatedtechnologiesConcisesummaryofthecasestudyThereappearstobealackofpublictrustinsomedigitaltechnologiesdevelopedtoaddressdifferentaspectsofCOVID-19,includingcontacttracingapps.
Toensurepublictrustandbuy-inofthesetechnologies,itisimportanttodetermineunderwhatcircumstancesthepublicconsiderstechnologicalsolutionssuchastheCOVID-19contacttracingapptobeappropriate.
Tothisend,theAdaLovelaceInstitute,togetherwithcollaborators,conductedarapidonlinedeliberationtoexploretheattitudesofmembersoftheUKpublictotheuseofdigitalCOVID-19technologies,includingtheNHScontacttracingapp.
Theprocesswaseffectiveinenablinganinformeddialoguewiththepublicinarapidlychanginganduncertainenvironment.
Theprocessdemonstratedthatitispossibletogatherpublicinputinashortspaceoftimeandcouldpotentiallybeappliedtootherfutureareasinwhichtimeislimited.
Althoughitistoosoontoindicatewhethertheengagementprocesshasimpactedonpolicyorregulation,theengagementexercisenonethelessidentifiedfourrequirementsthatwouldhelptoensurepublictrustandbuy-inregardingfutureCOVID-19technologies(i.
e.
providingatransparentevidencebase;offeringindependentreviewofthetechnology;clarifyingdatause,rightsandresponsibilities;andaddressingtherisksandneedsofvulnerablegroups).
Theserequirementsfedintoachecklistaimedatgovernment,policymakersandtechnologydeveloperstohelpthemwiththefuturedevelopment,designanduseofCOVID-19technologies.
TechnologyareaSector/organisationPublicengagementtechniquesCountryandtimescaleDigitaltechnologiesdevelopedtocombatPublicHealthDeliberativedialogueusinganonlineUnitedKingdom,May-June2020145COVID-19(e.
g.
contacttracingapp)platform(ZoomandEngagementHQ)WhatisthetechnologicalinnovationThetechnologicalinnovationconsistedofarangeofCOVID-19technologiesthatgovernmentsacrosstheworldaredevelopingtohelpwiththesurveillance,preventionandcontroloftheCOVID-19pandemic.
Thisincludessymptomtrackers,digitalcontacttracingapps,andpublichealthidentitysystems,aswellasbroaderdatacollectionanddatasharinginfrastructures(suchastheNHSDataStore).
Thediscussion,however,centredontheUK'sdigitalcontacttracingapp(beingpilotedintheIsleofWightatthetime).
WhatwasthepurposeofthepublicengagementThereappearstobealackofpublictrustinsomedigitaltechnologiesdevelopedtoaddressdifferentaspectsofCOVID-19,includingcontacttracingapps,andtheiruseofpersonaldata(Gardner2020).
Thisisproblematicbecausetheefficacyofdigitalcontacttracingappsreliesonmassadoption(AdaLovelaceInstitute2020).
Thiswillrequiretrustandbuy-infromthepublicregardingthedecisionsmadeinrelationtotheapp.
TheaimofthepublicengagementwastodetermineunderwhichcircumstancesthepublicconsiderstechnologicalsolutionssuchastheCOVID-19contacttracingapptobeappropriate(AdaLovelaceInstitute2020).
This,inturn,intendedtoidentifytherequirementsneededtomaketheUKGovernmentcontacttracingapptrustedandjustifiedinthepublic'sview.
HowwasthepublicengagementcarriedoutThepublicengagementconsistedofarapidonlinedeliberativeprocesswith28membersofthepublic(AdaLovelaceInstituteetal.
2020).
Theprocesstookplaceoverathree-weekperiod,inMayandJune2020.
Itwasconductedonlineusingtwoonlineplatforms,ZoomandEngagementHQ(AdaLovelaceInstituteetal.
2020).
Thestakeholdersinvolvedincludedtheprojectteamrepresentingfourorganisations(theAdaLovelaceInstitute,BangtheTable,InvolveandTraverse);28randomlyselectedmembersofthepublicfromtwourbanandrurallocations(CamdenandKent);62andfivespecialistsfromacademiaandthethirdsectorwhowereexpertsindataandtechnologypolicy(AdaLovelaceInstituteetal.
2020).
Eachengagementconsistedofone90-minutesessionperweek.
Theprocesswas62Therecruitmentapproachconsistedofpromotingtheprojectwithanadvertusingtheteam'slocalnetworksinCamdenandKent,throughmutualaidgroupsandlocalcommunityorganisations.
Potentialparticipantscompletedashortdemographicsformtoenableselectionofthemostdiversesampleavailable.
146deliberative,inthatparticipantswerefirstpresentedwithinformationaboutthetopic(e.
g.
digitalcontracttracingandtheNHSapp)fromtheexpertspeakers(whospokefor10minutesineach90-minutesession)andwerethenabletoaskquestions(usingtheZoomchatfunctionratherthanverbally)fortheremainderofthetime.
TheclosedplatformEngagementHQwasalsousedforalimitedtime(thefirsttwoweeks)toenableparticipantstoaccessmaterials(e.
g.
speakerslidesandvideorecordings)andtakepartinactivities(e.
g.
theabilitytointeractandhaveconversationswiththeotherparticipants)(AdaLovelaceInstituteetal.
2020).
Thewholeprocesswasdesignedtotakeplacerapidlyalongsidethepolicydevelopinginreal-timeinresponsetoCOVID-19duringthelockdown.
HoweffectivewasthepublicengagementThiswasapilotprojectthatsuccessfullydemonstratedthefeasibilityofconductingadeliberativedialogueonlineonarapidlyevolvingsubject(asevidencedintheabilitytoproduceconsideredfindingsfromthepublicinashortspaceoftimeandoutlinedinthelessonsbelow)(CS10-01;CS10-02).
Goingforwards,akeybenchmarkofchangewouldbetodeterminewhetherpolicymakersadopttheseapproachesinthefuture.
Althoughitistoosoontoindicatewhethertheengagementprocesshasimpactedonpolicyorregulation,theengagementexercisenonethelessidentifiedfourrequirementsthatwouldhelptoensurepublictrustandbuy-inregardingfutureCOVID-19technologies.
Thesewereto:provideatransparentevidencebase;offerindependentreviewofthetechnology;clarifydatause,rightsandresponsibilities;andaddresstherisksandneedsofvulnerablegroups(AdaLovelaceInstitute2020).
Thesefindingsfedintothedevelopmentofachecklist(basedaroundthefourrequirementsoutlinedabove)aimedatgovernment,policymakersandtechnologydeveloperstohelpthemwiththefuturedevelopment,designanduseofCOVID-19technologies,includingcontacttracingapps,toensurepublictrustandbuy-in(AdaLovelaceInstitute2020).
Thepublicengagementexercisehasnotbeenformallyevaluatedbuttheauthors,togetherwithfeedbackfromasurveysenttoparticipants,provideanumberofkeyobservationsontheprocess(AdaLovelaceInstituteetal.
2020).
Theengagementwasdeliberative,withgoodqualityconversationsabletotakeplacedespitetheonlineformat(AdaLovelaceInstituteetal.
2020).
Participantsindicatedthat,overall,theyweregivensufficienttimetocontributetheirviews(AdaLovelaceInstituteetal.
2020).
Theonlineprocessworkedrelativelywell,usingtoolsthatwereinexpensivetosetupandeasytouse,includingtwoonlineplatforms–ZoomandEngagementHQ.
Thedeliberativeexerciserequiredexperiencedfacilitators,whichlikelyaddedtothecost.
147Havingbothsynchronous(everyoneparticipatingatthesametimein,e.
g.
videochat)andasynchronous(interreactinginyourowntimein,e.
g.
anonlineforum)engagementworkedparticularlywellaccordingtooneinterviewee(CS10-02).
Havingtheengagementtakeplaceovermultiplesessionsenabledparticipantstoreflectandembedlearningbetweensessions(CS10-02).
Thereweremixedviewsregardingthedifferenttoolsandmethodsthatwereused(e.
g.
somepreferringthegroupdiscussionsessions,otherspreferringthechatfunction);however,overallparticipantsreportedthatthevarietyoftoolshelpedtosupportparticipantengagementandretention.
Theinformationpresentedbytheexpertswasclearandeasytounderstand.
Someparticipantssuggestedthathavingtheinformationandthetopicsinadvancewouldhavegiventhemtimeto'mullitover'andallowedthemtoengagebetter(AdaLovelaceInstituteetal.
2020).
Limitationsoftheengagementprocessincluded:Workingonarapidlyevolvingpolicyareawithatighttimeframemeantthattherewasnotimetogetpolicymakersonboard(CS10-01;CS10-02).
Foranengagementprocesstohaveimpacttypicallyrequiresengagementandbuy-infrompolicymakers(CS10-01;CS10-02).
Communicatingandgaugingparticipants'understandingofthetopic.
Theonlinenatureoftheengagementmadeitmorechallengingtocheckparticipants'understandingoftheinformationreceived(AdaLovelaceInstituteetal.
2020).
Theengagementwasbroadlyinclusivebutunlikelytoberepresentativeofthepopulationintermsofdemographics.
Althoughtheprocessincludedparticipantsfrombothurbanandruralareas(i.
e.
CamdenandKent),thesamplewaslimitedto28membersofthepublic(AdaLovelaceInstituteetal.
2020).
Inaddition,althoughsomeparticipantsenjoyedtheonlineformat,thetighttimeframemeantthattheprocessrecruitedindividualswhowerealreadycomfortableonline.
Feedbackfromparticipantssuggestedtheyhadmixedviewsabouttheonlineprocess,withsomepreferringitandotherspreferringface-to-face.
Whereastheonlineformatfacilitatedparticipationforsomepeopleduetotheshortertimecommitmentcomparedtotraditionaldeliberativeprocesses,noteveryonefeltcomfortableparticipatingonlineandusingonlinetoolssuchasZoom.
What'lessons'canbelearntfromthisexampleRapidonlinedeliberationisafeasiblepublicengagementtechnique:Itisfeasibletoconductinformedpublicconversationonlinewitharapidlyevolvingpolicycontext(CS10-01;CS10-02).
148Rapidonlinedeliberationworkswellwithaclearlydefinedandarticulatedquestion:Workinginatighttimeframeisperhapslesswellsuitedtoexploremorecomplexorsystemicquestions(CS10-01).
Thefast-pacedapproachmaynotworksowellifdealingforquestionsinwhichthereisconsiderabledifferenceofopinion,aneedtobuildtrustacrossthegroupwhengoingintoadeliberativeprocessonaparticularissue,andwherecomingtoaconsensusorameaningfulresolutionmayrequireseveralweeksandmonths,ratherthandays(CS10-02)(AdaLovelaceInstituteetal.
2020).
Policymakersupportandbuy-inisimportant:Thetighttimeframesmeanttherewaslesstimeforpolicymakerstobeinvolved.
Goingforwards,however,itisimportanttohaveclarityaroundthetopicandpolicymakerbuy-inandengagement,includinghowtheworkwillfeedintothedecision-makingprocess(CS10-01;CS10-02).
Onlinedeliberationcanbeinclusive:Althoughitprovideschallengesarounddigitalinclusion,itisalsoameansofincludingindividualsthatface-to-facemethodsexclude(CS10-01;CS10-02).
Itcanbeinclusiveinwaysthattakeintoaccounttimepressuresonpeople'sschedules.
149AnnexB:OverviewofgeneralpublicengagementtechniquesTable4providesanoverviewofaselectionofpublicengagementtechniquesthatexistandhavebeenusedincontextsbeyondtechnologicalinnovation.
Eachpublicengagementtechniquehasbeencategorisedaccordingtothreehigh-levelcategories:63(1)Communicate,whichischaracterisedbythedeliveryofinformationtothepublictoinformoreducate(thesetechniquesareone-way);(2)Consult,whichischaracterisedbyboththedeliveryofinformationtothepublicandthecollectionofinputfromthepublictolisten,gainknowledgeandunderstanding(thesetechniquesareone-way);and(3)Participate,whichischaracterisedbycollaborationandco-creationwiththepublic(thesetechniquesaretwo-way).
Thislistisnotintendedtobeanexhaustiveguide.
Itisalsoimportanttonotethattheappropriatetypeoftechniquewilldependontheobjectiveoftheexercise.
Table4:OverviewofgeneralpublicengagementtechniquesPublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipate21st-CenturyTownMeetingThisapproachbringstogetherbetween500-1000peoplefordiscussionoflocal,regionalornationalissues.
Theparticipantsaresplitintogroupswith10-12peopleineachandanetworkedcomputerisusedtocollectvotesfromparticipantsduringtheevent.
Participantsarealsogivenindividualkeypadtovoteonquestionsorthemes.
ThedatafromtheX(Involve2018)63Asnotedpreviously,thesethreecategoriesbroadlymapontoIAP2spectrumofpublicparticipation(IAP22018)thatconsistsofthefollowingcategories:inform,consult,involve,collaborate,andempower.
'Communicate'and'consult'roughlycorrespondtotheIAP2categories'inform'and'consult',respectively.
Thecategory'participate'broadlyincludesthethreeIAP2spectrumcategories'involve','collaborate'and'empower'.
150PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatevotingexercisesissortedgeographicallyandsharedwithstakeholdersattheendoftheevent.
ActionResearch/PlanningThisapproachaimsattransformativeactionbyinvolvingthepublicinlivingconditions,everydayproblems,andtheirenvironment,inordertochangetheseconditions.
Itacknowledgesdiverseformsofknowledge,bothscientificandcitizenknowledgetopoliticallyempowerpeople.
Theyarenormallystructuredinfivephases:(1)ameetingwithstakeholdersandcitizenstodiscussissues;(2)topicanddesignworkshops;(3)brainstormingsessionfortheworkshopideas;(4)analysisofideas;and(5)theagreedproposalsarepublishedinafinalreport.
x(Engage20202014;Involve2018;Ward2009)AppreciativeinquiryQuestionsareusedtobuildavisionforthefuture,byfocussingonpastandpotentialfuturesuccesses.
Thequestionsencouragebetween5-15participantsandtheirnetworkstotellstoriesabouttheirexperienceofwhatworks.
Theprocesshasfourstages:(1)aX(Involve2018)151PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatediscoverystage:participantsareasked'whatworks';(2)adreamstage:participantsareaskedtoimaginepositivefeaturesidentifiedinthediscoverystage;(3)designstage:participantsidentifystepstorealisethedreamsidentifiedinthepreviousstage;and(4)destinystage:focussesontheimplementationofpreviousideas.
Arts-basedTechniquestoengagethepublicthatspanthevisualarts,performingarts,games,artinstallationsandothertechniques.
X(NCCPE2019)BriefingworkshopsWorkingsessionsthataimtoestablishaprojectagendaorbrief.
Usersofaprojectareinvitedtoattendaworkshopthatnormallylasts1.
5hours.
Thesameworkshopcanbeheldwithdifferentstakeholdergroups,isfacilitatedbyafacilitator(s),andrecordsaremadeofkeypointsandissues.
X(Engage20202014;Ward2009)Charrette/Designcharrette/EnquirybydesignIntenseandhands-onworkshop/sessionthatbringstogetherdiversestakeholderstoexploredesignoptionsforcertainissues.
TheyoftenX(Engage20202014,Involve2018;TamarackInstitute152PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateinvolveintensiveworkshopsforstakeholders,suchaspolicymakers,expertsandpublic,tocollaborate.
Animportantcomponentisintegratingdesignactivitiesatanearlystageforresearchproposalsand/orimplementationplans.
2017;Ward2009)CitizencompassCitizens,whoarerandomlyselected,evaluategovernmentworkbasedoncriteriathatcitizenshavedeveloped.
Theythenproposemeasuresforfuturegovernmentwork.
Itoffersaplatformforcitizenstoteachpoliticianswhatthepublicthinksaboutpoliticalissuesandprovidesrecommendationsforpolicymaking.
X(Engage20202014)CitizenAdvisoryGroups/CommitteesInvolves10-30membersofthepublicwhoformacommitteetoadviseandinformdecisionmakingoveralongertimeperiod.
Theycantakemanydifferentformsbasedontherequirementsofthegroup.
X(Involve2018;Ward2009)CitizenscienceInclusionofthepublicinscientificresearchbycollectingoranalysingdataand/oraskingquestionstoX(Engage20202014)153PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatefacilitatediversecontributions.
Itisoftenahierarchicalandorganizedprocessthatareinitiatedandsupervisedbyscientists.
Citizens'juryItusuallyinvolves12-24participantsofarepresentativesampleandlastbetween2-7days.
Theopinionsandinsightsofcitizensarerevealed,andacommondecisionismade.
Theyaremoreapplicablewhereapolicyproblemcanbesolvedindiverseways.
X(Engage20202014;Involve2018;TamarackInstitute2017;Ward2009)Citizens'summitsLarge-scaledeliberativepublicmeeting(with200to5000participants)toexplorecitizens'attitudesaboutpoliticalprioritiesandpotentialactionsprovidedonabasisthatisinformed.
Theobjectiveistoprovideinspirationandadviceforpoliticaldecisionmakingwherelarge-scaledecision-makinghappensbyvoting.
Itprovidesaclearindicationofattitudesandimpliessomedegreeofcommitmentbypolicymakers.
X(Engage20202014;Involve2018;TamarackInstitute2017)154PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateCitizens'assemblyCitizenbodythatdeliberatesonanissueorissueswithbetween50-250participantsthatoftenlastsoverseveralweekends.
Theobjectiveistousearepresentativesampleofthepublicwhocanlearnaboutanissue,assessoptionsandmakerecommendationsindependentlyofpolicymakers.
Itusuallyhasthreephases:learning,deliberationanddecisionmaking.
Itinvolvesreportingphasewithrecommendationsthatispresentedtopolicymakersorcitizensthroughareferendum.
X(Engage20202014;Involve2018)Citizens'hearingTheobjectiveistocreatediscussionsandinformcitizens.
Itgathers20-25citizensinaone-daydiscussionandusesbrainstorming,dialogue,prioritisation,reasoningandvoting.
Citizensindependentlyformulatetheirownsuggestionsandideasandpresentthemtopolicymakers.
X(Engage20202014)155PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateCitizens'panelRepresentativeandconsultativebodyforlocalresidentsusedtoassesstheiropinionsandpreferences.
Theycanrangebetweenafewhundredorseveralthousandparticipants.
Whentherearemorethan1,000participantssub-groupsareoftenformed.
Thegroupisoftenrenewedthroughouttheprocesssothepopulationsampleisrepresentative.
X(Engage20202014;Involve2018;Ward2009)CitizenVisionsofScience,TechnologyandInnovation(CIVISTI)Basedonthenotionthatthedefinitionofrelevantandforward-lookingresearchandinnovationagendascanbeimprovedbyconsultingcitizens(often25)atthestageofdevelopment.
Itusescitizens'concernaboutsocietaldevelopmenttodevelopresearchprogrammepriorities.
Itdoesnotdevelopreal-worldmodels,butaskscitizenswhatadesiredfutureshouldlooklike.
X(Engage20202014)Civic/PublicdialogueStructuredformatforpublicdialoguesthataimstocreateunderstandingamongdiversepeople.
Itcanbuildbroadconsensusandcommitmentonissuesthatarecomplexand/orcontroversial.
ItisnotX(Engage20202014;Involve2018)156PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateonemethod,butconstitutesarangeofmethods.
Themaintypesinclude:i)publicinquiries,ii)openpublicconsultations,andiii)selectiveparticipation.
CommunityappraisalSurveysoflocalneedsandissues.
Manyincludethefollowingsteps:(1)havingasteeringcommitteeforoversight;(2)writingupaquestionnaireforhouseholds;(3)responsesaresortedandreportedon;(4)distributethereportinthecommunity;(5)actionsareagreedwithrelevantbodies;and(6)developmentsaremonitoredandreportedtothecommunity.
X(Involve2018)Community-based(participatory)researchInvolvesthecommunityinallstagesoftheresearchprocess(fromsettingquestions,framing,doingresearch,interpretingresults,andcommunication)toenablebetterunderstandingandanimprovementofacertainissue.
Itcanbecombinedwithactionstoimplementfindings,whichresultsinparticipatoryactionresearch.
Itinvolvesco-learningandtransferofexpertisebyallresearchX(Engage20202014)157PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatepartners,shareddecisionmaking,andcollectiveownershipofprocessesandproductsinvolvedinresearch.
CommunityindicatorprojectsCommunitiesdevelopavisionforasustainablefutureanduseindicatorstheyhavedevelopedtotracktheprogresstowardsthatfuture.
X(TamarackInstitute2017)CommunitymappingParticipatoryprocessforthepublictomapsocial,ecologicalandeconomicassets,andhistoricaleventsoftheircommunity.
Itinvolvesanaccessibleandgraphicwaytolearnaboutperceptionsofaplaceandisoftenusedforvisionprocesses.
X(TamarackInstitute2017)ConsensusconferenceAimistoexpandonandenrichdebateonatopicthatissociallycontroversial.
Agroupof10-30randomlyselectedcitizensgivetheirperspectivesonaspecifictechnologicalproblemorchallengearea.
Theydiscuss,consultexpertsandcreaterecommendationsinaconferencethatisoftenthreetofourdayslong.
Itoftentakesplaceintwostages:(1)ameetingwithexpertsandX(Involve2018;Ward2009)158PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatestakeholderstoachieveconsensus;and(2)aconferencewherethemainfindingsarepresentedtothepublic.
ConsensusvotingNormallylastsfromhalfadaytoonedayandisusedtoidentifyconsensususingabalancedvotingsystem.
Everyoneinthegroupcansuggestaproposal,alistofissuesisdrawnup,theyvoteontheirpreferences,andvotesarecounted.
X(Engage20202014;Involve2018)ConversationcaféInformaldialoguewherepeopleareinvitedtodiscussanissueinaninformalsetting.
Itnormallyhas10-12participantsandlastsbetween10-12hours.
Theprocesstakesplaceinfourrounds,namely:(1)Round1:atalkingobjectispassedaroundforparticipantdiscussion;(2)Round2:participantsusethetalkingobjecttoexpandontheircomments;(3)Dialogue:openandspiritedconversationisencouraged;and(4)FinalRound:usingtheobject,participantshighlightwhattheyfoundmeaningful.
X(Engage20202014;Involve2018)159PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateCo-productionServiceprovidersandusersworktogethertoreachacollectiveoutcome.
Itisanapproachtoservicedesignanddecision-makingratherthanaspecificmethodofpublicengagement.
x(Involve2018)Crowdwise(similartoconsensusvoting)Communityparticipationmethodthatencouragesconsensus-baseddecisions.
Consensusdevelopinstagesusingacombinationofdiscussionandvotesonasetofpre-definedoptions.
Thenumberofcitizensdiffertoencourageconsensusindiversecontexts.
Itisoftenusedtosetpriorities,allocatebudgets,orrespondtoconsultations.
X(Engage20202014;Involve2018)CrowdsourcingProcesstogatherideas,functions,services,orcontactsfromalargeandundefinednetworkofpeople.
Theprincipleofopennessunderpinstheexerciseandusesbottom-upprocesstofindgoalsthatareoftentop-down.
Thisapproachtendstohappenonline.
X(Engage20202014;Involve2018;TamarackInstitute2017;Ward2009)ScienceweekMethodtocommunicatesciencetoaset,targetaudience.
TheaimistocreateenthusiasmforX(Engage20202014)160PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatescience,technologyandhealthforchildrenandyouth,andtodevelopinterestinsciencecurriculainprimary,secondaryanduppersecondarylevelsoftheschoolsystem.
Deepdemocracy(theLewismethod)Advancedgroupfacilitationmethodcarriedoutoveronetotwodaysinintensedialoguestoaccessandbringoutknowledgethatalreadyexistswithinagroupandtoidentify'creativepotential'thatmightresultfromconflict.
X(Engage20202014)DeliberativemappingTechniquethatcombinesvariedapproachesandinvolvesbothexperts(around20participants)andthepublic(around40participants)toratediversepolicyoptionsagainstasetofcriteria.
Theobjectiveistocreateaprocessthatallowsformoredemocratic,robustandaccountabledecisionmakingthatreflectspublicsvalues.
Thecitizensandexpertsdiscussissuesseparatelyandthentogetherataworkshop.
X(Engage20202014;Involve2018)161PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateDeliberativeworkshops/PublicDialogueWorkshops/DeliberativePolicyWorkshopsDialogueeventswith10-12participantsthatfocusonin-depthandinformeddiscussionsofissuesthatareeithercomplexorcontroversial.
Theaimistocollectsocialintelligencetoinformpolicy,anticipateregulation,exchangeperspectives,orraiseawareness.
Theyhavealsobeenusedforresearchagendadevelopmentandtocreateobjectivesthatreflectpublicviews.
Theyweredevelopedoutoffocusgroupsandotherin-depthanddeliberativealternatives.
X(Engage20202014;Involve2018)DeliberativeonlineforumDiscussionsinonlineandweb-basedforumsbetweeninformedindividualsaboutissuesofconcernthatleadtoconsensusandcollectivedecisions.
Ithasthreemainelements:(1)communicativeoradiscussionspaceonline;(2)majorandminordiscussionspacesdefinedbythelikelyimpactthatparticipantscanhaveonapoliticaloutcome;and(3)politicalcultureandideologytoreflectthesocio-politicalcontextx(Engage20202014;TamarackInstitute2017)162PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateDeliberativepollsTechniquethatcombinesdeliberationinsmallgroupspickedbasedonrandomscientificsamplingtoensurepublicconsultationonpublicpolicyandelectoralissueswith100-600participants.
X(Engage20202014;Involve2018;Ward2009)Democscardgame/PlayDecideCard-gameandpolicyexplorationtoolforsmallgroupsofpeopletoengagewithpolicyissuesthatarecomplex.
Theaimistoinformparticipantsaboutatopic,enablethemtoexpresstheirviews,seekcommongroundwithotherparticipants,andprovidetheirpreferredpolicyoptions.
Itdoesnotrequirespeakersorexpertsasrequiredinformationisprovidedoncards.
X(Engage20202014;Involve2018)DialogueCoversdifferentmethodsthathasthegoaltoshareopinionsanddeveloprelationships.
Itcaninvolveafewpeopleorseveralhundredandaimstoinvolveallstakeholderswhocanhaveanimpactonorareinfluencedbyanoutcome.
X(Engage20202014;Involve2018)DigitalStorytellingParticipantscreatestoriesoftheirlifecollectedusingimagesandthataresharedusingonlinedevices.
IthasX(Engage20202014;TamarackInstitute163PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatesevenelements:apointofview,dramaticquestion,emotionalcontent,narrativevoice,thesoundtrack,economyandpacing.
2017;Ward2009)DistributeddialogueDecentralisedapproachthataimstodevelopdiscussionsthatareongoingandembeddedonatopic.
Someoftheengagementisorganisedbyparticipantsorgroupsthemselves.
Itengagesresearchcommunities,stakeholdersandthepublicforstrategyandpolicydevelopment.
Itofteninvolvesseveraldialogueeventsorganisedbyresearchersandinterestedparties,indifferentgeographicalareasandusingdiversemediums,suchdeliberativeengagementexercisesindifferentregions,devolvedactivities,settingupanonlineforum,andworkingwithexistingnetworks.
X(Engage20202014;Involve2018)E-conference/E-panelTemporaryonlineforumthatusestechnologiesforengagementonaspecifictopic.
Itcanbeusedforeducationalsessions,businessmeetingsorotherevents.
E-panelsareusedforcouncilsandotherX(Engage20202014;Involve2018;Ward2009)164PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateorganisationsforregularonlineconsultationswithaspecificgroupofcitizens.
E-petitionsOnlinepaperpetitionsthatareusedtoillustratesupportforanopiniontoputpressureongovernment.
X(Engage20202014;Involve2018)ExperientialTechniquesthatenablethepublictoexperienceatechnology(eitheratestprototypeintherealworld,interactiveapplications,orusingVRsimulations).
X(Woutersetal.
2019)FeedbackkioskBoothsthatarestaticandplacedinanyspaceforpeopletogiveelectronicfeedbackonservices.
Theyofteninvolveelectronically-operatedtouchscreendevicestocollectfeedbacktoimprovepublicservices.
X(Engage20202014;Involve2018)FestivalForumtoengagethepublicandmakeuseofdiverseformatsforpublicengagement,includingtalks,discussions,workshops,hands-onactivities,performance.
X(NCCPE2020a)165PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateFocusgroupGuideddiscussionofasmallgroupofthepublictogatherinformationonaparticulartopic.
Qualitativetechniquetodeterminepeople'spreferencesorforthemtoevaluatestrategiesandconcepts.
Participantsareselectedbasedoncertaincharacteristicsandworkingroupsof6-12peopleforapproximatelytwohours.
X(Engage20202014;Involve2018;TamarackInstitute2017;Ward2009)FishbowlconversationTechniquewheretheroomset-uprequiresspeakerstositinthecentreoftheroomora'fishbowl'wheretheotherparticipantssitaroundtheminacircleandlistentotheconversation.
Theaimoftheconversationistoincreaseparticipationandunderstandingofanissue.
X(Involve2018;Ward2009)FutureSearchAimstoencourageparticipantstothinkaboutanissueinanewway.
Participantscomefromdiversestakeholdergroupsandareaskedtoopenuptheirperspectivesonnewideasandactionsthatcangainwidespreadsupport.
TheconferenceaimstofindacommonbasisthatallparticipantscansupportandX(Engage20202014;Involve2018;Ward2009)166PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatenormallylastsforthreedayswith25-99participants.
FutureworkshopTechniquetoplanandformavisionofthefuture,andtosetoutandprioritisethestepsrequiredtoachievethevision.
Theworkshopscontributetogoalandproblemdefinitionandusuallyinvolves1-25participants.
Ithasthreephases:(1)criticalanalysisphaseofthesituation/technology;(2)visionaryphasethatbuildsonthefirstphasetocreatevisions;and(3)animplementationphasetoturnvisionsintoactions.
X(Engage20202014;GovernmentOfficeforScience2017;Involve2018)GraphicrecordingTechniqueofcapturingtheideasofparticipantsonalargepieceofpaperusingimages,word,artworks,andcolour.
Itisoftenusedtorecordmeetingsandotherevents.
X(Engage20202014;Involve2018)HackathonDesignsprint-likeeventsthatcanlastbetweenafewhoursandaweekwherepeopleusetechnologytoeitherimproveorbuildnewsoftwareinacollaborativeway.
SometimestheyareundertakentoachieveaX(Engage20202014;Involve2018;TamarackInstitute2017)167PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatespecificgoal,buttheycanalsobeusedtoexploreopen-endedandinnovativeideasledbycitizensorthepublic.
InterviewStructuredconversationtogatherinformationfromthepubliconaparticulartopic.
Exploresviews,normativepositions,beliefs,experiences,andmotivationsofanindividualparticipant.
Theyprovideanin-depthunderstandingofacertaintopic.
Threeformatsexist,namely:(1)structuredinterviewswithasetofpre-determinedquestions;(2)semi-structuredwithseveralkeyquestions,butthatalsoallowstheinterviewertodivergefromthepre-determinedquestionstoexploreandelaborateoncertainissues;and(3)unstructuredthatstartswithanopenquestanddevelopsaccordingtoresponses.
X(Engage20202014;TamarackInstitute2017;Ward2009)Localissuesforum/AreaforumTheaimofthetechniqueistoprovideagreatervoiceinlocaldecisionstoencouragepublicparticipationinlocalpublicpolicymaking.
Itisanonlineorin-personpubliccommonswherethepublicorelectedofficialscan:askX(Engage20202014;Involve2018)168PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatequestions,makepublicannouncements,networkwithothercitizens,monitorlocalopinionandaskforpublicinput.
MassexperimentInvolvesvolunteeringcitizensinscientificresearchfordatacollectionpurposesinascientificproject.
Itisausefulmethodfordatacollectionthatrequiresagreatnumberofindividualcontributionsthatarespatiallydispersed.
X(Engage20202014)Mostsignificantchange(MSC)Formofparticipatoryevaluationandmonitoring.
Participantsdescribethechangesthatwillberecordedandanalysethedata.
Itoccursthroughouttheprogrammecycleandprovidesinsighttohelpmanageaprogramme,whilstprovidingdataonimpactandoutcomes.
X(TamarackInstitute2017)Mysteryshopping"Mysteryshoppingawayofauditingservicesthroughtheinvolvementoftraineduservolunteers.
Mysteryshoppershavebeendescribedas'under-cover'serviceusers.
"(Engage20202014;Involve2018;Ward2009)MultiCriteriaDecisionAnalysis(MCDA)ToolappliedincomplexdecisionmakingprocessestoidentifyasinglemostX(Engage20202014)169PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatepreferredoption,rankoptions,short-listoptions,ortomakeadistinctionbetweenacceptableandunacceptablepossibilities.
PublicconsultationTechniqueusedtoaskgroupsofpeopletodiscusstheiropiniononissues.
Unlimitednumbersofparticipantscanbesentinformationaboutthesubject,downloaditonline,andrespondviaemailorcommentonthewebsite.
X(Engage20202014;Involve2018)OnlinediscussionTakesplaceonlineandsupportuserstohaveconversationswithoneormorepeoplebytypingmessages.
Caninvolvebetweenonetoover500hundredparticipants.
X(Engage20202014;Involve2020)OpenSpace(Technology)Techniquefortheorganisationofparticipanteventforanunlimitednumberofparticipantsthatlastsbetweenonetofivedays.
Ithasthreeparts:(1)introductiontotheplenum;(2)thesessions;and(3)afinalroundwiththewholeplenum.
X(Engage20202014;Involve2018;TamarackInstitute2017;Ward2009)OpenHouseEventsthatpresentinitiativetoawiderpublicandsecureX(TamarackInstitute170PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateareactioninaninformalmanner.
Theyaremoreinformalthanatraditionalexhibitionandarelessstructuredthanworkshops.
Thepubliccanattendtheeventatanytimeatasetlocationforaspecifiedtimeanditcanlastbetweenafewhourstoseveralweeks.
Differentoptionswillbedisplayedusinginteractivedisplays.
2017;Ward2009)Opinionpoll/surveyQuantitativesurveythatmeasurestheopinionofasampleofpeopleonaparticulartopic.
X(Engage20202014;Involve2018;TamarackInstitute2017)ParticipatoryappraisalBroadempowermentapproachthataimstodevelopcommunityknowledgeandencouragesgrassrootactions.
Itoftenusesseveralvisualmethodssothatparticipantscanuseothermeanstocommunicate.
X(Engage20202014;Involve2018)ParticipatorybudgetingUmbrellatermthatencompassestheinvolvementofcitizenstoexercisepowerorinfluenceoverlocalbudgets,investmentprioritiesandeconomicspending.
ItcanbeX(Engage20202014;Involve2018;TamarackInstitute2017)171PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipaterunasaone-offorcyclicalprocess.
ParticipatorydesignIncludesvarioustoolssuchasconsultations,workshops,anddesignworkbooks.
Oftencarriedoutwithcitizenswhoareconcernedaboutaspecificissue.
Itstartswithaconsultationphasedwithindividualsandcommunityorganisations,followedbyaninteractivedesignprocess,whichincludesfieldtestswithusersoftechnologiesordevicesunderdevelopment.
X(Engage20202014)ParticipatoryGeographicInformationSystemsSociallearningtoolthatusesvisualrepresentationforfacilitationoftheevent.
Itisoftenusedfororganisationsinvolvedinparticipationandlocalstakeholderswhoareinterestedinlocalissues.
X(Involve2018)Participatorysensing/Volunteersensing/CitizenobservatoryTechniquesthatinvolvevolunteersfordatacollectionforresearch.
ItisfacilitatedbyICTplatformsandofteninvolveshandhelddevicessuchassmartphonesandisoftenusedforcitizenscience.
x(Engage20202014)ParticipatorystrategicplanningConsensus-buildingworkshopmethodthataimstobringcommunitiestogethertoexplainhowtheyenvisionX(Involve2018)172PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatethedevelopmentoftheircommunityororganisationinthenextyears.
Thenumberofparticipantsvariesbetweenfiveand50andtheeventtakesplaceovertwodays.
Therearefourstagestotheprocess:(1)creatingavisionforthefutureofthecommunity/organisation;(2)discussingthepotentialthreatsthatcreatebarrierstoreachingthevision;(3)agreeingmethodstoreachthevisionsandovercomingobstacles;and(4)implementationplanning.
ParticipatoryvideoSetofapproachesthatinvolvecommunitiesandgroupstoshapeandcreatetheirownfilms.
X(Engage20202014;Involve2018)PerspectiveworkshopSWOT-inspiredworkshoptoexploremyths,generatenewperspectives,andcreateguidelinesonatechnologyortechnologicaldevelopment.
Itoftenhas36-48participantsandlastsforoneandahalfdayandcombinesgroupandplenarysessions.
X(Engage20202014)PlanningforrealeventsFocussedonthe3-Dmodelofalocalarea,whereparticipantsdiscussstrengthsandweakness,andsuggestX(Engage20202014;Involve2018)173PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatehowtheywouldlikethecommunitytodevelop.
Pop-updemocracyTemporaryandsite-specificinstallationsforlocalandcivicparticipation.
Thegoalistocreatespacesandopportunitiesforlocalactivismandparticipationinthecommunity.
X(Involve2018)QmethodologyResearchtoolusedtogaininsightsintodiverseperspectivesandcanusedtoselectparticipantsforfurtherdiscussionabouttherelevantissues.
Itinvolvesthreemainstages:(1)definitionoftheconcourse:definingthesumofallstatementsaboutanissue;(2)interviewsandperspectiveidentificationwheretheissuesfromthefirststagearepresentedtoparticipantsandtheyareaskingtorankthestatements(Q-sortoftheQ-set);and(3)analysisandconclusions:statisticalanalysistofindcorrelationsbetweenindividualviewpoints.
X(Engage20202014)ReflexiveInteractiveDesignStakeholders,consumers,NGOsandcitizensdefinethemostimportantcharacteristicsofasustainableproduction-X(Engage20202014)174PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateconsumptionsystemandthendesignaproductionsystemtomeetthesedemands.
Carriedoutinseveralstages:(1)interviewswithnumerousstakeholdersinthesector;(2)collectivesystemanalysisinaworkshop;and(3)designatelierswhereinterestedparticipantsworktogetherfortwodaystoidentifytheimportantcharacteristicsofasustainableproductionsystemandtodesigntheproductionsystem.
Scenarioworkshop/ScenariotestingInstrumentusedforparticipatoryplanningusingdialogueandparticipationwithstakeholders,localcitizens,expertsandpolicymakers.
Theaimsaretofacilitatedialogue,exchangeexperienceandknowledgeaboutbarriersandenablers,improveunderstandingoftheissue,andfacilitateconsensusonthesolutionsproposedbytheinvolvedgroups.
Itisoftenatwo-daymeetingwith25-30stakeholders.
X(Engage20202014;TamarackInstitute2017)SciencecaféEventorganisedinaninformalinformationtocreateadialoguewithparticipantsX(Engage20202014)175PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipatefromthepublicandacademia.
Anexpertpresentsasubjectconciselyandopenlyandthenopensthefloorfordiscussion.
Themoderatoraimstofacilitatesharingofawiderangeofissuesrelatedtothediscussedsubject.
Science/ForumTheatreTechniquesthatusecreativeapproachestobringcomplexissuesalivewithaudienceswhowouldnottakepartinmoretraditionalprocesses.
Itencouragesaudienceinteractionandexploresdifferentapproachestodealwithanissue.
X(Engage20202014;Involve2018)Sentimentanalysis/mappingTechniquetogatherandanalysedataonpublicperceptions.
X(SmartMobilityLivingLab2020)SeriousgamingTechniquesthatserveastoolstogaincomplexknowledgeandstimulatereal-lifeeventsand/orprocesses,andtoprovidetheparticipantwithaproblem-solvingenvironmentfortraining.
Itcanalsobeusedtodevelopinnovativeproductsandservices.
X(Engage20202014)176PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateSociallisteningTechniquetocapture'natural'unsolicitedopinionsusingtextminingorcontentanalysis.
Examplesincludeanalysisofcommentsonsocialmedia(e.
g.
FacebookandTwitter).
X(Calabreseetal.
2019;Kelleretal.
2017)Socialmediadiscussion/TownHallSocialmediaplatformsareusedtoconveneagrouponlineonideas,solutions,andquestionsrelatedtoaparticulartopic.
Thedialoguesareoftenpostedwithahashtag(#)tokeepdialoguesspecifictoanevent.
X(TamarackInstitute2017)SurveyToolusedtogatherquantitativeorqualitativeinformationfromagivenpopulation.
Canbeconductedfacetofaceoronline.
X(Participedia2021a)Usercommittee/Citizens'panelTechniquethatinvolvesregularmeetingsbetweenusersandotherorcitizensandotherstakeholderstoformallymonitoranddirecttheresearchandinnovationprocessorchangestothelocalcommunity.
X(Engage20202014;TamarackInstitute2017)VideoVideodesignedtoeducateandinformthepubliconaparticulartopic.
X(Middleton2017)177PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateProvidesinformationonaparticulartopicinamannerthatisengagingandinteresting.
WorldCaféTechniquethatusesaninformalIsettingtoengagegroupswithinorganisationsandinthepublicspace.
Itfollowssevencoredesignprinciples:(1)setthecontext;(2)createahospitablespace;(3)explorequestionsthatmatter;(4)encouragecontributionsfromeveryone;(5)connectandlearnfromdiverseperspectives;(6)listentogetherforinsights,patternsanddeeperreflection;and(7)collectandshareshareddiscoveries.
x(Engage20202014;Involve2018)World-WideViewsDeliberativetechniquethataimstoengagecitizensindebatesaboutimportantandchallengingissuestogiveadvicetopolicymakers.
Citizensatnumeroussites(with100participantsateachmeeting)discussthesamepolicy-relatedquestionsforthesameissueonagivenday.
Citizensaregivenmaterialbeforehandandduringtheeventandthenvoteonpredefinedquestions.
Theviewsarereportedonlinex(Engage20202014;Involve2018)178PublicengagementtechniqueDescriptionHigh-levelcategorySourceCommunicateConsultParticipateforcomparisonandthenanalysedandpresentedtopolicymakers.
WorkshopIntensiveandstructuredmeeting,whichisledbyasessionleaderandfacilitatortoworkthroughanissueordevelopsolutions.
Aninteractiveworkshopenvironment,effectivegroupdynamics,visualaidsandfacilitatedsessionsaimtogatherhighqualityinformationoverasettimeperiodinordertomeetpre-determinedgoals.
x(Engage20202014;TamarackInstitute2017)WrittenconsultationWaytogatheroutsideopinionsanddiverseperspectivesonanissue.
Theyarecreatedtoengagewithpartiestogatherintelligence,ideasandviewpointsonanytypeofissue.
x(Engage20202014;Involve2018)Source:RANDEuropeanalysis179AnnexC:DescriptionofmethodologicalapproachInthissection,weprovideadetaileddescriptionofthemethodologicalapproachweadoptedinthestudy.
Thefollowingfigureprovidesanoverviewoftheresearchapproach.
Figure1OverviewofmethodologyandapproachSource:RANDEuropeAstheabovefigureshows,thestudywasconductedoverthreephases.
Phase1ofthestudyincludedafocussedreviewoftheliteraturewhichwascomplementedwithconsultationswithpublicengagementexperts,torefinethescopeoftheresearchquestions.
Attheconclusionofphase1,along-listofcasestudieswasidentifiedPhase2ofthestudyfocussedontheselectionoftencasestudiesfromthelong-listidentifiedattheendofphase1.
Fortheshort-listedcasestudies,thestudyteamreviewedadditionaldocumentsrelevanttothecasestudiesandconductedinterviewswithstakeholdersassociatedwitheachcasestudy.
Thefindingsfromthedocumentreviewandtheinterviewswerethenincorporatedaspartofthecasestudywrite-ups.
180Phase3ofthestudycoveredcross-analysisofthefindingsfromthedeskresearchinPhase1andthecasestudiesinPhase2.
Thiscross-analysiswasconductedviaavirtualsynthesisworkshopbythecoremembersofthestudyteam.
Theresultingfindingsformthebasisofthenarrativepresentedinthisreport.
Eachofthesephasesisdiscussedinmoredetailbelow.
C.
1.
Phase1:DeskresearchC.
1.
1.
FocussedreviewoftheacademicandgreyliteratureWeconductedafocussedreviewoftheavailableacademicandgreyliteratureonpublicengagementtechniques,technologicalinnovation,andtheeffectivenessandimpactofthetechniques.
Aspartofthisreview,wedrewonsomeofthekeyprinciplesofasystematicreviewtotakearobustandreplicableapproachtosearchingandreviewingtheliterature.
Inordertoensurerapidturnaroundofthefindings,weadoptedapragmaticapproachtothescopeandcoverageofliterature,limitingstudyinclusionbyusingarangeofcriteriathatwereadjustedinresponsetothevolumeofliteratureidentified,enablingthemtobeconductedwithinamorelimitedtimeframe.
Theremainingsectionssetoutourapproachacrossfourtasks:ConductingsearchesScreeningExtractionAnalysisConductingsearchesWedevelopedasearchstrategywithexpertinputfromRANDKnowledgeServicesandBEIS/BRE.
OursetofsearchtermsandoursearchcriteriaarepresentedinTable5andTable6.
ThesearchwasconductedinScopus.
Thepublicationtimeframewasrestrictedto2016onwardstocaptureliteraturefromthepastfiveyears,andonlyhigh-qualityacademicpublications(bookchapters,articles,reviewsandconferenceproceedings)publishedinEnglishwereconsidered.
Atotalof820potentiallyrelevantstudieswereidentifiedforscreening.
Inadditiontotheformalsearches,weconductedtargetedsearchesofthegreyliterature.
SearcheswereconductedinGoogleandGoogleScholarusingasimilarsearchstrategyandinclusion/exclusioncriteria(Table6)butalsotargetingkeypublicengagementtechniques/methodsand/oraspecifictechnologicalinnovationtowhichthepublicengagementhadbeenapplied.
Anexampleofatargetedsearch181includes:("publicengagement"AND(technologyORtechnologiesOR"technologicalinnovation").
Beyondtheformalsearches,wealsoidentifiedadditionalliteraturethrough'snowballing'.
Thisisaprocessinwhichasmallnumberofadditionalrelevantstudiesarefoundthroughthequickreviewofthereferencelistofstudiesidentifiedforinclusionattheendofthescreeningstage.
Fromthese,atotalof57potentiallyrelevantarticleswereidentifiedforscreening.
Table5SearchtermsusedinthefocussedliteraturereviewSearchcategorySearchtermsNotes1Publicengagement(publicORcitizenORuserORlayORcommunity)w/2(engagementORscienceORcrowdsourc*ORinvolve*ORparticipationORperception*ORopinion*ORdialogue*ORattitude*ORunderstandingORawarenessORacceptabilityORperspective*)ThissetofsearchtermsseekstoidentifydifferentformsofpublicengagementAND2Technologicalinnovationtechnolog*w/2(newORinnovat*ORnovelORemerg*OR"cuttingedge"OR"state-of-the-art"ORfuture)ThissetofsearchtermsseekstoidentifytechnologicalinnovationsornewandemergingtechnologiesAND3RegulationgovernanceORregulat*ORpolic*ORlegislat*ORethic*ORrisk*ORtrust*ORmistrust*ORdistrust*ORskeptic*ORoversightThissetofsearchtermsseekstoidentifyexamplesthatconsidersomeformoflinkbetweenpublicengagementandimpacts182SearchcategorySearchtermsNotesonregulation,governanceorpublictrustScreeningWescreenedarticlesbytitleandabstractforrelevanceagainstpredefinedinclusionandexclusioncriteria(Table6).
Table6InclusionandexclusioncriteriaCriterionIncludeExcludeRationaleTopicrelevanceStudiesaddressingpublicengagementtechniquesorapproachesinrelationtotechnologicalinnovation/emergingtechnologies,andanyevidenceoftheirimpactoreffectivenessStudiesaddressingpublicengagementtechniquesorapproachesincontextsoutsideoftechnologicalinnovation(e.
g.
communicationofresearch)WeproposetofocusonthekeyaspectsstipulatedinthespecificationGeographicallocationAllcountriesN/AWewillnotrestrictthesearchtoanyparticularcountriestoprovideaglobaloverviewofpublicengagementtechniquesYearofpublication2016onwards2015orearlier(withsomeexceptions,e.
g.
,anyseminalstudiesreferencedWeproposethistimeperiod(5years)inordertooptimisetheidentificationof183CriterionIncludeExcludeRationaleinmultiplepublicationswithinthe5-yeartimeframe)novelandinnovativepublicengagementtechniquesacrossaspectrumofrelevanttechnologiesandlimitspuriousresults(thisisparticularlyrelevantintheareaoftechnologicalinnovationwhereresearchandknowledgeisadvancingquickly)StudytypePeer-reviewedjournalpublications,conferenceproceedings,greyliteraturewithclearauthorshipDocumentswithoutclearorganisationalauthorship,theoreticalwork,letters,editorials,commentsoropinionpieces,bookreviews,sub-PhDlevelthesesThe'studytype'selectioncriteriaareintendedtooptimisethequalityofsourcesintheliteraturesearchLanguageEnglishOtherlanguagesItisexpectedthatliteraturesearchesapplyingtheEnglish-languagesearchtermswillyieldmostlyEnglish-languagesources184Theinclusion/exclusioncriteriawereappliedinthreestages:Criteriawereappliedonthetitles.
Thoseappearingtofitthecriteria,orwheretherewasuncertainty,wereincluded.
Theabstractsofthesetitleswereread,andinclusioncriteriaappliedagain.
Thosefittingthecriteriaorthosewheretherewasuncertaintywereincluded.
Inthesefirsttwostages,wewereoverinclusivetoavoidexcludingpotentiallyrelevantstudies.
Reviewersretrievedfullreportsofstudiespassingthefirstround.
Eachcriterionwasthenappliedagainatthefulltextlevel.
Followingscreening,atotalof42studieswereidentifiedasmeetingtheinclusioncriteria.
ExtractionInthisstage,informationwasextractedfromeachincludedpublicationtofacilitatecross-analysisagainstthekeystudyquestionsandthemes,andthequalityofthestudiesincludedassessedtoinformthatanalysis.
Followingpiloting,researchersindependentlyrecordeddataabouteachselectedpapermeetingtheinclusioncriteria,includingbothgeneralinformationonthepublicationandinformationontheelementsofeachstudyquestionitaddressed.
WecapturedinformationfromeachincludedstudyinastandardtemplateinExcelcoveringthedifferentelementsofourconceptualframeworkandinlinewiththeaimsandobjectivesofthisstudy:GeneralinformationabouttheevidencesourceAuthor,yearandstudytitleBriefsummaryStudytype(e.
g.
experimental/randomisedcontrolledtrial,cohort/longitudinal,cross-sectional/factorial,meta-analysis,synthesis)Methodology(e.
g.
qualitativedatacollection,quantitativedatacollection,mixedmethods)Publicationtype(e.
g.
journalarticle,review,bookchapter)Coverageofstudy(temporal,geographic)EvidenceonpublicengagementtechniquesandapplicationsinrelationtonewtechnologiesandtechnologicalinnovationMethodofpublicengagementtechnique(s)usedinthestudy185ThenewtechnologyortechnologicalinnovationthatwasthefocusoftheengagementThesectorthatthetechnologyortechnologicalinnovationwasbeingappliedtoThecountryinwhichtheengagementprocesstookplaceThestakeholdergroup(s)involvedintheengagementHowandwhenthepublicengagementtechniquewasapplied–informationaboutthepublicengagementprocessandapplicationEvidenceontheimpactofpublicengagementtechniquesTheimpactoroutcomeofthepublicengagementEvidenceontheeffectivenessofpublicengagementtechniquesandanylearningsWastheeffectivenessofthepublicengagementtechnique(s)(formally)assessedEvidenceoftheeffectivenessofthetechnique(howandwhywasiteffectiveornoteffective)Evidenceonthefollowingcriteriainrelationtothepublicengagementtechnique,e.
g.
cost,burden,speedofdeployment,inclusiverepresentationAnykeylessonslearnedfromany(formal)assessmentofeffectivenessofthepublicengagementtechnique(s)OthercommentsAnalysisTheevidencewasbroughttogetherusingaframeworksynthesisapproachbasedontheframeworksetoutinourevidenceextractionapproach.
Eachelementoftheframeworkwasexploredinitiallybyamemberofthestudyteamtoidentifythekeytrendsandissuesemerging.
Thefindingswerethendiscussedwithothermembersoftheteamandthenfurtherexplored.
Throughaniterativeprocessofanalysisanddiscussionwewereabletoidentifyasetofkeyemergingfindingsthataresetoutinthisdocument.
Aspartoftheextractionandanalysisofthefocussedreviewoftheliterature,wecollatedalong-listofpotentialstudiesasperthestudyobjectives.
Thislistwasfurtheraugmentedbasedontheinputsreceivedfromthepublicengagementexpertswithwhomwespokeaspartofthescopingconsultations.
186C.
1.
2.
ScopingconsultationsWeconductedsevenscopingconsultationswithindividualsinvolvedinorwithknowledgeofpublicengagementaroundtechnology.
Theaimofthisengagementprocesswasto:(1)strengthenourbaselineunderstandingof,andinsightsspecificto,thepublicengagementtechniquesfortechnologicalinnovationintheUK;(2)gatherinformationrelatedtothesuccess/effectivenessofengagementinitiatives;and(3)obtainexamplesofpublicengagementfromthesestakeholdersthatcouldfeedintothecasestudiesinPhase2ofthestudy.
Interviewswereconductedbytelephoneore-mail.
Theinterviewsweresemi-structured,therebyensuringasimilarsetofquestionswereaskedofallintervieweesbutallowingforemergentissuestobeexplored.
Theorganisationsweengagedwithareasfollows:AdaLovelaceInstitute;Involve;Nesta;RoyalSociety;SocietyInside;UKRI;andaUK-basedHigherEducationInstitution(HEI).
AllconsultationswereconductedundertheprinciplesofinformedconsentinlinewiththerequirementsoftheEUGeneralDataProtectionRegulationrequirementsandtheEthicalAssuranceforSocialResearchinGovernmentprinciplesAttheconclusionofPhase1,aninterimreportprovidingtheemergingfindingsfromthefocussedreviewofliteratureandscopingconsultationswasprovidedtoBEIS/BREforreview.
C.
2.
Phase2:CasestudiesC.
2.
1.
CasestudyselectionAsindicatedintheprevioussection,aspartofPhase1ofthestudy(focussedreviewandscopingconsultations),along-listofpotentialcasestudieswascompiledbasedontheresultsofthefocussedliteraturereviewandscopingconsultationsshowcasingaspectrumofreal-worldpublicengagementtechniquesinrelationtotechnologicalinnovation.
Theexamplesspanneddifferenttechnologyareas,countries,sectors,organisations,timeperiodsandpublicengagementtechniques.
Foreachoftheseexamples,wecollatedthefollowinginformationtomaketheselectionforthefinalcasestudies:technologyarea,sector,organisation(s),timescale,publicengagementtechnique(s)/method(s),andanyinformationrelatedtothepotentialeffectiveness(fornon-UKexamples,wealsonotedthecountry).
Initialrecommendationsfromthelong-listofpotentialcasestudiesTheinitiallong-listcontained36potentialcasestudies.
Onthebasisofthesecasestudies,threecoremembersofthestudyteamparticipatedinavirtualworkshoptoconductaholisticassessmentofthelong-listwithaviewtoarrivingataninitialsetof187recommendedcasestudiestodevelopfurther.
Inthisworkshop,eachofthecasestudieswasexaminedfortheirfitwiththestudyobjectives,rangeofpublicengagementtechniquesused,technologicalinnovationscovered,evidenceofeffectivenessandtheavailabilityofstakeholdersforaninterview.
Inaddition,thesectorsimpactedandthecountry/regioninwhichthepublicengagementwascarriedoutwerealsoconsidered.
Anothercriterionforshort-listingcasestudieswasthestrengthofevidenceintheliteratureassociatedwitheachcasestudy;someexampleshadmoreextensiveandrigorousevidenceassociatedwiththemthanothers.
Basedonthesefactors,thecoremembersofthestudyteamassignedapossiblerecommendationof'Yes',No',and'Maybe'toeachcasestudyforfurtherin-depthinvestigation.
Outofthepotential36casestudies,11casestudieswereidentifiedas'Yes',13weremarkedas'No',and12werehighlightedas'Maybe'forfurtherconsiderationanddiscussionwithBEIS/BRE.
Discussionandshort-listingofthecasestudieswithBEIS/BREOnthebasisoftheinitiallong-listofcasestudiesandpotentialrecommendationsforinclusion/exclusion,wehadadetaileddiscussionwithBEIS/BRE.
Aspartofthisdiscussion,thelong-listofcasestudieswasexaminedforvariouscriteria,includingnovelpublicengagementtechniques,countryoforigin,potentialavailabilityoftheevidence,andthetechnologicalinnovationcoveredbythepublicengagementtechniques.
Atthisstage,basedontheevidencefromthefocussedliteraturereviewandscopingconsultationswehadidentifiedtwoadditionalpotentialcasestudiesaspartofthelong-list(whichwasnow38casestudies),whichwerealsodiscussed.
BEIS/BREdecideduponaninitialshort-listof10casestudiesonthebasisofwhichweapproachedstakeholdersassociatedwiththecasestudiesforinterviews.
Duetounavailabilityofstakeholdersforinterviews,oneofthecasestudiesintheinitialshort-listwasdeemedunattainableandneededtobereplaced.
Weproposedfourpotentialalternativesfromourinitiallong-list,includingthreefurther,previouslyunidentifiedcasestudiesinthelong-list(whichwasnow41casestudies).
AnnexAcontainsthefinalshort-listof10casestudiesagreedwithBEIS/BRE.
Thelong-listofcasestudiesisincludedinAnnexDforreference.
C.
2.
2.
InterviewswithcasestudystakeholdersWiththeshort-listofcasestudiesfinalised,weapproachedmultiplestakeholdersassociatedwitheachofthecasestudiesforaninterview.
Inordertoensureabalancedperspectiveontheeffectivenessofthepublicengagementtechniquesusedandimpactachieved,wherepossiblewetargetedtheorganisersofthepublicengagementexercisealongwithparticipantsandtechnologypartners.
188Theinterviewswereconductedinasemi-structuredformatundertheprinciplesofinformedconsentaspertheGDPRandtheEthicalAssuranceforSocialResearchinGovernmentprinciples.
Toprotecttheanonymityoftheinterviewees,theyhavenotbeennamedinthisreport.
Acrossthe10casestudies,weconducted20interviewswithvariousstakeholdersassociatedwiththecasestudies.
Weincludebelowtheindicativesetofquestionscoveredwiththeintervieweesaspartofthediscussion.
Theintervieweradaptedthequestionsinlinewiththeinterviewee'sexperienceandexpertise.
WhatwasthepurposeofthepublicengagementHowwasthepublicengagementcarriedoutHoweffectivewasthepublicengagement(e.
g.
intermsofinfluencingregulation,alternativestoregulation,publictrust,newbusinessmodels)WhatlessonscanbelearnedfromthisexampleArethereanykeypiecesofliteraturethatweshouldincludeorfocusonregardingthisexample(thiscanincludeacademicliteratureorrelevantreportsfromorganisations)C.
2.
3.
ReviewofselecteddocumentationandarticlesrelatedtocasestudiesInadditiontotheinterviews,wealsoconductedhighlytargetedsearchestoidentifydocumentationsandarticlesrelevanttoeachcasestudy.
Thesesearchesfocussedonidentifyingreportsandpeer-reviewedarticlesdiscussingevaluationofthepublicengagementexerciseandalsoanyevidenceofeffectivenessofthepublicengagementtechniqueemployed.
Thesearchesandthereviewofthedocumentationwascarriedoutinparallelwiththestakeholderinterviews.
Wealsodrewontheinputsofthestakeholdersinterviewedtoidentifyrelevantliterature.
ThearticlesreviewedarecitedaspartofthecasestudydescriptionsincludedinAnnexA.
Thefindingsfromtheinterviewswithrelevantstakeholdersandthedocumentreviewforthecasestudieswereusedtowritecasestudydescriptions.
Alongwiththeinsightsgatheredaspartofthefocussedliteraturereview,thecasestudydescriptionswereusedforthesynthesisworkshopandthefinalreportwritingaspartofPhase3ofthestudy.
189C.
3.
Phase3:AnalysisC.
3.
1.
SynthesisworkshoptotriangulateevidencefromPhases1and2CoremembersofthestudyteamparticipatedinavirtualworkshopaimedateffectivesynthesisoftheevidencefromPhases1and2.
Theevidencefromthefocussedliteraturereview,intermsofthedifferenttypesofpublicengagementtechniques,thediscussionintheliteratureontheeffectivenessofthetechniques,andthesuitabilityofthetechniquesdependingontheobjectivesoftheengagementexercise,wascross-analysedwiththeobservationsfromthecasestudiesontheuseofthetechniquesinpractice.
Foreachcasestudy,weconsideredthepublicengagementtechniqueused,thetechnologyarea,thegeographicalcontext,thedurationoftheexercise,effectivenessintermsofengaginggeneralpublic(andsustainingtheengagement),changeinreal-worldpractice,andpotentialimpact(e.
g.
intermsofincreasedpublictrustorimprovedunderstandingofthetechnologicalinnovation).
Thecross-analysisalsoconsideredpotentialgapsintheevidence,inparticularsomecasestudiesappeartohaveemployedmultipletechniquesandtheextenttowhichthetechniquesworkedeffectivelyinconjunctionwitheachotherwasalsoacrucialaspectofthediscussion.
GuidedbytheevidenceinthefocussedreviewoftheliteratureinPhase1,theworkshopcoveredpossibletypologiesofthepublicengagementtechniquesandhowthesealignedwiththefocusofthestudyonpublicengagementinrelationtotechnologicalinnovation.
Althoughimplicationsofthetechniquesbeyondtechnologicalinnovationwereconsidered,theworkshopoutcomewastoclustertheevidencefromPhase1andPhase2toidentifykeylessonslearntandinformthechoicesavailabletoregulatorsandpolicymakers.
Theoutcomeoftheworkshopintermsofclusteringoftheevidenceandpotentialgapsinithasbeenusedtostructurethenarrativepresentedinthemaintextofthisreport.
C.
3.
2.
FinalreportwithkeyfindingsAspartofthefinalreporting,wecollatedthefindingsfromPhase1andPhase2ofthestudyonthebasisofthethreehigh-levelresearchquestionsthatguidethisstudy.
Foreachquestion,theinsightsfromtheliteraturereviewformedthebaselineofthefindingsreported.
Wedrewonthecasestudiesforadditionalpracticalinsighttounderstandtheeffectivenessofthetechniquesandcontextinwhichthetechniqueismostlikelytobeuseful.
Thekeyfindingshavebeengroupedtoenablequickdiscoveryoftherangeofpublicengagementtechniquesemployedfortechnologicalinnovation,evidenceofimpactofthedifferentengagementtechniques,andanyformalassessmentsontheeffectivenessofthetechniques.
Wehaveusedmessage-ledheadingsinthesub-190sectionstocommunicatethefindingsinasuccinctmannerforanon-expertreader.
Detailedcasestudydescriptionshavebeenincludedintheannextoenableareadertounderstandspecificuseofthetechniquesinmoredepthifrequired.
C.
4.
CaveatsoftheanalysisTheanalysispresentedhereissubjecttosomecaveatsrelatedtotheapproach,thescopeoftheliteratureandtheanalysis.
Keylimitationsareoutlinedbelowandthefindingsfromthisstudyshouldbeconsideredbearingtheselimitationsinmind.
Thestudyconductedafocussedreviewratherthanasystematicreview.
Thisplaceslimitationsonthecoverageofliteratureincludedinthestudy.
Theremaybeimportantstudiesthathavenotbeenincluded,eitherbecausetheywerenotidentifiedthroughoursearchstrategyorbecausetheyfelloutsideofourinclusioncriteria(e.
g.
bydate).
Wenotethatthevolumeofliteratureinthefieldherehasbeenasignificantchallengeandwehavehadtoplaceadditionallimitationsonthescopeofourapproachtomakethestudyfeasiblewithintimeframes(e.
g.
byfocusingonmorerecentliterature).
Weareconfident,however,basedontheapproachtakenandourscopingconsultationprocess,thatouranalysisprovidesafairandrelativelyrepresentativepictureofthecurrentstateoftheevidence.
Becauseofthecomplexityandrichnessoftheliterature,itislikelythatthereareanumberofelementsevenwithinthestudiesincludedthathavenotbeenfullyexploredwithinthescopeofthisreport.
Wehavebalancedlengthandcomplexitywithcomprehensiveness,aimingtoprovideanoverallpictureofthekeyemergingissueswithafocusonthestudyquestionsidentifiedforthiswork.
Whileweaimedtoensurethatwecapturedasmanyrelevantandinterestingexamples,thelistofcasestudieswasnotintendedtobedefinitiveorexhaustive.
Rathertheyservedasconcrete,illustrativeexamplesofhowpublicengagementtechniqueshasbeenusedinthecontextoftechnologicalinnovation.
Thefinalsetofcasestudieshighlightaspectrumofpublicengagementtechniquesrangingfromtraditionaltechniquestomoreatypicaltechniques,inrelationtotechnologicalinnovation.
Inaddition,theexamplesspanneddifferenttechnologyareas,sectors,organisations,andtimeperiods.
191AnnexD:Long-listofcasestudiesillustratingtheuseofpublicengagementtechniquesinthecontextoftechnologicalinnovationTitle,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementPublicviewsofmitochondrialreplacementtherapyYear:2012Country/region:UnitedKingdomMitochondrialreplacementtherapySector:MedicalPublicengagementtechnique:Deliberative:workshopBriefdescription:TheHumanFertilisationandEmbryologyAuthorityusedamulti-methodapproach,includingworkshops,apublicsurvey,openmeetingsandfocusgroupstodeterminepublicacceptabilityofmitochondrialreplacementtherapy.
PublicviewsofmachinelearningYear:2017Country/region:UnitedKingdomMachinelearningSector:Technology/ITPublicengagementtechnique:Onlinesurvey,anddeliberativemethods:publicdialogueanddiscussiongroups,onlinecommunityBriefdescription:TheRoyalSocietyconductedapublicengagementexerciseconsistingofaquantitativesurvey,publicdialogueanddiscussioneventsandanonlinecommunityonmachinelearningtoraiseawareness,tounderstandviewsheldbythepublicandidentifykeysocial,ethical,scientificandtechnicalissues.
PublicattitudestowardsGMfoodsGMfoodsSector:AgriculturePublicengagementtechnique:Face-to-facesurvey192Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementYear:2019Country/region:UnitedKingdomBriefdescription:BritishsocialattitudessurveyaskedaboutattitudestowardsGMfoods(e.
g.
iftheUKshouldgrowGMfood,iftheadvantagesoutweightherisks).
PublicattitudestowardsnanotechnologyYear:2014Country/region:UnitedKingdomNanotechnologySector:Technology/ITPublicengagementtechnique:Face-to-facesurveyBriefdescription:SurveyaskedwhethertheadvantagesofnanotechnologyoutweighstherisksPublicattitudestowardsAI(generaluseofAI)Year:2019Country/region:GlobalAISector:Technology/ITPublicengagementtechnique:OnlinesurveyusingonlineplatformBriefdescription:GlobalsurveyonattitudestowardsAI(e.
g.
concernaboutuseofAI,whetherAIshouldbebanned).
Thesurveywasconductedin27countriesviaGlobalAdvisor,theonlinesurveyplatformofIpsos.
PublicattitudestowardsAI(ethicaluseofAI)Year:2019Country/region:UnitedKingdomAISector:Technology/ITPublicengagementtechnique:Deliberativei.
e.
citizens'juryBriefdescription:TheRSAandDeepMindconvenedacitizens'jurytodeliberateontheethicaluseofAI,andinparticular,itsusetohelpmakedecisions.
PublicattitudestowardsAIandassociatedtechnologiesAISector:Technology/ITPublicengagementtechnique:Deliberative:workshopBriefdescription:193Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementYear:2018Country/region:UnitedKingdomInvolveandNestadeliveredaone-dayworkshoptoexplorehowmembersofthepublicmightapproachsomeofthebiggestquestionsaroundAIandassociatedtechnologies.
Publicengagement,awareness-raising,communicationandthetestingofpoliciesYear:2016Country/region:EUNanotechnologySector:MedicalPublicengagementtechnique:Seriousgame:scenariosBriefdescription:TheScenarioExplorationSystem,designedbytheEuropeanCommission,isaboardgamedesignedtogetpeoplethinkingaboutthefuture,fromdifferentviewpoints(policymaker,citizen,businessorcivilsocietyorganisation).
PublicperceptionsoftheuseofvirtualrealityinhealthcareYear:2016Country/region:USAVirtualrealitySector:MedicalPublicengagementtechnique:SociallisteningBriefdescription:ResearchersfromtheCedars-SinaiMedicalCenterinLosAngelesanalysedover2,400commentsonavideoonFacebookabouttheuseofVRinhealthcare.
PublicdebateontheregulationofUberinTaiwanYear:2015Country/region:TaiwanRide-hailingapps(Uber)Sector:TransportPublicengagementtechnique:Onlinedeliberativeandcrowdsourcingplatform:opensource,onlinesurveytoolBriefdescription:ThevTaiwanplatformisbasedonpol.
is,anopensource,onlinesurveytooldesignedtoengagethepublicinlarge-scaledeliberation.
In2015,the194Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementplatformorganisedadebateontheregulationofUberinTaiwan,attracting1,737participants.
PublicdebateaboutgoverningartificialintelligenceYear:2017Country/region:GlobalAISector:Technology/ITPublicengagementtechnique:CrowdsourcingBriefdescription:In2017theAIInitiativelauncheda7-monthonlineGlobalCivicDebate.
Thesoftwareituses(Assembly)usesalgorithmstostructureandorganiseparticipantcontributions.
PublicperspectivesonmoraldecisionsmadebyAIYear:2018Country/region:USADriverlesscarsSector:TransportPublicengagementtechnique:SeriousgameBriefdescription:MIThavedevelopedMoralMachine,anonlineplatformforgatheringhumanperspectivesonmoraldecisionsmadebymachineintelligence,suchaself-drivingcars.
PublicparticipationinclimatepolicyYear:2013Country/region:UnitedKingdomDecarbonisingtheeconomySector:EnergyPublicengagementtechnique:OnlineexperimentalplatformBriefdescription:TheDepartmentforEnergyandClimateChangehavedevelopedanonlinetoolthatusesgamifiedandscenariobasedmethodsforthepublictoinputonclimatechangepolicies.
PublicattitudestowardsgenomicdatasharingYear:OngoingCountry/region:UnitedKingdomGenomeeditingSector:MedicalPublicengagementtechnique:OnlinesurveywitheducationalvideosBriefdescription:The'YourDNA,YourSay'projectisaglobalonlinesurveygatheringpublicattitudestowardsgenomicdatasharing.
195Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementPublicdialogueongenomeeditingYear:2016Country/region:UnitedKingdomGenomeeditingSector:MedicalPublicengagementtechnique:Deliberative:publicdialogueBriefdescription:TheNuffieldCouncilonBioethicsandSciencewiseorganisedaworkshopwithpolicymakers,fundersanddialoguespecialistsongenomeeditingtechniques.
PublicawarenessonsyntheticbiologytechnologyYear:2012Country/region:CanadaSyntheticbiologytechnologySector:MedicalPublicengagementtechnique:Deliberative:scientificcaféBriefdescription:TheauthorsconductedfiveScienceCafésacrossCanadatogaugepublicawarenessofsyntheticbiologytechnologyanditspotentialapplications.
ExploringdeterminantsofsolarpoweradoptionYear:2014Country/region:CanadaSolarpowerSector:EnergyPublicengagementtechnique:SurveyBriefdescription:Nationalsurveyof2065Canadianresidentstoidentifythedeterminantsoftechnologyadoptionintentionwiththeexemplarycaseofrooftopsolar.
Publicacceptanceof,andattitudestowards,driverlessvehiclesYear:2017Country/region:UnitedKingdomAutonomousvehiclesSector:TransportPublicengagementtechnique:Experiential:livetrialsBriefdescription:GATEwayProjectoverseenbytheTransportResearchLaboratoryandotherpartnerstotestprototypeautomatedvehiclesinareal-worldenvironment.
196Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementGMNationYear:2002Country/region:UnitedKingdomGMcropsSector:AgriculturePublicengagementtechnique:Deliberative:publicmeetings&workshopsBriefdescription:Apublicengagementprocessoverseenbyagovernment-appointedsteeringboardtodiscussgeneticmodificationoffoodcrops.
NanoJuryUKYear:2005Country/region:UnitedKingdomNanotechnologySector:MultiplePublicengagementtechnique:Deliberative:citizen'sJuryBriefdescription:This'citizen'sjury'broughttogethertwentypeople,chosentorepresentabroadcrosssectionofsocietybutalsoinclusiveofanumberofethnicitiesandreligions,todiscussissuessurroundingnanotechnology.
Publicconsultationonpreimplantationgeneticdiagnosis(PGD)Year:2005Country/region:UnitedKingdomPGDfornon-medicalsexselectionSector:MedicalPublicengagementtechnique:PublicconsultationBriefdescription:In2005,theHumanFertilisationandEmbryologyAuthorityranaconsultationonoptionsfortheregulationofsexselectionthroughpreimplantationgeneticdiagnosisandothermethods(suchasspermsorting).
DialogueononlinetargetingYear:2019Country/region:UnitedKingdomAISector:Technology/ITPublicengagementtechnique:Deliberative:workshop;OnlinesurveyBriefdescription:TheCentreforDataEthicsandInnovation,withsupportfromSciencewise,ranadialoguewhichexploredpublicattitudestowardsonlinetargeting.
AI360Year:2019Country/region:DenmarkAISector:Technology/ITPublicengagementtechnique:Deliberative:workshopandcitizenconsultationBriefdescription:AI360,aninitiativewithintheHumanBrainProject,wasanew,two-stageexperimentinpublicengagement.
Thisinnovatedontheclassic,deliberativedialoguemodelthat197Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementDBThadpreviouslyappliedtoavarietyofothertopicsinscienceandtechnology.
ConsumerAttitudestowardsEmergingTechnologiesYear:2019Country/region:UnitedKingdomNewfoodtechnologiesSector:FoodPublicengagementtechnique:Deliberative:publicdialogueBriefdescription:TheFoodStandardsAgencyconductedfourpublicdialogueeventstoexploretheviewsofmembersofthepublictowardsfouremergingtechnologies:GMfoods,nanotechnologyinfoods,foodfromclonedanimalsandculturedmeat.
PublicdialogueongenomicmedicineYear:2018Country/region:UnitedKingdomGenomeeditingSector:MedicalPublicengagementtechnique:Deliberative:PublicdialogueBriefdescription:GenomicsEnglandranapublicdialogue,withsupportfromSciencewise,toexplorepublicattitudestomainstreaminggenomicmedicineintotheNHS.
PublicdialogueondatascienceYear:2015Country/region:UnitedKingdomDatascienceSector:GovernmentpolicymakingandserviceprovisionPublicengagementtechnique:Deliberative:workshops;SurveyBriefdescription:TheGovernmentDataSciencePartnershipranapublicdialogueondatascienceethics,includingdeliberativeworkshops,asurveyandanonlineengagementtool/quiz,toinformaDataScienceEthicalFramework.
EPSRCpublicdialogueonQuantumTechnologiesYear:2017QuantumtechnologiesSector:Notspecified/multiplePublicengagementtechnique:Deliberative:workshopBriefdescription:KantarMediacarriedoutworkshopstobetterunderstandhowthepublicviewsandfeelsabouttheQuantumTechnologies(QTs)currentlybeingdevelopedbyresearchersandtheirpotentialapplications.
198Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementCountry/region:UnitedKingdomCitizendeliberationsonclimateandenergyYear:2009Country/region:GlobalClimatechangeSector:Sustainability/climatechangePublicengagementtechnique:Deliberative:onlineplatformBriefdescription:WorldWideViewsonGlobalWarming,heldonSeptember26,2009,wasthefirst-everglobeencompassingdemocraticdeliberationinvolvingroughly4000citizensin38countries.
ViewsonthefutureofneuralinterfacesYear:2019Country/region:UnitedKingdomNeuralinterfacesSector:Notspecified/multiplePublicengagementtechnique:Deliberative:publicdialogueBriefdescription:TheRoyalSocietyranaseriesofpublicdialogueslookingatneuralinterfacestoinformtheSociety'sthinkingontheregulatoryandethicalissuesassociatedwiththeemergingfield.
ByoLogyc:exploringfuturescenariosthroughimmersiveexperiencedesignYear:2012-13Country/region:CanadaBiotechnologySector:ConsumerHealthPublicengagementtechnique:Experiential:immersiveexperienceBriefdescription:ByoLogycwasanexperientialfuturesproject,whichexploredthepossibleimpactsofbiotechnologyinthecontextofconsumerhealth.
Theprojectdevelopedafictionalcompany,fictionalemployeesandliveeventsthatparticipantscouldengagewith.
PublicattitudestofacialrecognitiontechnologyYear:2019FacialrecognitiontechnologySector:Multiple(surveillance)Publicengagementtechnique:SurveyBriefdescription:AdaLovelaceInstitutecommissionedYouGovtoundertakethefirstsurveyofitskindtounderstandpublicattitudesintheUKtotheemergingpublicandprivate199Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementCountry/region:UnitedKingdomsectordeploymentoffacialrecognitiontechnology.
CollaborativestorytellingthatexplorestheimpactofclimatechangeYear:N/ACountry/region:GlobalClimatechangeSector:Sustainability/climatechangePublicengagementtechnique:CrowdsourcingBriefdescription:TheFutureCoaststorytellingprojectdevelopedbyUSgamesdesigner,KenEklund,incollaborationwiththePoLARInstituteatColumbiaUniversity:aninternationalonlinedigitalstorytellingprojectwhichasksaudiencestoimagineandcreatevoicemailsfromarangeofpossibleclimaticallychangedfutures.
CAVpublicacceptabilitydialogueYear:2018Country/region:UnitedKingdomConnectedandautonomousvehiclesSector:TransportPublicengagementtechnique:Deliberative:publicdialogueBriefdescription:TheUKGovernmentinpartnershipwithSciencewise,fundedbyUKResearchandInnovation,commissionedTraversetodeliverasetofpublicdialoguesonattitudestowardsconnectedandautomatedvehicles(CAVs).
ExploringpublicattitudestowardsgeneeditingYear:2018Country/region:UnitedKingdomGeneeditingSector:MedicalPublicengagementtechnique:Deliberative:workshopsBriefdescription:GeneGapwasaprojecttoexplorepeople'sattitudestowardsgeneediting.
FivefacilitatedworkshopswererunwithdiversecommunitiesacrosstheUK.
BioMetricMirrorYear:2018-19Country/region:AustraliaFacialanalysistechnologySector:Multiple(surveillance)Publicengagementtechnique:Experiential:interactiveapplicationBriefdescription:BiometricMirror,acasestudythatexploredopinionsabouttheethicsofanemergingtechnology.
Theinteractiveapplication200Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementdistinguisheddemographicandpsychometricinformationfrompeople'sfacialphotosandpresentedspeculativescenarioswithpotentialconsequencesbasedontheirresults.
ImmersiveenvironmentsinthecontextofaninnovationprocessforsmartcitiesYear:2015-16Country/region:FranceSmartcitiesSector:UrbangovernancePublicengagementtechnique:Experiential:immersiveexperienceBriefdescription:FabLivingLabPlatform:immersiveenvironmenttoengagedifferentstakeholders.
Citizens'workshopsarehostedandsupportedbytheLF2Lplatform.
TheLF2LinnovationprocesswasusedinthecontextofSmartCitypolicies.
RapidonlinedeliberationonCOVID-19technologiesYear:2020Country/region:UnitedKingdomCOVID-19contacttracingappSector:PublichealthPublicengagementtechnique:Deliberative:onlineusingZoomBriefdescription:Traverse,AdaLovelaceInstitute,InvolveandBangtheTabletrialledanewmodelofrapid,onlinedeliberationthatwouldenablecitizenstofeedintothedevelopmentofpolicyformovingoutoflockdown.
UKCitizens'BiometricCouncil:aCitizens'AssemblyontheUseofFacialRecognitionTechnologyYear:2020BiometricsSector:Multiple(surveillance)Publicengagementtechnique:Deliberative:citizens'assemblyBriefdescription:Followinganationalsurveythatfoundmajoritysupportfortheregulationoffacialrecognitiontechnology,theAdaLovelaceInstitutehasannouncedtheestablishmentofacitizens'assemblytoreviewtheevidencearoundbiometrics.
201Title,year,andcountry/regionTechnology/innovation,andsectorPublicengagementtechniqueandbriefdescriptionoftheengagementCountry/region:UnitedKingdomBioenergyDistributedDialogueYear:2012-14Country/region:UnitedKingdomBioenergySector:EnergyPublicengagementtechnique:Deliberative:distributeddialogueBriefdescription:BBSRC,onbehalfoftheUKResearchCouncils,undertookapublicdialogue(distributeddialogue)tohelpensurethatcontemporarypublicviews,concernsandaspirationsweretakenintoaccountbyresearchfundersandresearchersintheareaofbioenergy.
CIMULACTYear:2015-16Country/region:EUMultipleSector:S&TPolicyPublicengagementtechnique:Deliberative:workshopsBriefdescription:LedbytheDanishBoardofTechnology,morethan1,000citizensmetduringthewinter2015-2016atnationalvisionworkshopswheretheyexpressedtheirdreamsforasustainableanddesirablefuture.
PublicDialogueonDronesYear:2015-16Country/region:UnitedKingdomDronesSector:TransportPublicengagementtechnique:Deliberative:workshopsBriefdescription:ApublicdialoguewascommissionedbytheDepartmentforTransportandtheMinistryofDefencewithsupportfromtheSciencewiseprogramme.
Itexploredthepublic'sunderstandingofdrones,attitudestowardscurrentusage,andpublicexpectations,aspirationsandconcernsaboutfutureusage.
202ReferencesAbdul,Ashraf,JoVermeulen,DandingWang,BrianLim,andMohanKankanhalli.
2018.
'TrendsandTrajectoriesforExplainable,AccountableandIntelligibleSystems:AnHCIResearchAgenda'.
InCHI'18:Proceedingsofthe2018CHIConferenceonHumanFactorsinComputingSystems,1–18.
AdaLovelaceInstitute.
2019.
BeyondFaceValue:PublicAttitudestoFacialRecognitionTechnology.
London:AdaLovelaceInstitute.
———.
2020.
ConfidenceinaCrisisBuildingPublicTrustinaContactTracingApp.
London:AdaLovelaceInstitute.
AdaLovelaceInstitute,Traverse,Involve&BangtheTable.
2020.
DeliberatingrapidlyandonlineaboutCOVID-19technologies.
Asof6January2021:https://www.
adalovelaceinstitute.
org/event/deliberating-rapidly-and-online-about-covid-19-technologies/Aicardi,Christine,LiseBitsch,SaheliDattaBurton,KathinkaEvers,MicheleFarisco,TaraMahfoud,NikolasRose,AchimRosemann,ArleenSalles,BerndStahl&IngaUlnicane.
2020.
'TrustandTransparencyinArtificialIntelligence(D12.
–.
4-SGA2)'.
sos-ch-dk-2.
exo.
io,21October.
Asof6January2021:https://sos-ch-dk-2.
exo.
io/public-website-production/filer_public/95/15/9515ee61-bc63-43df-aad6-14a5649bc5d3/d1254_d794_d51_sga2_m24_accepted_200731.
pdfAllan,Sarah.
2016.
'9StepstoGettingPublicEngagementRight'.
involve.
org.
uk,17February2016.
Asof6January2021:https://www.
involve.
org.
uk/resources/blog/opinion/9-steps-getting-public-engagement-rightAltamirano-Allende,C.
,&C.
Selin.
2016.
'SeeingtheCity:PhotographyasaPlaceofWork'.
JournalofEnvironmentalStudiesandSciences6(3):460–69.
Alvarez,W.
M.
,M.
AngelDeMiguel,F.
Garcia&C.
Olaverri-Monreal.
2019.
'ResponseofVulnerableRoadUserstoVisualInformationfromAutonomousVehiclesinSharedSpaces'.
In2019IEEEIntelligentTransportationSystemsConference(ITSC),Auckland,NewZealand,2019,3714–19.
Armstrong,H.
,C.
Gorst&J.
Rae.
2019.
'RenewingRegulation:"AnticipatoryRegulation"inanAgeofDisruption'.
nesta.
org.
uk,6March2019.
Asof6January2021:https://www.
nesta.
org.
uk/report/renewing-regulation-anticipatory-regulation-in-an-age-of-disruption/Arnaldi,S.
2018.
'RetoolingTechno-MoralScenarios.
ARevisitedTechniqueforExploringAlternativeRegimesofResponsibilityforHumanEnhancement'.
NanoEthics12(3):283–300.
203Arning,K.
,J.
vanHeek&M.
Ziefle.
2018.
'AcceptanceProfilesforaCarbon-DerivedFoamMattress.
ExploringandSegmentingConsumerPerceptionsofaCarbonCaptureandUtilizationProduct'.
JournalofCleanerProduction188:171–84.
Awad,Edmond,SohanDsouza,RichardKim,JonathanSchulz,JosephHenrich,AzimShariff,Jean-FranoisBonnefon&IyadRahwan.
2018.
'TheMoralMachineExperiment'.
Nature563(7729):59–64.
Bainbridge,WilmaA.
,PhillipIsola&AudeOliva.
2013.
'TheIntrinsicMemorabilityofFacePhotographs'.
JournalofExperimentalPsychology:General142(4):1323–34.
Balaram,B.
,T.
Greenham&J.
Leonard.
2018.
ArtificialIntelligence:RealPublicEngagement.
London:RoyalSocietyfortheencouragementofArts,ManufacturersandCommerce.
Bandewar,S.
V.
S.
,F.
Wambugu,E.
Richardson&J.
V.
Lavery.
2017.
'TheRoleofCommunityEngagementintheAdoptionofNewAgriculturalBiotechnologiesbyFarmers:TheCaseoftheAfricaHarvestTissue-CultureBananainKenya'.
BMCBiotechnology17(1).
BBSRC(BiotechnologyandBiologicalSciencesResearchCouncil).
2005.
'NanotechnologyDialogue'.
bbsrc.
ukri.
org,October2005.
Asof6January2021:https://bbsrc.
ukri.
org/engagement/dialogue/activities/activities-nanotechnology/———.
2013.
'BioenergyPublicDialogue'.
bbsrc.
ukri.
org,December2013.
Asof6January2021:https://bbsrc.
ukri.
org/engagement/dialogue/activities/bioenergy-dialogue/BEIS(DepartmentforBusiness,Energy&IndustrialStrategy).
2019.
'RegulationfortheFourthIndustrialRevolution'.
gov.
uk,Researchandanalysis,11June2019.
Asof6January2021:https://www.
gov.
uk/government/publications/regulation-for-the-fourth-industrial-revolution———.
2020.
'PublicAttitudestoScience'.
gov.
uk,Researchandanalysis,17July2020.
Asof6January2021:https://www.
gov.
uk/government/publications/public-attitudes-to-science-2019Bellamy,R.
,&J.
Lezaun.
2017.
'CraftingaPublicforGeoengineering'.
PublicUnderstandingofScience26(4):402–17.
Bellamy,Rob,JasonChilvers&NaomiE.
Vaughan.
2016.
'DeliberativeMappingofOptionsforTacklingClimateChange:CitizensandSpecialists"Openup"AppraisalofGeoengineering'.
PublicUnderstandingofScience25(3):269–86.
Bennett,D.
,&T.
Radford.
2017.
'PublicPerceptionsofNanotechnologies:LessonsfromGeneticallyModifiedFoods'.
InNanotechnologiesinFood:Edition2,editedbyQasimChaudhry,LaurenceCastle,RichardWatkins,60-80.
London:RoyalSocietyofChemistry.
Bibri,S.
E.
2018.
'BackcastinginFuturesStudies:ASynthesisedScholarlyandPlanningApproachtoStrategicSmartSustainableCityDevelopment'.
EuropeanJournalofFuturesResearch6(13).
204Bitkina,O.
V.
,H.
Jeong,B.
C.
Lee,J.
Park,J.
Park&H.
K.
Kim.
2020.
'PerceivedTrustinArtificialIntelligenceTechnologies:APreliminaryStudy'.
HumanFactorsandErgonomicsInManufacturing30(4):282–90.
Bitsch,Lise,&NicklasBang.
2020.
'BringingVoicesfromSocietytoBearontheECWhitePaperOnArtificialIntelligence-AEuropeanApproachtoExcellenceandTrust'.
tekno.
dk,DanishBoardofTechnology,2020.
Asof6January2021:https://tekno.
dk/wp-content/uploads/2015/10/Feedback_EC_white_paper_AI_DBT-5.
pdfBitsch,Lise,NicklasBangBdum,NathanCampion&MarieLouiseJrgensen.
2020.
'D12.
3.
1:InsightintoEuropeans'InformedandConsideredViewsonArtificialIntelligence–ResultsfromEuropeSayonAI'.
ec.
europa.
eu,Research&Innovation,ParticipantPortal,2October2020.
Asof6January2021:https://ec.
europa.
eu/research/participants/documents/downloadPublicdocumentIds=080166e5d4318bcc&appId=PPGMSBontoux,L.
,J.
A.
Sweeney,D.
Bengtsson&A.
Rosa.
2016.
'TheJRCScenarioExplorationSystem-FromStudytoSeriousGame'.
JournalofFuturesStudies20(3):93–108.
https://jfsdigital.
org/2016-2/vol-20-no-3-march-2016/r-e-p-o-r-t/the-jrc-scenario-exploration-system-from-study-to-serious-game/Bontoux,L.
,J.
A.
Sweeney,A.
B.
Rosa,A.
Bauer,D.
Bengtsson,A.
Bock,B.
Caspar,M.
Charter,E.
Christophilopoulos,F.
Kupper,C.
Macharis,C.
Matti,M.
Matrisciano,J.
Schuijer,A.
Szczepanikova,T.
vanCriekinge&R.
Watson.
2020.
'AGameforAllSeasons:LessonsandLearningsfromtheJRC'sScenarioExplorationSystem'.
WorldEconomicReview12(1):81–103.
Boudet,H.
S.
2019.
'PublicPerceptionsofandResponsestoNewEnergyTechnologies'.
NatureEnergy4(6):446–55.
Brodie,E.
,E.
Cowling,N.
Nissen,A.
E.
Paine,V.
Jochum&WarburtonD.
2009.
'UnderstandingParticipation:ALiteratureReview.
'sp.
gov.
tr,PathwaysthroughParticipation,2009.
Asof6January2021:http://www.
sp.
gov.
tr/upload/Sayfa/47/files/Pathways-literature-review-final-version.
pdfBuckley,J.
A.
,P.
B.
Thompson&K.
P.
Whyte.
2017.
'Collingridge'sDilemmaandtheEarlyEthicalAssessmentofEmergingTechnology:TheCaseofNanotechnologyEnabledBiosensors'.
TechnologyinSociety48:54–63.
Burgess,J.
,J.
Chilvers,J.
Clark,R.
Day,J.
Hunt&S.
King.
2004.
CitizensandSpecialistsDeliberateOptionsforManagingtheUK'sIntermediateandHighLevelRadioactiveWaste:AReportoftheDeliberativeMappingTrial,June–July2004.
London:DepartmentforEnvironment,FoodandRuralAffairs.
Burgess,M.
M.
,J.
D.
Mumford&J.
V.
Lavery.
2018.
'PublicEngagementPathwaysforEmergingGMInsectTechnologies'.
BMCProceedings12:Article12.
205Busby,Amy,AliDigby&EmilyFu.
2020.
'QuantumTechnologiesPublicDialogue'.
epsrc.
ukri.
org,2020.
Asof6January2021:https://epsrc.
ukri.
org/newsevents/pubs/epsrc-quantum-technologies-public-dialogue-full-report/CabinetOffice.
2018.
'ConsultationPrinciples:Guidance'.
gov.
uk,Guidance,17July2012,updated19March2018.
Asof6January2021:https://www.
gov.
uk/government/publications/consultation-principles-guidanceCalabrese,C.
,J.
Ding,B.
Millam&G.
A.
Barnett.
2019.
'TheUproarOverGene-EditedBabies:ASemanticNetworkAnalysisofCRISPRonTwitter'.
EnvironmentalCommunication14(7):954-70.
Cameron,David,&KellyMaguire.
2017.
'PublicViewsofMachineLearning'.
royalsociety.
org,Projects,2017.
Asof6January2021:https://royalsociety.
org/~/media/policy/projects/machine-learning/digital-natives-16-10-2017.
pdfsource=post_pageCDEI(CentreforDataEthicsandInnovation).
2020.
'CDEIReviewofOnlineTargeting'.
gov.
uk,IndependentReport,4February2020.
Asof6January:https://www.
gov.
uk/government/publications/cdei-review-of-online-targetingCentreforPublicImpact.
2019.
'BuildingConsensusandCompromiseonUberinTaiwan'.
centreforpublicimpact.
org,18September2019.
Asof6January2021:https://www.
centreforpublicimpact.
org/case-study/building-consensus-compromise-uber-taiwan/Chhetri,Netra,RajivGhimire,MelissaWagner&MengWang.
2020.
'GlobalCitizenDeliberation:CaseofWorld-WideViewsonClimateandEnergy'.
EnergyPolicy147(December):111892.
CIMULACT.
2021a.
'CIMULACT&Foresight'.
cimulact.
eu.
Asof6January2021:http://www.
cimulact.
eu/approach/cimulact-foresight/———.
2021b.
'CIMULACT&RRI'.
cimulact.
eu.
Asof6January2021:http://www.
cimulact.
eu/approach/cimulact-rri/———.
2021c.
'CIMULACT&Visualisation'.
cimulact.
eu.
Asof6January2021:http://www.
cimulact.
eu/approach/cimulact-visualisation/———.
2021d.
'Objectives'.
cimulact.
eu.
Asof6January2021:http://www.
cimulact.
eu/objectives/Clift,J.
2019.
'ABlueprintforDynamicOversight:HowtheUKcantakeagloballeadinemergingscienceandtechnologies'.
welcome.
org,March2019.
Asof6January2021:https://wellcome.
org/sites/default/files/blueprint-for-dynamic-oversight.
pdf206Collier,D.
2016.
'PublicDialogueonDronesintheUK:EvaluationReport'.
sciencewise.
org.
uk,July2016.
Asof6January2021:https://sciencewise.
org.
uk/wp-content/uploads/2018/07/Drones-evaluation-report.
pdfDagorne,E.
,&N.
Gudowsky.
2018.
'Deliverable5.
1.
IspirationCatalogueforConsultingDifferentGroups'.
teknologiradet.
no,CIMULACT,MissionsPubliques,28March2018.
Asof6January2021:https://teknologiradet.
no/wp-content/uploads/sites/105/2019/01/D5.
1_Inspiration-Catalogue-for-consulting-different-groups-compressed.
pdfDascal,Julieta,MarkReid,WaguihWilliamIsHak,BrennanSpiegel,JenniferRecacho,BradleyRosen&ItaiDanovitch.
2017.
'VirtualRealityandMedicalInpatients:ASystematicReviewofRandomized,ControlledTrials'.
InnovationsinClinicalNeuroscience14(1–2):14–21.
Datta,Anupam,ShayakSen&YairZick.
2016.
'AlgorithmicTransparencyviaQuantitativeInputInfluence:TheoryandExperimentswithLearningSystems'.
In2016IEEESymposiumonSecurityandPrivacy(SP),SanJose,CA,598–617.
Davies,G.
,J.
Burgess,M.
Eames,S.
Mayer,S.
Staley,andA.
Stirling.
2003.
DeliberativeMapping:AppraisingOptionsforClosing"theKidneyGap".
London:WellcomeTrust.
DeepMind.
2021a.
'About'.
deepmind.
com.
Asof6January2021:https://deepmind.
com/about———.
2021b.
'ExploringtheReal-WorldImplicationsofAI'.
deepmind.
com.
Asof6January2021:https://deepmind.
com/about/ethics-and-societyDegeling,C.
,S.
M.
Carter,A.
M.
VanOijen,J.
McAnulty,V.
Sintchenko,A.
Braunack-Mayer,T.
Yarwood,J.
Johnson&G.
L.
Gilbert.
2020.
'CommunityPerspectivesontheBenefitsandRisksofTechnologicallyEnhancedCommunicableDiseaseSurveillanceSystems:AReportonFourCommunityJuries'.
BMCMedicalEthics21(1):Articlenumber31.
Drew,C.
2016.
'DataScienceEthicsinGovernment'.
PhilosophicalTransactionsoftheRoyalSocietyA:Mathematical,PhysicalandEngineeringSciences374(2083).
Dupont,L.
,J.
Hubert,C.
Guidat&M.
Camargo.
2019.
'UnderstandingUserRepresentations,aNewDevelopmentPathforSupportingSmartCityPolicy:EvaluationoftheElectricCarUseinLorraineRegion'.
TechnologicalForecastingandSocialChange142:333–46.
Dupont,L.
,L.
Morel&M.
Pallot.
2016.
'ExploringtheAppropriatenessofDifferentImmersiveEnvironmentsintheContextofanInnovationProcessforSmart-Cities'.
In2016InternationalConferenceonEngineering,TechnologyandInnovation/IEEEInternationalTechnologyManagementConference(ICE/ITMC),Trondheim,Norway,2016,1-9.
207Eaton,Weston,WynneWright,KyleWhyte,StephenP.
Gasteyer&PatJ.
Gehrke.
2014.
'EngagementandUncertainty:EmergingTechnologiesChallengetheWorkofEngagement'.
JournalofHigherEducationOutreachandEngagement18(2):151–78.
Ekstrand,MichaelD.
,RezvanJoshaghani&HodaMehrpouyan.
2018.
'PrivacyforAll:EnsuringFairandEquitablePrivacyProtections;Proceedingsofthe1stConferenceonFairness,AccountabilityandTransparency'.
ProceedingsofMachineLearningResearch8:35–47.
Engage2020.
2014.
'D3.
2PublicEngagementMethodsandTools'.
engage2020.
eu,Media,October2014.
Asof6January2021:http://engage2020.
eu/media/Fact-sheet-on-the-current-praxis-of-methods-tools-and-instruments.
pdfEngasser,F.
,&T.
Saunders.
2018.
'HowtoUseGamestoInvolvethePublicinDecisionsaboutResearchandInnovationPolicy'.
nesta.
org.
uk,17August2018.
Asof6January2021:https://www.
nesta.
org.
uk/blog/how-use-games-involve-public-decisions-about-research-and-innovation-policy/EUScienceHub.
2019.
'ScenarioExplorationSystem(SES)'.
ec.
europa.
eu,EUScienceHub,Research,3December2019.
Asof6January2021:https://ec.
europa.
eu/jrc/en/research/foresight/sesEuropeanCommission.
2017.
InterimEvaluationofHorizon2020.
Luxembourg:PublicationsOfficeoftheEuropeanUnion.
———.
2020.
'Horizon2–20-WorkProgramme2018-2020'.
Brussels,Belgium:PublicationsOfficeoftheEuropeanUnion.
Felt,Ulrike,SimoneSchumann,ClaudiaG.
Schwarz&MichaelStrassnig.
2014.
'TechnologyofImagination:ACard-BasedPublicEngagementMethodforDebatingEmergingTechnologies'.
QualitativeResearch14(2):233–51.
FernándezMedina,K.
,&R.
Jenkins.
2017.
'GATEway:PublicPerceptionsofaLast-MileDriverlessShuttle'.
gateway-project.
org.
uk,GATEwayProject,November2017.
Asof6January2021:https://gateway-project.
org.
uk/wp-content/uploads/2018/06/D3.
7_TRL-Workshop-Findings-Report.
pdfFoodStandardsAgency.
2020.
'ConsumerAttitudestowardsEmergingTechnologies'.
food.
gov.
uk,Researchprojects,21April2020.
Asof6January2021:https://www.
food.
gov.
uk/research/research-projects/consumer-attitudes-towards-emerging-technologies-0Fung,Archon.
2006.
'VarietiesofParticipationinComplexGovernance'.
PublicAdministrationReview66:66–75.
208Garcia,R.
,P.
Quiepo,O.
Kammona&I.
Malsch.
2019.
'D4.
3NANOFuturesWorkingGroupforSocietalEngagement'.
NANO2ALL,SocietalEngagementonResponsibleNanotechnology.
ec.
europa.
eu,Research,26February2019.
Asof6January2021:https://ec.
europa.
eu/research/participants/documents/downloadPublicdocumentIds=080166e5c2ad4341&appId=PPGMSGardner,Allison.
2020.
'Contact-TracingApps:There'sNoEvidenceThey'reHelpingStopCOVID-19'.
theconversation.
com,21October2020,12:08BST.
Asof6January2021:https://theconversation.
com/contact-tracing-apps-theres-no-evidence-theyre-helping-stop-covid-19-148397Garrett,Bernie,TarniaTaverner,WendyMasinde,DianeGromala,ChrisShaw&MichaelNegraeff.
2014.
'ARapidEvidenceAssessmentofImmersiveVirtualRealityasanAdjunctTherapyinAcutePainManagementinClinicalPractice'.
TheClinicalJournalofPain30(12):1089–98.
Gunashekar,Salil,SarahParks,JoeFrancombe,Camillad'Angelo,Gemma-ClaireAli,PaminaSmith,DanielaRodriguez-Rincon,MarleneAltenhofer&GordonR.
McInroy.
2019.
OversightofEmergingScienceandTechnology:LearningfromPastandPresentEffortsaroundtheWorld.
SantaMonica,Calif.
:RANDCorporation.
RR-2921-WT.
Asof6January2021:https://www.
rand.
org/pubs/research_reports/RR2921.
htmlHarrowD,GheerawoR.
2018.
'D3.
6AutoValetParkingTrialDesignforAcceptanceandAdoption:GATEwayProjectTrial2RCAWorkshopReport'.
gateway-project.
org.
uk,2018.
Asof6January2021:https://gateway-project.
org.
uk/wp-content/uploads/2018/06/D5.
3b_RCA_Trial-2-Auto-Valet-Parking.
pdfHauthal,Eva,DirkBurghardt,CarolynFish&AmyL.
Griffin.
2020.
'SentimentAnalysis'.
InternationalEncyclopediaofHumanGeography(SecondEdition)2020:169-77.
Hebáková,L.
,T.
Ratinger,L.
Jansa&I.
Vanurová.
2018.
'Deliverable5.
3–ReportontheImpactoftheProject'.
cimulact.
eu,Publications,31May2018.
Asof6January2021:http://www.
cimulact.
eu/wp-content/uploads/2018/06/CIMULACT-D5.
3.
pdfHelbing,Dirk,BrunoFrey,GerdGigerenzer,ErnstHafen,MichaelHagner,YvonneHofstetter,Jeroenvandenhoven,RobertoZicari,andAndrejZwitter.
2019.
'WillDemocracySurviveBigDataandArtificialIntelligence:EssaysontheDarkandLightSidesoftheDigitalRevolution'.
InTowardsDigitalEnlightenment,editedbyDirkHelbing,73–98.
Cham:Springer.
Hennen,L.
2013.
'ParliamentaryTechnologyAssessmentinEuropeandtheRoleofPublicParticipation'.
InPublicEngagemetnandEmergingTechnologies,editedbyK.
O'Doherty&Einsiedel.
E,27–44.
Vancouver:UBCPress.
Henry,Christine.
2018.
'Aspeculativefictionthread.
Inresponseto@mediatecture's#biometricmirroranditsdeconsentmechanism.
abc.
net.
au/triplej/progra…(by@jolauder.
h/t@boingboing;feelslike@doctorowyou'dmaybeappreciatethis)'.
twitter.
com,209@christinelhenry(blog),31July2018.
Asof6January2021:https://twitter.
com/christinelhenry/status/1024345651436507142Hienuki,S.
,Y.
Hirayama,T.
Shibutani,J.
Sakamoto,J.
Nakayama&A.
Miyake.
2019.
'HowKnowledgeaboutorExperiencewithHydrogenFuelingStationsImprovesTheirPublicAcceptance'.
Sustainability(Switzerland)11(22):63339.
Hoffman,H.
G.
,J.
N.
Doctor,D.
R.
Patterson,G.
J.
Carrougher&T.
A.
Furness.
2000.
'VirtualRealityasanAdjunctivePainControlduringBurnWoundCareinAdolescentPatients'.
Pain85(1–2):305–9.
Hoffman,H.
G.
,A.
Garcia-Palacios,D.
R.
Patterson,M.
Jensen,T.
Furness&W.
F.
Ammons.
2001.
'TheEffectivenessofVirtualRealityforDentalPainControl:ACaseStudy'.
Cyberpsychology&Behavior:TheImpactoftheInternet,MultimediaandVirtualRealityonBehaviorandSociety4(4):527–35.
Horlick-Jones,Tom,JohnWalls,GeneRowe,NickPidgeon,WouterPoortinga&TimO'Riordan.
2006.
'OnEvaluatingtheGMNationPublicDebateabouttheCommercialisationofTransgenicCropsinBritain'.
NewGeneticsandSociety25(3):265–88.
Hsiao,Y.
,S.
Lin,A.
Tang,D.
Narayanan,andC.
Sarahe.
2018.
'VTaiwan:AnEmpiricalStudyofOpenConsultationProcessinTaiwan'.
osf.
io,SocArXivPapers,4July2018.
IAP2.
2018.
'IAP2SpectrumofPublicParticipation'.
cdn.
ymaws.
com,2018.
Asof6January2021:https://cdn.
ymaws.
com/www.
iap2.
org/resource/resmgr/pillars/Spectrum_8.
5x11_Print.
pdfICO(InformationCommissioner'sOffice)&TheAlanTuringInstitute.
2021.
'ExpainingDecisionsMadewithAI'.
ico.
org.
uk.
Asof6January2021:https://ico.
org.
uk/for-organisations/guide-to-data-protection/key-data-protection-themes/explaining-decisions-made-with-artificial-intelligence/InstitutCatalàdeNanociènIiaiNanotecnologia.
2020.
'Nano2AllCitizensDebate(Barcelona,June2017)'.
youtube.
com,11May2020.
Asof6January2021:https://www.
youtube.
com/watchv=iTAxdipw6WUInvolve.
2018.
'Methods'.
involve.
org.
uk,2018.
Asof6January2021:https://www.
involve.
org.
uk/resources/methodsIpsosMORI.
2019.
'APublicDialogueonGenomicMedicine:TimeforaNewSocialContract'.
ipsos.
com,25April2015.
Asof6January2021:https://www.
ipsos.
com/ipsos-mori/en-uk/public-dialogue-genomic-medicine-time-new-social-contract-reportKarakas,C.
2019.
'HorizonEurope–SpecificProgrammeImplementingtheFrameworkProgramme'.
europarl.
europa.
eu,EuropeanParliamentBriefingDocument,EULegislationinProgress2021-2027MFF,May2019.
Asof6January2021:210https://www.
europarl.
europa.
eu/RegData/etudes/BRIE/2018/628297/EPRS_BRI(2018)628297_EN.
pdfKeller,M.
S.
,H.
J.
Park,M.
E.
Cunningham,J.
E.
Fouladian,M.
Chen&B.
M.
R.
Spiegel.
2017.
'PublicPerceptionsRegardingUseofVirtualRealityinHealthCare:ASocialMediaContentAnalysisUsingFacebook'.
JournalofMedicalInternetResearch19(12).
doi:10.
2196/jmir.
7467Kupper,F.
,&J.
W.
Schuijer.
2016.
'D3.
1DialogueMethodology:ModeratorManual'.
NANO2ALL,SocietalEngagementonResponsibleNanotechnology.
nano2all.
eu,December2016.
Asof6January2021:http://www.
nano2all.
eu/wp-content/uploads/files/D3.
1%20Dialogue%20methodology.
pdf———.
2018a.
'D3.
3.
ResponsibleInnovationAgendasatNationalLevel'.
NANO2ALL,SocietalEngagementonResponsibleNanotechnology.
nano2all.
eu,23July2018.
Asof6January2021:http://www.
nano2all.
eu/wp-content/uploads/2018/12/NANO2ALL-685931-D3.
3-Responsible-Innovation-Agenda-at-national-level.
pdf———.
2018b.
'D3.
4.
ResponsibleInnovationAgendaatEuropeanLevel'.
NANO2ALL,SocietalEngagementonResponsibleNanotechnology.
nano2all.
eu,10July2018.
Asof6January2021:http://www.
nano2all.
eu/wp-content/uploads/files/NANO2ALL%20-%20685931%20-%20D3.
4%20Responsible%20Innovation%20Agenda%20at%20European%20level-submitted.
pdfLezaun,J.
,&L.
Soneryd.
2007.
'ConsultingCitizens:TechnologiesandElicitationoftheMobilityofPublics'.
PublicUnderstandingofScience16:279–207.
Li,Angela,ZorashMontao,VincentJ.
Chen&JeffreyI.
Gold.
2011.
'VirtualRealityandPainManagement:CurrentTrendsandFutureDirections'.
PainManagement1(2):147–57.
Malloy,KevinM.
,&LeonardS.
Milling.
2010.
'TheEffectivenessofVirtualRealityDistractionforPainReduction:ASystematicReview'.
ClinicalPsychologyReview30(8):1011–18.
MartinKing.
2019.
'BuildingConsensusandCompromiseonUberinTaiwan'.
centreforpublicimpact.
org,TacklingChallengesTogether,CaseStudy,18September2019.
Asof6January2021:https://www.
centreforpublicimpact.
org/case-study/building-consensus-compromise-uber-taiwan/McCambridge,Jim,JohnWitton&DianaR.
Elbourne.
2014.
'SystematicReviewoftheHawthorneEffect:NewConceptsAreNeededtoStudyResearchParticipationEffects'.
JournalofClinicalEpidemiology67(3):267–77.
McCool,Skye.
2019.
'CAVPublicAcceptabilityDialogue,EngagementReport'.
assets.
publishing.
service.
gov.
uk,TraversereportfortheDepartmentforTransport,5August2112019.
Asof6January2021:https://assets.
publishing.
service.
gov.
uk/government/uploads/system/uploads/attachment_data/file/837958/cav-public-acceptability-dialogue-engagement-report.
pdfMcStay,Andrew.
2016.
'EmpathicMediaandAdvertising:Industry,Policy,LegalandCitizenPerspectives(theCaseforIntimacy)'.
BigData&Society3(November).
MERGEGreenwich.
2018.
'Autonomousvehicleride-sharingservices:Willtheymakecitiesgreener,moreefficientandmoreaccessible'MergeGreenwich.
static1.
squarespace.
com.
Asof6January:https://static1.
squarespace.
com/static/5ae87244b105982d79d73cf5/t/5be2dc0d4ae2375f2749156b/1541594145674/MERGE+Greenwich+Final+Report.
pdfMiah,A.
2017.
'Nanoethics,ScienceCommunication,andaFourthModelforPublicEngagement'.
NanoEthics11(2):139–52.
Middleton,Anna.
2017.
'YourDNA,YourSay'.
TheNewBioethics:AMultidisciplinaryJournalofBiotechnologyandtheBody23(1):74–80.
Mil,A.
V,H.
Hopkins&S.
Kinsella.
2019.
'FromOurBraintotheWorld:ViewsontheFutureofNeuralInterfaces'.
semanticscholar.
org.
Asof6January2021:https://www.
semanticscholar.
org/paper/From-our-brain-to-the-world%3A-views-on-the-future-of-Mil-Hopkins/53fd40b4e90cd0aeee2c1835825c0867605210b3Morris,LinzetteDeidré,QuinetteAbegailLouw&LynetteChristineCrous.
2010.
'FeasibilityandPotentialEffectofaLow-CostVirtualRealitySystemonReducingPainandAnxietyinAdultBurnInjuryPatientsduringPhysiotherapyinaDevelopingCountry'.
Burns:JournaloftheInternationalSocietyforBurnInjuries36(5):659–64.
Müller,Jrg,JulianeExeler,MarkusBuzeck,andAntonioKrüger.
2009.
'ReflectiveSigns:DigitalSignsThatAdapttoAudienceAttention'.
InPervasiveComputing.
Pervasive2009.
LectureNotesinComputerScience,editedbyH.
Tokuda,M.
Beigl,A.
Friday,A.
J.
B.
Brush&Y.
Tobe,17–24.
Berlin:Springer.
NANO2ALL.
2017.
'SettingPrioritiesforNewEuro-BrazilianCooperationinResponsibleResearchandInnovation'.
nano2all.
eu,NANO2ALL,SocietalEngagementonResponsibleNanotechnology,30October2017.
Asof6January2021:http://www.
nano2all.
eu/2017/10/30/setting-priorities-for-new-euro-brazilian-cooperation-in-responsible-research-and-innovation/———.
2018a.
'NANO2ALLatB.
Debate'.
nano2all.
eu,8October2018.
Asof6January2021:http://www.
nano2all.
eu/2018/10/08/nano2all-at-b-debate/———.
2019a.
'D4.
4E-MRSWorkingGroup"OutreachSocietalPromotion"'.
ec.
europa.
eu,NANO2ALL,SocietalEngagementonResponsibleNanotechnology,26April2019.
Asof6January2021:212https://ec.
europa.
eu/research/participants/documents/downloadPublicdocumentIds=080166e5c2475ea8&appId=PPGMS———.
2019b.
'D5.
5ReportonthePublicUnderstanding,AttitudesandFearsofNanotechnologyResearchfortheNano2AllEthicsBoard'.
ecsite.
eu,NANO2ALL,SocietalEngagementonResponsibleNanotechnology,2019.
Asof6January2021:https://www.
ecsite.
eu/sites/default/files/nano2all_report_on_the_public_understanding_attitudes_and_fears_of_nano.
pdf———.
201'c.
'26FEBRUARY:INTERNATIONALCONGRESS"NANOTECHNOLOGYINEVERYDAYLIFE"'.
nano2all.
eu,NANO2ALL,SocietalEngagementonResponsibleResearch,18February2019.
Asof6January2021:http://www.
nano2all.
eu/2019/02/18/26-february-international-congress-nanotechnology-in-everyday-life/———.
2021a.
'OtherNANO2ALLResults'.
nano2all.
eu,NANO2ALL,SocietalEngagementonResponsibleNanotechnology.
Asof6January2021:http://www.
nano2all.
eu/resources/other-nano2all-results/———.
2021b.
'Roadmap:ForaMoreInclusiveNanotechnologyDevelopmentinEurope'.
nano2all.
eu,WPContent.
Asof6January2021:http://www.
nano2all.
eu/wp-content/uploads/2019/05/roadmap_shortversion.
pdfNatCen.
2021.
'BritishSocialAttitudes37'.
bsa.
natcen.
ac.
uk,NatCenSocialResearch.
Asof6January2021:https://www.
bsa.
natcen.
ac.
ukNationalAuditOffice.
2014.
'UsingAlternativestoRegulationtoAchievePolicyObjectives'.
nao.
org.
uk,Report,30June2014.
Asof6January2021:https://www.
nao.
org.
uk/report/using-alternatives-to-regulation-to-achieve-policy-objectives/NCCPE(NationalCo-ordinatingCentreforPublicEngagement).
2018.
'GenomeEditingResources'.
publicengagement.
ac.
uk,2018.
Asof6January2021:https://www.
publicengagement.
ac.
uk/sites/default/files/overview_of_resources_live_2.
pdfNCCPE(NationalCo-ordinatingCentreforPublicEngagement).
2019.
'ScienceResearchEngagementandtheArts'.
publicengagement.
ac.
uk,2019.
Asof6January2021:https://www.
publicengagement.
ac.
uk/sites/default/files/publication/science_research_engagement_and_the_arts.
pdf———.
2020a.
'Festivals'.
publicengagement.
ac.
uk,2020.
Asof6January2021:https://www.
publicengagement.
ac.
uk/do-engagement/choose-method/festivals———.
2020b.
'WhatIsPublicEngagement'.
publicengagement.
ac.
uk,2020.
Asof6January2021:https://www.
publicengagement.
ac.
uk/about-engagement/what-public-engagement213———.
2020c.
'WhyDoesPublicEngagementMatter'.
publicengagement.
ac.
uk,2020.
Asof6January2021:https://www.
publicengagement.
ac.
uk/about-engagement/why-does-public-engagement-matterNavid,E.
L.
,&E.
F.
Einsiedel.
2012.
'SyntheticBiologyintheScienceCafé:WhatHaveWeLearnedaboutPublicEngagement'.
JournalofScienceCommunication11(14).
Newson,A.
J.
,S.
DeLacey,D.
K.
Dowling,S.
Murray,C.
M.
Sue,D.
R.
Thorburn,L.
Gillam&C.
Degeling.
2019.
'PublicAttitudestowardsNovelReproductiveTechnologies:ACitizens'JuryonMitochondrialDonation'.
HumanReproduction34(4):751–57.
OECD(OrganisationforEconomicCo-operationandDevelopment).
2001.
MeasuringProductivity,OECDManual:MeasurementofAggregateandIndustry-LevelProductivityGrowth.
oecd.
org,2021.
Asof6January2021:http://www.
oecd.
org/sdd/productivity-stats/2352458.
pdf———.
2017.
'OpenResearchAgendaSetting'.
OECDScience,TechnologyandInnovationPolicyPapers.
OECD.
———.
2021.
'AlternativestoTraditionalRegulation'.
oecd.
org.
Asof6January2021:https://www.
oecd.
org/gov/regulatory-policy/42245468.
pdfParticipedia.
2021a.
'Survey'.
participedia.
net.
Asof6January2021:https://participedia.
net/method/565———.
2021b.
'TheUKCitizens'BiometricCouncil:ACitizens'AssemblyontheUseofFacialRecognitionTechnology'.
participedia.
net.
Asof6January2021:https://participedia.
net/case/6086Persaud,A.
,S.
Desine,K.
Blizinsky&V.
L.
Bonham.
2019.
'ACRISPRFocusonAttitudesandBeliefstowardSomaticGenomeEditingfromStakeholderswithintheSickleCellDiseaseCommunity'.
GeneticsinMedicine21(8):1726–34.
Pfeiffer,J.
,A.
Gabriel&M.
Gandorfer.
2020.
'UnderstandingthePublicAttitudinalAcceptanceofDigitalFarmingTechnologies:ANationwideSurveyinGermany'.
AgricultureandHumanValues,JournaloftheAgriculture,Food,andHumanValuesSociety2020.
Phillips,D.
,&RobertsE.
2016.
'GATEway:WorkshopAnalysis'.
GATEwayProject.
RoyalCollegeofArt.
PintofScience.
2020.
'PintofScience'.
pintofscience.
co.
uk,2020.
Asof6January2021:https://pintofscience.
co.
uk.
PytlikZillig,L.
M.
,&A.
J.
Tomkins.
2011.
'PublicEngagementforInformingScienceAndTechnologyPolicy:WhatDoWeKnow,WhatDoWeNeedtoKnow,andHowWillWeGetThere'ReviewofPolicyResearch28(2):197–217.
214Ramos,J.
,J.
A.
Sweeney,K.
Peach&S.
Laurie.
2019.
'OurFutures:BythePeople,forthePeople'.
nesta.
org.
uk,Reports,11November2019.
Asof6January2021:https://www.
nesta.
org.
uk/report/our-futures-people-people/Ramseyer,R.
,F.
Cimmino,L.
Emery,S.
Grezes,V.
Grezes,B.
Nanchen,E.
Simon&E.
Fragniere.
2019.
'UsingPhenomenologytoAssessRiskPerceptionofaNewTechnologyinPublicTransportationtheCaseoftheAutonomousVehiclesasMobilityasaService(MaaS)inSwitzerland'.
In20183rdInternationalConferenceonSystemReliabilityandSafety(ICSRS),Barcelona,Spain,2018,289-293.
Rask,M.
,S.
Maciukaite-Zviniene,L.
Tauginiene,V.
Dikcius,K.
Matschoss,T.
Aarrevaara&L.
d'Andrea.
2012.
InnovativePublicEngagement:AConceptualModelofPublicEngagementinDynamicandResponsibleGovernanceofResearchandInnovation.
Brussels:PE2020,EuropeanUnion.
Asof6January2016:https://pe2020.
eu/wp-content/uploads/2016/05/Innovative-Public-Engagement-FINAL-1.
pdfRempel,E.
S.
,Barnett,J.
,&Durrant,H.
2018.
'PublicengagementwithUKgovernmentdatascience:Propositionsfromaliteraturereviewofpublicengagementonnewtechnologies'.
GovernmentInformationQuarterly35(4):569–578.
Rosa,A.
B.
,N.
Gudowsky,P.
Warnke&N.
Hauser.
2018.
'Deliverable5.
2-ReportonComparisonofResearchTopicsfromCIMULACTwithThosefromExpertOrientedForesightStudies'.
cimulact.
eu,2018.
Asof6January2021:http://www.
cimulact.
eu/wp-content/uploads/2018/04/D5.
2_Report-on-comparison-expert-oriented-foresight-studies-compressed.
pdfRotolo,Daniele,DianaHicks&BenR.
Martin.
2015.
'WhatIsanEmergingTechnology'ResearchPolicy44(10):1827–43.
RSA.
2019.
'DemocratisingDecisionsaboutTechnology:AToolkit'.
thersa.
org,2019.
Asof6January2021:https://www.
thersa.
org/globalassets/reports/2019/democratising-decisions-tech-report.
pdfSalisbury,J.
,&B.
Nicholas.
2005.
'ReviewofPublicEngagementintheDevelopmentandOversightofEmergingTechnologies("ScienceandSociety")'.
researchgate.
net,Publication,July2005.
Asof6January2021:https://www.
researchgate.
net/publication/228555382_Review_of_public_engagement_in_the_development_and_oversight_of_emerging_technologies_%27science_and_society%27Sample,M.
,S.
Sattler,S.
Blain-Moraes,D.
Rodríguez-Arias&E.
Racine.
2020.
'DoPublicsShareExperts'ConcernsaboutBrain–ComputerInterfacesATrinationalSurveyontheEthicsofNeuralTechnology'.
ScienceTechnologyandHumanValues45(6):1242–70.
Saunders,T.
2018a.
'SevenPrinciplesforPublicEngagementinResearchandInnovationPolicymaking'.
nesta.
org,30August2018.
Asof6January2021:https://www.
nesta.
org.
uk/report/seven-principles-public-engagement-research-and-innovation-policymaking/215———.
2018b.
'SixCreativeWaystoEngagethePublicinInnovationPolicy'.
nesta.
org,28February2018.
Asof6January2021:https://www.
nesta.
org.
uk/blog/six-creative-ways-to-engage-the-public-in-innovation-policy/Sciencewise.
2021a.
'CaseStudy:BioenergyDistributedDialogue'.
sciencewise.
org.
uk.
Asof6January2021:https://sciencewise.
org.
uk/wp-content/uploads/2018/08/CS-BioenergyFinal.
pdf———.
2021b'CaseStudy:DroneUsage'.
sciencewise.
org.
uk.
Asof6January2021:https://sciencewise.
org.
uk/wp-content/uploads/2018/10/Drones-Case-Study-FINAL.
pdf———.
2021c.
'CaseStudy:GenomicMedicine'.
sciencewise.
org.
uk.
Asof6January2021:https://sciencewise.
org.
uk/wp-content/uploads/2018/08/Genomics-England-Case-Study-FINAL.
pdf———.
2021d.
'CaseStudy:MedicalFrontiers:DebatingMitochondrialReplacement'.
sciencewise.
org.
uk.
Asof6January2021:https://sciencewise.
org.
uk/wp-content/uploads/2018/08/Mitochondrial-Replacement-Therapy-Case-Study-1.
pdf———.
2021e.
'OurImpact'.
sciencewise.
org.
uk.
Asof6January2021:https://sciencewise.
org.
uk/projects-and-impacts/impacts/Selin,C.
,K.
CampbellRawlings,K.
deRidder-Vignone,J.
Sadowski,C.
Altamirano-Allende,G.
Gretchen,S.
R.
Davies&D.
H.
Guston.
2017.
'ExperiementsinEngagement:DesigningPublicEngagementwithScienceandTechnologyforCapacityBuilding'.
PublicUnderstandingofScience26(6):634–49.
Setl,M.
,&Smith,G.
2018.
'Mini-publicsanddeliberativedemocracy'.
InTheOxfordHandbookofDeliberativeDemocracy,editedbyAndreBchtiger,JohnS.
Dryzek,JaneMansbridge&MarkWarren,300-314.
Oxford:OxfordUniversityPress.
Shabanpour,R.
,N.
Golshani,A.
Shamshiripour&A.
K.
Mohammadian.
2018.
'ElicitingPreferencesforAdoptionofFullyAutomatedVehiclesUsingBest-WorstAnalysis'.
TransportationResearchPartC:EmergingTechnologies93:463–78.
Sheikh,Sanah.
2013.
'MedicalFrontiers:DebatingMitochondriaReplacement:DeliberativePublicWorkshops'.
London:OfficeforPublicManagement.
Asof6January2021:https://sciencewise.
org.
uk/wp-content/uploads/2018/08/Mito-Annex_II-public_deliberative_workshops.
pdfSimon,J.
,T.
Bass,V.
Boelman&G.
Mulgan.
2017.
'DigitalDemocracy:TheToolsTransformingPoliticalEngagement'.
nesta.
org.
uk,20February2017.
Asof6January2021:https://www.
nesta.
org.
uk/report/digital-democracy-the-tools-transforming-political-engagement/Singh,A.
2019.
'PatientAI:TowardsaHuman-CenteredCultureofTechnologicalInnovationintheNHS'.
thersa.
org,30October2019.
Asof6January2021:https://www.
thersa.
org/reports/patient-ai-innovation-nhs216SmartMobilityLivingLab.
2020a.
'FirstUKAVTrialofItsKindGetsThumbsUpfromLondonParticipants'.
smartmobility.
london,30January2020.
Asof6January2021:https://smartmobility.
london/news/first-uk-av-trial-of-its-kind-gets-thumbs-up-from-london-participantsSmartMobilityLivingLab.
2020b.
'GreenwichAutomatedTransportEnvironment(GATEway)'.
smartmobility.
london,April2020.
Asof6January2021:https://smartmobility.
london/trials-blog/gatewayrq='GATEway'Smol'kin,A.
A.
,P.
A.
Zhigareva,E.
A.
Makarova,A.
A.
Kiryukhina,E.
A.
Milova&D.
A.
Khaltourina.
2018.
'AttitudestowardsAgingPrevention:ResultsofaFocus-GroupStudy'.
AdvancesinGerontology8(1):71–78.
Stewart,MargaretC.
,&ChristaL.
Arnold.
2018.
'DefiningSocialListening:RecognizinganEmergingDimensionofListening'.
InternationalJournalofListening32(2):85–100.
Stilgoe,J.
L.
,S.
J.
Lock&J.
Wilsdon.
2014.
'WhyShouldWePromotePublicEngagementwithScience'PublicEngagementinScience,SpecialIssue23(1):4–15.
Sutcliffe,H.
,R.
Lindner&C.
vonKameke.
2020.
'TrustandTechGovernance:TowardsaMoreEngaged,Collaborative,CommunicativeApproach'.
rri-tools.
eu,2020.
Asof6January2021:https://rri-tools.
eu/-/trust-amp-amp-tech-governance-towards-a-more-engaged-collaborative-and-communicative-approachTang,Audrey.
2016.
'UberRespondstovTaiwan'sCoherentBlendedVolition'.
blog.
pol.
is,23May2016.
Asof6January2021:https://blog.
pol.
is/uber-responds-to-vtaiwans-coherent-blended-volition-3e9b75102b9bTheAlternative.
2019.
'WeDreamofaConsensus-BasedPolitics-butvTaiwanIsUsingDigitalTech,asWellasComfyMeals,toMakeItHappen'.
thealternative.
org.
uk,24October2019.
Asof6January2021:https://www.
thealternative.
org.
uk/dailyalternative/2019/10/26/vtaiwan-shows-how-consensus-can-be-builtTheRSA.
2019.
'DemocratisingDecisionsaboutTechnology:AToolkit'.
thersa.
org,24October2019.
Asof6January2019:https://www.
thersa.
org/reports/democratising-decisions-technology-toolkit———.
2021.
'ForumforEthicalAI'.
thersa.
org.
Asof6January2021:https://www.
thersa.
org/projects/archive/economy/tech-and-society/forum-for-ethical-aiTong,Simon.
2017.
'BridgingtheFinalMetres:PublicFeedbackonaLastMileDriverlessDeliveryService:AGATEwayCAVTrial'.
gateway-project.
org.
uk,4December2017,GATEwayProject,PublishedProjectReportPPR851.
Asof6January2021:https://gateway-project.
org.
uk/wp-content/uploads/2018/06/D5.
3c_Trial-3-Public-Feedback-Survey-Report.
pdf217Tournay,V.
,M.
Jacomy,A.
Necula,A.
Leibing&A.
Blasimme.
2020.
'ANewWeb-BasedBigDataAnalyticsforDynamicPublicOpinionMappinginDigitalNetworksonContestedBiotechnologyFields'.
OMICSAJournalofIntegrativeBiology24(1):29–42.
TRLpublish.
2018a.
'D1.
3GATEwayProject:FinalReport'.
gateway-project.
org.
uk,GATEwayProject,8June2018.
Asof6January2021:https://gateway-project.
org.
uk/d1-3-gateway-project-final-report/TRLpublish.
2018b.
'D3.
6CommonplaceSentimentMapping'.
gateway-project.
org.
uk,GATEwayProject,8June2018.
Asof6January2021:https://gateway-project.
org.
uk/d3-6-commonplace-sentiment-mapping/Uber.
2021.
'UberX'.
uber.
com.
Asof6January2021:https://www.
uber.
com/gb/en/ride/uberx/UKERC(UKEnergyResearchCentre).
2021.
'SocietalEngagementObservatory'.
ukerc.
ac.
uk.
Asof6January2021:https://ukerc.
ac.
uk/research/see/Venverloo,T.
,F.
Duarte,T.
Benson,Q.
Bitran,A.
D.
Beldad,R.
Alvarez&C.
Ratti.
2020.
'EvaluatingtheHumanExperienceofAutonomousBoatswithImmersiveVirtualReality'.
JournalofUrbanTechnology.
vonSchomberg,R.
2011.
TowardsResponsibleResearchandInnovationintheInformationandCommunicationTechnologiesandSecurityTechnologyFields.
Luxembourg:PublicationsOfficeoftheEuropeanUnion.
vTaiwan.
2021.
'VTaiwan'.
info.
vtaiwan.
tw.
Asof6January2021:https://info.
vtaiwan.
twWatermeyer,R.
,&G.
Rowe.
2013.
'EvaluationoftheProject:"MitochondriaReplacementConsultation"'.
involve.
org.
uk,July2013.
Asof6January2021:https://www.
involve.
org.
uk/sites/default/files/field/attachemnt/Mitochondria-evaluation-FINAL-2013.
pdfHamlyn,Becky,MartinShanahan,HannahLewis,EllenO'Donoghue,TimHanson.
2015.
'FactorsAffectingPublicEngagementbyResearchers:AStudyofBehalfofaConsortiumofUKPublicResearchFunders'.
welcome.
org,December2015.
Asof6January2021:https://wellcome.
org/sites/default/files/wtp060033_0.
pdfWelp,M.
,A.
delaVega-Leinert,S.
Stoll-Kleemann&C.
C.
Jaeger.
2006.
'Science-BasedStakeholderDialogues:TheoriesandTools'.
GlobalEnvironmentalChange16(2):170–81.
Wilsdon,James,&RebeccaWillis.
2004.
'See-throughScience:WhyPublicEngagementNeedstoMoveUpstream'.
Demos.
researchgate.
net,2004.
Asof6January2021:doi:10.
13140/RG.
2.
1.
3844.
3681Winickoff,D.
E.
,L.
Jamal&N.
R.
Anderson.
2016.
'NewModesofEngagementforBigDataResearch'.
JournalofResponsibleInnovation3(2):169–77.
218Wouters,N.
,R.
Kelly,E.
Velloso,K.
Wolf,H.
S.
Ferdous,J.
Newn,Z.
Joukhadar&F.
Vetere.
2019.
'BiometricMirror:ExploringValuesandAttitudestowardsFacialAnalysisandAutomatedDecision-Making'.
InDIS'19:Proceedingsofthe2019onDesigningInteractiveSystemsConference,447–61.
Zhao,Y.
,C.
Fautz,Hennen.
L,K.
R.
Srinivas&Q.
Li.
2015.
'PublicEngagementintheGovernanceofScienceandTechnology'.
InScienceandTechnologyGovernanceandEthics,editedbyMiltosLadikas,SachinChaturvedi,YandongZhao&DirkStemerding,39–51.
Cham:Springer.
Asof6January2021:https://link.
springer.
com/chapter/10.
1007/978-3-319-14693-5_4