simulation Providing information for decision making: Contrasting descriptionand Journal of Applied Research in Memory and Cognition

ContentslistsavailableatScienceDirect

Journal

of

Applied

Research

in

Memory

and

Cognition

j ou rn a l h o m e p a g e :w w w . e l s e v i e r . c o m / l o c a t e / j a r m a c

Original

Article

Providing

information

for

decision

making:

Contrasting

description

and

simulation

Robin

M.

Hogarth

_,

_Emre

_Soyer

a_{UniversitatPompeuFabra,DepartmentofEconomics&Business,Barcelona,Spain}

b_{OzyeginUniversity,FacultyofBusiness,Istanbul,Turkey}

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received17October2013

Accepted22January2014

Availableonline30January2014

Keywords: Decisionmaking Uncertainty Description Experience Simulation Storytelling

a

b

s

t

r

a

c

t

Providinginformationfordecisionmakingshouldbeliketellingastory.Youneedtoknow,first,what youwanttosay;second,whomyouareaddressing;andthird,howtomatchthemessageandaudience. However,datapresentationsfrequentlyfailtofollowthesesimpleprinciples.Toillustrate,wefocuson presentationsofprobabilisticinformationthataccompanyforecasts.Weemphasizethattheproviders ofsuchinformationoftenfailtorealizethattheiraudienceslackthestatisticalintuitionsnecessaryto understandtheimplicationsofprobabilisticreasoning.Wethereforecharacterizesomeofthesefailings priortoconceptualizingdifferentwaysofinformingpeopleabouttheuncertaintiesofforecasts.We discussandcomparethreetypesofmethods:description,simulation,andmixturesofdescriptionand simulation.Weconcludebyidentifyinggapsinourknowledgeonhowbesttocommunicateprobabilistic informationfordecisionmakingandsuggestdirectionsforfutureresearch.

©2014SocietyforAppliedResearchinMemoryandCognition.PublishedbyElsevierInc.Thisisan openaccessarticleundertheCCBY-NC-NDlicense(http://creativecommons.org/licenses/ by-nc-nd/4.0/).

1. Introduction

UponarrivingincontinentalEuropeintheearly13thcentury,

FibonacciconvincedpeoplethattheHindu-Arabicnumerical

sys-tem wassuperior to Roman numerals for making calculations,

maintainingquantitativerecordsandconveyinginformation.His

workessentiallytransformedthelanguageinwhichanalyseswere

conducted and communicatedand thereby contributed

signifi-cantly toboth science and everyday life (Savage, 2009).Better

understandingof quantitative analyseseventually ledto better

judgmentsanddecisions.

Weproposethat providinginformationtohelppeoplemake

decisionscanbelikenedtotellingstories.First,theprovider–or

storyteller–needstoknowwhatheorshewantstosay.Second,

itisimportanttounderstandcharacteristicsoftheaudienceasthis

affectshowinformationisinterpreted.Andthird,theprovidermust

matchwhatissaidtotheneedsoftheaudience.Moreover,whenit

comestodecisionmaking,theprovidershouldnottelltheaudience

whattodo.Instead,thelattershoulddrawitsownconclusions.That

is,asinawell-craftedstory,theaudienceshouldbefreetointerpret

theoutcomeswithoutbeingtoldthe“message”directly(i.e.,what

todo).

∗ Correspondingauthor.Tel.:+34935422561.

E-mailaddresses:robin.hogarth@upf.edu(R.M.Hogarth),

emre.soyer@ozyegin.edu.tr(E.Soyer).

Inthispaper,wearguethatourstorytellingmetaphordoesnot

capturehowinformationistypicallypresentedfordecision

mak-inginappliedsettings.However,themetaphorcapturesprinciples

thatcanimprovedecisionmakers’understandingofthesituations

theyfaceandtheirsatisfactionwiththealternativestheyselect.1

Ouraimistohighlightandprovideaperspectiveaboutthese

prin-ciples,givenpossiblemethodsofcommunicatinginformationfor

decisionmaking.Weconsiderthestandardmethodofdescription

anduseitasabenchmarktodiscussthebenefitsandweaknesses

ofanalternativeapproach:providingexperiencethrough

simula-tions.Finally,wereflectonpossiblehybridtechniquesthatmerge

descriptionsandsimulations.Tomakeourideasconcrete,we

con-centratehereonthepresentationofinformationaboutuncertainty

associatedwithtakingdifferentactions.However,webelievethat

theprinciplesapplyacrossawiderangeoftypesofproblems.

Our interest in this issue was stimulated by a survey we

conducted of how economists interpret the results of

regres-sion analysis (Soyer &Hogarth, 2012). In this study,academic

economists from prestigious universities answered questions

aboutmakingdecisionsinlightoftheresultsofasimpleregression

analysis.Theeconomistsweregiventheoutcomesoftheregression

1_{Weemphasizethatweusetheterm“story”inametaphoricalmanner.Most}

forecastsare,ofcourse,notstoriesinthattheylackcharactersandplotsthatevolve

acrosstime.However,bothforecastsandstoriesrequiretransmittinginformation

inanaccessiblemanner.

http://dx.doi.org/10.1016/j.jarmac.2014.01.005

2211-3681/©2014SocietyforAppliedResearchinMemoryandCognition.PublishedbyElsevierInc.ThisisanopenaccessarticleundertheCCBY-NC-NDlicense(http://

analysisina typical,tabularformatand thequestionsinvolved

interpretingtheprobabilisticimplicationsofspecificactionsgiven

theestimationresults.Hence,theparticipantshadavailableallthe

informationnecessarytoprovidecorrectanswers,butingeneral

theyfailedtodoso.Althoughtheiranswerswereinfluencedbythe

uncertaintiesconcerningthemodel’sregressioncoefficients,they

tendedtoignoretheuncertaintyinvolvedinpredictingthe

depend-entvariableconditionalonvaluesoftheindependentvariable.As

suchtheyvastlyoverestimatedthepredictiveabilityofthemodel.

Oursurveyalsoinvolvedanothergroupofsimilareconomistswho

onlysawabivariatescatterplotofthedata.Theseeconomistswere

accurateinansweringthesamequestions.

Nowacademiceconomiststypicallydonotusetheresultsof

regressionanalysisfor decision makingpurposes andthus

per-hapsoursurveywas“unfair”.However,since theseeconomists

werestatisticalexperts(econometricians),theirpoorperformance

raisestheissueofwhatpeoplereallyunderstandwhentheyconsult

dataprovidedfordecisionmaking.Second,thatonegroupmade

accurateanswersafteronlyseeingascatterplotsuggeststhatsuch

displayscouldbeusedforbetterdecisionmaking.However,itis

notclearthatthissuggestionwouldbeacceptedbecause,despite

theaccuracyoftheiranswers,membersofthisgroupcomplained

bitterlythattheydidnothaveenoughinformationtoanswerthe

questionsadequately.

Asanexerciseinprovidinginformationfordecisionmaking,our

surveywasafailure.Thestorydidnotmatchwiththeaudience.

Inparticular,thestoryinthiscase(regressionresults)was

engi-neeredbytheanalyst,whoseprincipalaimistypicallynottobe

understood(intermsofimprovingjudgmentsanddecisions)but

justtobeheard(published).Ifnothingelse,ourstudyshowedthat

differentdescriptionsofthesameinformation,leadpeopletodraw

differentconclusions–aphenomenonthathasbeendocumented

manytimesintheliterature(Hogarth,1982).

2. Probabilisticforecasts–issuesandchallenges

Inthis paper,we considerthecommunicationof

probabilis-ticforecasts.Inessence,thismeansthattheanalystprovidesthe

decisionmakerwithaprobabilitydistributionoverpossiblefuture

outcomesofavariableofinterest.Thesecancovermany

differ-enttypesofapplications.Consider,forexample,simpleforecasts

involvingtheweather(e.g.,“Willitraintomorrow?”)asopposedto

morecomplicatedissuessuchaseffectsofclimatechange(Budescu,

Por,&Broomell,2012).Intheeconomicdomain,onecanenvisage

forecastsinvolvingsalesandinventories,aswellasoutcomesof

investments.Inpolitics,probabilisticforecastscancoverelections,

tradingdisputes,andsoon.

Weemphasizethisrangeofapplicationsbecauseanalystsand

decisionmakersmayhavequitedifferentconceptionswhenthey

consideradescriptionofadecisionmakingsituation.In

particu-lar,themeaningofprobabilityisnotcleartomanyinthatitdoes

notnecessarilymapintopeople’sexperience.Forexample,imagine

thatadecisionmakeristoldthattheprobabilityofraintomorrow

is0.3.Now,let’sassumeitdoesnotrainthenextday.Howshould

sheinterpretthemeaningoftheforecast?Wasitcorrector

incor-rect?Ourbemuseddecisionmakerisnotsurebecauseraincould

onlyoccurornotoccurandasingletrialcannotrevealwhethera

0.3probabilityestimateisappropriate(Lopes,1981).Ontheother

hand,forastatisticallysophisticatedanalyst,the0.3estimatecan

beinterpretedasapersonal“degreeofbelief”(supported

intellec-tuallybyaBayesianbettingparadigm)orasthelimitofalong-run

relativefrequency(imaginingmanydayswhentheweather

condi-tionswereidentical,i.e.,asafrequentiststatistician).

Giventheseissues,shouldanalystssimplyforgetabout

numer-ical estimates and instead use verbal statements that describe

feelingsofuncertainty?Indeed,severalstudiesshowthatverbal

expressionsofprobability(e.g.,phrasessuchas“unlikely”,“almost

certain”,andsoon)canberelativelyeffective(see,e.g.,Budescu&

Wallsten,1985).However,verbalexpressionsdonothaveexactly

thesamemeaning fordifferent peopleandit is problematicto

generalizefromtheseresults.

Afurtherprobleminprovidingforecastsintheformof

prob-abilities tostatistically naïvedecision makers is that thelatter

maymake assumptionsofwhichtheanalystsareunaware.Ina

revealingstudy,Gigerenzer,Hertwig,vandenBroek,Fasolo,and

Katsikopoulos(2005)askedpeoplewhattheythoughtwasmeant

byweatherforecastsoftheform“theprobabilityofraintomorrow

is30%”.Therewasawiderangeofdifferentinterpretations

includ-ingthepossibilityofhavingrainduring30%ofthedayand30%of

theregionreceivingrainduringthatday.

Atonelevel,theseinterpretationsareamusing.However,itcan

bearguedthatthestatementtherespondentswereaskedto

inter-pretwasambiguous.Whatismissingisclarificationofhowone

wouldknowwhetherornotithadrainedonthemorrow.Lacking

thisinsight,itispossibleforpeopletohaveseveralinterpretations

evenifstatisticalexpertswouldnotthinkofthem.Statementsof

probabilitiesofeventsshouldbeaccompaniedbyoperational

def-initionssuchthattheoccurrenceornon-occurrenceoftheevents

cannotbedisputed.Forexample,ifapersonmakesabetconditional

ontheoccurrenceoftheevent,heorsheshouldnotsubsequently

beabletoavoid responsibilitybychangingthedefinitionofthe

event.2

Finally,peoplemaydiffernotonlyinstatisticalexpertisebutalso

expertiseconcerningtheissueathand,e.g.,meteorologistsknow

muchmoreabouttheweatherthannon-meteorologists.Whatis

uncleariswhethersuchsubstantiveexpertiseaffectstheabilityto

interpretprobabilisticforecasts.

3. Humaninformationprocessing:strengthsand weaknesses

Weassumethat,priortoprovidingprobabilisticforecasts,the

analystshavemadetheappropriateanalyses.Thisbeingsaid,we

nowconsidersomehumanstrengthsandweaknessesin

informa-tionprocessingsinceitisimportanttounderstandthefactorsthat

helpandhinderpeopleinthetaskofinterpretinginformation.

Althoughresearchinpsychologyandneurosciencecanleadone

tomarvelatthecapacityofthehumanmind,fromourperspective,

therearelimitations.In particular,limitsonprocessingcapacity

restrictourabilityto“takein”alltheinformationthatmaybe

rel-evanttoaproblem.Atanypointintime,wecanonlyperceivea

smallfractionofwhatisactuallyinourvisualfield.Thus,anything

thatattractsattentionisimportantand therealityin whichwe

operateisboundbythisattentionalfilter.Indeedtheliteratureis

repletewithexamplesofhowminorshiftsinthepresentationof

informationcanchangeaperson’sconclusions(Einhorn&Hogarth,

1981;Hogarth,1982).Toovercomesuchproblems,through

expe-riencepeoplehavedevelopedskillsinseekingspecificinformation

inparticularsituationssothatattentioncanbeguided

appropri-ately.Indeed,thisliesattheheartofexpertise(Ericsson&Smith,

1991).

Asecondlimitationisthatpeopleoftenfailtoconsiderrelevant

informationpreciselybecauseitdoesnotformpartofthe

infor-mationpresentedandtheylacktheabilitytorecognizethisfact.

Considerpublicationbias(Ioannidis,2005).Academicpublications

makeinformationpartofthepublicdomain;easilyreachableby

allconsumersofknowledge.If certainanalyses(thosethatfind

statisticallysignificanteffects)aremorelikelytobepublishedthan

others(thosethatfailtofindeffects),thenthestory,thatis,the

con-clusionfromtheavailableevidenceisasbiasedastheevidenceon

whichitisbased.Imagine,forinstance,aphysiciancontemplating

whethertoprescribeacertaindrug.Say,amongthe20publications

thatcontainvalidtestsofthedrug’seffectiveness,17demonstrate

positiveresults.Theverdictseemsclear(17vs.3).If,however,there

arealso15unpublishedmanuscriptswithvalidtests,ofwhich13

shownoeffects,thestorynowchanges(19vs.16)andthedecision

becomesharder(forcasesonselectivepublication,seeGoldacre,

2013;andTurner,Matthews,Linardatos,Tell,&Rosenthal,2008).

Another important dimension is the distinction between

whethertheinformationispresentedatonce–asinthetypical

descriptionofaproblem–orwhetherithasbeenacquiredacross

time.Asanexampleofthelatter,imaginedecidingbetweentwo

restaurantsthatarewellknowntoyou.Inessence,youalreadyhave

estimatesof“howgood”bothrestaurantsarebasedonpast

expe-riences.Moreover,foreachtheestimatesarebasedonaggregating

yourexperiencesacrosstime.Thatis,yourestimatesarebasedon

sequentialupdatingoftheimpressionsofyourdifferentvisitsas

opposedtoaccumulatingalltheimpressionsatonetime.Indeed,

withasequentialupdatingprocess,allthatyouneedtoremember

isyourlastoverallimpression.

Inshort,limitsoninformationprocessingarenotsoimportant

whendealingwithwhatwecallsequentialdata.Indeed,numerous

studieshavedemonstratedthatpeopleareremarkablyeffectiveat

extractingaggregatefrequenciesfromsingleeventsthattheyhave

previouslyexperiencedandencodedsequentially(Hasher&Zacks,

1979,1984;Zacks&Hasher,2002).3_{Moreover,thisappearstobe}

awell-developedskillinthatitshowslittlevariationacrossthe

lifecycleandisusedinmanydifferenttasks.Thefactthatithas

beenobservedinseveralnon-humanspeciesalsosuggeststhatit

iswellanchoredinevolution.Fromourperspective,itisimportant

becauseitshowsthetypeoftaskenvironmentinwhichhumans

canovercomeprocessingcapacityconstraints.

In summary, although subject to attentional shifts, human

informationprocessingreactsaccuratelytoinformationobserved

sequentiallyacrosstime.However,thesystemisdeficientinthat

itoperatestoo“literally”.Manystudieshaveshown thatpeople

treatthedatatheyseeasrepresentativeoftheprocessesthat

gen-eratethem,thatis,theyare“naïve”statisticians(Juslin,Winman&

Hansson,2007;Peterson&Beach,1967).Asaconsequence,people

makeinferentialerrorsdue,interalia,to“smallsample”effectsand

thefailuretorealizethatsamplescanbebiasedindifferentways.

Indeed,thisfailuretorecognizeand correctforbiaseshasbeen

labeleda lackof“meta-cognitive”ability(Fiedler,2000;Fiedler,

Brinkmann,Betsch,&Wild,2000).

Clearly,lackofmeta-cognitiveabilitycoupledwiththeinability

totakeaccountofmissingdatameansthatpeople’sjudgmentsare

oftendefective.Moreover,theyaretypicallyunawareofthisfact.

InThinkingFastandSlow,Kahneman(2011)discussesthenotion

ofWYSIATI(acronymfor“whatyouseeisallthereis”),i.e.,the

tendencyofhumanstobasetheirjudgmentspredominantly on

theinformationthat isreadilyavailable.Moreover,humansare

inclinedtoconsidersuchavailableyetpotentiallybiased

informa-tionasthewholestory.Recognizingtheseissues,manyattempts

havebeenmadetohelppeoplemakemoreaccuratejudgments.

3_{Toillustrate,considerbeingaskedhowmanytimesyouhavebeentothecinema}

inthelastthreemonths.Mostpeoplecananswerthisquestion(albeitnotalways

completelyaccurately)despitethefactsthat(a)theydidnotknowtheyweregoing

tobeaskedthisparticularquestion,and(b)whentheyattendedthecinematheydid

notmakeaconsciousefforttorecordthefrequencyoftheirvisits.Ofcourse,wedo

notclaimthatpeoplehaveperfectmemory.Surveyresearchers,forexample,have

documentedseveralpatternsofsystematicdistortions(see,e.g.,Bradburn,Rips,&

Shevell,1987).

Fig.1.Twowaysofcalculatingtheprobabilitythatawomanwhotestspositive

inmammographyscreeningactuallyhasbreastcancer(positivepredictivevalue).

Theleftsideillustratesthecalculationwithconditionalprobabilities,andtheright

sidewithnaturalfrequencies.Thefourprobabilitiesatthebottomofthelefttree

areconditionalprobabilities,eachnormalizedonbase100.Thefourfrequenciesat

thebottomoftherighttreearenaturalfrequencies.Thecalculationusingnatural

frequenciesissimpler(smilingface)becausenaturalfrequenciesarenotnormalized

relativetobaseratesofbreastcancer,whereasconditionalprobabilities(orrelative

frequencies)are,andneedtobemultipliedbythebaserates(theformulatocalculate

thepositivepredictivevalueisknownasBayes’srule).

AdaptedfromFig.3inGigerenzeretal.(2007,p.56).Copyright2007bySage

Pub-lications.ReprintedbypermissionofSagePublications.

4. Varietiesofdecisionaids

Almostalldecision aidsinvolve changinghow problemsare

describedtohelppeoplemake“better”decisions.Therangeofaids

variesfromcomplexdecisionanalytictechniques(involving

deci-siontrees,multi-attributeutilityfunctions,andsoon)tosimply

reframingproblemsinordertodirectattentioninparticularways.

Thelatterapproachisparticularlyinterestinginthatitcombines

psychologicalinsightsofhowpeopleprocessinformationwithan

understandingofthetaskstheyface.

AgoodexampleistheworkofGigerenzerandHoffrage(1995)

whoexploredhowtohelppeoplemakeso-calledBayesian

updat-inginferences.Imagine,forinstance,assessingtheprobabilityof

havingaparticulardiseasegivenapositivetestresult.Thetypical

presentationusedinthedescriptionoftheseproblemsprovides

thecomponentprobabilitiesthatshouldbecombinedbyBayes’

theorem,i.e.,thepriorprobabilityofhavingthediseaseandthe

sensitivityandspecificityofthetest.Mostpeople,however,are

quiteconfusedabouthowtocombinethisinformationcorrectly.

GigerenzerandHoffragearguedthatsuchconfusionwasnot

sur-prisinggiventhatpeople’sexperienceisnotintheformofthese

componentprobabilitiesbutcanbemoreaccuratelyrepresented

byfrequencydata.Thus,ifproblemsweredescribedintermsof

“naturalfrequencies”peoplewouldbothunderstandthedata

bet-terandbeabletoperformthenecessarycalculationsmoreeasily–

seeFig.1.

ThatpeoplecanlearntodoBayesiancalculationsusingthe

nat-uralfrequency methodhasbeen welldocumented(Gigerenzer,

Gigerenzer,2001).However,thisisbutonesolutiontoonespecific

typeofinferentialproblem.

Morerecently,ThalerandSunstein(2008)highlightedchanging

themannerinwhichdecisionproblemsaredescribedinapopular

bookentitledNudge.Theirmainargumentisthat thereisoften

considerableleewayinhowchoicesaredescribed.Thus,decisions

canbeimprovedifathirdpartyredefinesproblemsandtellsthe

storyinanalternativeway.Onestrikingexampleinvolveshelping

peopletoinvestmoreinpensionplansbycommittingtosaveout

offutureincreasesinsalaryasopposedtoreducingcurrentsalary

(Thaler &Benartzi,2004).Sincemany pensionplansare

under-funded,such“nudges”areimportant.

Theimplication of thisapproach is that increatinga nudge

somebodyhastoknowwhatis“good”forthedecisionmakerand

thiscanraiseethicalissuesthatThalerandSunstein(2008)

recog-nize.Indeed,theydescribetheirapproachaslibertarianpaternalism.

Thatis,althoughoneproblempresentationformathasbeen

cho-sen“paternalistically”forthedecisionmaker(atleastonehastobe

chosen),peoplearestill“free”tomaketheirownchoicesandcan

changetheproblemformatiftheywant.

5. Analternativeapproach

Recentworkinthepsychologyofjudgmentanddecisionmaking

hashighlightedthefactthattheinformationhumansusefor

deci-sionmakingcanbeconceptualizedashavingtwodistinctsources

(Hertwig,Barron, Weber,& Erev, 2004;Hertwig &Erev, 2009).

Ontheonehand,peopleacquireinformationaboutjudgmentand

decisionproblemsthroughdescriptionoftheparticularsofthe

sit-uationsinvolved.ThisisexactlythecaseoftheBayesianinference

problemsofGigerenzeretal.(2007)describedaboveandthework

ofThalerandSunstein(2008).

Onthe otherhand,peoplealsolearn aboutthespecific

fea-turesofproblemsthroughexperienceofpastinstances.Forexample,

imaginetheownerofasupermarketwhowondershowmany

cus-tomerswillenterthestoreonaparticulardayoftheweek.Whereas

theownercouldhavesomedescriptionofthisproblem,heorshe

undoubtedlyhashadexperienceofthissituationfromthepastand

cancalluponthisexperiencetoestimatethenumberofcustomers.

Inbrief,peoplecanlearnaboutthecharacteristicsofjudgment

anddecisionproblemsthroughdescriptionorexperienceand,of

course,mixturesofthetwo.Mostdecisionaidinghasfocusedon

exploringeffectsofdifferentproblemdescriptionsand,ashasbeen

shown,isimportantbecausehumanjudgmentsanddecisionsareso

sensitivetodifferentaspectsofdescriptions.Atthesametime,this

verysensitivityisproblematicinthatdifferenttypesofjudgments

anddecisionsseemtoneeddifferentsolutions(eventhoughsome

conceptssuchas“lossaversion”havefoundwideapplicability).

Thereisaneedtofindmethodswithmoregeneralapplication.

We suggestexploiting thewell-recognizedhumanability to

encodefrequencyinformation.Thatis,weconsiderthepossibility

oftransformingproblemssothatpeoplelearnaboutthemthrough

experienceasopposedtodescription.Wearguethatsince

deci-sionandjudgmentproblemsinvolveoutcomes(i.e.,gains,losses,

estimates,etc.),beingabletodescribeaproblemalsoimplies

hav-ingtheinformationnecessarytobuildasimulationmodelthatcan

beusedtogenerate“outcomes” througha processthat wecall

“simulatedexperience”.

6. Experiencevs.description

Fromtheviewpointofstorytelling,simulatedexperience

essen-tiallyallowsadecisionmakertoliveactivelythroughadecision

situationasopposedtobeingpresentedwithapassive

descrip-tion.Asimpleexamplewouldbetheproblemofestimatingthe

Table1

Presentationofaregressionanalysis:Samplestatisticsandestimationresults.

Samplestatistics

Samplemean Standarddeviation

Changeinincome 8.4 7.9

Investment1 11.1 5.8

Investment2 9.6 5.2

Estimationresults(dependentvariable:changeinincome)

Coefficientestimate Standarderror

Constant −0.1 0.15

Investment1 0.5 0.20*

Investment2 0.3 0.05*

R2 _0.21

Numberofobservations(n) 1000

(DataoriginallyreportedinHogarth&Soyer,2011,Fig.5,p.445).

*_{Statisticalsignificanceat95%confidencelevel.}

probability ofobtaining,say a sumof four,when two diceare

cast.Thissituationcanbedescribed byproviding thestructure

and parameters of the problem (two six sided dice, identical

shapes,havingdotsoneachsiderepresentingnumbersfromone

tosix),suchthat thecorrectanswer canbecalculated

(analyt-icalapproach).Alternatively, anexperiential approachcouldbe

taken,castingthedicemanytimesandobservingtheprevalence

ofacertainoutcome(sumoffour)amongmanysuccessivetrials

(experientialapproach).

Inthisparticularexample,thedescriptioniseasytounderstand

andtheanalyticalapproachwouldproduceapreciseandaccurate

answer.Theexperientialapproach,ontheotherhand,wouldbe

timeconsuming,leadtoalesspreciseresponseandrequirethe

physicalavailabilityof twodice.However,imagine whatwould

happeniftheproblemweremorecomplicated.Forexample,

con-siderthetaskofestimatingtheprobabilitythattheproductofthe

outcomesislargerthaneightwhenoneofthedicehasfoursides

insteadofsix.Now,relativetoananalyticalapproach,basing

judg-mentsonexperiencewouldstarttobecomeeasierandlesserror

proneduetolackofexpertiseinprobabilitytheory.Parenthetically,

wenotethatthedifferencebetweenresolvingproblemsthathave

beendescribedasopposedtoexperiencedisrelatedtoBrunswik’s

(1952)distinctionbetweentheuseofcognitionandperception.

Intheformer,peoplecanbequiteaccurateintheirresponsesbut

theycanalsomakelargeerrors.Inthelatter,theyareunlikelyto

behighlyaccuratebuterrorsarelikelytobesmall.

BothLejarraga(2010)andHogarthandSoyer(2011)suggest

thatasthecomplexityofproblemsgrows,experientialapproaches

leadtoimprovementsinjudgmentsanddecisions.Insuchcases,

peoplealsotrusttheirexperiencesmorethantheiranalytical

intu-itions.Consider, forinstance,a scenariowheredecision makers

canmaketwopossibleinvestmentsand thenhavetojudgethe

probabilityofdifferentpossibleconsequencesoftheiractions,such

asendinguplosingmoney, earningmoreorless thanacertain

amount,or earningmore thansomeonewhodidnotmakeany

investments.Whenaregressionanalysisisconductedtodetermine

theyieldsoftheinvestments,thedescriptionoftheoutputswould

usuallyincludetwotables,onewithdescriptivestatisticsandother

withestimationresults(seeTable1).

Alternatively,asimulationcanbebuiltbasedontheestimated

modeltoallowuserstoentertheirdecisions(investments)and

observemodelpredictionsasoutcomes(seeFig.2).Acomparison

ofjudgmentsusingthesetwoapproachesrevealsthat,regardless

ofstatisticalsophistication, simulatedexperienceleadstomore

accurateperceptionsabouttheuncertainties.Description,onthe

otherhandleadstoanillusionofpredictability,wherethe

Fig.2.Simulationinterfaceforanestimatedinvestmentmodel.Eachtimethe

SIM-ULATEbuttonisclicked,apredictedoutcome(thechangeinincome)isshownbased

onboththeuser’sinputs(investmentchoices)andtheestimatedparametersofthe

model.

mainlythroughtheaveragecoefficientestimates.Hence,simulated

experienceprovidesausefulwayofcommunicatingsuch

probabil-ityestimates(Armstrong,2012;Hogarth&Soyer,2011;Hogarth

&Soyer, in press-a, in press-b). In order toidentify when this

approachshouldbepreferredtodescription,however,weneedto

discusstheprosandconsofsimulationmethodologyinthecontext

ofjudgmentanddecisionmaking.

7. Advantagesofsimulatedexperience

Letusfirstconsidertheadvantages.

7.1. Simulationtechnology

Therearenotechnologicalbarrierstosimulatingcomplex

pro-cesses.Justadecadeago,buildingsimulationsandusingthemwas

adifficultandslowendeavor.Today,thetechnologyallowsforthe

constructionofsimulationsforvirtuallyanydecisionscenarioand

canfunctionquiterapidly.

7.2. Easeofuse

Ourinvestigationsshowthatpeoplerelateeasilytothetaskof

simulatingsequentialoutcomesandinteractseamlesslywith

sim-ulations.Moreover,asproblemsgrowmorecomplex,theyprefer

theirexperientialintuitionsovertheiranalyticalcalculations.

7.3. Statisticalknowledge

Simulatedexperiencecanbeemployedevenwhenthedecision

makerhaslimitedknowledgeofstatistics.Forinstancethe

descrip-tioninTable1isforeigntoanyonenotfamiliarwithregression

analysis.Ontheotherhand,asimulationbasedonsuchresultscan

beusedbyanyonewhohaspreviouslyoperatedacalculatorora

simplespreadsheet.

7.4. Freedomofchoice

Iftheexperienceproducedbythesimulationiskind(i.e.,

abun-dant, unbiased and immediate, Hogarth, 2001, 2010) then the

experientialapproachprovidesthedecisionmakerwithacomplete

pictureoftheoutcomesofaprocessratherthanaframethatnudges

hertoaparticularchoice.Clearly,simulatedexperienceneedsa

simulationtobebuilt(byathirdpartyorinsomecaseseventhe

decisionmaker)butthisisnotaparticularstatementoftheproblem

thathasbeenselectedbysomebodyelseintheguiseoflibertarian

paternalism.

7.5. Activeparticipationinthedecisionprocess

Relatedly,simulatedexperiencemakessurethat,bybuilding

thesimulationorsimplybysamplingexperience,decisionmakers

activelyparticipateinthedecisionmakingprocess.Thiscanleadto

increasedunderstandingoftherelationbetweentheiractionsand

theconsequentoutcomes.

8. Disadvantagesofsimulatedexperience

Simulatedexperiencealsohassomedisadvantagesandprovides

severalchallengesforfutureresearch.

8.1. Samplesize

Simulatedexperienceallowsdecisionmakerstodeterminethe

number of trials they wish to experience. Hence, sample size

becomesanimportantfactorintheprecisionofjudgments.This

leadstoaskingwhataffectspeople’schoiceofsamplesizeandwhat

constitutestheoptimalnumberofobservations(Hertwig&Pleskac,

2010).

8.2. Rareevents

The literature on the so-called “decision-experience gap”

suggeststhatwhereasKahnemanandTversky’s(1979)

description-basedprospecttheoryimpliesoverweightingofsmallprobabilities,

decisionsbasedonexperienceareconsistentwithunderweighting

(Hertwigetal.,2004).Bydefinition,rareeventswillnotbe

expe-riencedofteninsimulationswiththeconsequencethatdecision

makersmightnotpaymuchattentiontothesewhenmaking

judg-mentsanddecisions.Onewaytoovercomethisproblemwouldbe

touseconditionalsimulations.Forexample,consideraBayesian

updatingproblem. Ifthesimulationisbuiltforincidents

condi-tionalonacertainpriorevent,thentherareoutcomeswouldbe

morevisible(e.g.,theprobabilityofhavingadisease,givena

pos-itiveresultinamedicaltest;orthepossibilityofalossbeyonda

certainamount,giventheoccurrenceofanaturaldisaster).

8.3. Riskattitudes

Someresearchhasshown thattheeffectof simulated

expe-rience,asopposedtodescription,islessriskaversioninchoice.

Kaufmann,Weber,andHaisley(2013)arguedthat,forinvestment

portfoliochoices,suchriskattitudesaremoreinlinewithrational

choicetheory.However,sincepeoplemayhavelegitimatereasons

tohavedifferentriskattitudesforsingleasopposedtorepeated

choices,theissueofhowriskattitudesareaffectedbysimulation

experienceneedstobeinvestigated.

8.4. Statisticalsophistication

Simulatedexperiencedoesnotrequirestatisticalsophistication

onthepartof theuser.Thus,its benefitsare largerin contexts

wheretheanalysisiscomplicatedandhardtodescribeto

some-onewithlimitedpriorknowledgeinstatistics.However,theuser

stillneedstobeintroducedtothesimulationinterfaceandtrained

initsoperation.Onekeyissueissamplesize(seealsoabove).In

ourexperiments,statisticallysophisticatedpeoplesampled

consis-tentlymoreinformation,which,inturn,ledtobetterjudgments.

Simulationtrainingshouldincludeinformationonthelawoflarge

numbers.

8.5. Trustinthemodel

Usersshouldtrustthemodelstheyuseforsimulatedexperience.

Atpresent,littleisknownaboutthedeterminantsofsuchtrust.

Thesame,ofcourse,isalsotruefordescriptions.Futureresearch

trustdescriptionsandexplanationsbyexpertsaswellas

experi-encesbasedonsimulations.Intermsofusersatisfaction,recent

analysesbyGoldstein,Johnson,andSharpe(2008)andKaufmann

etal.(2013)suggestthatsamplingincreasesbothcomprehension

andsatisfactionaboutdecisionsunderuncertainty.

8.6. Simulationbuilding

Themethodofsimulatedexperiencerequiresbotha

simula-torbasedontheunderlyingdecisionsituationandauser-friendly

interface. Although popular spreadsheet software, such as MS

Excel,canprovideviablebasesforsuchsimulations,onestillneeds

someproficiencyinprogrammingtobuildareliabledecisiontool.

Tosolvethis,aplatformofsimulationscanbecreatedthatincludes

asetofsimulationsforavarietyofrelevantsituations.Moreover,

sucha platform mightalsoincorporatemodules thatusers can

transformandcombinetocreatecustomsimulations.Forinstance,

differentmodulesmightallowusers toselectfroma varietyof

modelspecifications(e.g.,distributionswithfattails,skewed

dis-tributions,etc.).

8.7. Knowledgetransfer

Thispaperdiscussesthesimulatedexperiencemethodologyas

ameanstocommunicateinformationtodecisionmakersandwe

havelimitedourdiscussionstomattersofpresentation.However,

aweaknessofsimulationtechnologycouldbethatthedecision

makerdoesnotgaininsightintotheproblemstructurethatcanbe

generalized.Ontheotherhand,inourexperimentsweobserved

that,whenabletoconsultboth a descriptionand a simulation,

people’sanalyticalcalculationsimproved(Hogarth,Mukherjee,&

Soyer,2013;Hogarth&Soyer,2011).Theirexperiencesprovideda

meanstochecktheaccuracyoftheiranalysis.

9. Simulatedexperienceinsteadofdescription

Descriptionis typically thedefault modeofproviding

infor-mationfordecisionmaking.Thus,whenshoulditbereplacedby

simulatedexperience?Ourdiscussiononexperiencevs.description

highlightstwomajorshortcomingsofeverydescription.First,it

alwaysinvolvesaframe,andthusdifferentdescriptionsmightlead

todifferentperceptionsanddecisions.Second,forsomeproblems

thatincorporateuncertaintiesorcomplexstructures,descriptions

mightbehardtoconstructorobscureacrucialpartofthestory.

Simulatedexperienceessentiallytakesdescriptionoutofthe

pic-ture;henceitshouldsubstitutedescriptionsinsituationswhere

theseshortcomingsareprevalent.

9.1. Uncertainties

Describinguncertaintiesinherentinadecisionsituationis

dif-ficult.Examplesincludesituationswhereoutcomesaresubjectto

regressiontowardthemeanorwhenavariableisacomplex

func-tionofmanyindependentvariablesandpredictionerrors.

In a series of experiments we have foundthat in scenarios

similartothatdescribed inTable1,descriptionslead peopleto

overestimate the predictability of the dependent variable. The

mainreasonbehindthisillusionofpredictabilityisthat

descrip-tionframesthequestionmainlyaroundaverageeffects(Hogarth&

Soyer,2011;Soyer&Hogarth,2012).Moreimportantly,ourresults

suggestthataugmentingthedescriptionwithexperience,e.g.,

let-tingdecisionmakersexperiencesimulatedoutcomesontopofa

description,doesnotleadtobetterperceptions.Thisisbecause

peopleareusedtorelyingprimarilyondescriptions.Itwasonly

whenweeliminatedthedescriptionandconstrainedthedecision

makerstoonlyvisualizesimulatedoutcomesthattheirjudgments

improved.

9.2. Complexstructures

Inproblemswithcomplexprobabilisticstructures,descriptions

maynotonlybehardtodecipher,theycouldalsomisleadone’s

analysisofthesituation.Considerforinstancethebirthdayproblem

(i.e.,Whatistheprobabilitythattwoormorepeoplehavethesame

birthdayinagroupofNpeople?)andthefamousMontyHall

prob-lem.Researchhasrepeatedlyshownthatattemptstosolvethese

problemsanalyticallyleadtobiasedanswerslargelybecause

peo-plefinditdifficulttounderstandtheproblemstructures(Hogarth

&Soyer,2011).

Inthesesituations,decisionmakerscanbeprovidedwith

sim-ulations that let them live through the problems many times

(learningthebirthdaysofmultiplegroupsofNpeopleinthe

birth-dayproblem,andselectingonedooramongthreeintheMontyhall

problem).Inthisway,userslearnabouttheoutcomeprobabilities

throughexperience,withoutresortingtoanybias-proneanalyses.

Similarly,simulationshavebeenshowntobeusefulformitigating

baserateneglect(Hayes,Newell,&Hawkins,2013),andassessing

probabilitiesofsuccessincompetitions(Hogarthetal.,2013).

10. Hybridapproaches

Inourexperiments,participantsoftenshoweddiscontentwhen

theylackedaccesstodescriptions,evenwhensuchaccessbiased

theirjudgments. Hence,advocatingfortheabolitionof

descrip-tionsincertainsituationsisnotrealistic.Also,consideringthatboth

descriptionandsimulationhaveadvantagesanddisadvantagesit

isimportanttoinvestigatewherethesemethodsmightbeused

togetherforoptimalstorytelling.

10.1. Graphs

Onetoolthatmergesdescriptionswithsimulationisagraph

ofsimulatedoutcomesthatresultfromthedescribedmodel.Such

aplotwouldfollow thedescriptionoftheproblemand include

individuallevel data (simulatedor real) that makes visiblethe

uncertainty inherent in the outcomes. Advances in computing

technologyhave facilitatedtheway wedesign and use

graphi-calillustrations.Recentresearchshowsthatinrelevantsituations,

suchasmedicaldecisionmaking,depictingindividualdatathrough

interactivegraphicsthatallowdecisionmakerstovisualize

simu-latedoutcomeshelpprobabilisticunderstanding(Ancker,Chan,&

Kukafka,2009;Ancker,Weber,&Kukafka,2011).Moreover,news

mediaandonlinecommunitiesmakeuseofabroadselectionof

diagramsandinfographicstoinformtheirreadersaboutawide

varietyofstatistics(foracomprehensivereviewandexamplesof

suchapproaches,seeSpiegelhalter,Pearson,&Short,2011).

Although plotting individual level data is increasingly

con-venient andstraightforward, a survey weconducted ofapplied

economicspublicationsinprestigiousacademicjournalsreveals

thatapproximatelyonly40%oftheanalysesthatcanprovidescatter

plotsactuallydoso(Soyer&Hogarth,2012).Moreover,apartof

thesedisplaysarelimitedtomainlyshowingaveragetrends.

Fig.3showshowprovidingsimulatedoutcomesatthe

individ-uallevelingraphswouldleadtobetterperceptionsofuncertainty.

Inallthreefigures,thestraightlinedepictstheestimatedaverage

relationbetweenvariablesXandY,whichisone-to-oneandthe

sameinallthreecases.ThedifferencebetweenFig.3aandbisdue

todifferentlevelsofuncertaintyattheindividuallevel;the

pre-dictabilityofYfromXislowerintheformerduetolargervariation

Fig.3.(a–c)TheaverageeffectofXonY(thestraightline).(aandb)Simulatedindividualobservations(n=300).Inthescenariodepictedin(a),thevarianceoftheuncertainty

ofYconditionalonvaluesofXislargerthanthatofthescenariodepictedin(b).(c)doesnotdepictsuchuncertainty.

levelishiddenandthegraphonlydisplaystheestimatedaverage

effect.

Aweaknessofthisvisualapproachisthatitoversimplifiesthe

analysis by reducing the wholestory toa graph between two

variables.Mostoftheanalysesinappliedworkinvolvemultiple

variablesandfactorsthatinteractincomplexfashiontoproduce

theoutcomes.However,wearguethatthepresenceofevena

sim-pleplot showingthepredicted outcomesgiven differentlevels

ofanindependentvariableconditionalonthemeanvalesofthe

remainingcontrolvariableswouldhelpthedecisionmakeravoid

developinganillusionofpredictability.

10.2. Add-onsimulations

Whenusedfordecisionmaking,presentingresultsintheform

illustratedinTable1leadstoillusionsofpredictability.Moreover,

theimportanceofthisfindingisaccentuatedbythefactthatthis

presentationmethodisprevalentinmostscientificstudies,and

especially in applied economics. Publications in these domains

mightthereforeconsiderfeaturingonlinesimulationsto

accom-panyarticlesthathavedecisionmakingorpredictionimplications.

Suchsimulationswouldbebasedontheestimatedmodelandallow

peopletolivethroughtheproblemmanytimesbysequentially

observingpotentialoutcomesgiventheirinputs.Moreover,this

wouldnotdisruptthecontentofthedescription.Indeed,these

add-onsimulationswouldconstituteamuchneededbridgebetween

scientificanalysesanddecisionmakingactivities.

10.3. Sparklines

In theircreator EdwardTufte’s words,sparklines are“small,

high-resolution graphics usually embedded in a full context of

words, numbers, images. (They) are datawords: data-intense,

design-simple,word-sizedgraphics”(Tufte,2006,p.47).Inother

words,sparklinesarewithin-text-plotsthatreplacewordswhen

conveyingstatisticalinformation.Theseplotsusuallyhaveno

num-bersoraxesonthem.Theirpurposeistomakethedataandits

distributionreadilyavailableforquickandeasyinspection.

Such items are still not employed in academic or popular

writings,even though theyprovide genuine insights aboutthe

data,withoutrequiringdiscussionabouttheirstatisticalproperties

throughaseparatetableorvisualdisplay.Considerthefollowing

twoexamples:

1.“The Lakers’2004 season was

theirlast with Shaq,when they reachedtheNBA finals and

losttoDetroit(notethelast 3losses whichsealed theirfate

inthefinals).Comparethosedaysofglorywiththeirabysmal

2005performance,withonly2winsin

thelast21games”(top,graylinesrepresentwins,bottom,black

onesrepresentlosses,Gheorghiu,2005).

2.Peopleshowdifficultyinjudgingthechancesofsuccessin

com-petitions.Whereastheprobabilitiesofwinningaredistributed

,thejudgmentsrevealthatdecisionmakersactasif

theyare (Soyer&Hogarth,inpreparation).

Thisresultsuggeststhatanefficientmethodologytopresent

informationtodecisionmakerswouldinclude acombinationof

descriptionandsimulation.

11. Discussionor“backtostorytelling”

We have discussedhow information gatheredby ananalyst

iscommunicatedtopartieswhowilluseit fordecision making.

Althoughourinitialobservationsresultfromcasualempiricism,

ourgeneralcontentionisthatsuchinformationisnotalways

com-municatedsatisfactorily.Moreover,wecontendthatmuchcould

begainedbyrespectingnormsofgoodstorytelling.Ataminimum,

thisinvolvesknowing:(1)preciselywhatonewantstosay,that

is,themessage;(2)relevantcharacteristicsofthedecisionmaker,

thatis,theaudience;and(3)howtomatchthemessagetothe

audience.Ingeneral,theanalystshouldnotbetellingthedecision

makerwhattodobutinsteadprovideinformationthatallowsthe

decisionmakertoreachhisorherownconclusions.

Asaspecificexample,wefocusedonprobabilisticinformation

that accompanies forecasts.Asnoted, this encompassesa wide

rangeofactivitiesrangingfromsimpleweatherforecaststo

out-comesofmedicalprocedures,financialdecisions,climatechange,

andsoon.Wepointedoutthatmanypeoplehavedeficientnotions

ofprobabilisticreasoning.For example,thereismuchignorance

aboutthe meaningof probability and what conclusionscan be

drawnfromdata.Wealsonotedthat,althoughlimitedas

informa-tionprocessors,humansareadeptatestimatingthefrequenciesof

eventstheyhaveexperiencedsequentially.However,theytendto

treatsamplesas“representative”andlack“meta-cognitive”ability

tocorrectforbiasedobservations.

To date, attempts to help people understand probabilistic

forecasts or reasoning (i.e., to match message and audience)

have mainly involved describing problems in alternative ways

(Gigerenzer&Hoffrage,1995;Thaler&Sunstein,2008).Instead,

wesuggestbuildingonpeople’sabilitytounderstandandencode

frequencydatabyhavingthemexperiencesimulatedoutcomesor,

what wecall“simulated experience”.Importantly,this requires

thatthedecisionmakerplaysanactiveroleinthecommunication

process.Moreover,inallthecasesthatwehaveexamined,such

simulatedexperiencedoesmeetthecriteriaofgoodstorytelling

Theuseofsimulatedexperiencetoinformdecisionmakersof

theprobabilisticimplicationsofpredictionsisinitsinfancy.Above,

weoutlineddifferentconceptualandmethodologicalchallenges

thatstillneedtoberesolved.Inourview,goodstorytellingwill

involveboth descriptionandsimulation,therebyharnessingthe

advantagesofboth.Atamoregenerallevel,thechallengeweface

istodevelopunderstandingofwheninformationshouldbe

pre-sentedasdescription,experience,orboth.Forexample,weknow

thatcomplexityfavorsexperienceoverdescriptionbutweneedto

beabletodefineboundaryconditions.

Finally,followingFibonacci’sadvicetoadopttheHindu-Arabic

numericalsystem,humanshaveprogressedsubstantiallyintheir

abilitytoencode,store,andmanipulatedata.Indeed,thecurrent

explosionof“bigdata”owesmuchtohavinganappropriate

numer-icalsystemandmuchwillbegainedbythosewhocaninterpret

thisnewtroveofinformation.However,sinceinterpretationwill

alwaysbeahumanactivity,futureprogresswilldependonhow

wecopewiththisbottleneck.Wesuggestthatexploitingpeople’s

abilitytoprocesssimulatedexperienceisonewaytoenhancethe

useofintuitioninorganizationaldecisionmaking.

Conflictofinterest

Theauthorsdeclarethattherearenoconflictsofinterest.

Acknowledgments

TheauthorsthankSamSavage,theeditors,GaryKlein,Mitra

Galesic,andananonymousreviewerfortheirconstructive

com-ments.Hogarthgratefullyrecognizesthefinancialsupportofthe

SpanishMinisteriodeEconomíayCompetitividad(Grant

ECO2012-35545).

References

Ancker,J.S.,Chan,C.,&Kukafka,R.(2009).Interactivegraphicsforexpressinghealth

risks:Developmentandqualitativeevaluation.JournalofHealthCommunication, 14(5),461–475.

Ancker,J.S.,Weber,E.U.,&Kukafka,R.(2011).Effectsofgame-likeinteractive

graph-icsonriskperceptionsanddecisions.MedicalDecisionMaking,31(1),130–142.

Armstrong,J.S.(2012).Illusionsinregressionanalysis.InternationalJournalof

Fore-casting,28,689–694.

Bradburn,N.M.,Rips,L.J.,&Shevell,S.K.(1987).Answeringautobiographical

ques-tions:Theimpactofmemoryandinferenceonsurveys.Science,236(4798), 157–161.

Brunswik,E.(1952).Theconceptualframeworkofpsychology.Chicago,IL:University

ofChicagoPress.

Budescu,D.V.,Por,H.-H.,&Broomell,S.B.(2012).Effectivecommunicationof

uncer-taintyintheIPCCreports.ClimateChange,113,181–200.

Budescu,D.V.,&Wallsten,T.S.(1985).Consistencyininterpretationofprobabilistic

phrases.OrganizationalBehaviorandHumanDecisionProcesses,36,391–405.

Einhorn,H.J.,&Hogarth,R.M.(1981).Behavioraldecisiontheory:Processesof

judgmentandchoice.AnnualReviewofPsychology,32,53–88.

Ericsson,K.A.,&Smith,J.(Eds.).(1991).Towardageneraltheoryofexpertise:Prospects

andlimits.NewYork,NY:CambridgeUniversityPress.

Fiedler,K.(2000).Bewareofsamples!Acognitive-ecologicalsamplingapproachto

judgmentbiases.PsychologicalReview,107,659–676.

Fiedler,K.,Brinkmann,B.,Betsch,T.,&Wild,B.(2000).Asamplingapproachtobiases

inconditionalprobabilityjudgments:Beyondbaserateneglectandstatistical format.JournalofExperimentalPsychology:General,129,399–418.

Gheorghiu,G.(2005).Sparkplot:Creatingsparklineswithmatplotlib.AgileTesting

[weblog].Retrievedfrom

http://agiletesting.blogspot.co.uk/2005/04/sparkplot-creating-sparklines-with.html

Gigerenzer,G.,&Hoffrage,U.(1995).HowtoimproveBayesianreasoningwithout

instructions:Frequencyformats.PsychologicalReview,102,684–704.

Gigerenzer,G.,Gaissmaier,W.,Kurz-Milcke,E.,Schwartz,L.M.,&Woloshin,S.

(2007).Helpingdoctorsandpatientsmakesenseofhealthstatistics.

Psycho-logicalScienceinthePublicInterest,8,53–96.

Gigerenzer,G.,Hertwig,R.,vandenBroek,E.,Fasolo,B.,&Katsikopoulos,K.V.(2005).

A30%chanceofraintomorrow:Howdoesthepublicunderstandprobabilistic weatherforecasts?RiskAnalysis,25(3),623–629.

Goldacre,B.(2013).Badpharma:Howdrugcompaniesmisleaddoctorsandharm

patients.RandomHouseDigital,Inc.

Goldstein,D.G.,Johnson,E.J.,&Sharpe,W.F.(2008).Choosingoutcomesversus

choosingproducts:Consumer-focusedretirementinvestmentadvice.Journalof ConsumerResearch,35,440–456.

Hasher,L.,&Zacks,R.T.(1979).Automaticandeffortfulprocessesinmemory.Journal

ofExperimentalPsychology:General,108,356–388.

Hasher,L.,&Zacks,R.T.(1984).Automaticprocessingoffundamentalinformation:

Thecaseoffrequencyoccurrence.AmericanPsychologist,39,1372–1388.

Hayes,B.K.,Newell,B.R.,&Hawkins,G.E.(2013).Causalmodelandsampling

approachestoreducingbaserateneglect.InM.Knauff,M.Pauen,N.Sebanz, &I.Wachsmuth(Eds.),Proceedingsofthe35thannualconferenceoftheCognitive ScienceSocietyAugust,(pp.567–572).

Hertwig,R.,&Erev,I.(2009).Thedescription–experiencegapinriskychoice.Trends

inCognitiveSciences,13,517–523.

Hertwig,R.,Barron,G.,Weber,E.U.,&Erev,I.(2004).Decisionsfromexperienceand

theeffectofrareeventsinriskychoice.PsychologicalScience,15,534–539.

Hertwig,R.,&Pleskac,T.J.(2010).Decisionsfromexperience:Whysmallsamples?

Cognition,115,225–237.

Hogarth,R.M.(2001).Educatingintuition.Chicago:TheUniversityofChicagoPress.

Hogarth,R.M.(2010).Intuition:Achallengeforpsychologicalresearchondecision

making.PsychologicalInquiry,21(4),338–353.

Hogarth,R.M.(Ed.).(1982).Questionframingandresponseconsistency:Newdirections

formethodologyofsocialandbehavioralscience,No.11.SanFrancisco,CA: Jossey-Bass.

Hogarth,R.M.,&Soyer,E.(2011).Sequentiallysimulatedoutcomes:Kindexperience

vs.non-transparentdescription.JournalofExperimentalPsychology:General,140, 434–463.

Hogarth,R.M.,&Soyer,E.(inpress-a).Communicatingforecasts:Thesimplicityof

simulatedexperience.JournalofBusinessResearch(inpress-a).

Hogarth,R.M.,&Soyer,E.(inpress-b).Improvingjudgmentsanddecisionsby

expe-riencingsimulatedoutcomes.InV.Reyna(Ed.),Neuroeconomics,judgmentand

decisionmaking.NewYork,NY:PsychologyPress(inpress-b).

Hogarth,R.M.,Mukherjee,K.,&Soyer,E.(2013).Assessingthechancesof

suc-cess:Naïvestatisticsversuskindexperience.JournalofExperimentalPsychology: LearningMemory,andCognition,39,14–32.

Ioannidis,J.P.A.(2005).Whymostpublishedresearchfindingsarefalse.PLoS

Medicine,2,696–701.

Juslin,P.,Winman,A.,&Hansson,P.(2007).Thenaïveintuitivestatistician:Anaïve

samplingmodelofintuitiveconfidenceintervals.PsychologicalReview,114, 678–703.

Kahneman,D.(2011).Thinking,fastandslow.NewYork,NY:Farrar,Straussand

Giroux.

Kahneman,D.,&Tversky,A.(1979).Prospecttheory:Ananalysisofdecisionunder

risk.Econometrica,47,263–292.

Kaufmann,C.,Weber,M.,&Haisley,E.(2013).Theroleofexperiencesamplingand

graphicaldisplaysonone’sinvestmentriskappetite.ManagementScience,59, 323–340.

Lejarraga,T.(2010).Whenexperienceisbetterthandescription:Timedelaysand

complexity.JournalofBehavioralDecisionMaking,23,100–116.

Lopes,L.L.(1981).Decisionmakingintheshortrun.JournalofExperimental

Psychol-ogy:HumanLearningandMemory,7,377–385.

Peterson,C.R.,&Beach,L.R.(1967).Manasanintuitivestatistician.Psychological

Bulletin,68,29–46.

Savage,S.L.(2009).Theflawofaverages:Whyweunderestimateriskinthefaceof

uncertainty.Hoboken,NJ:JohnWiley&Sons,Inc.

Sedlmeier,P.,&Gigerenzer,G.(2001).TeachingBayesianreasoninginlessthantwo

hours.JournalofExperimentalPsychology:General,130,380–400.

Soyer,E.,&Hogarth,R.M.(2012).Theillusionofpredictability:Howregression

statisticsmisleadexperts.InternationalJournalofForecasting,28,695–711.

Soyer,E.,&Hogarth,R.M.(2014).Whataretheoddsofwinning?Amodelandpossible

heuristics.Manuscriptinpreparation.

Spiegelhalter,D.,Pearson,M.,&Short,I.(2011).Visualizinguncertaintyaboutthe

future.Science,333(6048),1393–1400.

Thaler,R. H.,&Benartzi, S.(2004).SaveMore TomorrowTM_{. Usingbehavioral}

economicsto increase employeesaving.Journal of PoliticalEconomy, 112, S164–S187.

Thaler,R.H.,&Sunstein,C.R.(2008).Nudge:Improvingdecisionsonhealth,wealth,

andhappiness.NewHaven,CT:YaleUniversityPress.

Tufte,E.R.(2006).Beautifulevidence(Vol.23)Cheshire,CT:GraphicsPress.

Turner,E.H.,Matthews,A.M.,Linardatos,E.,Tell,R.A.,&Rosenthal,R.(2008).

Selec-tivepublicationofantidepressanttrialsanditsinfluenceonapparentefficacy. NewEnglandJournalofMedicine,358(3),252–260.

Zacks,R.T.,&Hasher,L.(2002).Frequencyprocessing:Atwenty-fiveyear