Judgment and Dec s on Mak ng, Vol. 15, No. 6, November 2020, pp. 926-938
Act vat ng reflect ve th nk ng w th
dec s on just f cat on and deb as ng
tra n ng
Ozan Isler
*Onurcan Y lmaz
#Burak Dogruyol
$Man pulat ons for act vat ng reflect ve th nk ng, although regularly used n the l terature, have not prev ously been systemat cally compared. There are grow ng concerns about the effect veness of these methods as well as ncreas ng demand for them. Here, we study f ve prom s ng reflect on man pulat ons us ng an object ve performance measure — the Cogn t ve Reflect on Test 2 (CRT-2). In our large-scale prereg stered onl ne exper ment (N = 1,748), we compared a pass ve and an act ve control cond t on w th t me delay, memory recall, dec s on just f cat on, deb as ng tra n ng, and comb nat on of deb as ng tra n ng and dec s on just f cat on. We found no ev dence that onl ne vers ons of the two regularly used reflect on cond t ons — t me delay and memory recall — mprove cogn t ve performance. Instead, our study
solated two less fam l ar methods that can effect vely and rap dly act vate reflect ve th nk ng: (1) a br ef deb as ng tra n ng, des gned to avo d common cogn t ve b ases and
ncrease reflect on, and (2) s mply ask ng part c pants to just fy the r dec s ons. Keywords: cogn t ve reflect on, t me delay, memory recall, dec s on just f cat on, deb as ng tra n ng
1 Introduct on
The d st nct on between reflect ve and ntu t ve th nk ng gu des a w de range of research quest ons n modern behav oral sc ences. The dual-process model of the m nd prov des the lead ng theoret cal framework for these quest ons by pos t ng that
cogn t on s based on two fundamentally d st nct types of processes (Evans & Stanov ch, 2013; Morewedge & Kahneman, 2010). Type 1 processes nclude the automat c, effortless, and ntu t ve th nk ng that we share w th our evolut onary ancestors, whereas Type 2 processes nclude the controlled, effortful, and reflect ve th nk ng spec f c to humans (Kahneman, 2011). Although the assumpt on of the dual-process model that the two cogn t ve dual-processes are ndependent has recently come under scrut ny (Baron, Scott, F ncher & Metz, 2015; B ałek & De Neys, 2016; Kle n, 2011; Pennycook, Fugelsang & Koehler, 2015; Thompson, Evans & Frank sh, 2009; Trémol ère & Bonnefon, 2014), t s well-establ shed that the relat ve extent of reflect on vs. ntu t on const tut ng a dec s on-mak ng process can nevertheless
strongly nfluence bel efs and behav ors (e.g., deolog cal, rel g ous, and consp rat onal bel efs, and econom c, moral, and health behav ors; Gerva s et al., 2018; Pennycook, Cheyne, Barr, Koehler & Fugelsang, 2013; Pennycook, Cheyne, Sel , Koehler & Fugelsang, 2012; Rand, 2016; Swam , Voracek, St eger, Tran & Furnham, 2014; Y lmaz & Isler, 2019; Y lmaz & Sar bay, 2017a, 2017b).
Surpr s ngly, the relat ve effect veness of reflect on and ntu t on man pulat ons used n behav oral research rema ns largely unknown (Horstmann, Hausmann & Ryf, 2009; Myrseth & Wollbrant, 2017). We are aware of only one (unpubl shed) exper mental compar son of ntu t on man pulat ons n cogn t ve performance (Deck, Jahed & Sheremeta, 2017), and no prev ous exper mental study that has systemat cally compared alternat ve reflect on man pulat ons. The presumed effect veness of reflect on man pulat ons used n the l terature can be quest oned s nce basel ne cogn t ve funct ons tend to be ntu t ve and mot vat ng people to pursue an effortful
act v ty such as reflect on can be d ff cult (e.g., Kahneman, 2011). Here, we prov de poss bly the f rst systemat c methodolog cal compar son of regularly used and prom s ng reflect on man pulat ons.
Another reason for the m ss ng methodolog cal ev dence s the frequent lack of control cond t ons, wh ch stems from a rel ance on exper mental compar sons of ntu t on and reflect on man pulat ons as the bas s for hypothes s test ng. W thout these controls, the quest on of whether exper mental results are due to act vat on of ntu t ve or reflect ve processes cannot be answered (e.g., Isler, Maule & Starmer, 2018; Rand, 2016). S m larly, stud es that rely on the two-response parad gm, where an n t al (relat vely more ntu t ve) response s el c ted before a second (relat vely less ntu t ve and more reflected) response, often lack a control cond t on (e.g., Bago & De Neys, 2017). As a recent except on, Lawson, Larr ck, and Soll (2020) employ slow and fast th nk ng prompts (w thout t me-l m ts) and f nd that slow th nk ng has l m ted pos t ve effect on cogn t ve performance compared to a control cond t on. G ven ts mportance, we also employ control cond t ons n the current study.
Stud es us ng ntu t on and reflect on man pulat ons often do not d rectly test whether cogn t ve processes were act vated n the ntended d rect ons. Wh le some have
checked the d rect effects of the r man pulat ons on cogn t ve performance (e.g., Deppe et al., 2015; Lawson et al., 2020; Y lmaz & Sar bay, 2016), subject ve self-report quest ons and behav oral measures such as response t mes are frequently rel ed on as alternat ve man pulat on checks (Rand, Greene & Nowak, 2012; Y lmaz & Isler, 2019). The lack of performance measures would be m slead ng f, rather than th nk ng reflect vely about the problem at hand, part c pants were to rely on the r own lay theor es about reflect on (Sar bay, Y lmaz & Körpe, 2020) or f they were to respond n soc ally des rable ways (Gr mm, 2010). Cons stent w th the ex stence of such
methodolog cal problems, Sar bay et al. (2020) found ntu t on and reflect on pr mes to affect self-reported th nk ng style but not actual performance n the commonly used Cogn t ve Reflect on Test (CRT, Freder ck, 2005). Even the regularly used object ve performance measures — such as when d fferences n response t mes are used to check whether t me-l m t man pulat ons have mpacted behav or (e.g., Isler et al., 2018; Rand et al., 2012) — may not always prov de d rect and conv nc ng ev dence about whether and how cogn t ve processes have been man pulated (Krajb ch, Bartl ng, Hare & Fehr, 2015).
Therefore, the effect of reflect on man pulat ons should be observed on
well-establ shed measures of cogn t ve performance — such as the CRT (Freder ck, 2005) and the CRT-2 (Thomson & Oppenhe mer, 2016). Prov d ng ev dence of the r ab l ty to pred ct the doma n-general features of reflect on, test scores on these two tasks have been shown to correlate w th a w de-range of cogn t ve performance measures n the lab (e.g., syllog st c reason ng and heur st cs-and-b ases problems) and n the f eld (e.g., standard zed academ c test scores and un vers ty course grades) (Lawson et al., 2020; Meyer, Zhou & Shane, 2018; Thomson & Oppenhe mer, 2016; Toplak, West, & Stanov ch, 2011). Numerous other w dely-used reason ng problems, such as the conjunct on fallacy (Tversky & Kahneman, 1983), probab l ty match ng (Stanov ch & West, 2008) and base rate neglect (Kahneman & Tversky, 1973), can also be used to measure the effects of man pulat ons on cogn t ve performance (e.g., Lawson et al., 2020). Among these alternat ves, we chose CRT-2 as our performance measure because part c pants are less l kely to be fam l ar w th t, thereby m n m z ng problems such as ce l ng effects, and because ts rel ance on numeracy sk lls s less than that of CRT, wh ch can confound the nterpretat on of scores (see d scuss on n Thomson & Oppenhe mer, 2016). Desp te these advantages, the CRT-2 arguably captures only some of the spec f c features of cogn t ve reflect on d rectly, such as attent on to deta l and careful read ng. Hence, the mmed ate effects of the reflect on man pulat ons found
n our study can be l m ted to these features of reflect on, as we further deta l n the D scuss on.
The ncreased rel ance on onl ne exper ments prov des another reason to study the effect veness of reflect on man pulat ons, namely, to test the r robustness n th s novel research env ronment. Onl ne labor markets such as Amazon Mechan cal Turk as well as profess onally ma nta ned research part c pant pools such as Prol f c have been
shown to prov de nternally val d exper mental tests n sett ngs less art f c al and more anonymous than the laboratory (Horton, Rand & Zeckhauser, 2011; Palan & Sch tter, 2018; Peer, Brand marte, Samat & Acqu st , 2017), but onl ne exper ments can also suffer from d osyncrat c drawbacks such as noncompl ance w th treatments and asymmetry n dropout rates (Arechar, Gächter & Molleman, 2018; Isler et al., 2018). These problems may be more acute for cogn t vely demand ng tasks such as the reflect on man pulat ons that we study here, espec ally n onl ne dec s on env ronments that can be d stract ng to part c pants (Dandurand, Shultz & On sh , 2008). For
example, prov d ng part c pants w th monetary ncent ves has been shown to result n h gh rates of compl ance w th t me-l m ts (Isler et al., 2018) and reflect ve th nk ng (Lawson et al., 2020) n onl ne exper ments. W th these cons derat ons n m nd, we compare f ve tasks that are s mple and fast enough to be used n onl ne exper ments, and we use monetary ncent ves to mot vate compl ance for the task nstruct ons. Numerous exper mental tasks for promot ng reflect ve th nk ng are currently n use. Some of these tasks, ntroduced n once-acceptable small-sample stud es, are now known to be unrel able. For example, the perceptual d sfluency method (e.g., the use of hard-to-read-fonts to promote reflect on), the scrambled sentence task that pr mes part c pants w th words such as “reason” and “rat onal”, and the task that a ms to pr me reflect on by show ng part c pants a p cture of Rod n’s The Th nker (Gerva s &
Norenzayan, 2012; Song & Schwarz, 2008) all fa led to man pulate reflect ve th nk ng n recent large-sample repl cat on attempts (Bakht , 2018; Deppe et al., 2015; Meyer et al., 2015; Sanchez, Sunderme er, Gray & Cal n-Jageman, 2017; S rota,
Theodoropoulou & Juanch ch, 2020). In add t on, researchers somet mes attempt to act vate reflect ve th nk ng by hav ng part c pants complete tasks (e.g., the CRT) that are or g nally des gned to measure th nk ng style, but the effects of such unestabl shed approaches tend to be unrel able too (Yonker, Edman, Cresswell & Barrett, 2016). Instead, to make the most use of our exper mental resources, we here focus on methods that are spec f cally des gned to man pulate reflect on and that are not known to be unrel able.
One of the most frequently used reflect on man pulat ons s to put t me-l m ts on dec s on-mak ng processes (Horstmann, Ahlgr mm & Glöckner, 2009; Maule, Hockey & Bdzola, 2000; Sp l opoulos & Ortmann, 2018). In th s method, part c pants n a t me pressure cond t on, prompted to dec de w th n a t me-l m t (e.g., 10 seconds), are compared to those n a t me delay cond t on, who are e ther asked to th nk or forced to wa t for a certa n durat on (e.g., 20 seconds) before subm tt ng dec s ons (Capraro, Schulz & Rand, 2019; Rand, 2016; Suter & Hertw g, 2011). Although the t me delay cond t on s assumed to nduce reflect ve answers relat ve to the t me pressure cond t on, the usual lack of a control cond t on w thout t me-l m ts proh b ts the
dent f cat on of whether t s t me pressure or t me delay that affects dec s on-mak ng. Only a few stud es have used control cond t ons to solate the nfluence of t me delay (e.g., Everett, Ingbretsen, Cushman & C kara, 2017). Nevertheless, the exact effect of t me delay arguably rema ns unclear even w th a control cond t on, as t may be d ff cult to d st ngu sh between ncreased rel ance on reflect ve processes and d lut on of emot onal responses (Neo, Yu, Weber & Gonzalez, 2013; Wang et al., 2011). G ven
ts prom nence as the most frequently used cogn t ve process man pulat on, we here use t me delay as one of our exper mental cond t ons, and we also explore the role of emot onal responses.
Another frequently used techn que for act vat ng reflect on s memory recall (Cappelen, Sørensen & Tungodden, 2013; Forstmann & Burgmer, 2015; Ma, L u, Rand, Heatherton & Han, 2015; Rand et al., 2012; Shenhav, Rand & Greene, 2012). In th s method, part c pants are usually asked to wr te a paragraph descr b ng a personal exper ence where rel ance on careful reason ng led to a good outcome, w th the expectat on that the expl c t pr m ng of these memor es would mot vate reflect on. Although a recent h gh-powered study fa led to f nd an effect of th s pr m ng method on a cogn t ve performance measure (Sar bay et al., 2020), th s null result may have been a result of the low rates of compl ance w th the task nstruct ons (see Shenhav et al., 2012). S m lar d ff cult es n ach ev ng h gh rates of compl ance have been observed when us ng t me-l m ts to act vate reflect on (T nghog et al., 2013), and monetary ncent ves have successfully been mplemented to resolve th s problem (Isler
et al., 2018; Kocher & Sutter, 2006). Bu ld ng on these f nd ngs, we adapt th s task to the onl ne context and, as w th other tasks tested n the study, use monetary ncent ves to mot vate compl ance.
In the th rd reflect on man pulat on that we test here, we s mply ask part c pants to just fy the r answers by wr t ng an explanat on of the r reason ng. Across mult ple stud es employ ng the class c As an d sease problem (M ller & Fagley, 1991; S eck & Yates, 1997; Takemura, 1994), the dec s on just f cat on task has been found to reduce fram ng effects effect vely. Ask ng for just f cat on or elaborat on was found to be even more effect ve than monetary ncent ves (V e der, 2011), and ts effect veness has been val dated across mult ple dec s on-mak ng contexts, nclud ng health (Almashat, Ayotte, Edelste n & Margrett, 2008) and consumer cho ce (Cheng, Wu & L n, 2014). Just f cat on prompts can mot vate reflect on by generat ng feel ngs of h gher levels of respons b l ty for one’s dec s ons as well as expectat ons of the r scrut ny by others. However, the effect veness of the just f cat on task has been quest oned (Belard nell , Bellé, S c l a & Steccol n , 2018; Leboeuf & Shaf r, 2003). Add t onal f nd ngs have suggested that the effect veness of dec s on just f cat on s task-dependent (Le st , Radun, V rtanen, Nyman, & Häkk nen, 2014) and that t may even harm dec s ons (Igou & Bless, 2007), espec ally n spec f c contexts prone to mot vated reason ng (Chr stensen, 2018; S eck, Qu nn & Schooler, 1999). G ven the prom s ng but m xed f nd ngs on the effect veness of the just f cat on task, we used th s s mple techn que as an alternat ve reflect on man pulat on.
For the fourth reflect on task tested here, we develop a novel tra n ng procedure for the onl ne context cons stent w th well-establ shed deb as ng pr nc ples (Lewandowsky, Ecker, Se fert, Schwarz & Cook, 2012). We mod fy a deb as ng tra n ng task that was prev ously tested n the laboratory w th prom s ng results (Y lmaz & Sar bay, 2017a, 2017b). The lab vers on of the task prov des part c pants w th a 10-m nute tra n ng on not c ng and correct ng cogn t ve b ases: t f rst el c ts the Cogn t ve Reflect on Test (Freder ck, 2005) and var ous base-rate problems (De Neys & Glum c c, 2008) and then prov des feedback on the correct answers and the r explanat ons (also see Morewedge et al., 2015; Stephens, Dunn, Hayes & Kal sh, 2020). Wh le prev ous stud es us ng deb as ng tra n ng have been successful (Sell er, Scopell t &
Morewedge, 2019), ts lengthy and compl cated exerc ses have so far precluded ts systemat c use n onl ne exper ments.
In short, alternat ve reflect on man pulat ons have not yet been exper mentally compared us ng an actual performance measure and behav oral research methods lack rel able reflect on man pulat ons that can be used n onl ne exper ments. Here, we use CRT-2 scores as the cogn t ve performance measure and compare the effects of f ve prom s ng man pulat ons on reflect ve th nk ng n a h gh-powered between-subjects exper ment. The f ve reflect on man pulat ons nclude the t me delay cond t on (R1), the memory recall task (R2), the dec s on just f cat on task (R3), and the deb as ng tra n ng (R4) descr bed above as well as a comb ned task that ncludes both the deb as ng tra n ng and the dec s on just f cat on tasks (R5). We compare these f ve reflect on cond t ons w th two control groups: the pass ve control cond t on (C1) where part c pants rece ved no treatment pr or to tak ng part n CRT-2, and the act ve control cond t on (C2) where part c pants were ass gned neutral read ng and wr t ng tasks to prov de comparab l ty w th the reflect on cond t ons.
Table 1: Overv ew of reflect on man pulat ons.
Man pulat on Task descr pt on Completed (as % of recru ted)
Pass ve control (C1): No man pulat ons or act ve controls 262 (99%)
Act ve control (C2): Neutral read ng and wr t ng task 255 (96%)
T me delay (R1): Th nk ng carefully for at least 20 seconds for each quest on 262 (99%) Memory recall (R2): Descr b ng a t me when reflect on was benef c al 210 (79%) Dec s on just f cat on (R3): Just fy ng answers to each quest on 256 (97%) Deb as ng tra n ng (R4): Learn ng about and descr b ng three common cogn t ve b ases 252 (95%)
R3 + R4 (R5): Comb nat on of deb as ng tra n ng and just f cat on 251 (95%) Us ng th s exper mental setup, we test three prereg stered hypotheses on the effect of
man pulat ons on reflect ve th nk ng as measured by the CRT-2 scores. F rst, we pred cted that the CRT-2 scores n the f ve reflect on cond t ons (R1 to R5) w ll be h gher than the two control cond t ons (C1 to C2). Second, we pred cted that the CRT-2 scores n cond t ons w th deb as ng tra n ng (R4 and R5) w ll be h gher than the reflect on cond t ons w thout deb as ng tra n ng (R1, R2 and R3) because they are based on proven deb as ng techn ques, nclud ng repeated explanat ons of cogn t ve b ases and warn ngs aga nst potent al future m stakes (Lewandowsky et al., 2012). Th rd, we expected that the comb nat on of deb as ng tra n ng and dec s on just f cat on man pulat ons can mot vate even h gher reflect on by prompt ng part c pants to apply deb as ng techn ques when prov d ng just f cat ons for the r dec s ons on the CRT-2
tems. Accord ngly, we pred cted that the CRT-2 scores n the deb as ng tra n ng cond t on w th just f cat on (R5) w ll be h gher than the deb as ng tra n ng cond t on w thout just f cat on (R4).
In add t on to test ng these hypotheses, we report var ous exploratory analyses. We nvest gate response t mes and study the role of task compl ance n dr v ng the treatment effects. We then contrast CRT-2 scores w th self-report measures of
reflect on. We conjectured that a d screpancy between these two measures, where self-reported reflect on s not supported by actual performance, could nd cate soc ally des rable respond ng. There s l m ted but suggest ve ev dence that reflect on
man pulat ons such as t me l m ts can nfluence affect (Isler et al., 2018; Maule et al., 2000). Therefore, we also explore whether the effects of treatments on cogn t ve performance al gn w th d fferences n effects on emot onal responses.
2 Method
Us ng a between-subjects des gn, we exper mentally compared f ve reflect on
man pulat ons and two control cond t ons. Part c pants were bl nd to the exper mental cond t ons, and each part c pant was randomly ass gned to one of seven cond t ons (see Table 1). The exper ment was prereg stered at the Open Sc ence Framework (OSF) (https://osf.io/6axuz). The exper mental mater als, the dataset, and the analys s code are ava lable at the OSF study s te (https://osf.io/k495r/).
2.1 Part c pants
Part c pants were recru ted onl ne v a Prol f c (http://www.prolific.co/, Palan & Sch tter, 2018) and recru tment was restr cted to fluent Engl sh-speak ng UK res dents who were 18 or older. As prereg stered, part c pants w th ncomplete data were
excluded from the dataset pr or to analys s (n = 107). None of the excluded part c pants had completed the CRT-2. Hence, the r nclus on n the analys s does not change the results. We analyze data from 1,748 un que part c pants w th complete subm ss ons (Mage = 33.58, SDage = 11.50; 71.1% female). In add t on to a part c pat on fee of £0.40, part c pants were pa d £0.20 for compl ance w th task nstruct ons.
2.2 Planned sample s ze
We planned for a powerful test (1-β = 0.90) to dent fy small effects of man pulat ons (f = 0.10) n a one-way ANOVA model w th seven cond t ons and standard Type I error rate (α = 0.05). Us ng G*Power 3.1.9.2 (Faul, Erdfelder, Buchner & Lang, 2009), we est mated our target sample s ze to nclude at least 1750 complete subm ss ons.
2.3 Procedure
To ncrease compl ance w th the exper mental tasks, part c pants were nformed that they would earn an add t onal £0.20 f they closely followed the task nstruct ons. F ve of the seven cond t ons were des gned to act vate cogn t ve reflect on (R1 to R5),
whereas the other two cond t ons were des gned as controls (C1 and C2). In all cond t ons, part c pants completed the Cogn t ve Reflect on Test (CRT-2; Thomson & Oppenhe mer, 2016), wh ch prov des a less fam l ar and less numer cal alternat ve to the or g nal CRT (Freder ck, 2005). CRT-2 ncludes four quest ons that are des gned to tr gger a spontaneous but ncorrect response and rel ance on cogn t ve reflect on s operat onal zed as res stance to th s n t al response (e.g., “If you’re runn ng a race and you pass the person n second place, what place are you n?”). Hence, nd v dual CRT-2 scores range from 0 to 4. Cronbach’s α for the four CRT-CRT-2 tems was .54, n l ne w th the or g nal CRT (Baron et al., 2015). As we next descr be n deta l, the reflect on man pulat ons were mplemented dur ng the 2 for R1 and R3 and before the CRT-2 for RCRT-2 and R4, whereas part c pants n R5 were exposed to reflect on man pulat ons both before and dur ng the CRT-2.
In the f rst reflect on man pulat on (R1), the t me delay cond t on, part c pants were asked to th nk for at least 20 seconds before answer ng each CRT-2 quest on. Each quest on screen d splayed a reflect on prompt (“Carefully cons der your answer”) and a t mer count ng up from zero seconds. Cons stent w th ts regular use (Bouwmeester et al., 2017; Isler et al., 2018; Rand, 2016; Rand et al., 2012), t was techn cally poss ble to subm t answers w th n 20 seconds, wh ch allows check ng that t me delay
nstruct ons mot vate behav or change (Horstmann, Hausmann, et al., 2009). The average rate of compl ance w th t me-l m ts across the four quest ons was 67%. The second reflect on cond t on (R2), the memory recall task, was based on Shenhav et al. (2012). Part c pants were told to wr te a paragraph descr b ng an ep sode when carefully reason ng through a s tuat on led them n the r ght d rect on and resulted n a good outcome. Adapt ng th s task to the onl ne sett ng, we asked part c pants to wr te four sentences rather than e ght-to-ten sentences as n the or g nal task. Desp te th s mod f cat on, whereas at least 95% of the n t ally recru ted part c pants completed the study n other cond t ons ( .e., answered all quest ons, nclud ng the survey), th s f gure was only 79% for R2. Among those who completed R2, the compl ance rate ( .e., the prevalence of part c pants who wrote four or more sentences) was 88.6%. Because exclus on of non-compl ant part c pants can jeopard ze nternal val d ty by annull ng random zat on (Bouwmeester et al., 2017; T nghog et al., 2013), we nclude them n our analyses cons stent w th our prereg stered ntent on-to-treat analys s plan. The th rd reflect on cond t on (R3) ncluded the just f cat on task, wh ch el c ted just f cat ons from part c pants s m lar to M ller and Fagley (1991). Spec f cally, on each of the four screens where answers to the CRT-2 quest ons were el c ted, part c pants were asked to just fy the r answers n a separate cell by prov d ng an explanat on of the r reason ng n one sentence or more. For each quest on, the answer to the CRT-2 quest on and ts just f cat on were subm tted s multaneously.
F gure 1: CRT-2 scores across the cond t ons. Sample s ze (n) and average number of correct answers on the Cogn t ve Reflect on Test-2 (Thomson & Oppenhe mer, 2016) n the control cond t ons (C1 to C2, gray bars) and the cogn t ve reflect on man pulat ons (blue bars): (R1) T me delay, (R2) Memory
recall, (R3) Dec s on just f cat on, (R4) Deb as ng tra n ng, and (R5) Deb as ng tra n ng w th dec s on just f cat on. Error bars show 95% conf dence ntervals. As the fourth reflect on cond t on (R4), we developed a novel tra n ng task for the onl ne context. The task was des gned to mprove v g lance aga nst three commonly observed cogn t ve b ases. Part c pants were asked to answer three quest ons. The f rst quest on was ntended to llustrate a semant c llus on: “How many of each an mal d d Moses take on the ark?” The second quest on nvolved a test of the base rate fallacy: “In a study, 1000 people were tested. Among the part c pants, there were 5 eng neers and 995 lawyers. Jack s a randomly chosen part c pant n th s study. Jack s 36 years old. He s not marr ed and s somewhat ntroverted. He l kes to spend h s free t me read ng sc ence f ct on and wr t ng computer programs. What s most l kely?” (Jack s a lawyer or eng neer). The th rd quest on was des gned to exh b t ava lab l ty b as: “Wh ch cause more human deaths?” (sharks or horses). After each quest on, the screen d splayed the correct answer, along w th an explanat on of the b as (see mater als at the OSF study s te). F nally, part c pants were asked to wr te four sentences summar z ng what they have learned n tra n ng, and they were nstructed to rely on reflect on dur ng the next task ( .e., the CRT-2).
We dev sed a f fth reflect on cond t on (R5) that comb ned dec s on just f cat on (R3) w th deb as ng tra n ng (R4). Part c pants f rst part c pated n the deb as ng tra n ng and then they were asked to just fy the r responses to the CRT-2 quest ons, as descr bed above. Hence, R5 promoted learn ng-by-do ng (Bruce & Bloch, 2012), the appl cat on of the lessons rece ved dur ng deb as ng tra n ng on CRT-2 quest ons.
Two control cond t ons were des gned to allow ns ghtful compar sons to the f ve reflect on cond t ons. The pass ve control cond t on (C1), where part c pants completed CRT-2 w thout any add t onal tasks, measures basel ne CRT-2 scores n the part c pant pool. In the act ve control cond t on (C2), part c pants were f rst asked to descr be an object of the r choos ng n four sentences before answer ng the CRT-2 quest ons. Th s neutral wr t ng task n C2 controls for any d rect effect that the act of wr t ng tself n R2, R4 and R5 may have on reflect on. S m larly, to ach eve comparab l ty between reflect on man pulat ons, part c pants n R1 and R3 were asked to complete the same neutral wr t ng task as n C2 pr or to beg nn ng CRT-2.
After the CRT-2, part c pants answered two quest ons on a 7-po nt L kert scale (1 = “not at all”, 7 = “a great deal”): 1) “To what extent d d you rely on your feel ngs or
ntu t ons when mak ng your dec s ons?”, and 2) “To what extent d d you rely on reason when mak ng your dec s ons?” The score on the f rst quest on was reversed and the average of the scores on the two quest ons const tuted the self-reported compos te
ndex of reflect on.
F nally, part c pants completed a survey, nclud ng the 20- tem Pos t ve and Negat ve Affect Schedule (PANAS; Watson, Clark & Tellegen, 1988) and a br ef demograph c quest onna re. The PANAS cons sted of two 10- tem scales measur ng pos t ve and negat ve affect. Part c pants were asked to nd cate the extent to wh ch they
exper enced each emot on tem dur ng the prev ous task ( .e., CRT-2) on a L kert scale rang ng from 1 (“very sl ghtly or not at all”) to 5 (“extremely”). Both pos t ve and negat ve affect scales revealed suff c ent nternal cons stency (both Cronbach’s αs = .89).
Table 2: Study conf gurat on and response t mes. M denotes the pos t on of any reflect on man pulat on n the study procedures ( .e., before or dur ng the el c tat on of the CRT-2). AC denotes the pos t on of any act ve controls ( .e., a neutral wr t ng task to control for the act of wr t ng; see Method). Mean RTs ( n seconds) across cond t ons nd cate the durat on of the CRT-2 task (“CRT-2”), study durat on except for CRT-2 RTs (“Other”), and the total study durat on (“Total”).
Man pulat on Before CRT-2 Dur ng CRT-2 CRT-2 Other Total
Pass ve control (C1) 75 183 257
Act ve control (C2) (AC) 77 291 368
T me delay (R1) (AC) M 91 290 380
Memory recall (R2) M 72 421 493
Dec s on just f cat on (R3) (AC) M 250 309 559
Deb as ng tra n ng (R4) M 82 471 553
R3 + R4 (R5) M M 221 476 697
3 Results
3.1 Conf rmatory tests
Overall, the deb as ng tra n ng, the just f cat on task, and the r comb nat on
s gn f cantly mproved performance on the CRT-2, whereas t me delay and memory recall were not helpful. The CRT-2 scores across the control and exper mental cond t ons are presented n F gure 1. A one-way ANOVA model revealed s gn f cant d fferences n CRT-2 scores across the cond t ons (F(6, 1741) = 15.75, p < .001, ηp2 = .051). As poshoc analys s, we conducted pa rw se compar sons us ng two-ta led t-tests, wh ch nd cated part al support for our n t al hypothes s that reflect on
man pulat ons ncrease performance on the CRT-2. As pred cted, CRT-2 scores n the just f cat on and deb as ng tra n ng cond t ons ( .e., R3, R4 and R5) were s gn f cantly h gher than both of the control cond t ons, C1 (Cohen’s d = 0.47, 0.52 and 0.54 respect vely, ps < .001) and C2 (d = 0.40, 0.45 and 0.47, ps < .001). In contrast, ne ther t me delay (R1) nor memory recall (R2) showed s gn f cant d fference from C1 (vs. R1: p = .537, d = 0.05; vs. R2: p = .610, d = 0.05;) or C2 (vs. R1: p = .721, d = 0.03; vs. R2: p = .682, d = 0.04). We also found part al support for our second hypothes s that deb as ng tra n ng s more effect ve than the other reflect on man pulat ons: CRT-2 scores n the cond t ons w th deb as ng tra n ng (R4 and R5) were s gn f cantly h gher than t me delay (R1 vs. R4: d = 0.47; R1 vs. R5: d = 0.49; ps < .001) and memory recall cond t ons (R2 vs. R4: d = 0.48; R2 vs. R5: d = 0.50, ps < .001) but not the just f cat on cond t on (R3 vs. R4: p = .704, d = 0.03; R3 vs. R5: p = .448, d = 0.07). Fa l ng to f nd conf rmatory ev dence for our f nal hypothes s, CRT-2 scores n the two cond t ons w th deb as ng tra n ng d d not s gn f cantly d ffer (R4 vs. R5: p = .681, d = 0.04). In other words, the comb nat on of deb as ng tra n ng w th just f cat on prov ded no clear added benef ts.
3.2 Exploratory analyses
Here, we f rst report the rema n ng ( .e., non-conf rmatory) pa rw se compar sons of exper mental cond t ons, and then explore d fferences n response t mes (RTs), task noncompl ance, self-reported reflect on, and self-reported emot ons across the cond t ons. No d fference n CRT-2 scores were dent f ed when compar ng the two control cond t ons (p = .324) and when compar ng t me delay w th memory recall (p = .944). The CRT-2 scores were h gher n the dec s on just f cat on cond t on than n the memory recall (p < .001). F nally, CRT-2 scores n the dec s on just f cat on cond t on were s gn f cantly h gher than the t me delay cond t on (p < .001).
To help explore response t mes (RTs), Table 2 nd cates the pos t on of the reflect on man pulat ons and the act ve controls n the study procedure as well as the mean RTs across the seven cond t ons. We use log-transformed RTs (base 10) to account for data skewness n all exploratory analyses that nvolve study durat on measures. RTs n both the CRT-2 and the overall study s gn f cantly d ffered across cond t ons (CRT-2: F(6, 1741) = 274.84, p < .001, ηp2 = .486; overall: F(6, 1741) = 161.26, p < .001, ηp2 = .357). As expected, pa rw se compar sons w th two-ta led t-tests nd cated that el c t ng just f cat ons dur ng CRT-2 ( .e., R3 and R5) ncreased CRT-2 RTs compared to all
other cond t ons (ps < .001) and that lack of reflect on man pulat ons or act ve controls ( .e., C1) decreased the rema n ng study durat on ( .e., exclud ng CRT-2 RTs) compared to all other cond t ons (ps ≤ .001). Wh le there was no d fference between the total study durat ons of R3 and R4 (p = .889), R1 was the fastest, R2 was the second fastest, and R5 was the slowest reflect on cond t on (ps ≤ .001). S nce careful reflect on requ res t me, the var at on n CRT-2 scores across the cond t ons could n part be dr ven by these RT asymmetr es. Cons stent w th th s conjecture, a l near regress on of the CRT-2 scores on two var ables that together const tute the total study durat on were both pos t ve and stat st cally s gn f cant (log of total RT on CRT-2: β = 0.189, p < .031, ηp2 = .003; log of rema n ng t me spent on the study: β = 0.260, p < .034, ηp2 = .003).
F gure 2: Self-reported reflect on across the cond t ons. Average scores on the self-reported compos te ndex of reflect on n the control cond t ons (C1 to C2, gray bars) and the cogn t ve reflect on man pulat ons (blue bars): (R1) T me delay, (R2) Memory recall, (R3) Dec s on just f cat on, (R4) Deb as ng tra n ng, and (R5) Deb as ng tra n ng w th dec s on just f cat on. Error bars show 95% conf dence ntervals.
F gure 3: PANAS scores across the cond t ons. Average self-reported pos t ve (left panel) and negat ve (r ght panel) affect scores n the control cond t ons (C1 to C2, gray bars) and the cogn t ve reflect on man pulat ons (blue bars): (R1) T me delay, (R2) Memory recall, (R3) Dec s on just f cat on, (R4) Deb as ng tra n ng, and (R5) Deb as ng tra n ng w th dec s on just f cat on. Error bars show 95% conf dence ntervals.
One reason why the t me delay cond t on fa led to s gn f cantly act vate reflect on may be non-compl ance w th the t me-l m ts. In R1, 44.7% of part c pants fa led to comply w th the 20-second t me-l m t n one or more of the four CRT-2 quest ons. S m larly, 21% of part c pants n the memory recall cond t on (R2) fa led to complete the study
and 11.4% of part c pants n R2 who completed the study fa led to wr te at least four sentences n the memory recall task. In pr nc ple, task noncompl ance could have weakened these reflect on man pulat ons, s nce CRT-2 scores were h gher among compl ant than among non-compl ant part c pants n both R1 (2.70 vs. 2.02, t(260) = 5.08, p < .001, d = 0.63) as well as R2 (2.50 vs. 1.64, t(208) = 3.85, p < .001, d = 0.78). However, these d fferences may also be due to part c pants’ th nk ng styles, as those who tend to be reflect ve ( .e., those w th h gher basel ne CRT-2 scores) are l kely to read the task nstruct ons more carefully. Hence, exclus on of non-compl ant part c pants from the analys s can b as results by annull ng random ass gnment (Bouwmeester et al., 2017; T nghog et al., 2013), and the appropr ate solut on would be to ncrease compl ance n future stud es, for example by us ng forced delay n R1 and stronger monetary ncent ves n R2.
Next, we explore the nfluence of exper mental man pulat ons on self-reported reflect on (F gure 2) and affect (F gure 3). A one-way ANOVA showed that the self-reported compos te ndex of reflect on s gn f cantly d ffered between the cond t ons (F(6, 1741) = 3.08, p = .005, η = .011). Pa rw se compar sons us ng two-ta led t-tests revealed that part c pants n cond t ons w th deb as ng tra n ng (R4 and R5), cons stent w th d fferences n CRT-2 performance, reported rely ng more on reason as compared to those n the pass ve control (R4 vs. C1: p = .029, d = 0.19; R5 vs. C1: p = .027, d = 0.20) and the memory recall cond t ons (R4 vs. R2: d = 0.32; R5 vs. R2: d = 0.32; all
ps < .001). As a further nd cat on of the fa lure of the memory recall cond t on (R2) n
act vat ng reflect on, self-reported reflect on was s gn f cantly lower n R2 as
compared to the act ve control and the t me delay cond t ons (R2 vs. C2: p = .022, d = 0.21; R2 vs. R1: p < .001, d = 0.26). No other s gn f cant d fference n self-reported reflect on was dent f ed between the exper mental cond t ons.
One-way ANOVA models of PANAS showed s gn f cant effect on pos t ve affect (F(6, 1741) = 5.25, p < .001, η = .018) but fa led to show effect of cond t ons on negat ve affect (F(6, 1741) = 2.05, p = .057, ηp2 = .007). In part cular, pa rw se compar sons us ng two-ta led t-tests nd cated that deb as ng tra n ng w th dec s on just f cat on (R5) s gn f cantly ncreased pos t ve affect as compared to the two controls (R5 vs. C1: p = .001, d = 0.29; R5 vs. C2: p < .001, d = 0.44) as well as the t me delay (R5 vs. R1: p = .047, d = 0.18), the memory recall (R5 vs. R2: p = .002, d = 0.29), and the dec s on just f cat on cond t ons (R5 vs. R3: p < .001, d = 0.36). T me delay (R1) and deb as ng tra n ng (R4) cond t ons also ncreased pos t ve affect compared to the act ve control (R1 vs. C2: p = .004, d = 0.26; R4 vs. C2: p = .002, d = 0.27) and the dec s on just f cat on cond t ons (R1 vs. R3: p = .040, d = 0.18; R4 vs. R3: p = .027, d = 0.20). All other pa rw se compar sons fa led to reach stat st cal s gn f cance.
4 D scuss on
In th s study, we a med to dent fy exper mental man pulat ons that can effect vely act vate reflect ve th nk ng. Compar ng f ve reflect on man pulat ons and two control cond t ons, we found that just fy ng answers to the CRT-2 (R3), rece v ng a br ef deb as ng tra n ng pr or to t (R4), and the comb nat on of the two methods (R5) s gn f cantly ncreased reflect ve th nk ng. Aga nst our expectat ons, no d fference n cogn t ve performance was found across these three reflect on man pulat ons. The onl ne vers ons of the two man pulat ons commonly used n the l terature — t me delay (R1) and memory recall (R2) — were not found to be effect ve n ncreas ng rel ance on reflect on, wh ch may have been due to h gh noncompl ance n R1 and h gh dropout rates n R2. On a pos t ve note, reflect on man pulat ons were not found to ncrease negat ve affect, and no soc ally des rable respond ng was found n these neffect ve man pulat ons, s nce the self-reported reflect on scores n these cond t ons were not h gher than the controls. Overall, our study solated two underut l zed treatments (R3 and R4) as effect ve reflect on man pulat ons appropr ate for the onl ne context and
nd cated that the two regularly used reflect on methods (R1 and R2) may not be effect ve w th the conf gurat ons used n th s study.
Are any of the successful reflect on man pulat ons preferable to the others? Our study revealed that R3, R4 and R5 ncreased rel ance on reflect on to a s m lar extent —
result ng n moderate effect s zes that d d not s gn f cantly d ffer from each other. As compared to cond t ons w th deb as ng tra n ng (R4 and R5), the cond t on w th only the dec s on just f cat on task (R3) has the advantage of nvolv ng a s mple prompt that
s easy to adm n ster w thout the need to teach expl c t rules for reflect on. On the other hand, compared to the cond t ons that use dec s on just f cat on (R3 and R5), the cond t on w th only the deb as ng tra n ng (R4) ach eved not only h gh scores but also fast responses n the CRT-2 that was subsequently el c ted. Therefore, the deb as ng tra n ng shows prom se n nduc ng cont nued act vat on of reflect on, but the longev ty of th s man pulat on, as well as alternat ve ways to strengthen t, should be further explored. L kew se, R5 (and to a lesser extent R4) resulted n h gher levels of self-reported pos t ve affect as compared w th the controls, suggest ng that deb as ng tra n ng and the appl cat on of ts lessons dur ng dec s on mak ng can ncrease pos t ve effect. Whether pos t ve affect n turn a ds reflect on s an open quest on that needs further exam nat on. Overall, we adv se that the best reflect on man pulat on s the one that s most appropr ate for the exper mental task at hand. For example, ask ng
just f cat ons for dec s ons n tasks that measure prosoc al ntent ons can mot vate soc ally des rable respond ng. For such tasks, deb as ng tra n ng can be preferable. In other research sett ngs, dec s on just f cat on can prov de a fast and effect ve reflect on man pulat on.
The present study suffers from var ous l m tat ons. Most mportantly, our results are l m ted by ts rel ance on CRT-2 as the sole cogn t ve performance measure. Wh le t s well-establ shed that the CRT-2 scores show s gn f cant pos t ve correlat ons w th other cogn t ve reflect on measures such as the CRT (Thomson & Oppenhe mer, 2016; Y lmaz & Sar bay, 2017c) or standard heur st cs-and-b ases quest ons (e.g., Lawson et al., 2020), t s currently unclear exactly what aspects of cogn t ve reflect on are d rectly captured by the CRT-2. The CRT-2 tems d ffer from the standard CRT tems by des gn, rely ng more on careful read ng than on numeracy (Thomson &
Oppenhe mer, 2016). In th s sense, the CRT-2 tems can be l kened to the so-called “stumpers” (Bar-H llel, Noah & Freder ck, 2018; Bar-H llel, Noah & Shane, 2019). On the other hand, wh le stumpers are d ff cult r ddles that “do not evoke a compell ng, but wrong, ntu t ve answer” (Bar-H llel et al., 2018), the ntu t ve answers on the CRT-2 are systemat cally wrong and can be used to d st ngu sh between ntu t ve and reflect ve th nk ng. For example, more than a th rd of the answers to the f rst CRT-2 quest on (“If you’re runn ng a race and you pass the person n second place, what place are you n?”) n the or g nal study by Thomson and Oppenhe mer (2016) was “f rst” and not “second”. These systemat c m stakes are probably n part due to careless read ng but also because correct response on th s tem requ res the log cal nference that pass ng the second person n a race mpl es the ex stence of another runner who s ahead of them both. Nevertheless, more research s needed to d st ngu sh between var ous cogn t ve performance tasks n the r ab l ty to measure d fferent aspects of reflect on (e.g., Erceg, Gal ć & Ružojč ć, 2020).
Secondly, our results are not conclus ve about the potent al of t me delay and memory recall tasks n ncreas ng reflect on. Our setup, where the memory recall task was shortened for the onl ne context and where the t me delay cond t on was not forced, may have weakened the man pulat ons. Low task compl ance n t me delay and h gh dropout rates n memory recall could have contr buted to th s fa lure. Hence, mproved methods are needed to test the super or ty of the dec s on just f cat on and the
deb as ng tra n ng tasks over t me delay and memory recall. For such tests, the standard vers on of the memory recall that requ res wr t ng of e ght sentences can be coupled w th h gher monetary ncent ves to mot vate task compl ance, and the alternat ve vers on of the t me delay cond t on that forces part c pants to wa t for a set per od can be used.
Th rdly, we cannot rule out the poss b l ty that the d rect effects of our successful reflect on man pulat ons on cogn t ve performance may have been l m ted. For
example, rather than act vat ng reflect on d rectly, the deb as ng tra n ng cond t on may have nd rectly mproved reflect on performance by ncreas ng test-tak ng ab l ty through exposure to quest ons that are s m lar to the CRT-2 or by ncreas ng
understand ng of the CRT-2 tems through more careful read ng. L kew se, the dec s on just f cat on task may be open to exper menter demand effects n some contexts. One
reason why we d d not f nd ev dence for soc ally des rable respond ng may be the fact that all part c pants were exposed to the CRT-2 pr or to report ng how much they reflected. Exposure to CRT-2 may have created a sense of rel ance on reflect on n the control cond t ons. Future stud es spec f cally des gned to study the role of soc ally des rable respond ng n reflect on man pulat ons are needed.
Overall, th s study f lls an mportant gap n the l terature by h ghl ght ng two effect ve man pulat ons (and the r comb nat on) for act vat ng reflect ve th nk ng. These methods can be eas ly mplemented n future research on dual-process models, nclud ng exper ments conducted onl ne. Some of the commonly used reflect on man pulat ons are recently shown to be neffect ve (e.g., Deppe et al., 2015; Meyer et al., 2015), and earl er f nd ngs based on these man pulat ons often fa l to repl cate (e.g., Sanchez et al., 2017). Hence, prev ous results based on unrel able reflect on
man pulat ons should be tested us ng mproved methods. Our f nd ngs nd cate that, rather than just rem nd ng people of the benef ts of reflect on (as n memory recall) or g v ng them t me to th nk (as n t me delay), prov d ng gu dance about how to reflect spec f cally (as n deb as ng tra n ng and dec s on just f cat on) can mprove cogn t ve performance. The methods advanced n th s study — dec s on just f cat on, deb as ng tra n ng and the r comb ned use — can serve th s purpose well.
References
Almashat, S., Ayotte, B., Edelste n, B., & Margrett, J. (2008). Fram ng effect deb as ng n med cal dec s on mak ng. Pat ent Educat on and Counsel ng, 71(1), 102–107. http://dx.doi.org/10.1016/j.pec.2007.11.004.
Arechar, A. A., Gächter, S., & Molleman, L. (2018). Conduct ng nteract ve exper ments onl ne. Exper mental Econom cs, 21, 99–131.
Bago, B., & De Neys, W. (2017). Fast log c?: Exam n ng the t me course assumpt on of dual process theory. Cogn t on, 158, 90–109.
http://dx.doi.org/10.1016/j.cognition.2016.10.014.
Bakht , R. (2018). Rel g ous versus reflect ve pr m ng and suscept b l ty to the conjunct on fallacy. Appl ed Cogn t ve Psychology, 32(2), 186–191.
http://dx.doi.org/10.1002/acp.3394.
Bar-H llel, M., Noah, T., & Freder ck, S. (2018). Learn ng psychology from r ddles: The case of stumpers. Judgment & Dec s on Mak ng, 13(1), 112–122.
Bar-H llel, M., Noah, T., & Shane, F. (2019). Solv ng stumpers, CRT and CRAT: Are the ab l t es related? Judgment and Dec s on Mak ng, 14(5), 620–623.
Baron, J., Scott, S., F ncher, K., & Metz, S. E. (2015). Why does the cogn t ve reflect on test (somet mes) pred ct ut l tar an moral judgment (and other th ngs)?
Journal of Appl ed Research n Memory and Cogn t on, 4(3), 265–284.
Belard nell , P., Bellé, N., S c l a, M., & Steccol n , I. (2018). Fram ng effects under d fferent uses of performance nformat on: An exper mental study on publ c managers.
Publ c Adm n strat on Rev ew, 78(6), 841–851.
http://dx.doi.org/10.1111/puar.12969.
B ałek, M., & De Neys, W. (2016). Confl ct detect on dur ng moral dec s on-mak ng: Ev dence for deont c reasoners’ ut l tar an sens t v ty. Journal of Cogn t ve Psychology,
28(5), 631–639. http://dx.doi.org/10.1080/20445911.2016.1156118.
Bouwmeester, S., Verkoe jen, P., Aczel, B., Barbosa, F., Begue, L., Branas-Garza, P., . . . Wollbrant, C. E. (2017). Reg stered repl cat on report: Rand, Greene, and Nowak (2012). Perspect Psychol Sc , 12(3), 527–542.
Bruce, B. C., & Bloch, N. (2012). Learn ng by do ng. In N. M. Seel (Ed.),
Encycloped a of the Sc ences of Learn ng (pp. 1821–1824). Boston, MA: Spr nger US.
Cappelen, A. W., Sørensen, E. Ø., & Tungodden, B. (2013). When do we l e? Journal
of Econom c Behav or & Organ zat on, 93, 258–265.
http://dx.doi.org/10.1016/j.jebo.2013.03.037.
Capraro, V., Schulz, J., & Rand, D. G. (2019). T me pressure and honesty n a decept on game. Journal of Behav oral and Exper mental Econom cs, 79, 93–99. http://dx.doi.org/10.1016/j.socec.2019.01.007.
Cheng, F.-F., Wu, C.-S., & L n, H.-H. (2014). Reduc ng the nfluence of fram ng on nternet consumers’ dec s ons: The role of elaborat on. Computers n Human Behav or,
37, 56–63. http://dx.doi.org/10.1016/j.chb.2014.04.015.
Chr stensen, J. (2018). Do just f cat on requ rements reduce mot vated reason ng n pol t c ans’ evaluat on of pol cy nformat on? An exper mental nvest gat on. An
Exper mental Invest gat on.(December 3, 2018).
Dandurand, F., Shultz, T. R., & On sh , K. H. (2008). Compar ng onl ne and lab methods n a problem-solv ng exper ment. 40(2), 428–434.
http://dx.doi.org/10.3758/brm.40.2.428.
De Neys, W., & Glum c c, T. (2008). Confl ct mon tor ng n dual process theor es of th nk ng. Cogn t on, 106, 1248–1299.
http://dx.doi.org/10.1016/j.cognition.2007.06.002.
Deck, C., Jahed , S., & Sheremeta, R. (2017). The effects of d fferent cogn t ve man pulat ons on dec s on mak ng. Econom c Sc ence Inst tute, Work ng Paper. Deppe, K. D., Gonzalez, F. J., Ne man, J. L., Jacobs, C., Pahlke, J., Sm th, K. B., & H bb ng, J. R. (2015). Reflect ve l berals and ntu t ve conservat ves: A look at the Cogn t ve Reflect on Test and deology. Judgment and Dec s on Mak ng, 10. Erceg, N., Gal ć, Z., & Ružojč ć, M. (2020). A reflect on on cogn t ve reflect on– test ng convergent/d vergent val d ty of two measures of cogn t ve reflect on.
Judgment and Dec s on Mak ng, 15(5), 741–755.
Evans, J. S., & Stanov ch, K. E. (2013). Dual-process theor es of h gher cogn t on: Advanc ng the debate. Perspect Psychol Sc , 8(3), 223–241.
http://dx.doi.org/10.1177/1745691612460685.
Everett, J. A. C., Ingbretsen, Z., Cushman, F., & C kara, M. (2017). Del berat on erodes cooperat ve behav or — Even towards compet t ve out-groups, even when us ng a control cond t on, and even when el m nat ng select on b as. Journal of
Exper mental Soc al Psychology, 73, 76–81.
Faul, F., Erdfelder, E., Buchner, A., & Lang, A. (2009). Stat st cal power analyses us ng G* Power 3.1: Tests for correlat on and regress on analyses. Behav or Research
Methods, 41, 1149–1160.
Forstmann, M., & Burgmer, P. (2015). Adults are ntu t ve m nd-body dual sts. Journal
of Exper mental Psychology: General, 144(1), 222–235.
Freder ck, S. (2005). Cogn t ve reflect on and dec s on mak ng. Journal of Econom c
Perspect ves, 19, 25–42.
Gerva s, W. M., & Norenzayan, A. (2012). Analyt c th nk ng promotes rel g ous d sbel ef. Sc ence, 336(6080), 493–496. http://dx.doi.org/10.1126/science.1215647. Gerva s, W. M., van Elk, M., Xygalatas, D., McKay, R. T., Aveyard, M., Buchtel, E. E., . . . R ekk , T. (2018). Analyt c athe sm: A cross-culturally weak and f ckle phenomenon? Judgment and Dec s on Mak ng, 13, 268–274.
Gr mm, P. (2010). Soc al des rab l ty b as. In J. Sheth & N. K. Malhotra (Eds.), W ley
nternat onal encycloped a of market ng. New York: John W ley & Sons.
Horstmann, N., Ahlgr mm, A., & Glöckner, A. (2009). How d st nct are ntu t on and del berat on? An eye-track ng analys s of nstruct on- nduced dec s on modes.
Judgment and Dec s on Mak ng, 4(5), 335–354.
Horstmann, N., Hausmann, D., & Ryf, S. (2009). Methods for nduc ng ntu t ve and del berate process ng modes. In G. A & W. C (Eds.), Foundat ons for trac ng ntu t on:
Challenges and methods (pp. 219–237). New York, NY: Psychology Press.
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The onl ne laboratory: conduct ng exper ments n a real labor market. Exper mental Econom cs, 14(3), 399– 425. http://dx.doi.org/10.1007/s10683-011-9273-9.
Igou, E. R., & Bless, H. (2007). On undes rable consequences of th nk ng: Fram ng effects as a funct on of substant ve process ng. Journal of Behav oral Dec s on
Mak ng, 20(2), 125–142. http://dx.doi.org/10.1002/bdm.543.
Isler, O., Maule, J., & Starmer, C. (2018). Is ntu t on really cooperat ve? Improved tests support the soc al heur st cs hypothes s. PLoS One, 13(1), e0190560.
http://dx.doi.org/10.1371/journal.pone.0190560.
Kahneman, D. (2011). Th nk ng, fast and slow. New York, NY: Farrar, Straus and G roux.
Kahneman, D., & Tversky, A. (1973). On the psychology of pred ct on. Psycholog cal
Rev ew, 80(4), 237–251.
Kle n, C. (2011). The dual track theory of moral dec s on-mak ng: A cr t que of the neuro mag ng ev dence. Neuroeth cs, 4(2), 143–162.
http://dx.doi.org/10.1007/s12152-010-9077-1.
Kocher, M. G., & Sutter, M. (2006). T me s money — T me pressure, ncent ves, and the qual ty of dec s on-mak ng. Journal of Econom c Behav or & Organ zat on, 61, 375–392.
Krajb ch, I., Bartl ng, B., Hare, T., & Fehr, E. (2015). Reth nk ng fast and slow based on a cr t que of react on-t me reverse nference. Nat Commun, 6, 7455.
http://dx.doi.org/10.1038/ncomms8455.
Lawson, M. A., Larr ck, R. P., & Soll, J. B. (2020). Compar ng fast th nk ng and slow th nk ng: The relat ve benef ts of ntervent ons, nd v dual d fferences, and nferent al rules. Judgment and Dec s on Mak ng, 15(5), 660.
Leboeuf, R. A., & Shaf r, E. (2003). Deep thoughts and shallow frames: On the suscept b l ty to fram ng effects. Journal of Behav oral Dec s on Mak ng, 16(2), 77– 92. http://dx.doi.org/10.1002/bdm.433.
Le st , T., Radun, J., V rtanen, T., Nyman, G., & Häkk nen, J. (2014). Concurrent explanat ons can enhance v sual dec s on mak ng. 145, 65–74.
http://dx.doi.org/10.1016/j.actpsy.2013.11.001.
Lewandowsky, S., Ecker, U. K., Se fert, C. M., Schwarz, N., & Cook, J. (2012). M s nformat on and ts correct on: Cont nued nfluence and successful deb as ng.
Psycholog cal Sc ence n the Publ c Interest, 13(3), 106–131.
http://dx.doi.org/10.1177/1529100612451018.
Ma, Y., L u, Y., Rand, D. G., Heatherton, T. F., & Han, S. (2015). Oppos ng oxytoc n effects on ntergroup cooperat ve behav or n ntu t ve and feflect ve m nds.
Neuropsychopharmacology, 40(10), 2379–2387.
Maule, A. J., Hockey, G. R. J., & Bdzola, L. (2000). Effects of t me-pressure on dec s on-mak ng under uncerta nty: changes n affect ve state and nformat on process ng strategy. Acta Psycholog ca, 104(3), 283–301.
Meyer, A., Freder ck, S., Burnham, T. C., Guevara P nto, J. D., Boyer, T. W., Ball, L. J., . . . Schuldt, J. P. (2015). D sfluent fonts don’t help people solve math problems. J
Exp Psychol Gen, 144(2), e16–30. http://dx.doi.org/10.1037/xge0000049.
Meyer, A., Zhou, E., & Shane, F. (2018). The non-effects of repeated exposure to the Cogn t ve Reflect on Test. Judgment and Dec s on Mak ng, 13(3), 246.
M ller, P. M., & Fagley, N. S. (1991). The effects of fram ng, problem var at ons, and prov d ng rat onale on cho ce. Personal ty and Soc al Psychology Bullet n, 17(5), 517– 522.
Morewedge, C. K., & Kahneman, D. (2010). Assoc at ve processes n ntu t ve judgment. Trends n Cogn t ve Sc ences, 14(10), 435–440.
http://dx.doi.org/10.1016/j.tics.2010.07.004.
Morewedge, C. K., Yoon, H., Scopell t , I., Symborsk , C. W., Korr s, J. H., & Kassam, K. S. (2015). Deb as ng dec s ons. Pol cy Ins ghts from the Behav oral and Bra n
Sc ences, 2(1), 129–140. http://dx.doi.org/10.1177/2372732215600886.
Myrseth, K. O. R., & Wollbrant, C. E. (2017). Cogn t ve foundat ons of cooperat on rev s ted: Commentary on Rand et al.(2012, 2014). Journal of Behav oral and
Exper mental Econom cs, 69, 133–138.
Neo, W. S., Yu, M., Weber, R. A., & Gonzalez, C. (2013). The effects of t me delay n rec proc ty games. Journal of Econom c Psychology, 34, 20–35.
http://dx.doi.org/10.1016/j.joep.2012.11.001.
Palan, S., & Sch tter, C. (2018). Prol f c.ac — A subject pool for onl ne exper ments.
Journal of Behav oral and Exper mental F nance, 17, 22–27.
http://dx.doi.org/10.1016/j.jbef.2017.12.004.
Peer, E., Brand marte, L., Samat, S., & Acqu st , A. (2017). Beyond the Turk:
Alternat ve platforms for crowdsourc ng behav oral research. Journal of Exper mental
Soc al Psychology, 70, 153–163.
Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2013). The role of analyt c th nk ng n moral judgements and values. Th nk ng & Reason ng,
20(2), 188–214. http://dx.doi.org/10.1080/13546783.2013.865000.
Pennycook, G., Cheyne, J. A., Sel , P., Koehler, D. J., & Fugelsang, J. A. (2012). Analyt c cogn t ve style pred cts rel g ous and paranormal bel ef. Cogn t on, 123(3), 335–346. http://dx.doi.org/10.1016/j.cognition.2012.03.003.
Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015). What makes us th nk? A three-stage dual-process model of analyt c engagement. Cogn t ve psychology, 80, 34– 72. http://dx.doi.org/10.1016/j.cogpsych.2015.05.001.
Rand, D. G. (2016). Cooperat on, fast and slow: Meta-analyt c ev dence for a theory of soc al heur st cs and self- nterested del berat on. Psychol Sc , 27(9), 1192–1206. http://dx.doi.org/10.1177/0956797616654455.
Rand, D. G., Greene, J. D., & Nowak, M. A. (2012). Spontaneous g v ng and calculated greed. Nature, 489(7416), 427–430.
http://dx.doi.org/10.1038/nature11467.
Sanchez, C., Sunderme er, B., Gray, K., & Cal n-Jageman, R. J. (2017). D rect repl cat on of Gerva s & Norenzayan (2012): No ev dence that analyt c th nk ng decreases rel g ous bel ef. PLoS One, 12(2), e0172636.
Sar bay, S. A., Y lmaz, O., & Körpe, G. G. (2020). Does ntu t ve m ndset nfluence bel ef n God? A reg stered repl cat on of Shenhav, Rand and Greene (2012). Judgment
and Dec s on Mak ng, 15(2), 193–202.
Sell er, A.-L., Scopell t , I., & Morewedge, C. K. (2019). Deb as ng Tra n ng Improves Dec s on Mak ng n the F eld. Psycholog cal Sc ence, 30(9), 1371–1379.
http://dx.doi.org/10.1177/0956797619861429.
Shenhav, A., Rand, D. G., & Greene, J. D. (2012). D v ne ntu t on: cogn t ve style nfluences bel ef n God. J Exp Psychol Gen, 141(3), 423–428.
http://dx.doi.org/10.1037/a0025391.
S eck, W. R., Qu nn, C. N., & Schooler, J. W. (1999). Just f cat on effects on the judgment of analogy. 27(5), 844–855. http://dx.doi.org/10.3758/bf03198537. S eck, W. R., & Yates, J. F. (1997). Expos t on effects on dec s on mak ng: Cho ce and conf dence n cho ce. Organ zat onal Behav or and Human Dec s on Processes, 70(3), 207–219. http://dx.doi.org/10.1006/obhd.1997.2706.
S rota, M., Theodoropoulou, A., & Juanch ch, M. (2020). D sfluent fonts do not help people to solve math and non-math problems regardless of the r numeracy. Th nk ng &
Reason ng, 1–18. http://dx.doi.org/10.1080/13546783.2020.1759689.
Song, H., & Schwarz, N. (2008). Fluency and the detect on of m slead ng quest ons: Low process ng fluency attenuates the Moses llus on. Soc al Cogn t on, 26(6), 791– 799.
Sp l opoulos, L., & Ortmann, A. (2018). The BCD of response t me analys s n exper mental econom cs. Exper mental Econom cs, 21, 383–433.
Stanov ch, K. E., & West, R. F. (2008). On the relat ve ndependence of th nk ng b ases and cogn t ve ab l ty. Journal of Personal ty and Soc al Psychology, 94(4), 672. Stephens, R. G., Dunn, J. C., Hayes, B. K., & Kal sh, M. L. (2020). A test of two processes: The effect of tra n ng on deduct ve and nduct ve reason ng. Cogn t on, 199, 104223. http://dx.doi.org/10.1016/j.cognition.2020.104223.
Suter, R. S., & Hertw g, R. (2011). T me and moral judgment. Cogn t on, 119(3), 454– 458. http://dx.doi.org/10.1016/j.cognition.2011.01.018.
Swam , V., Voracek, M., St eger, S., Tran, U. S., & Furnham, A. (2014). Analyt c th nk ng reduces bel ef n consp racy theor es. Cogn t on, 133(3), 572–585. http://dx.doi.org/10.1016/j.cognition.2014.08.006.
Takemura, K. (1994). Influence of elaborat on on the fram ng of dec s on. The Journal
of Psychology, 128(1), 33–39. http://dx.doi.org/10.1080/00223980.1994.9712709. Thompson, V. A., Evans, J., & Frank sh, K. (2009). Dual process theor es: A metacogn t ve perspect ve. Ar el, 137, 51–43.
Thomson, K. S., & Oppenhe mer, D. M. (2016). Invest gat ng an alternate form of the cogn t ve reflect on test. Judgment and Dec s on Mak ng, 11, 99–113.
T nghog, G., Andersson, D., Bonn, C., Bott ger, H., Josephson, C., Lundgren, G., . . . Johannesson, M. (2013). Intu t on and cooperat on recons dered. Nature, 498(7452), E1–2; d scuss on E2–3. http://dx.doi.org/10.1038/nature12194.
Toplak, M. E., West, R. F., & Stanov ch, K. E. (2011). The Cogn t ve Reflect on Test as a pred ctor of performance on heur st cs-and-b ases tasks. Memory & Cogn t on,
39(7), 1275–1289. http://dx.doi.org/10.3758/s13421-011-0104-1.
Trémol ère, B., & Bonnefon, J.-F. (2014). Eff c ent k ll–save rat os ease up the cogn t ve demands on counter ntu t ve moral ut l tar an sm. Personal ty and Soc al
Tversky, A., & Kahneman, D. (1983). Extens onal versus ntu t ve reason ng: The conjunct on fallacy n probab l ty judgment. Psycholog cal Rev ew, 90, 293–315. V e der, F. M. (2011). Separat ng real ncent ves and accountab l ty. 14(4), 507–518. http://dx.doi.org/10.1007/s10683-011-9279-3.
Wang, C. S., S vanathan, N., Narayanan, J., Ganegoda, D. B., Bauer, M., Bodenhausen, G. V., & Murn ghan, K. (2011). Retr but on and emot onal regulat on: The effects of t me delay n angry econom c nteract ons. Organ zat onal Behav or and Human
Dec s on Processes, 116(1), 46–54. http://dx.doi.org/10.1016/j.obhdp.2011.05.007. Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and val dat on of br ef measures of pos t ve and negat ve affect: the PANAS scales. J Pers Soc Psychol, 54(6), 1063–1070. http://dx.doi.org/10.1037//0022-3514.54.6.1063.
Y lmaz, O., & Isler, O. (2019). Reflect on ncreases bel ef n God through self-quest on ng among non-bel evers. Judgment and Dec s on Mak ng, 14(6), 649–657. Y lmaz, O., & Sar bay, S. A. (2016). An attempt to clar fy the l nk between cogn t ve style and pol t cal deology: A non-western repl cat on and extens on. Judgment and
Dec s on Mak ng, 11, 287–300.
Y lmaz, O., & Sar bay, S. A. (2017a). Act vat ng analyt c th nk ng enhances the value g ven to nd v dual z ng moral foundat ons. Cogn t on, 165, 88–96.
http://dx.doi.org/10.1016/j.cognition.2017.05.009.
Y lmaz, O., & Sar bay, S. A. (2017b). Analyt c thought tra n ng promotes l beral sm on contextual zed (but not stable) pol t cal op n ons. Soc al Psycholog cal and Personal ty
Sc ence, 8, 789–795. http://dx.doi.org/10.1177/1948550616687092.
Y lmaz, O., & Sar bay, S. A. (2017c). The relat onsh p between cogn t ve style and pol t cal or entat on depends on the measures used. Judgment and Dec s on Mak ng,
12(2), 140–147.
Yonker, J. E., Edman, L. R. O., Cresswell, J., & Barrett, J. L. (2016). Pr med analyt c thought and rel g os ty: The mportance of nd v dual character st cs. Psychology of
Rel g on and Sp r tual ty, 8(4), 298–308. http://dx.doi.org/10.1037/rel0000095. *
Centre for Behav oral Econom cs, Soc ety and Technology, School of Econom cs and F nance, Queensland Un vers ty of Technology, Br sbane, Austral a. Ema l: ozan. sler@qut.edu.au. ORCID: 0000-0002-4638-2230.
#
Department of Psychology, Kad r Has Un vers ty, Istanbul, Turkey. ORCID: 0000-0002-6094-7162.
$
Department of Psychology, Alt nbas Un vers ty, Istanbul, Turkey. ORCID: 0000-0002-3469-590X.
Th s research s funded by Alt nbas Un vers ty Research Fund Grant Number PB2018-FALL-IISBF-3. Ozan Isler and Onurcan Y lmaz acknowledge support from the Th nk Forward In t at ve.
Copyr ght: © 2020. The authors l cense th s art cle under the terms of the Creat ve Commons Attr but on 3.0 L cense.
