close
Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Apr 21;11(4):76.
doi: 10.3390/jintelligence11040076.

Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test

Nick Byrd  1 , Brianna Joseph  2 , Gabriela Gongora  3 , Miroslav Sirota  4
Affiliations

Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test

Nick Byrd et al. J Intell. .

Abstract

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of mathematical reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of 'reflection' (as deliberate and conscious).

Keywords: cognitive reflection test; heuristics and biases; judgment- and decision-making; psychometrics; think-aloud protocol analysis.

PubMed Disclaimer

Conflict of interest statement

Phonic audio survey service was provided in exchange for beta testing its platform and features (Study 2).

Figures

Figure A1
Figure A1
Practice effect tests in Study 1 (N = 99). Performance did not increase but decreased during the survey. The magnitude of this decrease was not dramatically different between conditions (Z-score). Error bands indicate standard error.
Figure A2
Figure A2
Practice effect test in Study 2 (N = 102). Performance did not increase or decrease during the survey. Error bands indicate standard error.
Figure 1
Figure 1
The effect of thinking aloud on (A) the number of lured responses and (B) the number of correct responses on the verbal cognitive reflection test (vCRT) in Study 1 (N = 99). Error bars represent a standard error.
Figure 2
Figure 2
Correlations between the “two-factor” coding of reflective responses (deliberate and/or conscious) and (A) the number of lured—or so-called “unreflective”—responses and (B) the number of correct—or so-called “reflective”—responses on the verbal reflection test (vCRT) in the think-aloud condition of Study 1 (N = 47) with gray standard error bands.
Figure 3
Figure 3
Correlations between consideration of lures and (A) acceptance of lures and (B) acceptance of correct responses on the verbal reflection test (vCRT) in Study 1 (N = 99). Gray bands represent a standard error.
Figure 4
Figure 4
The number of (A) lured responses and (B) correct responses on the verbal reflection test (vCRT) among participants in the think-aloud condition of Study 1 (N = 47) depending on their unsolicited self-report of familiarity with the vCRT with standard error bars.
Figure 5
Figure 5
Correlations between the “two-factor” coding of reflective responses (deliberate and/or conscious) and (A) the number of lured—or so-called “unreflective”—responses and (B) the number of correct—or so-called “reflective”—responses on the verbal reflection test (vCRT) performance in Study 2 (N = 102). Gray bands represent a standard error.
Figure 6
Figure 6
Correlations between consideration of lures and (A) acceptance of lures and (B) acceptance of correct responses on the verbal reflection test (vCRT) performance Study 2 (N = 102). Gray bands represent a standard error.
Figure 7
Figure 7
The number of (A) lured responses and (B) correct responses on the verbal reflection test (vCRT) among participants in Study 2 (N = 102) depending on their unsolicited self-report of familiarity with the vCRT. Error bars represent a standard error.
See this image and copyright information in PMC

References

    1. Attali Yigal, Bar-Hillel Maya. The False Allure of Fast Lures. Judgment and Decision Making. 2020;15:93–111. doi: 10.1017/S1930297500006938. - DOI
    1. Bago Bence, De Neys Wim. The Smart System 1: Evidence for the Intuitive Nature of Correct Responding on the Bat-and-Ball Problem. Thinking & Reasoning. 2019;25:257–99. doi: 10.1080/13546783.2018.1507949. - DOI
    1. Ball Linden J., Stevens Alexandra. Evidence for a Verbally-Based Analytic Component to Insight Problem Solving. Proceedings of the Thirty-First Annual Conference of the Cognitive Science Society. 2009;31:1060–65.
    1. Białek Michał, Pennycook Gordon. The Cognitive Reflection Test Is Robust to Multiple Exposures. Behavior Research Methods. 2018;50:1953–59. doi: 10.3758/s13428-017-0963-x. - DOI - PubMed
    1. Blech Christine, Gaschler Robert, Bilalić Merim. Why Do People Fail to See Simple Solutions? Using Think-Aloud Protocols to Uncover the Mechanism behind the Einstellung (Mental Set) Effect. Thinking & Reasoning. 2020;26:552–80. doi: 10.1080/13546783.2019.1685001. - DOI

LinkOut - more resources