Belief in & interventions against misinformation

Jan Pfänder & Sacha Altay

2025-06-27

People worry about misinformation

But people might not worry enough about information

Newman, N., Fletcher, R., Kalogeropoulos, A., & Nielsen, R. K. (2018). Reuters Institute Digital News Report 2018.

There has been a lot of research on people’s belief in misinformation

Pennycook, G., Binnendyk, J., Newton, C., & Rand, D. G. (2021). A Practical Guide to Doing Behavioral Research on Fake News and Misinformation. Collabra: Psychology, 7(1), 25293.

67 papers


194’438 participants


303 effect sizes


40 countries

Discernment = Accuracy(true news) − Accuracy(false news)

For 298 of the 303 effect sizes, people rated true news as (a lot) more accurate than false news


On the whole, people did well in discerning true from false news…


but of course they still made some errors.


Skepticism bias = Error(true news) - Error(false news)

For 203 of 303 effect sizes, people made (slightly) more errors on true news than on false news.

Interim conclusion


People discern rather well between true and false news


If they err, they tend to be more skeptical of true news than gullible towards false news

Many studies have tested individual-level interventions to reduce belief in misinformation

See for example: Basol, M., Roozenbeek, J., & Linden, S. van der. (2020). Good News about Bad News: Gamified Inoculation Boosts Confidence and Cognitive Immunity Against Fake News. Journal of Cognition

Guess, A. M., Lerner, M., Lyons, B., Montgomery, J. M., Nyhan, B., Reifler, J., & Sircar, N. (2020). A digital media literacy intervention increases discernment between mainstream and false news in the United States and India. Proceedings of the National Academy of Sciences

Brashier, N. M., Pennycook, G., Berinsky, A. J., & Rand, D. G. (2021). Timing matters when correcting fake news. Proceedings of the National Academy of Sciences

But…


The effectiveness of these interventions has typically been evaluated in terms of discernment only


Some misinformation interventions might be harmful if they foster general skepticism of news

Modirrousta-Galian, A., & Higham, P. A. (2023). Gamified inoculation interventions do not improve discrimination between true and fake news: Reanalyzing existing research with receiver operating characteristic analysis. Journal of Experimental Psychology: General.

How effective are misinformation interventions ? A(nother) meta-analysis

Individual Participant Data (IPD) Meta-analysis

  1. Identify relevant studies

  2. Download raw data

  3. Clean and bring into common format

  4. Analyze in a unifying framework (Signal Detection Theory)

Signal Detection Theory




Measures:

  • sensitivity (d’) (~discernment)
  • response bias (c) (~skepticism bias)

Outcomes:

  • \(\Delta d'\) = d’(treatment) - d’(control)
  • \(\Delta c\) = c (treatment) - c (control)

What we have so far…

paper_id experiment_id Participants News headlines Intervention
1 altay_2023 1 198 16 FalseLabel
2 badrinathan_2021 1 818 14 literacy
3 bago_2022 2 331 24 suppression
4 bago_2022 3 1502 24 suppression
5 bago_2022 4 1520 24 suppression
6 brashier_2021 1 397 36 veracity_labels
7 brashier_2021 2 422 36 veracity_labels
8 clayton_2020 1 429 9 false_no_warning
9 dias_2020 1 190 24 highlight_banner
10 dias_2020 2 923 24 highlight_banner
11 guess_2020 1 4907 12 literacy
12 guess_2020 2 3744 12 literacy
13 guess_2020 3 3273 12 literacy

Figure 1: Forest plot for delta d. The figure displays all effect sizes. Effects are weighed by their sample size. Effect sizes are calculated as z-scores. Horizontal bars represent 95% confidence intervals. The average estimate is the result of a multilevel meta model with clustered standard errors at the paper level.

Figure 2: Forest plots for delta c. The figure displays all effect sizes. Effects are weighed by their sample size. Effect sizes are calculated as z-scores. Horizontal bars represent 95% confidence intervals. The average estimate is the result of a multilevel meta model with clustered standard errors at the paper level.

Conclusion

  • It is not (yet) clear how effective individual-level misinformation interventions are:
    • do they help people discriminate better?
    • do they just make people more skeptical?
    • which are the most/least effective interventions?

Thank you

Jan Pfänder

Sacha Altay

Slides and our pre-registration are available here:

Backup slides

Countries

Political concordance

Political concordance


Alignment between personal political stance and the political slant of the news


For example, pro-republican news rated by republicans are coded as concordant

Comparison SDT and Discernment/ Skepticism bias

Selection bias

Signal Detection theory

Table 1: Accuracy ratings in Signal Detection Theory terms
Participant response
Stimulus Accurate Not Accurate SDT Metric
True news (target) Hit Miss Hit rate (HR) = Hits / (Hits + Misses)
False news (distractor) False alarm Correct rejection False alarm rate (FAR) = False Alarms / (False Alarms + Correct Rejections)


\[ d' = z(HR) - z(FAR) \]


\[ c = -\frac{1}{2}(\text{zHR} + \text{zFAR}) \]