We hypothesize that each test will uniquely explain a statistically significant proportion of the variance in each self-report of mentalizing. We also hypothesize that the Hinting Task and the Faux-pas Test will demonstrate a higher incremental validity compared to the RMET and the Frith-Happé Animations (Quesque & Rossetti, 2020).
Because the mentalizing tests and self-report measures inherently entail measurement errors, we will compute a latent variable from the combined measures and examine the incremental validity of combined performance. This will test the idea that using multiple tests provides a more comprehensive assessment of mentalizing abilities, reinforcing the importance of multimethod approaches in both research and clinical neuropsychological evaluations.
Last, based on previous results suggesting that self-report mentalizing measures may capture the perceived performance (rather than the actual performance) at mentalizing tasks, we will explore the incremental validity of both performance and perceived performance at mentalizing tasks against self-reported mentalizing measures. We hypothesize that the performance at mentalizing tests predicts variance in self-report measures beyond that explained by perceived performance.
Finally, we hypothesize that performance at mentalizing tests, when combined into a latent variable, will explain unique variance in self-report measures of psychosocial functioning (loneliness and broad psychosocial functioning), beyond the variance explained by performance at reasoning and language tests. Furthermore, we explore the contribution of three distinct latent variables —mentalizing tests, perceived performance, and self-reports— on psychosocial functioning outcomes, while controlling for performance in reasoning and language tasks. We hypothesize that perceived performance and self-reports of mentalizing will explain unique variance in psychosocial functioning.
library(MASS)
IRI_REAS = .05
IRI_LANG = .05
MZtest_IRI_1 = .25
MZtest_IRI_2 = .15
REAS_LANG_1 = .40
REAS_LANG_2 = .30
MZtest_REAS_1 = .20
MZtest_LANG_1 = .30
MZtest_REAS_2 = .10
MZtest_LANG_2 = .20
simulate_analysis <- function(MZtest_REAS, MZtest_LANG, MZtest_IRI, REAS_LANG, IRI_REAS, IRI_LANG, n_participant = 320, n_simulations = 10000) {
correlation_matrix <- matrix(NA, nrow = 4, ncol = 4)
rownames(correlation_matrix) <- c("MZtest", "REAS", "LANG", "IRI")
colnames(correlation_matrix) <- c("MZtest", "REAS", "LANG", "IRI")
diag(correlation_matrix) <- 1.0
correlation_matrix["MZtest", "REAS"] <- MZtest_REAS
correlation_matrix["REAS", "MZtest"] <- MZtest_REAS
correlation_matrix["MZtest", "LANG"] <- MZtest_LANG
correlation_matrix["LANG", "MZtest"] <- MZtest_LANG
correlation_matrix["MZtest", "IRI"] <- MZtest_IRI
correlation_matrix["IRI", "MZtest"] <- MZtest_IRI
correlation_matrix["REAS", "LANG"] <- REAS_LANG
correlation_matrix["LANG", "REAS"] <- REAS_LANG
correlation_matrix["REAS", "IRI"] <- IRI_REAS
correlation_matrix["IRI", "REAS"] <- IRI_REAS
correlation_matrix["IRI", "LANG"] <- IRI_LANG
correlation_matrix["LANG", "IRI"] <- IRI_LANG
mu = c(0, 0, 0, 0)
inc_val_tot = p_value_total = rep(NA, n_simulations)
for (i in 1:n_simulations) {
dat_1 <- as.data.frame(MASS::mvrnorm(n=n_participant, mu=mu, Sigma=correlation_matrix))
colnames(dat_1) <- c("Mentalizing", "REAS", "LANG", "IRI")
dat_1$ID = 1:nrow(dat_1)
res_crude = lm(IRI ~ REAS + LANG, data = dat_1)
res_ToM = lm(IRI ~ REAS + LANG + Mentalizing, data = dat_1)
INC_crude = summary(res_crude)$r.squared
INC_ToM = summary(res_ToM)$r.squared
p_value_total[i] = as.numeric(anova(res_crude, res_ToM)[2, "Pr(>F)"])
inc_val_tot[i] = INC_ToM - INC_crude
}
return(data.frame(
power = sum(p_value_total < 0.05)/n_simulations * 100,
incremental_validity = mean(inc_val_tot)))
}
result_1_best_scenario <- simulate_analysis(MZtest_REAS_1, MZtest_LANG_1, MZtest_IRI_1, REAS_LANG_1, IRI_REAS, IRI_LANG)
result_2_worst_scenario <- simulate_analysis(MZtest_REAS_2, MZtest_LANG_2, MZtest_IRI_2, REAS_LANG_2, IRI_REAS, IRI_LANG)
print(result_1_best_scenario)## power incremental_validity
## 1 99.38 0.06181584print(result_2_worst_scenario)## power incremental_validity
## 1 71.78 0.02282091Klein, H., Springfield ,Cassi R., & and Pinkham, A. E. (2024). Measuring social cognition within the university: The Social Cognition Psychometric Evaluation (SCOPE) battery in an undergraduate sample. Applied Neuropsychology: Adult, 31(5), 866–873. https://doi.org/10.1080/23279095.2022.2082875
Mansueto, G., Schruers, K., Cosci, F., van Os, J., Alizadeh, B. Z., Bartels-Velthuis, A. A., van Beveren, N. J., Bruggeman, R., Cahn, W., de Haan, L., Delespaul, P., Meijer, C. J., Myin-Germeys, I., Kahn, R. S., Schirmbeck, F., Simons, C. J. P., van Haren, N. E. M., & van Winkel, R. (2019). Childhood adversities and psychotic symptoms: The potential mediating or moderating role of neurocognition and social cognition. Schizophrenia Research, 206, 183–193. https://doi.org/10.1016/j.schres.2018.11.028
Meinhardt-Injac, B., Daum, M. M., & Meinhardt, G. (2020). Theory of mind development from adolescence to adulthood: Testing the two-component model. British Journal of Developmental Psychology, 38(2), 289–303. https://doi.org/10.1111/bjdp.12320
Müller, S., Wendt, L. P., & Zimmermann, J. (2023). Development and Validation of the Certainty About Mental States Questionnaire (CAMSQ): A Self-Report Measure of Mentalizing Oneself and Others. Assessment, 30(3), 651–674. https://doi.org/10.1177/10731911211061280
Murphy, B. A., & Lilienfeld, S. O. (2019). Are self-report cognitive empathy ratings valid proxies for cognitive empathy ability? Negligible meta-analytic relations with behavioral task performance. Psychological Assessment, 31(8), 1062–1072. https://doi.org/10.1037/pas0000732
Sty in the Mind’s Eye: A Meta-Analytic Investigation of the Nomological Network and Internal Consistency of the “Reading the Mind in the Eyes” Test—Anne Frieda Doris Kittel, Sally Olderbak, Oliver Wilhelm, 2022. (n.d.). Retrieved 8 April 2025, from https://journals.sagepub.com/doi/10.1177/1073191121996469
Wechsler, D. (2012). Wechsler Adult Intelligence Scale—Fourth Edition. https://doi.org/10.1037/t15169-000
Wendt, L. P., Zimmermann, J., Spitzer, C., & Müller, S. (2024). Mindreading measures misread? A multimethod investigation into the validity of self-report and task-based approaches. Psychological Assessment, 36(5), 365–378. https://doi.org/10.1037/pas0001310
Yeung, E. K. L., Apperly, I. A., & Devine, R. T. (2024). Measures of individual differences in adult theory of mind: A systematic review. Neuroscience & Biobehavioral Reviews, 157, 105481. https://doi.org/10.1016/j.neubiorev.2023.105481
George arrives at Angela’s office after a long, hot trip on the motorway. Angela immediately begins to talk about some business ideas. George interrupts Angela to say:
“Ugh, it has been a long, hot trip on the motorway.”
• 1st question: What does George really want to say when he says this?
• 2nd hint: George continues to say, “I am thirsty.”
Marissa is going to the bathroom to have a shower. Anna just had a bath. When Marissa realises that the bathroom is dirty, she calls Anna and says:
“Couldn’t you find the cleaner, Anna?”
• 1st question: What does Marissa really want to say when she says this?
• 2nd hint: Marissa continues to say, “Sometimes you’re very lazy, Anna.”
Jacob is going to the supermarket with his mother. They arrive at the sweets section and Jacob says:
“Look, those sweets look very good.”
• 1st question: What does Jacob really want to say when he says this?
• 2nd hint: Jacob continues to say, “I’m hungry, mum.”
Hinting Item Scoring:
- 2 = Correct answer with first hint
- 1 = Correct answer with second hint
- 0 = Paraphrase of what character said or No response
=> 20 = Total possible points
Notes on scoring:
- If the respondent gives a 1-point response on the first hint, you may
score a 1 and move on to the next question.
- If the respondent gives a 0-point response on the first hint, present
the second hint.
- No further question is asked after the second hint.
- You may re-read a story once if the respondent requests it.
RMET Item Scoring:
- 1 = Correct answer
- 0 = Incorrect answer
=> 36 = Total possible points
MCQ-categorization
Which of these 3 categories best fits the film clip you have just seen?
No interaction
Physical interaction
Mental interaction
MCQ-feelings (correct answers in bold)
Coaxing:
How do you think the Little Triangle feels at the end of the clip?
How do you think the Big Triangle feels at the end of the clip?
Mocking:
How do you think the Little Triangle feels at the end of the clip?
How do you think the Big Triangle feels at the end of the clip?
Seducing:
How do you think the Little Triangle feels at the end of the clip?
How do you think the Big Triangle feels at the end of the clip?
Surprising:
How do you think the Little Triangle feels at the end of the clip?
How do you think the Big Triangle feels at the end of the clip?
Jeanette bought her friend Anne a crystal bowl for a wedding gift. Anne had a big wedding and there were a lot of presents to keep track of. About a year later, Jeanette was over one night at Anne’s for dinner. Jeanette dropped a wine bottle by accident on the crystal bowl, and the bowl shattered.
“I’m really sorry, I’ve broken the bowl,” said Jeanette. “Don’t worry,” said Anne, “I never liked it anyway. Someone gave it to me for my wedding.”
• 1st question: Did someone say something they shouldn’t have said? (Tests for detection of faux pas.)
• 2nd question: Who said something they shouldn’t have said? (Tests for understanding of faux pas.)
• 3rd question: Why shouldn’t they have said it? (Requires understanding mental state of listener.)
• 4th question: Why did they say it? (Requires understanding mental state of speaker.)
• 5th question: Did Anne remember that Jeanette had given her the crystal bowl?
• 6th question: How do you think Jeanette felt?
• 7th question: What is Jeanette giving Anne for her wedding?
• 8th question: How was the bowl broken?
Questions 7 and 8 are control questions to assess the good understanding of the story. Questions 2 to 6 are only asked if the participant answers “yes” on the first question.
Faux-pas Test Scoring:
Control stories :
=> 10 = Control stories possible points
Faux-pas stories :
=> 30 = Faux-pas stories possible points
=> 40 = Total possible points
I sometimes find it difficult to see things from the “other guy’s” point of view.
I try to look at everybody’s side of a disagreement before I make a decision.
I sometimes try to understand my friends better by imagining how things look from their perspective.
If I’m sure I’m right about something, I don’t waste much time listening to other people’s arguments.
I believe that there are two sides to every question and try to look at them both.
When I’m upset at someone, I usually try to “put myself in his shoes” for a while.
Before criticizing somebody, I try to imagine how I would feel if I were in their place.
IRI - PT Scoring:
7-point scale :
=> 49 = Total possible points
I often have tender, concerned feelings for people less fortunate than me.
Sometimes I don’t feel very sorry for other people when they are having problems.
When I see someone being taken advantage of, I feel kind of protective towards them.
Other people’s misfortunes do not usually disturb me a great deal.
When I see someone being treated unfairly, I sometimes don’t feel very much pity for them.
I am often quite touched by things that I see happen.
I would describe myself as a pretty soft-hearted person.
IRI - EC Scoring:
7-point scale :
=> 49 = Total possible points
I know my innermost wishes and desires.
I can tell whether another person is at peace with themselves.
I know how other people will react to something.
I understand why certain things make me happy.
I know what I am trying to achieve with my behavior.
I can tell when a person in a group is feeling awkward.
I know why I am interested in certain things.
I can tell when other people don’t give their honest opinions.
I understand my feelings.
I know when other people are hiding their thoughts.
I know what my virtues are.
I can tell when other people are just pretending to find something funny.
I know why I have a strong opinion on a subject.
When I’m in a bad mood, I know the reason why.
I know if a person is trustworthy.
I know the reasons for my behavior.
I can tell when other people are taking advantage of someone.
I can tell if another person is bored by what I am saying
I know how a person feels when I look at their face.
I know what the best decision is for me in a difficult life situation.
7-point scale :
People’s thoughts are a mystery to me
I don’t always know why I do what I do
When I get angry I say things without really knowing why I am saying them
If I feel insecure I can behave in ways that put others’ backs up
Sometimes I do things without really knowing why
Strong feelings often cloud my thinking
7-point scale :
I have a hard time reading people’s emotions
I can tell when someone is afraid
It’s obvious when people are pretending to be happy
I usually understand why people feel the way they do
I have a hard time figuring out what someone else is feeling
I can tell when people are about to lose their temper
I can usually predict how someone will feel
I can usually tell how people are feeling
I am not good at understanding other people’s emotions
I can usually guess what’s making someone angry
People don’t have to tell me when they’re sad, I can see it in their faces
I find it hard to tell when someone is sad
5-point scale :
I find it easy to put myself in somebody else’s shoes
I sometimes find it difficult to see things from other people’s point of view
I sometimes try to understand my friends better by imagining how things look from their perspective
I can usually understand another person’s viewpoint, even if it differs from my own
4-point scale :
[Incremental validity of each test, against IRI-PT]. A commonality analysis will assess incremental validity of each mentalizing test in explaining variance in IRI-PT scores while controlling for language and reasoning. Bootstrap estimation (10,000 simulations) will provide 95% confidence intervals, and nested linear models will provide insight into the p-value. This will provide the percentage of variance (R²) explained by each test.
Exploratory analysis 1 [comparison of incremental validity between tests, against IRI-PT]. We will compare the incremental R² of each test by computing pairwise differences, and apply bootstrapping (10,000 resamples) to assess statistical significance. This will identify the tests that have the largest incremental validity in IRI-PT scores.
Exploratory analysis 2 [incremental validity of each test, against each mentalizing questionnaire]. A commonality analysis will assess incremental validity of each mentalizing test in explaining variance in each mentalizing self-report while controlling for language and reasoning.Bootstrap estimation (10,000 simulations) will provide 95% confidence intervals, and nested linear models will provide insight into the p-value.
Exploratory analysis 3 [comparison of incremental validity between tests, against each mentalizing questionnaire]. We will compare the incremental R² of each test on each self-report at a time by computing pairwise differences, and apply bootstrapping (10,000 resamples) to assess statistical significance. This will identify the tests that explain the largest variance in each questionnaire.
Exploratory analysis 4 [incremental validity of a combination of all tests, against each questionnaire]. A commonality analysis will assess incremental validity of a latent variable of all mentalizing tests in explaining variance of each self-report of mentalizing while controlling for language and reasoning. Bootstrap estimation (10,000 simulations) will provide 95% confidence intervals, and nested linear models will provide insight into the p-value. This could inform us on the importance of assessing mentalizing using more than one test when performing a clinical neuropsychological assessment but also on which self-reports better reflect mentalizing performance.
Exploratory analysis 5 [incremental validity of each test, against a combination of all mentalizing self-reports]. A commonality analysis will assess incremental validity of each mentalizing test in explaining variance in a latent variable of all self-reports of mentalizing while controlling for language and reasoning. Bootstrap estimation (10,000 simulations) will provide 95% confidence intervals, and nested linear models will provide insight into the p-value.
Exploratory analysis 6 [comparison of perceived performance and mentalizing tests performance in explaining unique variance in mentalizing self reports]. A commonality analysis will assess incremental validity of each mentalizing test in explaining variance in a latent variable of all self-reports of mentalizing while controlling for perceived performance, language and reasoning. Bootstrap estimation (10,000 simulations) will provide 95% confidence intervals, and nested linear models will provide insight into the p-value. This will facilitate the identification of the variance of self-reports explained by mentalizing task performance that has not already been explained by perceived performance.
Exploratory analysis 7 [incremental validity of a combination of all tests, against each measure of psychosocial functioning (loneliness and broad psychosocial functioning)]. A commonality analysis will assess incremental validity of a latent variable of all mentalizing tests in explaining variance in each measure of psychosocial functioning while controlling for language and reasoning. Bootstrap estimation (10,000 simulations) will provide 95% confidence intervals, and nested linear models will provide insight into the p-value. This exploratory analysis will investigate whether a latent variable of all tests explains unique variance in psychosocial functioning measures.
Exploratory analysis 8 [incremental validity of tests, perceived performance, mentalizing questionnaire, against each measure of psychosocial functioning (loneliness and broad psychosocial functioning)]. This analysis extends the previous one by incorporating perceived performance on tests and scores from mentalizing questionnaires in explaining each measure of social functioning. This will allow us to isolate the variance in psychosocial functioning explained by three latent variables—mentalizing self-reports, perceived performance, and mentalizing tests—while controlling for the other factors, along with reasoning and language.
library(psych)
library(MASS)
library(yhat)
library(boot)
set.seed(12134)
r = runif(64, 0.1, 0.4)
mat = matrix(r, nrow=8)
diag(mat) <- 1
mu=c(19, 17, 34, 16, 38, 6, 9, 11)
dat_1 <- as.data.frame(mvrnorm(n=19, mu=rep(0,8), Sigma=mat))
colnames(dat_1) <- c("RMET", "FauxPas", "Hinting", "Frith", "Iri", "UCLA3", "language", "reasoning")
dat_1$ID = 1:nrow(dat_1)
## Commonality analysis
res_lm = lm(Iri~RMET+FauxPas+Frith+Hinting+language+reasoning, data=dat_1)
res_dom <- calc.yhat(res_lm)
boot.out = boot(data=dat_1, yhat::boot.yhat, 1000,
lmOut=res_lm, regrout0=res_dom)
boot_results = booteval.yhat(res_dom, boot.out,bty="perc",.95,3)results_filtered <- boot_results$combCIaps["language,reasoning",
c("RMET.Inc", "FauxPas.Inc", "Frith.Inc", "Hinting.Inc")]
print(results_filtered)## RMET.Inc FauxPas.Inc Frith.Inc
## language,reasoning 0.049(0.000,0.391) 0.266(0.003,0.639) 0.113(0.002,0.357)
## Hinting.Inc
## language,reasoning 0.121(0.000,0.501)results_filtered <- boot_results$combCIincDiff["language,reasoning",
c("RMET.Inc-FauxPas.Inc", "RMET.Inc-Frith.Inc", "RMET.Inc-Hinting.Inc", "FauxPas.Inc-Frith.Inc", "FauxPas.Inc-Hinting.Inc", "Frith.Inc-Hinting.Inc")]
print(results_filtered)## RMET.Inc-FauxPas.Inc RMET.Inc-Frith.Inc
## language,reasoning -0.217(-0.572,0.304) -0.064(-0.315,0.312)
## RMET.Inc-Hinting.Inc FauxPas.Inc-Frith.Inc
## language,reasoning -0.072(-0.341,0.212) 0.153(-0.128,0.473)
## FauxPas.Inc-Hinting.Inc Frith.Inc-Hinting.Inc
## language,reasoning 0.145(-0.400,0.585) -0.008(-0.381,0.281)library(ggplot2)##
## Attaching package: 'ggplot2'## The following objects are masked from 'package:psych':
##
## %+%, alphalibrary(ggsignif)## Warning: package 'ggsignif' was built under R version 4.1.3data_plot <- data.frame(
Test = c("RMET", "Hinting", "Frith", "FauxPas"),
Incrementality = c(0.090, 0.082, 0.085, 0.101)
)
comparisons <- list(
c("RMET", "Hinting"),
c("RMET", "Frith"),
c("Frith", "Hinting"),
c("RMET", "FauxPas"),
c("Frith", "FauxPas"),
c("Hinting", "FauxPas")
)
annotations <- c("p = ns", "p = ns", "p = ns", "p = ns", "p = ns", "p = ns")
y_positions <- c(0.09,
0.13,
0.1,
0.19,
0.11,
0.16)
ggplot(data_plot, aes(x = Test, y = Incrementality, fill = Test)) +
geom_bar(stat = "identity", color = "black", width = 0.2) +
geom_signif(
comparisons = comparisons,
annotations = annotations,
y_position = y_positions,
tip_length = 0.03,
textsize = 4
) +
labs(
title = "Incrementality of Predictors",
x = "Predictors",
y = "Incrementality (R² Difference)"
) +
theme_minimal() +
theme(
plot.title = element_text(hjust = 0.5),
legend.position = "none"
) +
ylim(0, 0.2)
Abell, F., Happé, F., & Frith, U. (2000). Do triangles play tricks? Attribution of mental states to animated shapes in normal and abnormal development. Cognitive Development, 15(1), 1–16. https://doi.org/10.1016/S0885-2014(00)00014-9
Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (n.d.). The ‘“ Reading the Mind in the Eyes ”’ Test Revised Version: A Study with Normal Adults, and Adults with Asperger Syndrome or High-functioning Autism.
Bertoux, M., Delavest, M., Souza, L. C. de, Funkiewiez, A., Lépine, J.-P., Fossati, P., Dubois, B., & Sarazin, M. (2012). Social Cognition and Emotional Assessment differentiates frontotemporal dementia from depression. Journal of Neurology, Neurosurgery & Psychiatry, 83(4), 411–416. https://doi.org/10.1136/jnnp-2011-301849
Clutterbuck, R. A., Callan, M. J., Taylor, E. C., Livingston, L. A., & Shah, P. (2021). Development and validation of the Four-Item Mentalising Index. Psychological Assessment, 33(7), 629–636. https://doi.org/10.1037/pas0001004
Corcoran, R., Mercer, G., & Frith, C. D. (1995). Schizophrenia, symptomatology and social inference: Investigating “theory of mind” in people with schizophrenia. Schizophrenia Research, 17(1), 5–13. https://doi.org/10.1016/0920-9964(95)00024-G
Fonagy, P., Luyten, P., Moulton-Perkins, A., Lee, Y.-W., Warren, F., Howard, S., Ghinai, R., Fearon, P., & Lowyck, B. (2016). Development and Validation of a Self-Report Measure of Mentalizing: The Reflective Functioning Questionnaire. PLOS ONE, 11(7), e0158678. https://doi.org/10.1371/journal.pone.0158678
Gosling, C. J., Colle, R., Cartigny, A., Jollant, F., Corruble, E., & Frajerman, A. (2024). Measuring loneliness: A head-to-head psychometric comparison of the 3- and 20-item UCLA Loneliness Scales. Psychological Medicine, 1–7. https://doi.org/10.1017/S0033291724002083
Müller, S., Wendt, L. P., & Zimmermann, J. (2023). Development and Validation of the Certainty About Mental States Questionnaire (CAMSQ): A Self-Report Measure of Mentalizing Oneself and Others. Assessment, 30(3), 651–674. https://doi.org/10.1177/10731911211061280
Nimon, K. F., & Oswald, F. L. (2013). Understanding the Results of Multiple Linear Regression: Beyond Standardized Regression Coefficients. Organizational Research Methods, 16(4), 650–674. https://doi.org/10.1177/1094428113493929
Spitzer, C., Zimmermann, J., Brähler, E., Euler, S., Wendt, L., & Müller, S. (2020). Die deutsche Version des Reflective Functioning Questionnaire (RFQ): Eine teststatistische Überprüfung in der Allgemeinbevölkerung. PPmP - Psychotherapie · Psychosomatik · Medizinische Psychologie, 71, 124–131. https://doi.org/10.1055/a-1234-6317
Tyrer, P., Yang, M., Tyrer, H., & Crawford, M. (2021). Is social function a good proxy measure of personality disorder? Personality and Mental Health, 15(4), 261–272. https://doi.org/10.1002/pmh.1513
Vachon, D. D., & Lynam, D. R. (2016). Fixing the Problem With Empathy: Development and Validation of the Affective and Cognitive Measure of Empathy. Assessment, 23(2), 135–149. https://doi.org/10.1177/1073191114567941
White, S. J., Coniston, D., Rogers, R., & Frith, U. (2011). Developing the Frith-Happé animations: A quick and objective test of Theory of Mind for adults with autism. Autism Research, 4(2), 149–154. https://doi.org/10.1002/aur.174