Data analytical stability in fMRI research: an application to cluster-wise inference in fMRI

In neuroscience, the localisation of task associated brain activation is an essential topological concern. Via non-invasive techniques, such as fMRI, further theorizing and clinical practice can be guided. Carp (2012) demonstrated the variability present in the method sections in fMRI studies. This methodological variability between studies limits reliability (Bennet & Miller, 2010) and hinders reproducible research. Therefore the evaluation of methods in fMRI should include data-analytical stability measures (e.g. Qiu et al., 2006), which quantify the variability due to choices in the methods. Here, we aim to determine the data-analytical properties of two competing approaches to cluster-wise inference (Gaussian random field and permutation) and evaluate the reproducibility of both methods next to its validity. We focus on the type I error, the data-analytical stability following inferential decisions and additionally explore the distribution of p-values. We investigate differences between both procedures via simulations and end with an illustration using real data.