The Use of Control Charts to Monitor and Manage Survey Data Collection Process from the Perspective of Measurement Errors

Survey quality is a complex concept encompassing three levels: survey product, survey process, and survey organization. The quality of survey data (product level) can be assessed by using various types of survey errors. In this study, we concentrate on measurement errors (product level) and the corresponding stage of the survey process where measurement errors occur: the data collection process (process level). In general the objective is to evaluate the data collection process quality from the perspective of measurement errors. To this end, we consider data that are created during the data collection process and that are also informative about the data collection process quality. In order to analyze these data, depending on their nature (univariate or multivariate) and measurement level (numerical or categorical), we explore the use of control charts combined with different statistical tools. 

We start with a widely used type of paradata--interview duration, and evaluate the use of univariate control charts from the commencement of survey data collection to monitor the distribution of interview duration (Chapter 2). By simulating the real conditions of an ongoing data collection process, this study investigates whether the results regarding the identification of interviews that are not ``in control'' (outliers) are reliable. We then move on to consider multiple numerical response quality indicators, which are calculated based on various characteristics of response behavior (e.g., straightlining). For simultaneously monitoring the quality indicators and identifying outliers, we present a procedure integrating kernel density estimation with the traditional Hotelling $T^2$ control chart (Chapter 3). Next we take a step further by also including categorical response quality indicators. To deal with the mixed data simultaneously, a procedure integrating a multivariate statistical tool--principal component analysis mix--and bootstrap resampling with the Hotelling $T^2$ chart (Chapter 4) is presented. We show how to identify and interpret an outlier, how to obtain the in-control data, and how the outliers differ from the in-control interviews. As a follow-up to this study, we further evaluate the performance of the procedure by using simulation studies (Chapter 5). The last part of this thesis turns to the role of survey organizations in the three-level concept of survey quality. We discuss the design and use of dashboards as a communication and information tool at the survey organization level to help identify potentially problematic cases (interviews and interviewers), and facilitate decisions about interventions (Chapter 6).

The results presented in this thesis highlight the possibilities and positive effects of using control charts to monitor and manage the data collection in a survey project in order to improve the quality of the data collection process and, ultimately, the quality of the survey data.