Invalid responding can negatively impact the quality of data collected in self-reported surveys [1]. It occurs when participants answer questions without sufficient effort [2] or attention [3] and has been associated with several factors including distraction [4], fatigue [5], and boredom [1]. Regardless of the underlying causes, during surveys participants can become unmotivated to respond accurately, consistently, or to consider item content or survey instructions [2]. This can result in random data patterns [4] where the participant responds unsystematically and every answer has an equal likelihood of being selected [6]. Non-random response patterns are also possible and may include repeated selection of the same response option (e.g. straightlining) or the use of other discernible patterns (e.g. diagonal lines) [4, 7]. In research, invalid responding may bias validity coefficients, estimated group differences [5], effect size estimates [8], and significance tests [7]. A small amount of invalid responding in a sample (5%) can exaggerate or mute correlations [1], and 10% can create an additional factor in models [1, 9]. It can lower or inflate reliability estimates [1, 9], and impacts internal and external validity [5, 10]. Finally, removing even a small proportion of these cases can improve scale properties [2].
Although there is extensive research on invalid responding, few addictions studies report rates of its occurrence in their samples [6]. This makes it difficult to estimate the overall frequency in addictions research, however, instances as high as 40% have been reported [3, 9]. This is concerning since there is widespread agreement that invalid responding is a serious threat to data quality [1], which can in turn impact research conclusions [7,8,9] and clinical decisions [5, 10]. While it is likely that most invalid responding is unintentional [4], these responses are nonetheless inaccurate, impact the quality of self-report data, and merit greater attention in addictions research.
It is also important to be aware of the risks of invalid responding in the context of Internet research, especially given the recent proliferation of online studies. For instance, one concern relates to the environment in which the participant completes the survey. Since the participant and researcher are separated, the researcher cannot control possible distractions or monitor participant motivation, attention, or engagement during the survey [1, 11]. Another concern is the possibility of fraudulent respondents [12, 13]. For example, individuals may intentionally misreport on surveys in order to ensure their enrollment in the study or may attempt to re-enroll by adjusting their responses to appear to be different people. Likewise, bots (automatic programs) seek out and complete surveys randomly or inconsistently. Although invalid responding is not unique to Internet administered surveys, there are unique contributing factors, which need to be considered by researchers.
Fortunately, a number of research methods have been developed to try to identify this phenomenon. These procedures can be divided into post-hoc data cleaning and direct measures applied during data collection. Post-hoc data cleaning is crucial for screening data for invalid responding. Visually inspecting data sets for anomalies such as univariate outliers, out-of-range data, and missing values can help identify suspicious patterns [3]. Frequency curves and norms data are also useful [3], as are more complicated methods such as, testing for multivariate outliers (Mahalanobis D) [1], and measures of internal consistency (i.e. Goldberg’s psychometric antonyms/synonyms, Jackson’s individual reliability index) [3]. Nonsensical answers can also identify suspicious cases (e.g. length of substance use greater than participant age), as can comparing pairs of answers for inconsistent responses [2]. Finally, response times have been used with some success. Shorter completion times are thought to indicate inattention (i.e. a participant does not read the question or consider their response) [7]. However, to varying degrees, these methods can be labour intensive or difficult to implement [1] depending on the type of scales used, the sample size, and survey length [3].
Post-hoc methods provide various means of screening data sets for invalid responding, however, deleting data post-hoc runs the risk of under powering a sample, altering relationships between variables, or reducing sample diversity [10]. Instead it has been suggested that early detection of invalid responding during data collection might reduce the risk of introducing these other biases [8]. Some more direct procedures have been tested with varying results. Adding bogus items [7] (e.g. “I was born on February 30th”), attention checks [7] (e.g. “To show I am paying attention, I will select ‘All the Time’”), and self-reported measure of diligence [1] are popular methods to assess whether participants are reading questions, following directions, and considering their answers.
The following secondary analysis uses data collected during a randomized controlled trial (RCT) of a brief intervention for risky cannabis use. To mitigate concerns related to invalid responding, select responses provided by the individual during the baseline interview were compared to responses provided during the eligibility screener. Those who failed to meet any one of four consistency checks were screened out (not randomized). The objective of this secondary analysis was to understand why individuals were screened out and investigate any differences compared to those who were screened in to the RCT.