Converting from face-to-face to postal follow-up and its effects on participant retention, response rates and errors: lessons from the EQUAL study in the UK

Abstract

Background

Prospective cohort studies are challenging to deliver, with one of the main difficulties lying in retention of participants. The need to socially distance during the COVID-19 pandemic has added to this challenge. The pre-COVID-19 adaptation of the European Quality (EQUAL) study in the UK to a remote form of follow-up for efficiency provides lessons for those who are considering changing their study design.

Methods

The EQUAL study is an international prospective cohort study of patients ≥65 years of age with advanced chronic kidney disease. Initially, patients were invited to complete a questionnaire (SF-36, Dialysis Symptom Index and Renal Treatment Satisfaction Questionnaire) at research clinics every 3–6 months, known as “traditional follow-up” (TFU). In 2018, all living patients were invited to switch to “efficient follow-up” (EFU), which used an abbreviated questionnaire consisting of SF-12 and Dialysis Symptom Index. These were administered centrally by post. Response rates were calculated using returned questionnaires as a proportion of surviving invitees, and error rates presented as the average percentage of unanswered questions or unclear answers, of total questions in returned questionnaires. Response and error rates were calculated 6-monthly in TFU to allow comparisons with EFU.

Results

Of the 504 patients initially recruited, 236 were still alive at the time of conversion to EFU; 111 of these (47%) consented to the change in follow-up. In those who consented, median TFU was 34 months, ranging from 0 to 42 months. Their response rates fell steadily from 88% (98/111) at month 0 of TFU, to 20% (3/15) at month 42. The response rate for the first EFU questionnaire was 60% (59/99) of those alive from TFU. With this improvement in response rates, the first EFU also lowered errors to baseline levels seen in early follow-up, after having almost trebled throughout traditional follow-up.

Conclusions

Overall, this study demonstrates that administration of shorter follow-up questionnaires by post rather than in person does not negatively impact patient response or error rates. These results may be reassuring for researchers who are trying to limit face-to-face contact with patients during the COVID-19 pandemic.

Background

Retention of participants is acknowledged as a major challenge in longitudinal follow-up studies, and attrition is more likely amongst older participants and those in poorer health [1]. Observational studies and randomised controlled trials (RCTs) suffered further disruption to recruitment, retention and follow-up during the COVID-19 pandemic, with delays to study reporting [2]. Meanwhile, studies employing follow-up approaches compatible with social-distancing guidelines were able to continue data collection. Here, we reflect on transition from a conventional face-to-face approach to one employing postal follow-up, within the UK arm of the EQUAL study (European QUALity Study). This change was made to improve study efficiency through reduced costs and use of resources, while also reducing patient burden. Not only was this approach found to improve falling response rates, but in retrospect it also provided a valuable lens into maintaining follow-up during the pandemic.

EQUAL is a prospective observational study which has recruited people aged over 65 with advanced chronic kidney disease (CKD; with eGFR ≤20 mL/min/1.73m²) from 2012 to present [3]. The study initially involved research-staff administering follow-up until 2017 in the UK, but a number of issues including slower than anticipated recruitment and progression to end-stage in the main study necessitated an extension and in 2018, all surviving UK participants were invited to switch from traditional follow-up (TFU) to efficient follow-up (EFU). This employed an abbreviated questionnaire, administered by post with linkage to the UK Renal Registry, which captures survival and quality assurance data for all individuals with CKD and receiving renal replacement therapy in the UK. The analyses presented here aimed to describe the impact of a switch to remote patient questionnaire follow-up on response rates and error rates in the EQUAL Study.

Methods

Study design and study population

The EQUAL study developed a prospective cohort of older people with advanced CKD in Germany, Italy, Poland, Sweden, the Netherlands and the UK. The main research question addresses when dialysis should ideally be initiated in elderly participants with CKD, based on uraemic signs and symptoms, and quality of life [4, 5]. Approval was obtained from the Medical Ethical Committees of the national co-ordinating centres and institutional review boards of the participating centres. Written informed consent was obtained from all participants.

Participants ≥65 years of age were included if their eGFR, as estimated by the Modification of Diet in Renal Disease equation, [6] had dropped for the first time to ≤20 mL/min/1.73m² during the previous 6 months. Participants were excluded if the drop in eGFR represented an acute kidney injury, or if they had previously received any form of renal replacement therapy. Recruitment began in 2012, with more than 1700 participants recruited by November 2020. Data on all 506 UK recruits are presented here.

Data collection

Clinical and demographic data were obtained from medical records and entered in a web-based clinical record form. Baseline data included ethnicity, primary renal disease, co-morbidities and eGFR. A weighted comorbidity score was calculated using the Charlson Comorbidity Index [7]. Physical examination for body mass index (BMI) and subjective global assessment was performed [8].

Until 2018, follow-up was conducted 3–6 monthly (depending on the level of kidney function), wherein staff assisted participants in completing questionnaires collecting symptoms of kidney disease (Dialysis Symptom Index [DSI]) [9], and quality of life (Short-Form 36 [SF-36]) [10]. DSI is available in the public domain; SF-36 was developed by the Research and Development Corporation (RAND) for the Medical Outcomes Study, and is available in the public domain [10]. During this “traditional” follow-up (TFU), other questionnaires were administered to participants including the Renal Treatment Satisfaction Questionnaire (RTSQ) [11], Illness Perception Questionnaire (IPQ) [12] and a questionnaire on Decision-Making in kidney disease (DM) which was developed by the EQUAL investigators. A licence was provided from Health Psychology Research Limited for RTSQ, and email permission was provided from the author of IPQ. The Decision-Making in kidney disease questionnaire is included as a supplementary file (Additional File 1).

In September 2018, the EQUAL protocol was amended in the UK to allow linkage to routine healthcare databases and administration of postal questionnaires (EFU). Local research teams contacted surviving participants to seek their consent to take part, with ongoing follow-up administered centrally through the UK Renal Registry. The first round of EFU questionnaires was administered in October 2019. The original questionnaire was amended to contain the SF-12 instead of SF-36, with 12 of the original 36 questions covering all validated indicators of quality of life: physical functioning, physical role, emotional role, bodily pain, social functioning, mental health, vitality, and general health [13]. A licence from QualityMetric Incorporated, LLC was provided for use of the SF-12 questionnaire. In order to shorten the EFU questionnaire, the RTSQ, IPQ and DM questionnaires were not included. Participants were asked to record the date on which they completed the questionnaire, and whether they had assistance from family or healthcare staff. The original questionnaire in TFU included 102 questions and 11 pages; the abbreviated EFU questionnaire included 80 questions and 8 pages. A freepost envelope was included for return of the questionnaire to the UK Renal Registry.

Data analysis

Six-monthly traditional (for all 504 initial participants) and subsequent efficient follow-up were analysed and compared. Due to the reconsenting process, EFU was viewed as a new study group and hence non-consenters were not included in the outcome statistics. To facilitate comparison, only the DSI and SF-36 (in TFU) and SF-12 (in EFU) questionnaires are presented. Throughout follow-up, response rates are calculated using returned questionnaires as a proportion of surviving invitees – deceased participants were censored. Baseline clinical and demographic data are reported as mean values with standard deviations for normally distributed continuous variables, median values with interquartile ranges (IQRs) for not normally distributed data, and as proportions for categorical variables.

Errors in EFU responses were coded under the following categories: a missing answer; missed double-page spread of questions; duplication of answers; and crossing answers out. Duplication of answers in the DSI was further clarified into (1) answering both “no” and “yes” for whether they experience the symptom (2); answering “no” alongside how much the symptom bothers them (3); answering ≥2 quantifiers for how much the symptom bothers them. Crossing answers out was further delineated into whether the participant has corrected the cross-out with another answer; or whether they left the crossed-out response uncorrected. Examples of these errors are given in Fig. 1. Although error rates were counted for both follow-up periods, this categorisation was not conducted for questionnaires completed during traditional follow-up. Error rates are presented as the average percentage of unanswered questions or unclear answers, of questions per returned questionnaire (unreturned questionnaires were censored for this calculation).

Fig. 1

Examples of errors counted in the Dialysis Symptom Index.

In the first three questions duplications are shown, with (1) answering both “no” and “yes” simultaneously, (2) answering “no” along with a quantifier for how much the symptom bothers them, and (3) answering two quantifiers together, respectively. The latter two questions show errors related to crossing answers out, with the first example showing a corrected error, and the last example showing an error crossed out but no correction

A comparison was also made for six-monthly TFU between all 504 initial participants and the 236 participants who were alive and invited to participate in EFU, regarding both response and error rates.

Results

Characteristics of the study cohort

Of the 506 UK EQUAL participants, 312 (62%) were male and the median age was 76.7 (inter-quartile range [IQR] 70.8–81.7) (Table 1). The majority were white (459, 91%), and the median eGFR was 18.7mls/min/1.73 m2 (IQR 16.5–19.9). A minority (34, 7%) had a university degree, with the majority attaining primary school education only (154, 30%), and 137 (27%) attaining a secondary school education. Most were married (205, 41%). The mean Charlson comorbidity index was 7.0 (standard deviation [SD] 1.8). Just 30 participants had no major co-morbidity listed and co-morbidity data were missing for 15 participants (3%). The most frequent co-morbidities were hypertension, diabetes and malignancy (76, 38 and 21% of individuals respectively). After CKD of unknown aetiology, diabetic nephropathy secondary to type 2 diabetes mellitus was the most common primary renal disease (87, 17.2%).

Table 1 Demographic and clinical characteristics of the study cohort

Of the 125 people who did not consent to EFU, a higher proportion than in the consenting group were of Black African ethnicity (8% Vs. 2%), and more were widowed (21% Vs. 15%). The majority of participants who did not consent to EFU had a primary renal disease (PRD) diagnosis of diabetes (20%), whereas the majority of those consenting to EFU had an unknown/missing primary renal disease (19%).

The 41 participants who consented to EFU but did not respond were older (77.0 [IQR 70.5–81.1] Vs. 73.2 [IQR 68.2–79.7]), a greater proportion were male (71% Vs. 58%) and had lower levels of post-primary education (36% Vs. 53%) than responders (Table 1).

Response rates during follow-up

Of the 504 original UK recruits, 236 were alive and participating in the study on September 1st 2018 and 111 consented to EFU. An additional 2 participants joined the study in 2018 and consented to EFU, totalling 113 participants consenting to EFU out of 238 invited (48%). 125 did not consent to continuing with the EQUAL study in EFU. Eleven consenting individuals died, and one withdrew before the first EFU questionnaire was administered in October 2019. Of the 101 recipients, 60 responded and returned a questionnaire (59%) (Fig. 2).

Fig. 2

Flowchart showing status of participants throughout EQUAL study.

Traditional follow-up – all participants (inclusive of consenters and non-consenters to subsequent efficient follow-up)

As participants were serially recruited between 2013 and 2017, the duration of TFU ranged from 0 to 48 months with a median TFU for the 504 participants of 26.8 months. Four participants died or withdrew after consenting, before returning their first questionnaire, leaving 500 patients to respond at month 0.

Response rates were calculated at each time point for people who were alive and had not finished follow-up or withdrawn from the study. Response rates throughout TFU for the 504 participants are shown in Fig. 3, with the status of all 504 participants shown at each timepoint, meaning the response rates at each timepoint are directly comparable. Response rates fell gradually every 6 months from 360/500 (72%) at baseline to 52/208 (25%) at 30 months. For those who were recruited earlier and hence took part in follow-up for longer, response rates continued to decline steadily to 0/2 (0%) at 48 months.

Fig. 3

Status of all 504 participants across traditional follow-up.

Status includes those who respond or do not respond to each questionnaire, and reasons for withdrawal. “Withdrawal from study” includes patient choice, being discharged from the renal clinic to their GP, moving to a renal centre not involved in the EQUAL study, or receiving a transplant. “Patient not available for follow-up” indicates how long patients have been followed up upon reaching conclusion of traditional follow-up in 2017. Percentages above each bar indicate response rates for participants still in the study at that timepoint

Follow-up for participants consenting to efficient follow-up (from traditional follow-up through to efficient follow-up)

As described above, in 2018 the remaining EQUAL participants were invited to consent to EFU, with 111 TFU participants opting to continue in the study. EFU was on average 29 months after the participants’ last TFU visit. Of the 111 TFU participants consenting to EFU, median TFU was 34 months, ranging from 0 to 42 months.

Focussing on TFU response rates for the subset of individuals who consented to EFU, response rates gradually decreased from 98/111 (88%) at baseline, to 23/76 (30%) at 30 months, which was the median end of this cohort’s TFU. Of participants who were recruited earlier to the study and therefore had longer TFU, response rates fell further to 3/15 (20%) at 42 months of traditional follow-up. As two individuals joined the study at the point of converting to EFU in 2018, 113 participants consented to EFU in total. Eleven of these individuals died and one withdrew prior to administration of the questionnaire to the remaining 101 participants. Figure 4 shows the response rates at each timepoint of TFU through to EFU for the 113 participants who consented to EFU. The status of all 113 participants is shown at each timepoint, making the response rates at each timepoint directly comparable. The introduction of the EFU questionnaire appears to boost response rates in this cohort from 30% at median end of TFU (30 months), to 59% (60/101) (Fig. 4).

Fig. 4

Status of 113 participants throughout TFU and EFU for those who consented to EFU.

“Patient not available for follow-up” indicates how long patients have been followed up upon reaching conclusion of TFU in 2017, and subsequently joining EFU in this cohort. Percentages above each bar indicate response rates for participants still in the study at that timepoint

To investigate whether patients who consented to EFU were better responders throughout or not, comparisons were made for their responses throughout TFU. Response rates throughout TFU were consistently higher in those who responded to the EFU questionnaire, compared with those who did not (Table 2).

Table 2 Comparison of response rates throughout follow-up based on response or non-response to EFU

Error rates during follow-up

Figure 5 is stratified to show sub-groups of responders and how their error rates reduced after introduction of EFU. In TFU across all 504 participants, error rates increased from 7% at baseline through to 18% at month 42. Error rates in the subset who consented to EFU were comparable to all 504 original participants, rising from 5% at baseline to 13% at month 42 (Fig. 5 and Additional file 2).

Fig. 5

Graph comparing error rates throughout follow-up between groups of EQUAL participants.

Error rates are the average percentage of unanswered questions or unclear answers, of questions per returned questionnaire (unreturned questionnaires were censored)

Most participants, 51/60 (85%), filled in the EFU questionnaire without assistance, with the remaining nine seeking help from a family member. Nevertheless, at introduction of EFU, errors were comparable with early TFU at 7%, and did not differ substantially between those who had help (5%) compared with those who completed the questionnaire alone (7%). From month 18 onwards, the participants who responded to EFU had the lowest error rates, with highest error rates in those who did not consent to EFU (Fig. 5). Age and duration of follow-up did not appear to have a significant bearing on the number of errors.

Categories of errors in the EFU questionnaire

The total number of errors made across all 60 questionnaires was 322. The most common error was leaving a single question blank (144/322, 45%). This was followed by errors specific to the DSI questionnaire, where when asked whether they have experienced a particular symptom in the past month, participants answered “no” as well as how much the symptom bothers them, which is only meant to be used for clarifying when answering “yes”, accounting for 23% (74/322) of all errors made. In 20% of errors (65/322), participants failed to complete whole double-page spreads of questions. Other categories of error were less frequent, including four cases where participants wrote or explained that questions regarding sexual function were not applicable (“N/A”) to them (Table 3).

Table 3 Categories of errors made in the EFU questionnaire

No significant difference was seen in TFU response and error rates between the 504 initial participants and the 236 participants who were alive and invited to consent to EFU.

Discussion

Our work demonstrates that the transition from conventional face-to-face research follow-up to postal follow-up resulted in increased response rates and reduced error rates, in an older cohort of people with advanced CKD. Despite falling response rates and rising error rates, a new focus on the follow-up, even remote, postal follow-up, can achieve response rates and error rates similar to baseline levels. Our findings are timely, as the global COVID-19 pandemic has meant that research studies with traditional clinic-based follow-up have had to pause in the interest of safety, whereas those with remote follow-up have succeeded in continuing with their research [2]. Investigators who are considering altering their study follow-up should be reassured by our results, which illustrate that a change to postal follow-up during an established observational study is feasible and may improve response rates and error rates.

Follow-up fatigue” and attrition are a common problem in research studies, with attrition rates varying from 5 to 70% [14]. Indeed, we observed this attrition during TFU in EQUAL, with participant response rates decreasing from 72 to 25% at 30 months. Maintaining a good participant response rate throughout study follow-up is important but difficult to achieve. There are many potential reasons why people may not complete a research study, including the perceived time burden, feeling under-valued by the researcher, and not being aware of how they are contributing and helping the medical community [15, 16]. The time burden associated with research may be a particularly significant factor for people with advanced CKD, whose clinical care requires frequent interactions with health care professionals for dialysis education, dialysis access investigations and procedures, alongside other speciality appointments to manage their co-morbidities. In addition, older, multi-morbid patients may be dependent on others to help them attend hospital appointments, making face-to-face follow-up less achievable [17].

The improved questionnaire response rates that we observed in EFU may be due to the shorter questionnaire, a preference for remote follow-up to avoid the burden of clinic visits, or due to the novelty of a change in follow-up. In a trial which directly compared different questionnaire administration methods in a cohort of older people, postal follow-up had higher response rates than face-to-face questionnaire administration approaches [18]. Also, research studies with older participants should not discount the possibility of using digital follow-up. In a German study of participant-reported outcomes in breast cancer, when asked about their preferences between digital or paper-based follow-up, a greater proportion of women from all age-groups preferred digital, including 87% of women aged 70–80 [19]. Indeed, studies which collected follow-up data in mobile phone apps found that the older people were the most highly engaged respondents [20, 21]. These findings may be contrary to the perceived stereotypes of older people.

A key problem with attrition in observational studies is that it can introduce bias, as participants who drop out from studies tend to differ from those who continue to participate [22]. People are more likely to withdraw from longitudinal studies if they are older, cognitively impaired, living alone and not married, have a lower socio-economic status or level of education, and/or are less socially active [22, 23]. Furthermore, people with deteriorating health or high symptom burden may be less likely to complete follow-up, thus skewing the final results [24, 25]. We noted that out of those who did not respond to EFU questionnaires, a greater proportion were older, co-morbid males, with lower educational levels. The disparity between males and females has been identified in many research studies, whereby women appear more willing to be involved and continue to respond [20, 26, 27]. The relationship between non-response and educational level is less well established; in contrast to our results, having higher levels of education have been found to correspond to non-response in some longitudinal studies [28]. Although the transition to EFU in our study improved response rates and so should reduce attrition bias, the change required all participants to be re-consented. It is widely recognised that people who consent to participate in research are different to those who do not [29,30,31]. Indeed, our results show that a greater proportion of people who consented to EFU were white and married when compared to those who did not consent. Therefore, with only 47% of people consenting to EFU, this additional re-consent step should be acknowledged as another stage in which bias could be introduced into the study and affect the external validity of the results.

“Follow-up fatigue” was also demonstrated by a consistent increase in questionnaire error rate in all participants during TFU, regardless of whether they consented or responded to EFU. We observed that introducing EFU halved the error rate and that the majority of participants completed the EFU questionnaire without assistance. This was an unexpected finding as we had anticipated that our older multimorbid cohort may have relied on the research nurse to guide them through the questionnaire, however this finding is consistent with other studies [18]. A plausible explanation is that participants feel more relaxed and under less time pressure when they complete the questionnaire at home and therefore make fewer errors. The most common error in the EFU questionnaire was leaving a question blank, and the beginning of the SF-12 questionnaire had the highest error rate. This is worth considering when designing remote questionnaires, to perhaps include more guidance for participants at the start of the questionnaire.

Our work has demonstrated that a pragmatic change in follow-up did not negatively impact questionnaire response or error rate, instead it appears to have had a positive effect. The main limitation of our work is that the questionnaire used in EFU included the SF-12, rather than the SF-36 which was used during TFU. Although the SF-12 is still a validated questionnaire which includes the 12 questions to cover all quality of life indicators, it is possible that the improvement which we observed in response rates may be due to the administration of a shorter questionnaire [32, 33]. Furthermore, the EQUAL study in the UK was converted to postal follow-up for efficiency and to facilitate longer follow-up without the availability of research nurses, rather than to permit direct comparisons between traditional clinic-based and remote forms of follow-up. As such, our work does not include any qualitative data on the participants’ opinion of the change in follow-up. In order to fully evaluate this, bespoke work would be necessary to compare different forms of follow-up, ideally including digital follow-up approaches. Finally, in our work we did not include a cost analysis, which is another key factor to consider when planning the follow-up strategy in a research study.

Conclusions

In conclusion, in this well-designed longitudinal cohort study of older participants with advanced CKD, response rates fell and error rates rose throughout clinic-based follow-up. With the introduction of a shorter postal questionnaire, response and error rates improved to levels resembling early follow-up in the study, and some participants responded despite not engaging towards the end of traditional follow-up. This suggests that even in older people with advanced CKD, altering the follow-up approach to post is acceptable and may provide more complete data than traditional follow-up. This is acutely relevant in this period of limited contact in the COVID-19 pandemic, as it provides hope that investigators can continue their follow-up studies remotely without sacrificing participant retention or accuracy of their responses.