Current issues in protocol registration for systematic review and meta-analysis studies: a survey of global researchers


 Background Although protocol registration of systematic reviews/meta-analysis (SR/MA) is still not mandatory, authors are strongly suggested to publish their SR/MA protocols prior to submitting their manuscripts for publication as recommended by the Cochrane guidelines for conducting SR/MAs. We aimed to assess awareness, obstacles, and opinions of SR/MA authors about the protocol registration process.Methods A cross-sectional survey study included all authors who published SR/MAs during the period from 2010 to 2016 were contacted for participation in our survey study. They were identified through the literature search of SR/MAs in Scopus database. Upon receiving their approval to join our study, an online questionnaire was sent via e-mail to each participant.Results A total of 270 authors' responses were complete and were included in the final analysis. Our results showed that PROSPERO was the most commonly used database for protocol registration (71.3%). The registration-to-acceptance interval in PROSPERO was less than one month (99.1%). Almost half of the authors (44.2%) did not register their protocols prior to publishing their SR/MAs and their lack of knowledge of its importance was the most commonly reported reason (44.9%). A significant proportion of respondents (37.4%) believed that people would steal their ideas from protocol databases, while only 5.3% reported that their SR/MA had been stolen. However, the majority (72.9%) of participants agreed that protocol registries have a role in preventing unnecessary duplication of reviews. Finally, 37.4% of participants agreed that SR/MA protocol registration should be mandatory.Conclusion Lack of knowledge about SR/MA protocol registration process seems to be the primary reason for not registering such protocols. Therefore, interventions for increasing awareness about the benefits of protocol registration among researchers would be welcome.

Results A total of 270 authors' responses were complete and were included in the final analysis. Our results showed that PROSPERO was the most commonly used database for protocol registration (71.3%). The registration-to-acceptance interval in PROSPERO was less than one month (99.1%).
Almost half of the authors (44.2%) did not register their protocols prior to publishing their SR/MAs and their lack of knowledge of its importance was the most commonly reported reason (44.9%). A significant proportion of respondents (37.4%) believed that people would steal their ideas from protocol databases, while only 5.3% reported that their SR/MA had been stolen. However, the majority (72.9%) of participants agreed that protocol registries have a role in preventing unnecessary duplication of reviews. Finally, 37.4% of participants agreed that SR/MA protocol registration should be mandatory.
Conclusion Lack of knowledge about SR/MA protocol registration process seems to be the primary reason for not registering such protocols. Therefore, interventions for increasing awareness about the benefits of protocol registration among researchers would be welcome.

Background
Systematic review studies are a subtype of literature reviews that aim to answer a defined research question by collecting and critically analyzing all empirical evidence published as primary research studies.(1, 2) Systematic reviews and meta-analyses (SR/MAs) are considered the highest level of evidence in medicine. (1) Recently, there has been an exponential increase in the number of published SR/MAs as compared to primary studies. (3) This trend is defensible giving that SR/MAs may be easier to perform, may require lower funding, and may be less time-consuming than comply primary research. (4) Along with this exponential growth, there is now an increased concern regarding "duplicate" or "overlapping" SR/MAs, where many SR/MAs addressing the same research question are being published by different authors, over a short period of time (5)(6)(7), and not necessarily reaching the same conclusions or including the same studies. (8) Although replication of prior research does not need to be negative, unnecessary and redundant duplication of studies mandate researchers to carefully review existing SR/MAs, and to identify the added value of any new work as an initial step. (9,10) However, overlapping studies are sometimes produced because researchers may not be aware of other ongoing, but still unpublished SR/MAs. Therefore, there are efforts to support the registration of SR/MA study protocols in publicly-available databases to allow researchers to identify existing protocols of ongoing reviews. (11,12) There are many register databases available for researchers to register their protocols in, as PROSPERO University of York, Cochrane Database of Systematic Reviews, Campbell Collaboration, but the most known between researchers is PROSPERO. In February 2011, the Centre for Reviews and Dissemination (CRD) at the University of York established PROSPERO -an international database for registering SR/MA protocols -that aims to promote high methodological standards, ensure transparency of review process, and reduce undesirable duplication.(13) PROSPERO has received wide acceptance among the research community, resulting in its continuous expansion.(13) PROSPERO registrations are increasing rapidly especially in the period 2011-2017 and expected this expansion will reach over 30,000 registrations by the end of 2017. (14) Although registering the protocols of SR/MAs is not mandated by medical journal editors, such as registration of clinical trial protocols, registration of SR protocols is now recommended by the National Institute for Health Research (NIHR). (12,(15)(16)(17) However, according to a recent study, the registration rate of SRs' protocols is still low (only 21%). (17) Meanwhile, an investigation conducted in 2019 revealed that the rate of registering the protocols of "dose-response" meta-analyses (DRMAs) was even lower, reaching a prevalence rate of 8.51% from a total of 529 investigated DRMAs. (18) This may partially be attributed to the fact that PROSPERO is an open-access register, which may raise the fear about the possibility of idea theft whereas somebody else reproduces original ideas from the published protocols.(13) Therefore, we conducted the current investigation to examine the knowledge and attitude of systematic reviews-publishing authors about the process of registering their protocols in PROSPERO. We also asked recruited participants about their perception of current solutions to issues related to the usage of PROSPERO as well as their recommendations for better protocol registries. To the best of our knowledge, there are no currently published studies in this regard so this will be the first study to address this gap in the literature.

Participants
Participants in this study were corresponding authors who had published SR/MA. Potential participants were identified via two steps. In the first step, we conducted a search for SR/MAs published in Scopus between 2010 -2016 to collect information about SR/MA authors. The search was performed in Scopus using the search term: (("systematic review" OR "systematic literature review" OR "metaanalysis") OR (Cochrane Database Syst Rev)), without any restriction on language, topic, and country of SR/MAs. Scopus restricts the number of references that can be obtained to a maximum of 2,000 in each year and our aim was to target at least 10,000 authors. So, we collected 14000 SR/MA manuscripts from seven-year periods (2010 -2016) according to our criteria. To make sure that we will reach that target number, we searched for SR/MAs published in the seven-year period. In the second step, from the retrieved search results, we collected 10,469 SR/MA manuscripts for different author emails and extracted data about their corresponding authors. From each of the 10,469 SR/MAs, we extracted authors' information such as names, study title, year of publication, author affiliations, and e-mail of corresponding authors using Endnote software to export these details from selected results in Scopus and exclude duplications.

Data collection
Data were collected via an online questionnaire, which was sent to authors via e-mail in three rounds starting from January 2017. After the initial e-mail, we sent two more reminders with a one-month period between each e-mail. We finished receiving e-mails from the participants in April 2017.
Consent for participation was obtained online from all participants included in the study. Failure to obtain such consent was considered one of the exclusion criteria in our study. We recorded the number of potential participants to whom our e-mails could not be delivered. Flow chart of participants' selection process is shown in Figure 1.

Development of the online questionnaire
The online survey was composed of a short section assessing demographic variables of participants, followed by a comprehensive questionnaire designed to gather information about authors' knowledge, opinions, practice, and fear of idea theft related to protocol registration.

Data analysis
Descriptive analysis was carried out using R software version 3.4.1. R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL: http://www.R-project.org/ and packages used are "psych" and "models".

Results
We downloaded 10,469 articles from Scopus and retrieved the e-mails from them technically (not manually). Each article contained one e-mail address. After removing duplicated emails, there were 7,599 e-mails available. Of them, 6,650 e-mails were successfully sent, while delivery for the others failed. We received a response to the survey from 275 authors. However, we excluded from the analysis five authors who responded in the survey that they are not giving a consent to participate, Finally, we included 270 SR/MA responses in our analysis (response rate = 4.0%).

Baseline and demographic characteristics
The majority of authors were men, with a mean age of 45.2 years ( Table 1). Most of the respondents had a research interest in clinical trials, followed by epidemiology, and diagnostic accuracy. On average, they had nine years of experience in SR/MAs. When assessing the highest impact factor (IF) for journals in which the authors managed to publish, the majority managed to publish in journals with IF ranging between 2 -5, followed by those with IF ranging between 5-10.

Protocol registration process
Upon assessing how often protocols were being registered, 44.2% of participants never registered their protocols before starting their SR/MA while only 10.1% reported that they registered all of their protocols. The most commonly reported reasons for not registering protocols were "not knowing that protocols should be registered" (44.9%), followed by "registration is not mandatory" (43%), "not knowing the benefits of registration" (35%), "registration is a time-consuming process" (32.7%), and fear that somebody will steal their ideas (24.3%). The majority of participants (77%) said that SR/MA protocol registration was not required at their institutes.
Meanwhile, PROSPERO and Cochrane Database of Systematic Reviews were the most commonly used databases for publication of the SR/MA protocols (71.3% and 45.3%, respectively). Regarding the registration-to-acceptance interval, 78.6% of authors who applied for registration indicated that their protocols were accepted by PROSPERO within one week of registration and 99.1% of protocols were accepted within one month. Only 1.6% of authors experienced rejection by PROSPERO upon submitting their SR/MA protocols.
Out of those who had registered their protocols, about 80% eventually managed to publish their manuscripts. The mean registration-to-submission interval was 11.1 ± 8.1 months. The two primary reasons for not publishing the registered SR/MA were (i) not completing the ongoing projects and (ii) not reaching a favorable conclusion. The details of the authors' responses to questions related to protocol registration are shown in Table 2.

Authors' opinions on registration protocol of SR/MA
A total of 76.7% of participants indicated that they believed that protocols improved the transparency, and 66.4% of them indicated that registration improved the quality of the SR/MAs. About 72.9% of the responders believed that protocol registration helped in avoiding unnecessary duplication of the reviews. However, 67.5% agreed that it was useful to have more than one SR/MA addressing the same research question to see whether the results would be consistent. Only 37.4% of participants agreed with making protocol registration mandatory for all SR/MAs and 49.2% of participants agreed that registered SR/MAs should have a priority in the publication process ( Figure   2).

Authors' opinions and suggestions to avoid duplications and idea theft
Among the responders, 37.4% believed that people were using the open-access databases that publish protocols to steal ideas of other researchers, 47.8% of them indicated that they have considered that the idea presented in their protocols could be stolen and 5.3% reported they personally experienced a situation where others stole ideas from their publicly registered protocol (Table 3).

Discussion
Our results show insufficient knowledge among SR authors about the importance of protocol registration, which was directly reflected in their practice and opinions. About half of the participants did not know that protocols should be registered, which was the major reason for not registering protocols. The authors' insufficient awareness about the importance of SR protocol registration could indicate a lack of appropriate and clear declaration about the importance of these databases in scientific fields. However, researchers should always be updated and eager to know more about the methodology of their study. Other reported reasons for not registering a SR protocol were: registration is not mandatory, carries no benefits, the process is time-consuming, and the fear of idea theft.
In this context, the myth that registration has no benefits or is time-consuming has already been debunked. (19) Before conducting any SR/MA, researchers should scan the field for any ongoing or completed reviews on the same topic. (20,21) However, many of the individual researchers or research groups do not register or publicly publish the protocols of their ongoing studies; further contributing to the problem of duplication. Accordingly, SR protocol registration has actually become an urgent need that has to be addressed by guideline developers and decision-makers. (19) Protocol registration is the way to alert different research groups that a related review is being conducted. This will not only help in preventing duplications, saving time and resources but could also open the gate for collaborative work among researchers with shared interests in a certain topic. (19,22) Protocols may take time to be developed; however, the step of writing the review's protocol is critical in order to make sure that all investigators are on the same page with avoiding discarded efforts or unintended bias. (19,22) Additionally, registration will enhance confidence in the reported results by knowing that the methodology was determined in advance and making sure it was not changed to suit the preference of the authors. (19,22) Furthermore, a positive association between prospective registration and the methodological quality of SRs has been found. (23) The revised assessment of multiple SRs (R-AMSTAR) of registered reviews was higher than that for non-registered ones.(23) This difference was maintained even after a sensitivity analysis was performed by excluding the registration-related item.
Similarly, the total preferred reporting items for SR/MAs (PRISMA) scores of registered reviews were significantly higher. However, this statistical significance faded away after excluding the item related to registration. Regarding the fear of idea theft, nearly half of the participants suggested that it will be safer for them if the details of the ongoing SR/MA were blocked until the completion of their studies. This option is already available in some databases in order to assure authors as well as to limit the possibility of idea theft problem.
On the other hand, nearly 20% of authors in our study have at least one paper that was registered but not published; either for not managing to finish the study or because they did not get results they anticipated. Furthermore, around 1% of our respondents never managed to publish any of their registered protocols. So, is it fair that someone keeps the idea forever for just thinking about it?. (25) A recent study by Tsujimoto et al. reported that around 26% of protocols registered in PROSPERO remained unpublished 5.4 years after registration. (26) Further, they reported that funding for SRs was a determining factor in the publication status of registered SRs. (26,27) As some of the responders in our sample suggested, a possible solution for unpublished papers for registered protocols, is adding an option to allow for direct official contact with authors documented by the database itself. Authors of the ongoing SR/MA may offer a collaboration or even confirm that they will not continue working on this idea. Another possible solution that was suggested by the responders and can be adopted by the databases registering SR protocols is to contact the authors in regular pre-defined intervals to make sure that they are still working on the registered study. If not, the authors can be given an option to either discarded the protocol from the database, to invite other potential authors to continue working on this topic, or to put up a notice that the authors are looking for collaborators that will help them complete the SR.
Some of our participants also suggested that collaboration between different databases registering protocols would be helpful in identifying duplicate ideas, as the same idea might be registered in two different databases by two different author teams. These potentially redundant protocols may pass screening unnoticed and get published, and the authors may find out too late that they have invested their time and efforts to the overlapping study.
The main limitation of this study was related to the low response rate (4.0%), raising the possibility of non-response bias. It is possible that many of the e-mails we sent ended up in the junk/spam folders, or that the e-mail addresses we used were functional but outdated and the authors do not use them anymore, as many e-mails were retrieved from manuscripts published 6 -8 years ago. Our pilot testing indicated that the length of our questionnaire can be completed within a reasonable time, and we can therefore only speculate that the length of the survey did not contribute to the low response rate.
Due to the limited sample size, our results are not necessarily generalizable. (28,29) However, it has to be emphasized that surveys conducted via e-mail generally have lower response rates when compared to in-person surveys (30-33), which is a common problem in all online surveys in general.

Ethics approval and consent to participate
We stated at the last sentence of the sent online cover letter of our questionnaire: "By completing and submitting this survey, you are indicating your consent to participate in the study. Your participation is appreciated." Responses were only included in our analysis if participants gave a 'yes' response to the following question 'Have you read information about the study given in the cover letter and agree to participate in this survey?'

Consent for publication
Not applicable.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.  SD: standard deviation, SR/MA: systematic review/meta-analysis.

(22)
Ever experienced a situation in which another group, who did not register their SR/MA protocol, publish before you have a paper based on a protocol identical to yours?, n=246 16 (6.5) If another group publishes a paper that is identical to your registered protocol before your team gets the publication and you find out their protocol registered after your protocol. What would be your next step?, n=190 Identify the similarity and difference between your review and published ones and keep working on your own review Contact with both authors and editors as they possibly used your ideas   Authors' opinions on registration protocol of systematic review/meta-analysis.

Supplementary Files
This is a list of supplementary files associated with this preprint. Click to download.