Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments

Eligibility criteria

Authors of studies included in a published systematic review on risk factors for severe outcomes in patients with influenza and published between 2008 and 2011 were included [7]. More remote publications were not considered, as the contact information was more likely no longer accurate and authors may not recall details of their studies.

Quality assessment: NOS and NOS-derived survey

The NOS assigns up to a maximum of nine points for the least risk of bias in three domains: 1) selection of study groups (four points); 2) comparability of groups (two points); and 3) ascertainment of exposure and outcomes (three points) for case–control and cohort studies, respectively (Additional file 1) [2]. Survey questions were developed based on the NOS’ questions covering all three domains so that authors could provide detailed information about their studies. Because the NOS questionnaires are slightly different between case–control and cohort studies, two surveys were created.

Vaccination status for influenza and antiviral treatment were defined as the most important covariates that defined comparability, consistent with the NOS assessment conducted by the reviewers in the systematic review [7].

The corresponding author and the first author were contacted by email and were provided with the link to the actual electronic survey. An online survey was provided to all authors where their name, responses and timestamp were securely documented. Authors were required to respond to all questions provided prior to submission and answered follow-up questions only when prompted to. An example of a follow-up question: For ascertainment of exposure, authors who claimed using structured interviews had to also report whether the interviewers were blinded to exposure status. A feedback form was provided at the end of the survey for authors to comment on the survey. A total of two reminders were sent to authors not completing the survey.

Analysis

Questions in the survey were matched to the original NOS items, so that points were awarded to the respective answers.

Differences in the NOS scores between the reviewers’ and authors’ assessment were compared with the Wilcoxon paired sign-rank test.

Inter-rater agreement Cohen’s kappa was used to compare agreement between the reviewers from the systematic review and the authors’ assessments. The scores were compared between raters across each NOS domain or item. Weighted kappa was performed for variables with more than two classifications (i.e., comparability had scores of 0, 1, and 2). A weighted kappa was also performed for the total NOS scores.

Agreement interpretation was based on established categorizations: poor (K < 0.00), slight (0.00-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), and almost perfect (0.81-1.00) [8].

MedCalc® was used for all weighted/non-weighted kappa analyses [9, 10] and for the Wilcoxon paired sign-rank test. Mean and standard deviation (SD), median and interquartile range (IQR) were reported.

Inter-rater reliability

The developers of the NOS evaluated the tool for face and criterion validity by comparing it with an established scale by Downs and Black [11]. Ten cohort and 10 case–control studies about the association between breast cancer and hormone replacement therapy were examined. Criterion validity showed strong and moderate agreement for cohort (intra-class correlation coefficient [ICC] = 0.88) and case–control studies (ICC = 0.62), respectively. Inter-rater reliability was also high for both cohort (ICC = 0.94) and case–control studies (ICC = 0.82).

To our knowledge, only two other studies have examined inter-rater reliability of NOS other than those conducted by the developers of the tool [3, 4]. Contrary to findings from developers of the tool, other studies have found overall low inter-rater reliability. In a study by Hartling [3], two reviewers applied NOS to 131 cohort studies included in eight meta-analyses on different medical topics. Inter-rater reliability between reviewers ranged from poor (K = −0.06, 95% CI = −0.20, 0.07) to substantial (K = 0.68, 95% CI = 0.47, 0.89), although eight out of nine of the NOS items had K values <0.50 [3]. Reliability for overall NOS score was fair (K = 0.29, 95% CI = 0.10, 0.47). A similar lack of agreement was found in a study applying the NOS to observational studies on cognitive impairment following electroconvulsive therapy for major depressive disorders [4]. Using inexperienced student raters, inter-rater reliability for cohort studies ranged from K = −0.14 (95% CI = −0.28, 0.00) to K = 0.39 (95% CI = −0.02, 0.81), and for case–control studies from K = −0.20 (95% CI = −0.49, 0.09) to K = 1.00 (95% CI = 1.00, 1.00). All nine of the NOS items and seven out of nine items had K values <0.50 for cohort and case–control studies, respectively. However, the findings of the latter study may also explain to some extent the disagreement between methodically trained reviewers and authors of study: In our study, the disagreement between raters may also have been attributable to the authors’ lack of experience applying the NOS.

Limitations

One limitation of our study was the relatively small sample size of 65 cohort studies. With a response rate of 36%, selection bias may be present as authors who responded to the survey might have a different level of expertise and interest in research methodology than non-responders. Also, all studies included were from one field of medicine. Notably however, the reviewer assessments of the risk of bias were similar for the included and excluded studies. It remains unclear whether the difference between the reviewer and author’s assessments were due to different information available to them for assessment, or whether authors’ lack of familiarity with the NOS tool resulted in this finding. Given that the one item (i.e., representativeness of the exposed cohort) that would not need to be downgraded based on lack of information was similarly low in agreement as items that would expect downgrading for lack of reporting, subjective interpretation was probably the primary driver resulting in the low agreement between reviewers and authors. This would emphasize, as suggested by Hartling [3] and Oremus [4], that training and detailed guidance is needed in order to appropriately apply the NOS. Although the survey explained in detail how the assessment should be conducted, lack of familiarity with the NOS tool by the authors may have negatively affected their performance.

Another limitation is the nature of the inter-rater reliability test, that is, some have considered kappa to be an overly conservative measure of agreement [12]. Since the kappa value depends on the observed categories’ frequencies, the test might underestimate agreement for a category. An example was the item selection of the non-exposed cohort, where 82% of the time both reviewers and authors agreed that the non-exposed were derived from the same population as the exposed (i.e., one point assigned). However, there were no occurrences where both raters agreed that the non-exposed were not derived from the same population as the exposed, which resulted in an underestimation of the agreement. A larger sample of studies may have circumvented this problem.