An evaluation of harvest plots to display results of meta-analyses in overviews of reviews: a cross-sectional study

Background Harvest plots are used to graphically display evidence from complex and diverse studies or results. Overviews of reviews bring together evidence from two or more systematic reviews. Our objective was to determine the feasibility of using harvest plots to depict complex results of overviews of reviews. Methods We conducted a survey of 279 members of Cochrane Child Health to determine their preferences for graphical display of data, and their understanding of data presented in the form of harvest plots. Preferences were rated on a scale of 0–100 (100 most preferred) and tabulated using descriptive statistics. Knowledge and accuracy were assessed by tabulating the number of correctly answered questions for harvest plots and traditional data summary tables; t-tests were used to compare responses between formats. Results 53 individuals from 7 countries completed the survey (19 %): 60 % were females; the majority had an MD (38 %), PhD (47 %), or equivalent. Respondents had published a median of 3 systematic reviews (inter-quartile range 1 to 8). There were few differences between harvest plots and tables in terms of being: well-suited to summarize and display results from meta-analysis (52 vs. 56); easy to understand (53 vs. 51); and, intuitive (49 vs. 44). Harvest plots were considered more aesthetically pleasing (56 vs. 44, p = 0.03). 40 % felt the harvest plots could be used in conjunction with tables to display results from meta-analyses; additionally, 45 % felt the harvest plots could be used with some improvement. There was no statistically significant difference in percentage of knowledge questions answered correctly for harvest plots compared with tables. When considering both types of data display, 21 % of knowledge questions were answered incorrectly. Conclusions Neither harvest plots nor standard summary tables were ranked highly in terms of being easy to understand or intuitive, reflecting that neither format is ideal to summarize the results of meta-analyses in overviews of reviews. Responses to knowledge questions showed some misinterpretation of results of meta-analyses. Reviewers should ensure that messages are clearly articulated and summarized in the text to avoid misinterpretation. Electronic supplementary material The online version of this article (doi:10.1186/s12874-015-0084-0) contains supplementary material, which is available to authorized users.

Should you wish to be notified of the publication results of this study, you may provide your e-mail address at the end of the survey. This information will be saved separately and cannot be associated with your responses to the survey.
At the end of the survey, you will also have the opportunity to enter your name in a draw for an iPad. You will have about a 1 in 200 chance of winning -good luck! Thank you.
Lisa Hartling, PhD Director, Cochrane Child Health Field Associate Professor, Department of Pediatrics University of Alberta

Explanation of Harvest Plots
These are harvest plots of six intervention comparisons for bronchiolitis for two outcomes. The comparisons are: glucocorticoid vs. placebo, bronchodilator vs. placebo, epinephrine vs. placebo, epinephrine vs. bronchodilator, 3% hypertonic saline vs. 0.9% saline, and chest physiotherapy vs. standard care.
The outcomes are clinical score and length of stay. Each row represents the outcomes for the specified comparison. Each 'plot' contains a bar showing the number of participants with the outcome for that comparison.
For example, the plot shows that: a) among approximately 350 participants, there was no difference between bronchodilator compared with placebo on length of stay; b) among over 200 participants, there was no difference between epinephrine compared with placebo on clinical score or length of stay; and c) among approximately 100 participants, chest physiotherapy was better than standard care at reducing clinical score at 1-3 days.
Bar color indicates quality of evidence (based on GRADE): -green for high quality; -yellow for moderate quality; and -red for low quality.
[Inline Image: "bronchiolitis_legend_10Dec2013.JPG"] www.project-redcap.org Please answer these questions with respect to the harvest plots above: What were the results for glucocorticoids vs. placebo There was a significantly longer length of stay (check all that apply): with glucocorticoids compared with placebo There was no difference in clinical score at 1 to 3 days There was a small difference in clinical score at 1 to 3 days There was no significant difference in length of stay between glucocorticoids and placebo What were the results for epinephrine vs. other There was a significant benefit for epinephrine bronchodilators (check all that apply): compared with other bronchodilators in terms of clinical score at 60 minutes and 120 minutes There was a significant benefit for other bronchodilators compared with epinephrine in terms of clinical score at 60 minutes and 120 minutes There was a significantly greater length of stay with epinephrine compared with other bronchodilators There was a significantly greater length of stay with other bronchodilators compared with epinephrine What were the results for 3% hypertonic saline vs. There was a significant difference with improved 0.9% saline (check all that apply): clinical score at 1-3 days for 0.9% saline There was a significant difference with improved clinical score at 1-3 days for 3% hypertonic saline There was a significantly longer stay with 3% hypertonic saline There was a significantly longer stay for 0.9% saline There was not enough evidence to compare 3% hypertonic saline with 0.9% hypertonic saline for length of stay _____________________________________________ I have seen this type of graph before.

Yes No
This type of graph is well-suited to summarize and   Table  This is a table of results for the same six intervention comparisons for bronchiolitis. [Inline Image: "bronchiolitis_table_10Dec2013_no gridlines.JPG"] Please answer these questions with respect to the table above: What were the results for glucocorticoids vs. placebo There was a significantly longer length of stay (check all that apply): with glucocorticoids compared with placebo There was no difference in clinical score at 1 to 3 days There was a small difference in clinical score at 1 to 3 days There was no significant difference in length of stay between glucocorticoids and placebo What were the results for epinephrine vs. other There was a significant benefit for epinephrine bronchodilators (check all that apply): compared with other bronchodilators in terms of clinical score at 60 minutes and 120 minutes There was a significant benefit for other bronchodilators compared with epinephrine in terms of clinical score at 60 minutes and 120 minutes There was a significantly greater length of stay with epinephrine compared with other bronchodilators There was a significantly greater length of stay with other bronchodilators compared with epinephrine What were the results for 3% hypertonic saline vs. There was a significant difference with improved 0.9% saline (check all that apply): clinical score at 1-3 days for 0.9% saline There was a significant difference with improved clinical score at 1-3 days for 3% hypertonic saline There was a significantly longer stay with 3% hypertonic saline There was a significantly longer stay 0.9% saline There was not enough evidence to compare 3% hypertonic saline with 0.9% hypertonic saline for length of stay  Could harvest plots be used in conjunction with Yes tables to display results from meta-analyses?
Yes, but it could be improved No Please describe how the harvest plots in conjunction with tables could be improved to best display results from meta-analyses. __________________________________ Please describe why harvest plots in conjunction with tables could not be used to display results from meta-analyses. __________________________________ Is there key information that is missing that would have been helpful in displaying the results from meta-analyses? Please describe in detail. __________________________________ Q17 Q18 Q19 Q20 Q21 www.project-redcap.org