Open Access
Open Peer Review

This article has Open Peer Review reports available.

Investigation of relative risk estimates from studies of the same population with contrasting response rates and designs

  • Nicole M Mealing1, 2Email author,
  • Emily Banks1, 3,
  • Louisa R Jorm1, 2,
  • David G Steel4,
  • Mark S Clements2 and
  • Kris D Rogers1
BMC Medical Research MethodologyBMC series ¿ open, inclusive and trusted201010:26

DOI: 10.1186/1471-2288-10-26

Received: 14 December 2009

Accepted: 1 April 2010

Published: 1 April 2010

Abstract

Background

There is little empirical evidence regarding the generalisability of relative risk estimates from studies which have relatively low response rates or are of limited representativeness. The aim of this study was to investigate variation in exposure-outcome relationships in studies of the same population with different response rates and designs by comparing estimates from the 45 and Up Study, a population-based cohort study (self-administered postal questionnaire, response rate 18%), and the New South Wales Population Health Survey (PHS) (computer-assisted telephone interview, response rate ~60%).

Methods

Logistic regression analysis of questionnaire data from 45 and Up Study participants (n = 101,812) and 2006/2007 PHS participants (n = 14,796) was used to calculate prevalence estimates and odds ratios (ORs) for comparable variables, adjusting for age, sex and remoteness. ORs were compared using Wald tests modelling each study separately, with and without sampling weights.

Results

Prevalence of some outcomes (smoking, private health insurance, diabetes, hypertension, asthma) varied between the two studies. For highly comparable questionnaire items, exposure-outcome relationship patterns were almost identical between the studies and ORs for eight of the ten relationships examined did not differ significantly. For questionnaire items that were only moderately comparable, the nature of the observed relationships did not differ materially between the two studies, although many ORs differed significantly.

Conclusions

These findings show that for a broad range of risk factors, two studies of the same population with varying response rate, sampling frame and mode of questionnaire administration yielded consistent estimates of exposure-outcome relationships. However, ORs varied between the studies where they did not use identical questionnaire items.

Background

The aim of most epidemiological studies is to obtain estimates that can be generalised to a population of interest. For surveys concerned with disease prevalence, the main means to achieve this is to draw a sample that is sufficiently representative of the target population. However, few surveys have perfect response rates and any level of nonresponse can potentially lead to biased estimates of prevalence [1, 2].

In contrast, much epidemiological practice is based around the principle that representativeness is not necessarily required for reliable estimates of relative risk based on internal comparisons within study populations [35]. Indeed, having a greater proportion of respondents in extreme categories compared to the population of interest may often be necessary, in order to yield sufficient information about specific exposure-outcome relationships [5]. A key issue is whether there is any nonresponse bias after conditioning on the covariates included in the analysis.

Cohort studies generally require more extensive data collection than one-off surveys, as well as the provision of identifying details and a long-term commitment to follow-up. While cohort studies often focus on selected population groups (e.g. occupational groups) [6, 7] and have relatively high response rates within these groups, recent response rates to population-based cohort studies are usually below 50% [812]. Furthermore, cohort study participants are generally healthier and more health conscious than non-participants [3, 1316]. Concern is often expressed at the low response rates for cohort studies, or the selectiveness of the group under study, and the generalisability of their results [17].

Direct empirical data to support the assumption that internal comparisons remain reliable, despite low response rates or highly selected study groups, is lacking. Furthermore, concerns are also expressed that elements of study design, such as sampling methods and use of postal questionnaires versus interviews, may influence the observed relationships [18]. This paper investigates whether or not cross-sectional estimates of exposure-outcome relationships are affected by survey aspects (response rate, sampling frame and mode of questionnaire administration) or the wording of questionnaire items, by comparing estimates computed from two independent studies of the same target population with divergent response rates and different designs.

Methods

The 45 and Up Study

The 45 and Up Study is a population-based cohort study of more than 260,000 men and women aged 45 years and over in New South Wales (NSW), Australia [10]. Participants were randomly selected from the database that is used to administer the national universal health insurance scheme (Medicare Australia), which has almost complete coverage of the population. Equal numbers of males and females were selected for participants less than 80 years old. Individuals aged 80 years or over and residents in rural areas were oversampled by a factor of two, males aged 80 years or over were oversampled compared to females and all residents in remote areas were completely enumerated. Participants entered the study by completing a baseline postal questionnaire and providing written consent to have their health followed over time. The study questionnaire is available at http://​www.​45andUp.​org.​au. The survey was available only in English. The current overall response rate to the baseline questionnaire is estimated to be 17.9% [10]. The final analytic sample consisted of 44,851 men and 52,961 women joining the study up to July 2008 after excluding 125 respondents who had a missing Accessibility Remoteness Index of Australia (ARIA+) [19] score.

Post-stratification estimation weights were assigned to the 45 and Up baseline survey to adjust the sample to account for the differences in selection probabilities and response rates and give consistency with 2006 population estimates produced by the Australian Bureau of Statistics (ABS) [20]. The post-strata were formed according to sex (male or female), remoteness (major city, inner regional, outer regional or remote) and age (five year age groups from 45-85 years or ≥85 years).

The NSW Population Health Survey

The NSW Population Health Survey (PHS) is an ongoing survey on the health of people in NSW using computer assisted telephone interviewing [21]. Independent samples of NSW households with private telephones are drawn each year using random digit dialling, and one person is randomly selected to participate in the survey. Informed consent was obtained from participants by their willingness to complete the telephone interview. The survey questionnaire is available at http://​www.​health.​nsw.​gov.​au/​publichealth/​surveys/​phs.​asp. The survey is administered in 6 languages. In 2006 participants were asked all survey questions and in 2007 they were asked a random subset of the survey questions. We report analyses of data for 5,766 men and 9,030 women aged 45 years or over who responded to the 2006 (n = 5,480) or 2007 (n = 9,316) PHS, with response rates of 59.3% [22] and 63.6% [23] respectively.

Weights were assigned to each year of data to adjust for the differences in the probability of selection within the household, number of residential telephone connections to the house and the varying sampling fraction between each of the 8 NSW area health services to provide estimates that were representative of the NSW population [21]. These area health services can include several remoteness categories. Post-stratification weights were also assigned according to sex (male or female) and age (five year age groups from 45-85 years or 85-110 years) using 2005 and 2007 mid-year population statistics released by the ABS for each area health service [22, 23]. After weighting, Indigenous people are slightly under-represented in the PHS sample, and Australian-born people slightly over-represented, compared to the overall NSW population [22, 23].

Questionnaire items

We obtained the original questionnaires from the 45 and Up Study and the 2006 and 2007 PHS and compared the wording of questions and response categories. We classified questionnaire items as highly comparable, moderately comparable or not comparable, based on whether the item was expected to yield identical, similar or non-comparable responses, respectively, for a given individual. Analyses focused on items considered highly or moderately comparable; items used in these analyses are compared in Additional file 1. All variables used in these analyses were derived from self-reported data except postcode (45 and Up Study only).

All analyses included all participants in both studies, unless otherwise stated. If one study only asked a sub-set of participants a question of interest then the same restriction was applied to the other study. Data are reported on falls in the past 12 months for participants aged 60 years and over, hysterectomy operation in females less than 70 years, mammography screening in the past two years for females less than 80 years and bowel screening in the past 5 years for all persons aged 50 years and over.

Questions on mammography screening and hysterectomy were only asked in the 2006 PHS and hypertension and bowel screening in the 2007 PHS.

Highly comparable questionnaire items

Remoteness was determined using the mean ARIA+ score for the postcode of the participant's residential address and categorised as major city, inner regional, outer regional or remote, according to the Australian Institute of Health and Welfare [24].

Self reported height and weight were used to calculate participants' body mass index (BMI) as weight in kilograms divided by the square of height in meters. BMI was categorised as underweight (BMI < 18.5 kg/m2), normal range (BMI 18.5-24.9 kg/m2), overweight (BMI 25.0-29.9 kg/m2) or obese (BMI ≥ 30 kg/m2) according to the World Health Organisation [25].

Participants were classified as having hypertension, diabetes and/or asthma if they reported that these conditions had ever been diagnosed by a doctor (both studies) or at a hospital (PHS only). Only participants who answered version two of the 45 and Up baseline questionnaire were asked if they had ever been diagnosed with asthma (n = 65,522).

Indicator variables were constructed for being born in Australia, missing all natural teeth, speaking a language other than English at home, having fallen in the past 12 months, having private health insurance (excluding Medicare) and having a hysterectomy.

Daily fruit consumption was grouped into participants who don't eat fruit, participants who eat fruit but less than two serves per day, and participants who eat two or more serves per day.

An indicator for females who were breast screened in the past two years was ascertained from responses to ever having a mammogram and the year of (45 and Up Study) or time interval (PHS) since their last mammogram.

Psychological distress was evaluated using the Kessler (K10) measure [26] ascertained as the sum of responses for 10 questions. If a respondent answered nine of the 10 items then the missing item was imputed as the average of the other nine responses. If a respondent answered less than nine items their K10 score was set to missing. Participants with a K10 score of 22 or greater were assigned as having high/very high levels of psychological distress and those with a score less than 22 as having low/moderate levels of psychological distress [27].

Moderately comparable questionnaire items

The wording of questions across the two studies differed for household income before tax (45 and Up Study included benefits, pensions and superannuation), bowel screening (screening tests varied by study) and current smoking status (45 and Up Study participants recorded whether they were 'regular smokers' currently without a definition for regular, whereas PHS participants recorded if they 'smoke daily' where smoking was defined to include cigarettes, cigars and pipes).

The response categories varied across the two studies for highest level of educational attainment (for these analyses similar categories were constructed) and self-rated health status (the PHS had an additional response category - for these analyses the categories "poor" and "very poor" on the PHS were combined).

Analysis

Before analyses commenced, twenty exposure-outcome pairs were selected for inclusion in our analyses. These were selected on the basis of demonstrating relationships across a wide range of domains of research interest. This consisted of i) ten pairs where both the exposure and the outcome variables were highly comparable across the two studies; and ii) ten pairs where the exposure and/or outcome variables were only moderately comparable across the two studies.

Unweighted and survey weighted prevalence estimates with 95% confidence intervals (CI) were calculated for each study for all highly and moderately comparable variables used in the exposure-outcome relationship analyses. Odds ratios (ORs) were used to approximate relative risks and logistic regression analyses were used to calculate the 20 pre-determined exposure-outcome relationships for each study; separated into highly and moderately comparable ORs. In each case two sets of ORs were calculated; namely the crude OR and the OR adjusting for age, sex and remoteness since these were the sampling variables common to both surveys. Unweighted and weighted comparisons of these two types of ORs by study are presented in Additional files 2 and 3 respectively (45 and Up Study) and Additional files 4 and 5 respectively (PHS).

In the figures, the squares and lines represent each OR estimate and CI, with the area of each square being proportional to the sample size used for each estimate.

Wald chi-square statistics were computed to test for differences in the log odds ratios between the two surveys for each of the 20 exposure-outcome pairs and compared to a chi-square distribution with degrees of freedom equal to the number of categories minus one in the exposure variable. Each study was modelled separately and then the Wald statistics were calculated by combining the two sets of estimated parameters, variances and covariances. Analyses were conducted with and without using sampling weights. With survey data the Wald test can be unreliable if the degrees of freedom on the estimated covariance matrix are small [28]. In this case the samples were large and the designs relatively simple.

Analyses were carried out using SAS, version 9 [29]. This study has the approval of the University of New South Wales Ethics Committee and the NSW Population and Health Services Research Ethics Committee.

Results

The distributions of social and demographic characteristics and of health risk factors and conditions in the two studies are shown in Tables 1 and 2, respectively. Younger persons and/or those living in major cities were under-represented in both surveys as were males in the PHS (the 45 and Up Study sample was stratified by sex). The prevalence confidence intervals were narrower on the 45 and Up Study compared to the PHS, because of the larger sample size.
Table 1

Social and demographic characteristics of the 45 and Up Study and the NSW PHS populations

 

45 and Up Study

PHS

Social and Demographics

      

Characteristics a

N

Crude % (95% CI)

Weighted b % (95% CI)

N

Crude % (95% CI)

Weighted c % (95% CI)

Sex

      

Male

48851

48.0 (47.7, 48.3)

48.0 (47.6, 48.4)

5766

39.0 (38.2, 39.8)

48.4 (47.4, 49.5)

Female

52961

52.0 (51.7, 52.3)

52.0 (51.6, 52.4)

9030

61.0 (60.2, 61.8)

51.6 (50.5, 52.6)

Age

      

45-49 years

12399

12.2 (12.0, 12.4)

18.9 (18.6, 19.2)

1803

12.2 (11.7, 12.7)

19.3 (18.3, 20.3)

50-54 years

16120

15.8 (15.6, 16.1)

17.1 (16.8, 17.4)

2043

13.8 (13.3, 14.4)

17.5 (16.6, 18.4)

55-59 years

17350

17.0 (16.8, 17.3)

15.9 (15.7, 16.2)

2188

14.8 (14.2, 15.4)

16.2 (15.4, 17.0)

60-64 years

15342

15.1 (14.8, 15.3)

12.6 (12.4, 12.8)

2252

15.2 (14.6, 15.8)

12.5 (11.9, 13.1)

65-69 years

12800

12.6 (12.4, 12.8)

10.1 (9.9, 10.3)

1939

13.1 (12.6, 13.6)

10.2 (9.6, 10.8)

70-74 years

9564

9.4 (9.2, 9.6)

8.3 (8.1, 8.5)

1710

11.6 (11.0, 12.1)

8.5 (8.0, 8.9)

75-79 years

7235

7.1 (6.9, 7.3)

7.3 (7.2, 7.5)

1490

10.1 (9.6, 10.6)

7.3 (6.9, 7.8)

80-84 years

7545

7.4 (7.2, 7.6)

5.4 (5.3, 5.6)

922

6.2 (5.8, 6.6)

5.9 (5.4, 6.3)

≥ 85 years

3457

3.4 (3.3, 3.5)

4.3 (4.1, 4.4)

449

3.0 (2.8, 3.3)

2.6 (2.3, 2.9)

Remoteness (ARIA+)

      

Major City

44146

43.4 (43.1, 43.7)

69.1 (68.8, 69.4)

6718

45.4 (44.6, 46.2)

60.6 (59.6, 61.6)

Inner Regional

36640

36.0 (35.7, 36.3)

22.7 (22.5, 23.0)

4351

29.4 (28.7, 30.1)

24.4 (23.5, 25.2)

Outer Regional

18926

18.6 (18.4, 18.8)

7.6 (7.5, 7.7)

3086

20.9 (20.2, 21.5)

12.2 (11.6, 12.7)

Remote

2100

2.1 (2.0, 2.1)

0.6 (0.5, 0.6)

456

3.1 (2.8, 3.4)

1.3 (1.2, 1.5)

Missing

0

-

-

185

1.3 (1.1, 1.4)

1.5 (1.2, 1.8)

Language other than English spoken at home

      

No

92230

90.6 (90.4, 90.8)

87.4 (87.1, 87.7)

13379

93.0 (92.6, 93.4)

89.5 (88.7, 90.3)

Yes

9580

9.4 (9.2, 9.6)

12.6 (12.3, 12.9)

986

6.9 (6.4, 7.3)

10.4 (9.6, 11.2)

Missing

2

0.0 (0.0, 0.0)

0.0 (0.0, 0.0)

17

0.1 (0.0, 0.2)

0.1 (0.0, 0.2)

Country of Birth

      

Australia

75821

74.5 (74.2, 74.7)

70.8 (70.5, 71.1)

11332

76.6 (75.9, 77.3)

71.1 (70.1, 72.2)

Not Australia

24964

24.5 (24.3, 24.8)

28.2 (27.9, 28.6)

3433

23.2 (22.5, 23.9)

28.7 (27.7, 29.7)

Missing

1027

1.0 (0.9, 1.1)

1.0 (0.9, 1.0)

24

0.2 (0.1, 0.2)

0.2 (0.1, 0.2)

Private Health Insurance

      

No

38300

37.6 (37.3, 37.9)

33.9 (33.6, 34.3)

6766

45.7 (44.9, 46.5)

41.5 (40.4, 42.5)

Yes

63508

62.4 (62.1, 62.7)

66.1 (65.7, 66.4)

7973

53.9 (53.1, 54.7)

58.2 (57.1, 59.3)

Missing

4

0.0 (0.0, 0.0)

0.0 (0.0, 0.0)

57

0.4 (0.3, 0.5)

0.3 (0.2, 0.4)

Educational Attainment

      

No School Certificate

12385

12.2 (12.0, 12.4)

10.9 (10.7, 11.1)

2017

13.7 (13.1, 14.2)

11.7 (11.1, 12.3)

School Certificate

22608

22.2 (22.0, 22.5)

20.3 (20.0, 20.6)

3732

25.3 (24.6, 26.0)

20.6 (19.8, 21.4)

Trade/Certificate/Diploma

32289

31.7 (31.4, 32.0)

31.0 (30.6, 31.3)

1631

26.5 (25.8, 27.2)

29.5 (28.5, 30.5)

Higher School Certificate

9787

9.6 (9.4, 9.8)

10.1 (9.9, 10.3)

3902

11.1 (10.6, 11.6)

10.9 (10.2, 11.6)

Tertiary Qualification

22802

22.4 (22.1, 22.7)

25.9 (25.6, 26.3)

3096

21.0 (20.3, 21.7)

25.0 (24.0, 26.0)

Missing

1941

1.9 (1.8, 2.0)

1.8 (1.8, 1.9)

360

2.4 (2.2, 2.7)

2.3 (2.0, 2.6)

Household Income

      

<$20,000 p.a

20633

20.3 (20.0, 20.5)

17.9 (17.6, 18.1)

4505

30.4 (29.7, 31.2)

22.2 (21.4, 22.9)

$20,000-$39,999 p.a

18386

18.1 (17.8, 18.3)

15.3 (15.1, 15.6)

3145

21.3 (20.6, 21.9)

20.3 (19.5, 21.1)

≥$40,000 p.a

39792

39.1 (38.8, 39.4)

44.6 (44.2, 44.9)

4768

32.2 (31.5, 33.0)

41.4 (40.3, 42.5)

Missing

23001

22.6 (22.3, 22.8)

22.2 (21.9, 22.5)

2378

16.1 (15.5, 16.7)

16.2 (15.4, 17.0)

ARIA+, Accessibility Remoteness Index of Australia; CI, Confidence Interval; p.a, per annum

a Characteristics above the line-break are highly comparable across the two surveys. Characteristics below the line-break are moderately comparable across the two surveys. Weighted prevalence estimates for age, sex and remoteness differ across the two studies since the weighting schemes vary.

b Weighted by age, sex and remoteness

c Weighted by area health service, probability of selection in the household and the number of residential connections to the house

Table 2

Health risk factors and conditions of the 45 and Up Study and the NSW PHS populations

 

45 and Up Study

PHS

Variable a

N

Crude % (95% CI)

Weighted b % (95% CI)

N

Crude % (95% CI)

Weighted c % (95% CI)

Fruit Consumption

      

Don't eat fruit

6620

6.5 (6.4, 6.7)

6.5 (6.4, 6.7)

321

6.1 (5.5, 6.8)

6.1 (5.2, 6.9)

< 2 serves per day

31707

31.1 (30.9, 31.4)

31.0 (30.7, 31.4)

1719

32.8 (31.6, 34.1)

33.4 (31.8, 35.1)

≥ 2 serves per day

57620

56.6 (56.3, 56.9)

56.7 (56.3, 57.0)

3135

59.9 (58.5, 61.2)

59.4 (57.7, 61.2)

Missing

5865

5.8 (5.6, 5.9)

5.7 (5.6, 5.9)

61

1.2 (0.9, 1.5)

1.0 (0.7, 1.3)

Teeth

      

Some/all natural teeth

88501

86.9 (86.7, 87.1)

88.0 (87.8, 88.2)

9321

86.7 (86.0, 87.3)

90.3 (89.7, 90.9)

No natural teeth

9953

9.8 (9.6, 10.0)

8.7 (8.5, 8.9)

1427

13.3 (12.6, 13.9)

9.7 (9.1, 10.3)

Missing

3358

3.3 (3.2, 3.4)

3.3 (3.2, 3.5)

6

0.1 (0.0, 0.1)

0.0 (0.0, 0.1)

Body Mass Index

      

Underweight

1461

1.4 (1.4, 1.5)

1.5 (1.4, 1.6)

230

2.1 (1.9, 2.4)

1.9 (1.6, 2.3)

Normal Range

35219

34.6 (34.3, 34.9)

35.6 (35.2, 35.9)

4214

39.0 (38.1, 40.0)

38.4 (37.2, 39.6)

Overweight

37373

36.7 (36.4, 37.0)

36.6 (36.2, 36.9)

3774

35.0 (34.1, 35.9)

36.0 (34.8, 37.2)

Obese

20271

19.9 (19.7, 20.2)

19.2 (18.9, 19.5)

2205

20.4 (19.7, 21.2)

20.5 (19.4, 21.5)

Missing

7488

7.4 (7.2, 7.5)

7.1 (6.9, 7.3)

373

3.5 (3.1, 3.8)

3.2 (2.8, 3.7)

Ever diagnosed with hypertension

      

No

72023

70.7 (70.5, 71.0)

72.1 (71.7, 72.4)

1189

52.1 (50.1, 54.2)

55.2 (52.6, 57.8)

Yes

25144

24.7 (24.4, 25.0)

22.9 (22.6, 23.2)

1086

47.6 (45.6, 49.7)

44.6 (42.0, 47.2)

Missing

4645

4.6 (4.4, 4.7)

5.0 (4.9, 5.2)

6

0.3 (0.1, 0.5)

0.2 (0.0, 0.4)

Ever diagnosed with diabetes

      

No

88056

86.5 (86.3, 86.7)

86.6 (86.3, 86.8)

9450

88.4 (87.8, 89.0)

89.2 (88.4, 90.0)

Yes

9111

8.9 (8.8, 9.1)

8.4 (8.2, 8.6)

1202

11.2 (10.6, 11.8)

10.5 (9.7, 11.2)

Missing

4645

4.6 (4.4, 4.7)

5.0 (4.9, 5.2)

36

0.3 (0.2, 0.4)

0.3 (0.2, 0.4)

Ever diagnosed with asthma

      

No

55034

84.0 (83.7, 84.3)

83.8 (83.5, 84.1)

8753

82.0 (81.3, 82.8)

83.2 (82.2, 84.1)

Yes

7653

11.7 (11.4, 11.9)

11.4 (11.1, 11.7)

1894

17.8 (17.0, 18.5)

16.7 (15.8, 17.6)

Missing

2835

4.3 (4.2, 4.5)

4.8 (4.6, 5.0)

22

0.2 (0.1, 0.3)

0.1 (0.1, 0.2)

Fallen in the past 12 months

      

No

42038

75.1 (74.8, 75.5)

74.1 (73.7, 74.6)

2957

77.3 (76.0, 78.7)

77.5 (75.9, 79.1)

Yes

10757

19.2 (18.9, 19.6)

20.0 (19.6, 20.4)

856

22.4 (21.1, 23.7)

22.1 (20.6, 23.7)

Missing

49017

5.6 (5.4, 5.8)

5.8 (5.6, 6.1)

11

0.3 (0.1, 0.5)

0.4 (0.1, 0.6)

Last Mammography Screening

      

Not within past 2 years

12622

26.2 (25.9, 26.6)

28.6 (28.1, 29.1)

1018

34.4 (32.7, 36.1)

36.7 (34.5, 39.0)

Within past 2 years

30056

62.5 (62.1, 62.9)

60.1 (59.6, 60.6)

1925

65.1 (63.4, 66.8)

62.8 (60.6, 65.1)

Missing

5413

11.3 (11.0, 11.5)

11.3 (11.0, 11.7)

15

0.5 (0.3, 0.8)

0.4 (0.2, 0.7)

Hysterectomy

      

No

29487

73.9 (73.4, 74.3)

76.6 (76.1, 77.0)

1583

71.1 (69.2, 73.0)

74.2 (72.0, 76.4)

Yes

10431

26.1 (25.7, 26.6)

23.4 (23.0, 23.9)

630

28.3 (26.4, 30.2)

25.0 (22.8, 27.2)

Missing

0

-

-

13

0.6 (0.3, 0.9)

0.8 (0.3, 1.3)

Psychological Distress

      

Low\Moderate

81542

80.1 (79.8, 80.3)

80.7 (80.4, 81.0)

9503

88.8 (88.2, 89.4)

88.9 (88.1, 89.7)

High\Very high

6663

6.5 (6.4, 6.7)

6.8 (6.6, 7.0)

1101

10.3 (9.7, 10.9)

10.2 (9.4, 11.0)

Missing

13607

13.4 (13.2, 13.6)

12.5 (12.2, 12.7)

100

0.9 (0.8, 1.1)

0.9 (0.7, 1.2)

Current Smoker

      

No

93685

92.0 (91.9, 92.2)

91.9 (91.7, 92.1)

9468

87.9 (87.3, 88.6)

88.1 (87.3, 89.0)

Yes

7575

7.4 (7.3, 7.6)

7.6 (7.4, 7.8)

1292

12.0 (11.4, 12.6)

11.8 (11.0, 12.7)

Missing

552

0.5 (0.5, 0.6)

0.5 (0.5, 0.6)

7

0.1 (0.0, 0.1)

0.1 (0.0, 0.1)

Last Bowel Screening

      

Not within past 5 years

49036

54.8 (54.5, 55.2)

55.9 (55.6, 56.3)

2497

53.5 (52.1, 54.9)

54.6 (52.7, 56.4)

Within past 5 years

35990

40.3 (39.9, 40.6)

39.2 (38.8, 39.6)

2098

45.0 (43.5, 46.4)

43.3 (41.5, 56.4)

Missing

4387

4.9 (4.8, 5.0)

4.9 (4.7, 5.0)

72

1.5 (1.2, 1.9)

2.2 (1.4, 2.9)

Self-reported Health Status

      

Excellent

14920

14.7 (14.4, 14.9)

15.5 (15.2, 15.8)

2433

17.8 (17.2, 18.5)

18.1 (17.2, 18.9)

Very Good

36046

35.4 (35.1, 35.7)

35.4 (35.0, 35.7)

3815

28.0 (27.2, 28.7)

28.5 (27.5, 29.5)

Good

33178

32.6 (32.3, 32.9)

32.1 (31.7, 32.4)

3990

29.2 (28.5, 30.0)

29.8 (28.7, 30.8)

Fair

11878

11.7 (11.5, 11.9)

11.4 (11.2, 11.6)

2265

16.6 (16.0, 17.2)

16.0 (15.2, 16.8)

Poor

2090

2.1 (2.0, 2.1)

2.0 (1.9, 2.1)

863

6.3 (5.9, 6.7)

5.8 (5.3, 6.3)

Very Poor d

N/A

  

239

1.8 (1.5, 2.0)

1.6 (1.3, 1.9)

Missing

3700

3.6 (3.5, 3.7)

3.7 (3.5, 3.8)

44

0.3 (0.2, 0.4)

0.3 (0.2, 0.4)

CI, Confidence Interval; NSW, New South Wales;

a Variables above the line-break are highly comparable across the two surveys. Variables below the line-break are moderately comparable but similar across the two surveys

b Weighted by age, sex and remoteness

c Weighted by area health service, probability of selection in the household and the number of residential connections to the house

d The category 'Very Poor' was only available on the NSW Population Health Survey

The weighted estimates of prevalence were similar across the two studies for variables such as age, sex, and remoteness (the variables used for weighting), country of birth, educational attainment, fruit consumption, body-mass-index and falls. However, the prevalence of speaking a language other than English at home and of holding private health insurance was higher in the 45 and Up Study compared to the PHS, while the prevalence of smoking, high/very high psychological distress, ever diagnosed with hypertension, ever diagnosed with diabetes and ever diagnosed with asthma was lower (Table 1; Table 2). The PHS tended to have less missing data than the 45 and Up Study, particularly for variables relating to mammography screening, K10 score and household income before tax. Prevalence estimates for self-rated health status varied across the two studies with the proportion who reported the lowest category of self-rated health status on the 45 and Up baseline questionnaire (i.e. "poor") being similar to the proportion who reported the lowest category on the PHS (i.e. "very poor")(Table 2).

The ten exposure-outcome relationships where both the exposure and the outcome variables were highly comparable across the two studies are presented in Figure 1, with ORs adjusted for age, sex and remoteness. The observed relationships were virtually identical between the two studies. For 8 out of the 10 relationships there was no significant difference between the results from the different studies. There was borderline evidence of a difference in the risk of falling according to BMI across the two studies (Wald test P = 0.04) and minor heterogeneity in the relationship of age to high/very high psychological distress was observed (Wald test P = 0.02). Similar observations were seen when the ORs from these ten relationships were calculated using sampling weights (Additional file 6).
https://static-content.springer.com/image/art%3A10.1186%2F1471-2288-10-26/MediaObjects/12874_2009_Article_429_Fig1_HTML.jpg
Figure 1

Odds ratios a by study where the exposure and outcome variables were highly comparable across the 45 and Up Study and the NSW PHS b . ARIA+, Accessibility Remoteness Index of Australia; CI, Confidence Interval; NSW, New South Wales; PHS, Population Health Survey. a Adjusting for age, sex and remoteness.b Black squares represent ORs with area inversely proportional to the sample size contributing to the OR and the corresponding line represents the 95% confidence interval. c P-value from Wald chi-square test: testing for a difference between the two studies for the specific exposure-outcome pair

The ten exposure-outcome relationships where the exposure and/or outcome variables were only moderately comparable across the two studies are presented in Figure 2, with ORs adjusted for age, sex and remoteness. Each exposure-outcome pair had a similar relationship pattern for both studies and all OR estimates were in the same direction and of similar magnitude, except when self-rated health status was the exposure variable. The relationships did not differ significantly for 4 out of 10 of the exposure-outcome associations. Significant but relatively minor differences in ORs were observed for smoking and educational attainment and pre-tax income in relation to psychological distress, private health insurance and remoteness of residence. In spite of the similarity in the shape of the relationship, substantial heterogeneity and large differences in ORs were observed for relationships with self-rated health (where the PHS had an additional response category, "very poor"). Similar observations were seen when the ORs from these ten relationships were calculated using sampling weights (Additional file 7).
https://static-content.springer.com/image/art%3A10.1186%2F1471-2288-10-26/MediaObjects/12874_2009_Article_429_Fig2_HTML.jpg
Figure 2

Odds Ratios a by study where either the exposure or outcome or both variables were only moderately comparable across the 45 and Up Study and the NSW PHS b . ARIA+, Accessibility Remoteness Index of Australia; CI, Confidence Interval; NSW, New South Wales; p.a, per annum; PHS, Population Health Survey. a Adjusting for age, sex and remoteness. b Black squares represent ORs with area inversely proportional to the sample size contributing to the OR and the corresponding line represents the 95% confidence interval. c P-value from Wald chi-square test: testing for a difference between the two studies for the specific exposure-outcome pair

Following adjustment for age, sex and remoteness, additional weighting of the OR for age, sex and remoteness did not change any of the ORs from the 45 and Up Study materially (i.e. no changes were >10%) (Additional files 2, 3). This is because the weighting is determined by the variables used in the logistic regression. Weighting the PHS resulted in some changes to the ORs because not all variables used to determine the weighting (i.e. household size to account for the selection of a person from each selected household and the 8 area health services) were used in the logistic regression (Additional files 4, 5). Weighting did not change the general nature of the observed relationships.

Discussion

Discussions around epidemiological methods often conclude that representativeness is not necessarily required for reliable estimates of relative risk based on internal comparisons within study populations [4]. By their nature, cohort studies tend not to be directly representative of the general population, however over time, their results have usually been shown to be both reproducible and generalisable to the larger population [6, 7]. Miettinen explains that "an empirical relation is not distorted by any manipulation of the distribution of the study base according to the elements in the occurrence relation - the determinant, the modifiers and/or confounders. For example, the empirical relation of body weight to gender does not depend on the gender distribution in the study base." [[5], p. 56]

It is generally accepted that in order to produce results that are generalisable, studies should exhibit sufficient variability in the determinant and modifiers to be studied and a limited range for confounders [4, 5]. Nevertheless the possibility of bias cannot be excluded, and empirical data on how exposure-outcome relationships might vary according to the degree of nonresponse are lacking.

Nonresponse is a form of self-selection. Selection solely by the exposure or outcome variable does not bias the estimates of ORs in logistic regression [14, 30, 31] and selection solely on the basis of covariates in the logistic regression also leads to unbiased ORs. Although evidence from simulation supports the principle of generalisability [12], specific scenarios may result in significant bias if selection criteria and dependent variables are closely related [32]. In particular, biases can occur if selection depends on both the exposure and outcome [14, 17].

We found that although some prevalence estimates varied between the two studies of the same population investigated here, exposure-outcome relationships did not differ materially, where the variables used were highly comparable. This was despite major differences between the studies, including varying response rates, sampling frames and modes of administration; the PHS had a smaller proportion of missing and invalid responses due to the nature of the computer assisted telephone interviewing system and it included respondents who completed the survey in languages other than English. It was not possible to definitively separate the individual effects of sampling frame, response rate and mode of administration, since response rates and aspects of study design are closely linked [33].

We were unable to locate other empirical comparisons of relative risk estimates in independent studies with divergent response rates and different study designs that were drawn from the same target population. Indirect evidence supporting our findings comes from studies that have observed consistent ORs in study respondents and non-respondents using linked data [34] and in initial cohort study participants and participants responding to a subsequent questionnaire [35]. Two studies found only small biases in relative risk estimates due to nonresponse, in cross-sectional ORs from a cohort study relating to cardiovascular disease [31], and in cohort analyses relating to reproductive outcomes [12]. One study found consistent ORs related to smoking in respondents recruited by postal survey and those recruited through postal and telephone surveys and home visits [18].

Having established the lack of any major differences attributable to response rate and study design (including sampling frame and mode of questionnaire administration), the comparison of exposure-outcome relationships containing moderately comparable variables across the two studies can be seen as illustrating the additional effect of the specific questionnaire items used. Our findings demonstrate that an apparently minor difference in the wording of questions can significantly influence measures of prevalence and estimates of risks. This emphasizes the critical importance of maintaining the consistency of survey questions if valid comparisons are to be made and is consistent with previous studies [3638]. Although most differences attributable to question wording resulted in minor heterogeneity, highly significant heterogeneity was evident for the question on self-rated health status, where the response categories varied across the two studies. However, despite differences between questionnaire items, the observed ORs would lead to similar conclusions regarding the nature of the exposure-outcome relationships.

One shortcoming of our study is the lack of strict gold-standard measures for the study variables. The PHS has a 40% nonresponse rate and may be subject to nonresponse bias. Under ideal circumstances we could use census data; however the Australian Census includes only very limited health data. Additionally, these findings relate to two large studies with considerable variability in the factors included in our analyses. This ensured that there were substantial numbers of participants from each study in each exposure-outcome category, and allowed for adjustment for multiple factors. Although these findings support the principle of generalisability of findings from a relatively select group of participants, it remains possible that they are less applicable to smaller, less heterogeneous studies. These findings relate to cross-sectional analyses; prospective, longitudinal analyses are less prone to the potential biases investigated here, since baseline selection cannot be influenced by outcome status.

Applying weights to survey data to calculate prevalence estimates that account for the differences in probability of selection is standard practice. However, use of sampling weights is less common when calculating relative risks from cohort study data; instead adjustments are usually made to account for potential confounders. The relative risk estimates adjusting for age, sex and remoteness from the 45 and Up Study were not altered materially by further weighting. Hence weighting did not appear to be necessary when the variables used in calculating the weights were used as covariates in the analysis. Weighting is potentially important in the PHS because of the role of household size and area health service in the weighting.

Conclusions

These findings show that broad ranges of exposure-outcome relationships estimated from two studies of the same population remained consistent regardless of the underlying response rate or mode of questionnaire administration. They provide empirical support for the basic epidemiological principle that results based on internal comparisons remain generalisable even when study subjects are drawn from a relatively select group. They emphasize the crucial importance of maintaining the consistency of question wording in order to permit comparisons between studies.

Declarations

Acknowledgements

The authors are grateful to Ms Margo Barr and Mrs Sandra Carlson for their expert assistance and advice on the NSW Population Health Survey and use of the NSW Population Health Survey data. We thank the men and women participating in the 45 and Up Study, and the NSW Department of Health for permitting use of the 2006 and 2007 NSW Population Health Survey Datasets.

The 45 and Up Study is run by The Sax Institute in collaboration with the Cancer Council of New South Wales; the New South Wales Division of the National Heart Foundation of Australia; the New South Wales Department of Health; beyondblue: the national depression initiative and the New South Wales Department of Ageing, Disability and Home Care. This work was also supported by the National Health and Medical Research Council [402810]. NM was employed by the NSW Department of Health on the NSW Biostatistical Officer Training Program at the time this work was conducted.

The concept of this study was developed as part of the Study of Economic and Environmental Factors in health (SEEF) project. The SEEF investigators are: Adrian Bauman, Sally Redman, Emily Banks, Mark Harris, Deborah Schofield, Tony McMichael, Hilary Bambrick, John Beard, Dorothy Broom, Julie Byles, Judith Clark, Bin Jalaludin, Louisa Jorm, Don Nutbeam, Bryan Rodgers, Heather Bowles, Jack Chen, PH Phongsavan, Lynne Parkinson.

Authors’ Affiliations

(1)
The Sax Institute
(2)
School of Medicine, University of Western Sydney
(3)
National Centre for Epidemiology and Population Health, Australian National University
(4)
Centre for Statistical and Survey Methodology, University of Wollongong

References

  1. Burkell J: The dilemma of survey nonresponse. Libr Inform Sci Res. 2003, 25 (3): 239-63. 10.1016/S0740-8188(03)00029-X.View Article
  2. Van Goor H, Verhage AL: Nonresponse and recall errors in a study of absence because of illness: An analysis of their effects on distributions and relationships. Quality & Quantity. 1999, 33 (4): 411-28.View Article
  3. Breslow NE, Day NE: Statistical methods in cancer research. Volume II--The design and analysis of cohort studies. IARC Sci Publ. 1987, 82: 1-406.
  4. Ponsonby AL, Dwyer T, Couper D: Is this finding relevant? Generalisation and epidemiology. Aust N Z J Public Health. 1996, 20 (1): 54-6. 10.1111/j.1467-842X.1996.tb01336.x.PubMedView Article
  5. Miettinen OS: Theoretical epidemiology: principles of occurrence research in medicine. 1985, New York: Wiley
  6. Doll R, Peto R, Boreham J, Sutherland I: Mortality in relation to smoking: 50 years' observations on male British doctors. BMJ. 2004, 328 (7455): 1519-10.1136/bmj.38142.554479.AE.PubMedPubMed CentralView Article
  7. Colditz GA, Manson JE, Hankinson SE: The Nurses' Health Study: 20-year contribution to the understanding of health among women. J Womens Health. 1997, 6 (1): 49-62. 10.1089/jwh.1997.6.49.PubMedView Article
  8. Lee C, Dobson AJ, Brown WJ, Bryson L, Byles J, Warner-Smith P, Young AF: Cohort profile: the Australian Longitudinal Study on Women's Health. Int J Epidemiol. 2005, 34 (5): 987-91. 10.1093/ije/dyi098.PubMedView Article
  9. Day N, Oakes S, Luben R, Khaw KT, Bingham S, Welch A, Wareham N: EPIC-Norfolk: study design and characteristics of the cohort. European prospective investigation of cancer. Br J Cancer. 1999, 80 (Suppl 1): 95-103.PubMed
  10. 45 and Up Study Collaborators: Cohort profile: the 45 and up study. Int J Epidemiol. 2008, 37 (5): 941-7. 10.1093/ije/dym184.PubMed CentralView Article
  11. Watts G: UK Biobank gets 10% response rate as it starts recruiting volunteers. BMJ. 2007, 334 (7595): 659-10.1136/bmj.39167.407616.DB.PubMedPubMed CentralView Article
  12. Nohr EA, Frydenberg M, Henriksen TB, Olsen J: Does low participation in cohort studies induce bias?. Epidemiology. 2006, 17 (4): 413-8. 10.1097/01.ede.0000220549.14177.60.PubMedView Article
  13. Eastwood BJ, Gregor RD, MacLean DR, Wolf HK: Effects of recruitment strategy on response rates and risk factor profile in two cardiovascular surveys. Int J Epidemiol. 1996, 25 (4): 763-9. 10.1093/ije/25.4.763.PubMedView Article
  14. Criqui MH: Response bias and risk ratios in epidemiologic studies. Am J Epidemiol. 1979, 109 (4): 394-9.PubMed
  15. Melton LJ, Dyck PJ, Karnes JL, O'Brien PC, Service FJ: Nonresponse bias in studies of diabetic complications - the Rochester Diabetic Neuropathy Study. Journal of Clinical Epidemiology. 1993, 46 (4): 341-8. 10.1016/0895-4356(93)90148-T.PubMedView Article
  16. Heilbrun LK, Nomura A, Stemmermann GN: The effects of nonresponse in a prospective-study of cancer - 15-year follow-up. Int J Epidemiol. 1991, 20 (2): 328-38. 10.1093/ije/20.2.328.PubMedView Article
  17. Greenland S: Response and Follow-up Bias in Cohort Studies. Am J Epidemiol. 1977, 106 (3): 184-7.PubMed
  18. Siemiatycki J, Campbell S: Nonresponse bias and early versus all responders in mail and telephone surveys. Am J Epidemiol. 1984, 120 (2): 291-301.PubMed
  19. Department of Health and Aged Care (GISCA): Measuring remoteness: accessibility/remoteness index of Australia (ARIA). Canberra. 2001
  20. Population Health Division: The health of the people of New South Wales - Report of the Chief Health Officer. Sydney: NSW Department of Health, Accessed 4 March 2009, [http://​www.​health.​nsw.​gov.​au/​publichealth/​chorep/​]
  21. Barr M, Baker D, Gorringe M, Fritsche L: NSW Population Health Survey: Description of Methods. 2008, Sydney: NSW Department of Health
  22. Centre for Epidemiology and Research: 2006 Report on Adult Health from the New South Wales Population Health Survey. 2007, Sydney: NSW Department of Health
  23. Centre for Epidemiology and Research: 2007 Report on Adult Health from the New South Wales Population Health Survey. 2008, Sydney: NSW Department of Health
  24. Australian Institute of Health and Welfare: Rural, Regional and Remote Health: A guide to remoteness classifications. Canberra. 2004
  25. WHO/IASO/IOTF: The Asia-Pacific perspective: redefining obesity and its treatment. 2000, Melbourne: Health Communications Australia
  26. Kessler RC, Andrews G, Colpe LJ, Hiripe E, Mroczek DK, Normand SL, Walters EE, Zaslavsky AM: Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002, 32 (6): 959-76. 10.1017/S0033291702006074.PubMedView Article
  27. Andrews G, Slade T: Interpreting scores on the Kessler Psychological Distress Scale (K10). Aust N Z J Public Health. 2001, 25 (6): 494-7. 10.1111/j.1467-842X.2001.tb00310.x.PubMedView Article
  28. Rao JNK, Thomas DR: Analysis of categorical response data from complex surveys: An appraisal and update. Analysis of survey data. Edited by: Chambers RL, Skinner CJ. 2003, New York: Wiley, 92-
  29. SAS Institute Inc: SAS version 9. NC, USA: Cary
  30. Scott AJ, Wild CJ: Fitting logistic-models under case-control or choice based sampling. Journal of the Royal Statistical Society Series B-Methodological. 1986, 48 (2): 170-82.
  31. Austin MA, Criqui MH, Barrett-Connor E, Holdbrook MJ: The effect of response bias on the odds ratio. Am J Epidemiol. 1981, 114 (1): 137-43.PubMed
  32. Brehm J: The phantom respondent: Opinion surveys and political representation. 1993, Michigan: The University of Michigan Press
  33. Perkins JJ, Sanson-Fisher RW: An examination of self- and telephone- administered modes of administration for the Australian SF-36. J Clin Epidemiol. 1998, 51 (11): 969-73. 10.1016/S0895-4356(98)00088-2.PubMedView Article
  34. Martikainen P, Laaksonen M, Piha K, Lallukka T: Does survey non-response bias the association between occupational social class and health?. Scand J Public Health. 2007, 35 (2): 212-5. 10.1080/14034940600996563.PubMedView Article
  35. Van Loon AJM, Tijhuis M, Picavet HSJ, Surtees PG, Ormel J: Survey non-response in the Netherlands: Effects on prevalence estimates and associations. Ann Epidemiol. 2003, 13 (2): 105-10. 10.1016/S1047-2797(02)00257-0.PubMedView Article
  36. Goodman R, Iervolino AC, Collishaw S, Pickles A, Maughan B: Seemingly minor changes to a questionnaire can make a big difference to mean scores: a cautionary tale. Soc Psychiatry Psychiatr Epidemiol. 2007, 42: 322-27. 10.1007/s00127-007-0169-0.PubMedView Article
  37. Kalton G, Schuman H: The effect of the question on survey responses: A review. Journal of the Royal Statistical Society Series A (General). 1982, 145 (1): 42-73. 10.2307/2981421.View Article
  38. Picavet HSJ, Bos Van den GAM: Comparing survey data on functional disability: The impact of some methodological differences. J Epidemiol Community Health. 1996, 50 (1): 86-93. 10.1136/jech.50.1.86.PubMedPubMed CentralView Article
  39. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:http://​www.​biomedcentral.​com/​1471-2288/​10/​26/​prepub

Copyright

© Mealing et al; licensee BioMed Central Ltd. 2010

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Advertisement