A questionnaire to measure melanoma risk, knowledge and protective behaviour: Assessing content validity in a convenience sample of Scots and Australians

Abstract

Background

The aim of this study was to assess the content validity of a questionnaire to measure melanoma risk, knowledge and protective behaviour in a convenience sample of Scots and Australians. Australia has the highest melanoma incidence worldwide but has developed a culture of skin cancer avoidance with a long history of skin cancer primary prevention campaigns of proven effectiveness. Scotland has lower incidence, but has shown a greater rate of increase between 1985 and 2007. There is an urgent need in Scotland, therefore, to identify those groups at greatest risk and provide them with effective preventative advice.

Method

A self-administered postal survey was completed by four groups formed from convenience samples in two geographical locations (Northeast Scotland and Western Australia). In univariate analysis scores on personal risk, level of concern, protective behaviour, and knowledge were compared by nationality, previous skin cancer diagnosis and personally knowing someone with melanoma. Multivariate linear regression analysis modelled the influence of potential predictor variables upon each of the scores.

Results

540 people completed the questionnaire, 273 Scots (50.6%). 133 (24.6%) Scots and 83 (15.4%) Australians previously had melanoma or non-melanoma skin cancer, whilst 120 (22.2%) Scots and 190 (35.2%) Australians personally knew someone with melanoma. Australians had higher knowledge (p < 0.001), level of concern (p < 0.001) and protective behaviour (p < 0.001) scores than the Scottish. Australian nationality was the strongest independent predictor of a higher knowledge score (p < 0.001), followed by a previous skin cancer diagnosis (p = 0.003), personal knowledge of someone with melanoma (p = 0.011), female gender (p = 0.005) and higher education status (p < 0.001) (R ² = 0.163).

Conclusion

The questionnaire detected higher levels of knowledge and skin cancer protective behaviours in Australians than in Scottish people. This was expected and supports the content validity of the questionnaire and its value as a future research tool in the Scottish population.

Background

The incidence of cutaneous melanoma is increasing in most Caucasian populations and is viewed as an increasing global public health problem[1, 2]. This is particularly so in Scotland where the incidence of melanoma is increasing rapidly, by 85% in men and 31% in women between 1983 and 2001[3, 4]. If current trends are maintained Scottish melanoma incidence rates will double every 10-20 years[5]. There is, therefore, a pressing need in Scotland to identify those most at risk of melanoma and provide them with appropriate preventative advice. Current evidence suggests those previously diagnosed with melanoma, those with a family history of melanoma, and young men of lower socioeconomic and educational status may benefit from targeted advice[6–10]. Other factors which could be important, however, such as place of residence and occupation, have yet to be explored in Scotland.

Australia has the highest worldwide incidence of melanoma (48.7 per 100,000 people in 2007, compared to 21.1 per 100,000 in Scotland)[11]. Consequently, efforts to prevent melanoma based on sun protection began in the 1960s, much earlier than elsewhere in the world[12]. Co-ordinated nationwide campaigns began with Slip! Slap! Slop! in 1980 and were followed by SunSmart, which continues at the current time, particularly emphasising the importance of reducing sun exposure in children[3, 13, 14]. Following sustained effort over three decades, formal evaluation suggests that such campaigns have contributed to changed cultural attitudes with respect to the sun and protective behaviour, and there is evidence of reduced skin cancer incidence amongst younger people born after 1960, when the campaigns first began[3, 13, 14].

In contrast melanoma prevention efforts in Scotland have been sporadic and focused on early detection and treatment rather than prevention[3, 6]. Since 2003, however, Cancer Research UK in conjunction with the UK government has run the SunSmart campaign, based upon the Australian model[6, 7]. This campaign, however, has yet to be formally evaluated and may not represent the best use of resources given the apparent urgency with which preventative behaviour would appear to need to be improved amongst those Scottish people at highest risk. To wait for cultural change to accrue over decades may not be the best way to approach Scotland's dramatically increasing melanoma incidence.

Our aim in conducting this pragmatic study was to assess the content validity of a questionnaire adapted to compare personal risk, knowledge and protective behaviour with respect to melanoma. Content validity has been defined as "whether or not the items sampled for inclusion on the tool adequately represent the domain of content addressed by the instrument"[15]. We did this by comparing responses from a convenience sample of people in Australia and Scotland. If proven valid, we believed that the questionnaire could provide an invaluable research tool in two respects: first, to identify those groups at greatest risk of developing melanoma: and second, to gauge the effect of targeted advice to high risk groups.

Methods

Setting

This study was conducted in Northeast Scotland and in Perth, Western Australia.

Ethical approval

Full ethical approval for the study was granted by the North of Scotland Research Ethics Committee (REC reference number: 08/S0801/13) on February 15 2008, and by the Human Research Ethics Committee of the University of Western Australia in April 2008.

Participants

The questionnaire was completed by a convenience sample of participants drawn from four population groups which we believed would differ with respect to melanoma risk and knowledge. The first group comprised people from Northeast Scotland that had been treated for melanoma and had participated in a randomised trial of GP-led follow-up for melanoma. The second group comprised people aged between 18 and 80 recruited through three general practices in Northeast Scotland who, according to their primary care-held medical records, had never been diagnosed with any form of skin cancer. The third group comprised people aged between 18 and 80 who attended one of three Perth-based GPs. The fourth group comprised people that had self-referred themselves to a commercial mole screening clinic in Western Australia. Australian people previously affected by melanoma and non-melanoma skin cancers were included in both the third and fourth groups.

Questionnaire and data collection

The questionnaire had been adapted by us, with permission, from an existing instrument resulting in a self-completion questionnaire comprising 28 questions[16, 17]. The first six questions gathered data on demographics and respondents' personal experience of melanoma. The remaining 22 questions collected information on four domains: respondents' personal risk of melanoma (7 questions); their level of concern about melanoma (i.e. how often they checked their own skin and how likely they were to seek medical advice about skin lesions) (4 questions); self-reported behaviour to protect themselves against melanoma (7 questions), and knowledge about melanoma (4 questions). For each of the 22 questions respondents were presented with a range of options, from which they selected one. Each individual response attracted a score. Scores within each domain were then summed to produce four overall scores, personal risk, level of concern, protective behaviour, and melanoma knowledge. The approach used to define the summary scores was intuitive, but grounded in relative effect sizes from epidemiological meta-analyses[18–24]. Because the questionnaire used multiple response items, which varied in the number of responses which people could assign, it would not have been valid to conduct conventional tests of internal consistency. The details of potential responses, the scores associated with each response, the way in which domain scores were calculated and their ranges are all shown in detail in Table 1. The questionnaire was mailed to participants in each of the four groups for self completion.

Table 1 Calculation of personal risk, level of concern, protective behaviour and knowledge summary scores from the study questionnaire

Data handling

Paper questionnaires were manually coded prior to entry into a Microsoft Access database. Data were subsequently exported into SPSS version 17.0 for statistical analysis.

Outcome measures

Outcome measures of the questionnaire are the four summary scores: protective behaviour, personal risk, level of concern and melanoma knowledge.

Statistical analysis

All statistical analysis was conducted using SPSS 17.0. Demographic variables were summarised using percentage (n) or mean (standard deviation) and compared across nationalities using either the chi square test or the unpaired t-test. In univariate analysis, since three of the summary scores (personal risk, protective behaviour and knowledge) were Normally distributed, they were compared between groups using the unpaired t-test. The other (level of concern) was non-Normally distributed and levels were compared using the Mann-Whitney U test. Multiple linear regression was used to adjust for gender, nationality, age, having a diagnosis of skin cancer, personally knowing someone with skin cancer, and educational status with each of the four summary scores (personal risk, level of concern, behaviour and knowledge scores) as the dependent variables.

Results

The demographic characteristics of those completing the questionnaire are summarised in table 2. The Scottish respondents were significantly older (p = 0.007) and of lower educational status (p < 0.001) compared with the Australian respondents. Significantly more of the Australian group personally knew someone with skin cancer (p < 0.001).

Table 2 Comparison of demographic characteristics between the Scottish and Australian participants

Table 3 summarises the univariate comparison of the four summary scores by nationality. There was no difference between Australians and Scots in personal risk score. The median level of concern summary score was significantly higher among the Australian participants (p < 0.001), as were the mean summary scores for protective behaviour (p < 0.001) and knowledge (p < 0.001). Tables 4 and 5 summarises data by a personal diagnosis of skin cancer (melanoma and non-melanoma skin cancer), and knowing someone personally with melanoma. Both Australians and Scots previously diagnosed with skin cancer (melanoma and non-melanoma skin cancer) had significantly higher personal risk and level of concern than their non-affected compatriots. In terms of protective behaviour there was no significant difference between affected and non-affected Australians, whereas Scots previously diagnosed with melanoma reported greater protective behaviour than non-affected Scots. Conversely, Australians previously diagnosed with skin cancer (melanoma and non-melanoma skin cancer) had greater knowledge than Australians not affected, but there was no significant difference between Scots irrespective of a prior melanoma diagnosis. Australians and Scots who knew someone else that had been affected by melanoma had significantly higher personal risk scores and knowledge than those who did not know someone affected with melanoma. Knowing someone with melanoma did not appear to significantly increase level of concern or protective behaviour in either Scots or Australians.

Table 3 Univariate comparison of the four summary scores by nationality

Table 4 Univariate comparison of the four summary scores by personal diagnosis of skin cancer

Table 5 Univariate comparison of summary scores by personal knowledge of someone with skin cancer

Multivariate regression analysis was then conducted to explore the effect of potential modifiers. In each case nationality, age, sex, educational status, a previous diagnosis of skin cancer (melanoma and non-melanoma skin cancer), and knowing someone else with melanoma were entered into the model (Table 6). Total model R² values were 0.191 for personal risk summary score; 0.303 for level of concern summary score; 0.179 for protective behaviour summary score, and 0.163 for knowledge summary score. Of factors that independently predicted a higher personal risk summary score, younger age, having a diagnosis of skin cancer (melanoma and non-melanoma skin cancer), personally knowing someone with melanoma and a higher level of education were all independent predictors. Neither gender nor nationality significantly predicted personal risk summary score. However, Australian nationality was an independent predictor of a higher level of concern summary score, along with having a previous diagnosis of skin cancer (melanoma and non-melanoma skin cancer), a higher level of education and lower age. Gender and personally knowing someone with melanoma did not independently predict the level of concern summary score. Australian nationality was an independent predictor of a high protective behaviour summary score, alongside older age, having had a diagnosis of skin cancer (melanoma and non-melanoma skin cancer) and a higher level of education. A higher level of education was an independent predictor of a high knowledge summary score, along with Australian nationality, personally knowing someone with skin cancer, having had a diagnosis of skin cancer (melanoma and non-melanoma skin cancer) and female gender. Age was not a significant predictor of knowledge summary score.

Table 6 Multiple linear regression of each summary score following adjustment for potential confounders (including age, gender, educational status, personal diagnosis of skin cancer, personally knowing someone with skin cancer)

Discussion

Summary of main findings

Univariate comparison of personal risk of melanoma scores by nationality revealed no difference between those Scots and Australians completing the questionnaire. In contrast, Australians' were more concerned about developing skin cancer, reported higher levels of protective behaviour and knew more about skin cancer than Scots and these differences remained following adjustment for other potential confounders. These results are what we expected to find and, therefore, appear to support the content validity of the questionnaire. Of note, the adjusted analysis suggests that in this sample, even Scots diagnosed and treated for melanoma knew less and reported less protective behaviour than unaffected Australians. Again, given the prevailing cultures in the two countries this is unsurprising, albeit somewhat concerning if should it be reproduced in a population study. The adjusted analysis also reveals that in this convenience sample, as expected, being diagnosed with any type of skin cancer increased concern, protective behaviour and knowledge. Knowing someone else diagnosed with melanoma had a lesser effect leading to increased concern and knowledge, but not self reported protective behaviour. Again, considering the convenience samples used these results are what we should have expected to find.

Context with other literature

We report a study seeking to confirm the content validity of a questionnaire in a convenience sample of participants and any comparisons with other literature must be correspondingly tempered and cautious. Nevertheless, the findings are consistent with reports that primary prevention efforts have succeeded in Australia, especially amongst younger people where increases in incidence appear to be levelling off[3]. Conversely, the view that primary prevention and education are currently suboptimal in Scotland are supported by the data, and is consistent with the continued rise in incidence here[10]. Two studies, conducted since the introduction of SunSmart in the UK in 2003, which lend further credence to our questionnaire should be mentioned here[6, 7]. Both found that young British men of lower socioeconomic and educational status knew least about skin cancer, whereas women, the elderly and those at highest risk were most likely to be concerned about it. They also reported sub-optimal practice of sun protection, although women, older people, those of higher educational status and those at highest risk appeared to be most proactive. Most worryingly, in 2005, 65% of respondents associated a sun tan with good health[6]. Perhaps in relation to this, Diffey et al (2009) reported that British people generally tend to underestimate their personal risk of skin cancer, and despite high levels of concern, as indicated by high levels of self reported self examination, continue to practice high risk activities such as sun-bathing without sunscreen, particularly younger people[7]. Although the data on self examination were encouraging, taken together these findings are all the more concerning since those taking part had a personal interest in skin cancer due to the recruitment method[7]. Both studies correspond closely to the findings presented here, further supporting the content validity of the questionnaire. Neither study explored the effect of personal experience (either having or knowing someone with skin cancer) on knowledge and behaviour. Our findings suggest that these are important and may point the way toward how educational strategies could be designed in future.

Strengths and limitations

We are quite clear that the purpose of this pragmatic study was to provide evidence of the content validity of a questionnaire to measure personal risk, knowledge and protective behaviour with respect to melanoma. We can make no claims beyond this. Participants were drawn purposively from groups between which we would have expected to have found differences in these domains. In this respect, we believe that the data appear to show the questionnaire can detect these differences. Furthermore, we believe that this has been an important exercise, since the way in which the questionnaire is designed, with multiple response items which vary the number of responses which people can assign, mean that it would not be valid to conduct conventional tests of internal consistency. Further, the questionnaire appears to have performed well amongst both Scottish and Australian people. In both populations it appears to have been understood and easily completed with very little missing data, despite a wide range of demographic characteristics. The questionnaire had been used before but the way we have used and analysed it, in two discrete geographical regions and computing four separate summary scores is novel. We believe that this method of analyzing the questionnaire can provide meaningful information on separate domains of a persons' risk of and attitudes to skin cancer and supports it use in future Scottish studies to determine where melanoma prevention advice should be targeted and subsequently evaluated.

Several limitations must be acknowledged. First and foremost, the four study groups were formed purposively as described. Therefore, although our data makes intuitive sense, we make no claims as to its external validity and generalizability to the general populations of both countries. That was not our purpose in conducting this study.

It would have been logical to have explicitly included a sample of Australian people treated for melanoma. We did not do this due to a concern that Australian people with melanoma are an over-researched group.

Data was not collected from all groups at the same time. Data was collected from Scottish melanoma patients about two years before the other Scottish group, with Australian data collection occurring at approximately the same time. Arguably, therefore this could mean that differences between the Scottish groups could have been attenuated by publicity and prevention campaigns in the interim, although differences were still striking. Also, of the people affected by skin cancer, all of the Scots had had melanoma, whereas the affected Australians comprised those affected by all skin cancers. If anything, this should have biased the study toward the Scots. With the caveat of the previously discussed methodological limitations the differences we demonstrate could, therefore, considerably underestimates the true values within the Scottish population.

Implications

Our data imply that the questionnaire used in this study has potential as a research tool to guide melanoma prevention efforts, certainly in Scotland. In future, and in carefully designed population studies, it could be administered to large numbers of people. The data thus generated could then be used to target resources for melanoma prevention in the most appropriate way. Firstly, baseline personal risk domain data could identify population groups with differential personal risk thereby allowing efforts to be targeted. In conjunction with baseline population data on level of concern, melanoma knowledge and self reported protective behaviour melanoma prevention efforts could be directly targeted where it is most needed. These data could also permit prevention efforts to be subtly tailored to achieve the maximum impact. Secondly, once prevention had been implemented and after a suitable period of time the questionnaire could be used to assess the impact of efforts, allowing subsequent redirection of effort if needed.

Conclusions

We have demonstrated that the questionnaire described provides meaningful data with respect to melanoma risk, concern, knowledge and self-reported protective behaviour and could be useful in future to target and assess the impact of melanoma prevention efforts.