A questionnaire to measure melanoma risk, knowledge and protective behaviour: Assessing content validity in a convenience sample of Scots and Australians
BMC Medical Research Methodology volume 11, Article number: 123 (2011)
The aim of this study was to assess the content validity of a questionnaire to measure melanoma risk, knowledge and protective behaviour in a convenience sample of Scots and Australians. Australia has the highest melanoma incidence worldwide but has developed a culture of skin cancer avoidance with a long history of skin cancer primary prevention campaigns of proven effectiveness. Scotland has lower incidence, but has shown a greater rate of increase between 1985 and 2007. There is an urgent need in Scotland, therefore, to identify those groups at greatest risk and provide them with effective preventative advice.
A self-administered postal survey was completed by four groups formed from convenience samples in two geographical locations (Northeast Scotland and Western Australia). In univariate analysis scores on personal risk, level of concern, protective behaviour, and knowledge were compared by nationality, previous skin cancer diagnosis and personally knowing someone with melanoma. Multivariate linear regression analysis modelled the influence of potential predictor variables upon each of the scores.
540 people completed the questionnaire, 273 Scots (50.6%). 133 (24.6%) Scots and 83 (15.4%) Australians previously had melanoma or non-melanoma skin cancer, whilst 120 (22.2%) Scots and 190 (35.2%) Australians personally knew someone with melanoma. Australians had higher knowledge (p < 0.001), level of concern (p < 0.001) and protective behaviour (p < 0.001) scores than the Scottish. Australian nationality was the strongest independent predictor of a higher knowledge score (p < 0.001), followed by a previous skin cancer diagnosis (p = 0.003), personal knowledge of someone with melanoma (p = 0.011), female gender (p = 0.005) and higher education status (p < 0.001) (R 2 = 0.163).
The questionnaire detected higher levels of knowledge and skin cancer protective behaviours in Australians than in Scottish people. This was expected and supports the content validity of the questionnaire and its value as a future research tool in the Scottish population.
The incidence of cutaneous melanoma is increasing in most Caucasian populations and is viewed as an increasing global public health problem[1, 2]. This is particularly so in Scotland where the incidence of melanoma is increasing rapidly, by 85% in men and 31% in women between 1983 and 2001[3, 4]. If current trends are maintained Scottish melanoma incidence rates will double every 10-20 years. There is, therefore, a pressing need in Scotland to identify those most at risk of melanoma and provide them with appropriate preventative advice. Current evidence suggests those previously diagnosed with melanoma, those with a family history of melanoma, and young men of lower socioeconomic and educational status may benefit from targeted advice[6–10]. Other factors which could be important, however, such as place of residence and occupation, have yet to be explored in Scotland.
Australia has the highest worldwide incidence of melanoma (48.7 per 100,000 people in 2007, compared to 21.1 per 100,000 in Scotland). Consequently, efforts to prevent melanoma based on sun protection began in the 1960s, much earlier than elsewhere in the world. Co-ordinated nationwide campaigns began with Slip! Slap! Slop! in 1980 and were followed by SunSmart, which continues at the current time, particularly emphasising the importance of reducing sun exposure in children[3, 13, 14]. Following sustained effort over three decades, formal evaluation suggests that such campaigns have contributed to changed cultural attitudes with respect to the sun and protective behaviour, and there is evidence of reduced skin cancer incidence amongst younger people born after 1960, when the campaigns first began[3, 13, 14].
In contrast melanoma prevention efforts in Scotland have been sporadic and focused on early detection and treatment rather than prevention[3, 6]. Since 2003, however, Cancer Research UK in conjunction with the UK government has run the SunSmart campaign, based upon the Australian model[6, 7]. This campaign, however, has yet to be formally evaluated and may not represent the best use of resources given the apparent urgency with which preventative behaviour would appear to need to be improved amongst those Scottish people at highest risk. To wait for cultural change to accrue over decades may not be the best way to approach Scotland's dramatically increasing melanoma incidence.
Our aim in conducting this pragmatic study was to assess the content validity of a questionnaire adapted to compare personal risk, knowledge and protective behaviour with respect to melanoma. Content validity has been defined as "whether or not the items sampled for inclusion on the tool adequately represent the domain of content addressed by the instrument". We did this by comparing responses from a convenience sample of people in Australia and Scotland. If proven valid, we believed that the questionnaire could provide an invaluable research tool in two respects: first, to identify those groups at greatest risk of developing melanoma: and second, to gauge the effect of targeted advice to high risk groups.
This study was conducted in Northeast Scotland and in Perth, Western Australia.
Full ethical approval for the study was granted by the North of Scotland Research Ethics Committee (REC reference number: 08/S0801/13) on February 15 2008, and by the Human Research Ethics Committee of the University of Western Australia in April 2008.
The questionnaire was completed by a convenience sample of participants drawn from four population groups which we believed would differ with respect to melanoma risk and knowledge. The first group comprised people from Northeast Scotland that had been treated for melanoma and had participated in a randomised trial of GP-led follow-up for melanoma. The second group comprised people aged between 18 and 80 recruited through three general practices in Northeast Scotland who, according to their primary care-held medical records, had never been diagnosed with any form of skin cancer. The third group comprised people aged between 18 and 80 who attended one of three Perth-based GPs. The fourth group comprised people that had self-referred themselves to a commercial mole screening clinic in Western Australia. Australian people previously affected by melanoma and non-melanoma skin cancers were included in both the third and fourth groups.
Questionnaire and data collection
The questionnaire had been adapted by us, with permission, from an existing instrument resulting in a self-completion questionnaire comprising 28 questions[16, 17]. The first six questions gathered data on demographics and respondents' personal experience of melanoma. The remaining 22 questions collected information on four domains: respondents' personal risk of melanoma (7 questions); their level of concern about melanoma (i.e. how often they checked their own skin and how likely they were to seek medical advice about skin lesions) (4 questions); self-reported behaviour to protect themselves against melanoma (7 questions), and knowledge about melanoma (4 questions). For each of the 22 questions respondents were presented with a range of options, from which they selected one. Each individual response attracted a score. Scores within each domain were then summed to produce four overall scores, personal risk, level of concern, protective behaviour, and melanoma knowledge. The approach used to define the summary scores was intuitive, but grounded in relative effect sizes from epidemiological meta-analyses[18–24]. Because the questionnaire used multiple response items, which varied in the number of responses which people could assign, it would not have been valid to conduct conventional tests of internal consistency. The details of potential responses, the scores associated with each response, the way in which domain scores were calculated and their ranges are all shown in detail in Table 1. The questionnaire was mailed to participants in each of the four groups for self completion.
Paper questionnaires were manually coded prior to entry into a Microsoft Access database. Data were subsequently exported into SPSS version 17.0 for statistical analysis.
Outcome measures of the questionnaire are the four summary scores: protective behaviour, personal risk, level of concern and melanoma knowledge.
All statistical analysis was conducted using SPSS 17.0. Demographic variables were summarised using percentage (n) or mean (standard deviation) and compared across nationalities using either the chi square test or the unpaired t-test. In univariate analysis, since three of the summary scores (personal risk, protective behaviour and knowledge) were Normally distributed, they were compared between groups using the unpaired t-test. The other (level of concern) was non-Normally distributed and levels were compared using the Mann-Whitney U test. Multiple linear regression was used to adjust for gender, nationality, age, having a diagnosis of skin cancer, personally knowing someone with skin cancer, and educational status with each of the four summary scores (personal risk, level of concern, behaviour and knowledge scores) as the dependent variables.
The demographic characteristics of those completing the questionnaire are summarised in table 2. The Scottish respondents were significantly older (p = 0.007) and of lower educational status (p < 0.001) compared with the Australian respondents. Significantly more of the Australian group personally knew someone with skin cancer (p < 0.001).
Table 3 summarises the univariate comparison of the four summary scores by nationality. There was no difference between Australians and Scots in personal risk score. The median level of concern summary score was significantly higher among the Australian participants (p < 0.001), as were the mean summary scores for protective behaviour (p < 0.001) and knowledge (p < 0.001). Tables 4 and 5 summarises data by a personal diagnosis of skin cancer (melanoma and non-melanoma skin cancer), and knowing someone personally with melanoma. Both Australians and Scots previously diagnosed with skin cancer (melanoma and non-melanoma skin cancer) had significantly higher personal risk and level of concern than their non-affected compatriots. In terms of protective behaviour there was no significant difference between affected and non-affected Australians, whereas Scots previously diagnosed with melanoma reported greater protective behaviour than non-affected Scots. Conversely, Australians previously diagnosed with skin cancer (melanoma and non-melanoma skin cancer) had greater knowledge than Australians not affected, but there was no significant difference between Scots irrespective of a prior melanoma diagnosis. Australians and Scots who knew someone else that had been affected by melanoma had significantly higher personal risk scores and knowledge than those who did not know someone affected with melanoma. Knowing someone with melanoma did not appear to significantly increase level of concern or protective behaviour in either Scots or Australians.
Multivariate regression analysis was then conducted to explore the effect of potential modifiers. In each case nationality, age, sex, educational status, a previous diagnosis of skin cancer (melanoma and non-melanoma skin cancer), and knowing someone else with melanoma were entered into the model (Table 6). Total model R2 values were 0.191 for personal risk summary score; 0.303 for level of concern summary score; 0.179 for protective behaviour summary score, and 0.163 for knowledge summary score. Of factors that independently predicted a higher personal risk summary score, younger age, having a diagnosis of skin cancer (melanoma and non-melanoma skin cancer), personally knowing someone with melanoma and a higher level of education were all independent predictors. Neither gender nor nationality significantly predicted personal risk summary score. However, Australian nationality was an independent predictor of a higher level of concern summary score, along with having a previous diagnosis of skin cancer (melanoma and non-melanoma skin cancer), a higher level of education and lower age. Gender and personally knowing someone with melanoma did not independently predict the level of concern summary score. Australian nationality was an independent predictor of a high protective behaviour summary score, alongside older age, having had a diagnosis of skin cancer (melanoma and non-melanoma skin cancer) and a higher level of education. A higher level of education was an independent predictor of a high knowledge summary score, along with Australian nationality, personally knowing someone with skin cancer, having had a diagnosis of skin cancer (melanoma and non-melanoma skin cancer) and female gender. Age was not a significant predictor of knowledge summary score.
Summary of main findings
Univariate comparison of personal risk of melanoma scores by nationality revealed no difference between those Scots and Australians completing the questionnaire. In contrast, Australians' were more concerned about developing skin cancer, reported higher levels of protective behaviour and knew more about skin cancer than Scots and these differences remained following adjustment for other potential confounders. These results are what we expected to find and, therefore, appear to support the content validity of the questionnaire. Of note, the adjusted analysis suggests that in this sample, even Scots diagnosed and treated for melanoma knew less and reported less protective behaviour than unaffected Australians. Again, given the prevailing cultures in the two countries this is unsurprising, albeit somewhat concerning if should it be reproduced in a population study. The adjusted analysis also reveals that in this convenience sample, as expected, being diagnosed with any type of skin cancer increased concern, protective behaviour and knowledge. Knowing someone else diagnosed with melanoma had a lesser effect leading to increased concern and knowledge, but not self reported protective behaviour. Again, considering the convenience samples used these results are what we should have expected to find.
Context with other literature
We report a study seeking to confirm the content validity of a questionnaire in a convenience sample of participants and any comparisons with other literature must be correspondingly tempered and cautious. Nevertheless, the findings are consistent with reports that primary prevention efforts have succeeded in Australia, especially amongst younger people where increases in incidence appear to be levelling off. Conversely, the view that primary prevention and education are currently suboptimal in Scotland are supported by the data, and is consistent with the continued rise in incidence here. Two studies, conducted since the introduction of SunSmart in the UK in 2003, which lend further credence to our questionnaire should be mentioned here[6, 7]. Both found that young British men of lower socioeconomic and educational status knew least about skin cancer, whereas women, the elderly and those at highest risk were most likely to be concerned about it. They also reported sub-optimal practice of sun protection, although women, older people, those of higher educational status and those at highest risk appeared to be most proactive. Most worryingly, in 2005, 65% of respondents associated a sun tan with good health. Perhaps in relation to this, Diffey et al (2009) reported that British people generally tend to underestimate their personal risk of skin cancer, and despite high levels of concern, as indicated by high levels of self reported self examination, continue to practice high risk activities such as sun-bathing without sunscreen, particularly younger people. Although the data on self examination were encouraging, taken together these findings are all the more concerning since those taking part had a personal interest in skin cancer due to the recruitment method. Both studies correspond closely to the findings presented here, further supporting the content validity of the questionnaire. Neither study explored the effect of personal experience (either having or knowing someone with skin cancer) on knowledge and behaviour. Our findings suggest that these are important and may point the way toward how educational strategies could be designed in future.
Strengths and limitations
We are quite clear that the purpose of this pragmatic study was to provide evidence of the content validity of a questionnaire to measure personal risk, knowledge and protective behaviour with respect to melanoma. We can make no claims beyond this. Participants were drawn purposively from groups between which we would have expected to have found differences in these domains. In this respect, we believe that the data appear to show the questionnaire can detect these differences. Furthermore, we believe that this has been an important exercise, since the way in which the questionnaire is designed, with multiple response items which vary the number of responses which people can assign, mean that it would not be valid to conduct conventional tests of internal consistency. Further, the questionnaire appears to have performed well amongst both Scottish and Australian people. In both populations it appears to have been understood and easily completed with very little missing data, despite a wide range of demographic characteristics. The questionnaire had been used before but the way we have used and analysed it, in two discrete geographical regions and computing four separate summary scores is novel. We believe that this method of analyzing the questionnaire can provide meaningful information on separate domains of a persons' risk of and attitudes to skin cancer and supports it use in future Scottish studies to determine where melanoma prevention advice should be targeted and subsequently evaluated.
Several limitations must be acknowledged. First and foremost, the four study groups were formed purposively as described. Therefore, although our data makes intuitive sense, we make no claims as to its external validity and generalizability to the general populations of both countries. That was not our purpose in conducting this study.
It would have been logical to have explicitly included a sample of Australian people treated for melanoma. We did not do this due to a concern that Australian people with melanoma are an over-researched group.
Data was not collected from all groups at the same time. Data was collected from Scottish melanoma patients about two years before the other Scottish group, with Australian data collection occurring at approximately the same time. Arguably, therefore this could mean that differences between the Scottish groups could have been attenuated by publicity and prevention campaigns in the interim, although differences were still striking. Also, of the people affected by skin cancer, all of the Scots had had melanoma, whereas the affected Australians comprised those affected by all skin cancers. If anything, this should have biased the study toward the Scots. With the caveat of the previously discussed methodological limitations the differences we demonstrate could, therefore, considerably underestimates the true values within the Scottish population.
Our data imply that the questionnaire used in this study has potential as a research tool to guide melanoma prevention efforts, certainly in Scotland. In future, and in carefully designed population studies, it could be administered to large numbers of people. The data thus generated could then be used to target resources for melanoma prevention in the most appropriate way. Firstly, baseline personal risk domain data could identify population groups with differential personal risk thereby allowing efforts to be targeted. In conjunction with baseline population data on level of concern, melanoma knowledge and self reported protective behaviour melanoma prevention efforts could be directly targeted where it is most needed. These data could also permit prevention efforts to be subtly tailored to achieve the maximum impact. Secondly, once prevention had been implemented and after a suitable period of time the questionnaire could be used to assess the impact of efforts, allowing subsequent redirection of effort if needed.
We have demonstrated that the questionnaire described provides meaningful data with respect to melanoma risk, concern, knowledge and self-reported protective behaviour and could be useful in future to target and assess the impact of melanoma prevention efforts.
MacKie R, Bray C, Vestey J, Doherty V, Evans A, Thomson D, Nicolson M: Melanoma incidence and mortality in Scotland 1979-2003. Brit J Cancer. 2007, 96: 1772-1777. 10.1038/sj.bjc.6603801.
Health Protection Agency: Health Protection in the 21st Century: Understanding the burden of disease, preparing for the future. 2005, Health Protection Agency, London
Whiteman DC, Bray CA, Siskind V, Green AC, Hole DJ, MacKie RM: Changes in the incidence of cutaneous melanoma in the west of Scotland and Queensland, Australia: hope for health promotion?. Eur J Cancer Prev. 2008, 17: 243-250. 10.1097/CEJ.0b013e3282b6fe3f.
ISD Scotland: Malignant melanoma of the skin: ICD-10 C43. Annual Incidence. 2007. 2011, Accessed February 28th, 2011, [http://www.isdscotland.org/isd/1048.html]
Garbe C, Leiter U: Melanoma epidemiology and trends. Clin Dermatol. 2009, 27: 3-9. 10.1016/j.clindermatol.2008.09.001.
Miles A, Waller J, Hiom S, Swanston D: SunSmart? Skin cancer knowledge and preventive behaviour in a British population representative sample. Health Educ Res. 2005, 20: 579-585. 10.1093/her/cyh010.
Diffey B, Norridge Z: Reported sun exposure, attitudes to sun protection and perceptions of skin cancer risk: a survey of visitors to Cancer Research UK's SunSmart campaign website. Brit J Dermatol. 2009, 160: 1292-1298. 10.1111/j.1365-2133.2009.09149.x.
Geller AC, Emmons K, Brooks DR, Zhang Z, Powers C, Koh HK, Sober AJ, Miller Dr, Li F, Haluska F, Gilchrest BA: Skin cancer prevention and detection practices among siblings of patients with melanoma. Am Acad Dermatol. 2003, 49: 631-638. 10.1067/S0190-9622(03)02126-1.
Geller AC, Brooks DR, Colditz GA, Koh HK, Frazier AL: Sun protection practices among offspring of women with personal or family history of skin cancer. Pediatrics. 2006, 117: e688-10.1542/peds.2005-1734.
MacKie R, Hauschild A, Eggermont A: Epidemiology of invasive cutaneous melanoma. Ann Oncol. 2009, 20 (Supplement 6):
Australian Government: Australian Institute of Health and Wellfare. 2011, Accessed 17th May 2011, [http://www.aihw.gov.au/acim-books/]
Coory M, Baade P, Aitken J, Smithers M, McLeod GRC, Ring I: Trends for in situ and invasive melanoma in Queensland, Australia, 1982-2002. Cancer Cause Control. 2006, 17: 21-27. 10.1007/s10552-005-3637-4.
Sinclair C, Foley P: Skin cancer prevention in Australia. Brit J Dermatol. 2009, 161: 116-123.
Dobbinson SJ, Wakefield MA, Jamsen KM, Herd NL, Spittal MJ, Lipscomb JE, Hill DJ: Weekend Sun Protection and Sunburn in Australia: Trends (1987-2002) and Association with SunSmart Television Advertising. Am J Prev Med. 2008, 34: 171-172. 10.1016/j.amepre.2007.11.002.
Waltz CF, Strickland OL, Lenz ER: Measurement in nursing and health research. 2005, New York: Springer Publishing Co., 3
Bataille V, de Vries E: Melanoma Part 1: Epidemiology, risk factors, and prevention. Brit Med J. 2008, 337: a2249-10.1136/bmj.a2249.
Jackson A, Wilkinson C, Pill R: Moles and melanomas - who's at risk, who knows, and who cares? A strategy to inform those at high risk. Brit J Gen Pract. 1999, 49: 199-203.
Murchie P, Iweuke FC: Comparing personal risk; melanoma knowledge and protective behaviour in people with and without melanoma: a postal survey to explore educational needs in North East Scotland. Journal of Cancer Education. 2010
Bataille V, Boniol M, De Vries ED, Severi G, Brandenberg Y, Sasieni P, Cuzick J, Eggermont A, Ringborg U, Grivegnée AR, Coebergh JW, Chignol MC, Doré JF, Autier P: A multicentre epidemiological study on sun bed use and cutaneous melanoma in Europe. Eur J Cancer. 2005, 41: 2141-2149. 10.1016/j.ejca.2005.04.038.
Gandini S, Sera F, Cattaruzza MS, Pasquini P, Abeni D, Boyle P, Melchi CF: Meta-analysis of risk factors for cutaneous melanoma: I. Common and atypical naevi. Eur J Cancer. 2005, 41: 28-44. 10.1016/j.ejca.2004.10.015.
Gandini S, Sera F, Cattaruzza MS, Pasquini P, Picconi O, Boyle P, Melchi CF: Meta-analysis of risk factors for cutaneous melanoma: II. Sun exposure. Eur J Cancer. 2005, 41: 45-60. 10.1016/j.ejca.2004.10.016.
Gandini S, Sera F, Cattaruzza MS, Pasquini P, Zanetti R, Masini C, Boyle P, Melchi CF: Meta-analysis of risk factors for cutaneous melanoma: III. Family history, actinic damage and phenotypic factors. Eur J Cancer. 2005, 41: 2040-2059. 10.1016/j.ejca.2005.03.034.
Pack JT, David J, Oppenheim A: The relation of race and complexion to the incidence of moles and melanoma. Ann NY Acad Sci. 2009, 100: 719-742.
Pfahlberg A, Kolmel K-F, Gefeller O, the Febim Study Group: Timing of excessive ultraviolet radiation and melanoma: epidemiology does not support the existence of a critical period of high susceptibility to solar ultraviolet radiation-induced melanoma. Brit J Dermatol. 2001, 144: 471-475. 10.1046/j.1365-2133.2001.04070.x.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/11/123/prepub
We wish to acknowledge the assistance and participation of general practitioners and their staff in Northeast Scotland and Western Australia. We are grateful to Dr Arthur Jackson for permission to adapt his original questionnaire. We are grateful to Mr Franklin Iwueke for inputting the questionnaire data. We wish to acknowledge the Scottish School of Primary Care who funded this project which was completed as part of Helen S Gillespie's BSc (Med Sci) thesis. We also acknowledge the University of Aberdeen who is the study sponsor.
The authors declare that they have no competing interests.
PM designed the study. JE and TW co-ordinated data collection in Western Australia, PM conducted data collection in northeast Scotland. HG conducted the analysis of the data under the direction of AL. HG and PM wrote the paper with comments on drafts from JE, TW and AL. All authors have read and approved the final manuscript.
About this article
Cite this article
Gillespie, H.S., Watson, T., Emery, J.D. et al. A questionnaire to measure melanoma risk, knowledge and protective behaviour: Assessing content validity in a convenience sample of Scots and Australians. BMC Med Res Methodol 11, 123 (2011). https://doi.org/10.1186/1471-2288-11-123
- Skin Cancer
- Summary Score
- Personal Risk
- Protective Behaviour