- Research article
- Open Access
A systematic review of studies with a representative sample of refugees and asylum seekers living in the community for participation in mental health research
BMC Medical Research Methodology volume 17, Article number: 37 (2017)
The aim was to review the literature to identify the most effective methods for creating a representative sample of refugee and asylum seeker groups living in the community to participate in health and mental health survey research.
A systematic search of academic and grey literature was conducted for relevant literature with ‘hidden’ groups published between January 1995 and January 2016. The main search used Medline, PsycINFO, EMBASE, CINAHL and SCOPUS electronic databases. Hidden groups were defined as refugees, asylum seekers, stateless persons or hard/difficult to reach populations. A supplementary grey literature search was conducted. Identified articles were rated according to a created graded system of ‘level of evidence for a community representative sample’ based on key study factors that indicated possible sources of selection bias. Articles were included if they were assessed as having medium or higher evidence for a representative sample. All full-text papers that met the eligibility criteria were examined in detail and relevant data extracted.
The searches identified a total of 20 publications for inclusion: 16 peer-reviewed publications and four highly relevant reports. Seventeen studies had sampled refugee and asylum seekers and three other hidden groups. The main search identified 12 (60.0%) and the grey search identified another eight (40.0%) articles. All 20 described sampling techniques for accessing hidden groups for participation in health-related research. Key design considerations were: an a priori aim to recruit a representative sample; a reliable sampling frame; recording of response rates; implementation of long recruitment periods; using multiple non-probability sampling methods; and, if possible, including a probability sampling component. Online social networking sites were used by one study. Engagement with the refugee and asylum seeker group was universally endorsed in the literature as necessary and a variety of additional efforts to do this were reported.
The strategies for increasing the likelihood of a representative sample of this hidden group were identified and will assist researchers when doing future research with refugee groups. These findings encourage more rigorous reporting of future studies so that the representativeness of samples of these groups in research can be more readily assessed.
Population health surveys are typically used to determine health status, healthcare needs and health service utilization patterns in particular populations. However, despite efforts to ensure findings are representative of the population of interest, certain “hidden” groups are inevitably excluded - people of refugee or asylum seeker (RAS) backgrounds being one such group [1–5]. Hidden groups present special challenges for the sample process including representative sampling and coverage, identification, contact, recruitment, and data collection [1–10]. Mental health issues are a problem for RAS populations worldwide [11, 12] and this can compound sampling challenges. There are likely to be issues around trust and safety, particularly for those with a history of torture and trauma, as well as concerns about stigma and privacy – especially if refugee status is undetermined. Low rates of health literacy in some refugees may also impact on the willingness and capacity to be involved in research and disclose information related to mental health. There is little high quality published evidence about mental health and health services use among RAS groups [13, 14]. Understanding the mental health needs of RAS groups living in the community, especially within communities with large RAS populations, is needed to inform service delivery to this vulnerable group .
Representative samples are subgroups of people that contain all the elements of interest from a target population . The sample frame represents a list of the target population from which the sample is selected, and ideally contains all elements in the target population. Sometimes the frame can consist of the entire target population, but this is uncommon. The sampling frame should be clearly defined and have measurable characteristics before a representative subgroup is sought. Gender representative samples, for instance, will endeavour to match the proportion of each gender as in the target population. The representativeness of the sample depends on the quality of the sampling frame . The lack of a sampling frame or rapidly changing frames for many RAS groups is a known barrier to conducting research with RAS populations [1–6].
By definition, a reliable frame and a representative sampling mechanism cannot be easily established for hidden groups . Inherent problems within these two important aspects of sampling are sources of selection bias . Even if the relevant frame is understood, for example by using a host country immigration records, methods to select RAS participants are often not conducive to representative sampling. Sampling techniques commonly used are those that promote participation in known RAS individuals such as convenience sampling, e.g., research participants selected because they are known to the researchers, or snowballing, e.g., research participants are sought through chain-referral by other research participants [2, 10]. Generally convenience and snowball samples are non-representative because the sampling coverage is limited to the contact circles of certain people and are thus subject to selection bias [10, 17].
This systematic review aimed to identify methods to achieve representative samples of RAS groups living in the community for participation in mental health survey research. However, since methods could be transferred from research seeking representative samples from other at risk groups that are characterized as hidden such as men who have sex with men [18, 19], this review also includes health-related research involving participants from other hidden groups, but only if the sampling methods were suggested as potentially transferable to other groups. Motivation for individuals to participate in health research can be different if a service is offered in exchange for participation ; for example, survey research on oral health behaviours that incentivizes participation using a free dental examination is likely to recruit individuals who want a dental examination. Whereas research involving only surveys about dental hygiene, without a service in exchange, might not attract the same participants. This systematic review concentrated on the latter strategy. The question that we aimed to answer through this systematic review was, “What are the most effective methods for creating a representative sample of RAS living in the community, to participate in health and mental health survey research?”
We followed the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement for conducting and reporting a systematic review [17, 21]. A systematic search of both academic and grey literature identified available studies that met the inclusion criteria. To ensure a comprehensive representation of the literature, we included papers that used qualitative, quantitative, mixed methods and case study methodologies, and cross-sectional, cohort, experimental and observational designs. We did not restrict the search to population surveys because these studies are resource-heavy and infrequently conducted within hidden groups. The review processes are given below. Also see the section describing author contributions for further details of who undertook the review tasks.
Main search strategy – peer-reviewed literature
Searches of the Medline, PsycINFO, EMBASE, CINAHL and SCOPUS electronic databases were conducted for English language papers published between January 1995 and January 2016. The following medical subject headings and keywords were sought: [“refugee*” OR “asylum seeker*” OR “stateless person*” OR “difficult to reach” OR “hard-to-reach” OR “hidden population*”] AND [“sampling”, “recruitment” OR “surveying” OR “sampling studies”]. Note the *asterisk indicated a word that was truncated during the search. The search purposely included papers not indexed with a RAS term, in case relevant papers had included this information elsewhere and was inclusive of all age groups, not restricted to ‘adults’ because this may have excluded papers that did not specify the age group of their sample. In addition, manual checks of the reference lists of retrieved papers and citation searches were conducted.
The initial searches were performed in January 2015 and subsequently re-run in PubMed and PsycINFO in January 2016 to identify additional relevant studies published in 2015. No additional studies were identified.
Initial inclusion criteria
To pass an initial screen, abstracts and titles needed to contain enough information to indicate that the study had focussed on health research and referred to methods of recruitment for hidden groups. Hidden groups were defined as refugees, asylum seekers, stateless persons or hard/difficult to reach populations such as men who have sex with men [18, 19].
Full text articles were retrieved for all records that passed the screen, or if exclusion could not be determined during the screen. Full text articles were then examined and met the eligibility for analysis criteria if they were:
Published in the English language
Peer-review publication (this condition was waived for the grey literature search)
Primary article providing original data
Focus on health and a community sample (not in-patient, etc.)
Focus on adults
Published January 1995 – January 2016
Study sample frame is a hidden population
Sample of RAS or another hidden sample with methods described to be potentially transferable to RAS
Reports the sampling technique in sufficient detail to replicate sampling
Focus on original data collection (not census, hospital administrative data etc.)
Articles were excluded if they were classified as an incomplete article (e.g., conference abstract, editorial, commentary or letter); offered a service to study participants; reported data already used in another included article; or were review articles.
Final inclusion criteria – and study quality assessment
Full text articles that met the initial inclusion criteria then underwent an assessment for study quality which consisted of an analysis for level of evidence for obtaining a representative sample. The final inclusion criteria required studies to have achieved a medium-high level evidence of obtaining a representative sample. This quality assessment process is described immediately below and summarised in Table 1.
Determining the level of evidence for a representative sample involved a quality assessment of each study for potential sources of bias. This assessment was performed by two authors (JE and KB) and differences were resolved through discussion and consensus. Sources of selection bias can particularly compromise the establishment of a representative sample . Other sources of bias exist, but the key obstacles to initially overcome are those related to selection bias. Our judgement criteria for assessing selection bias was adapted from Higgins and Green . We produced a graded system of ‘level of evidence for a community representative sample’ based on key study factors that indicated possible sources of selection bias . Table 1 outlines this graded system.
Grey literature search strategy
The novel, and we think unique, grey literature search parameters described below were devised from input from an expert advisory committee that included senior researchers experienced in the sourcing of refugee publications.
The systematic search of grey literature centred on 23 countries: Australia, Canada and United States of America plus the top 20 countries in the UNHCR global rankings of highest refugee “third country resettlement” intake per 1000 inhabitants in 2010 , (see Additional file 1: Appendix A). For each of these 23 countries, a maximum of 2 hours per country was allocated to search the Internet for relevant literature. Instructions given to the research assistants who performed this task were to: identify the website for the Department of Health in that country and spend a maximum of one hour searching this website using the same search terms as in the main search; and next, identify any website for the statistical department in that country and again spend a maximum of 1 hour searching this website for relevant articles.
This search was supplemented with a general Internet search using Google and Google Scholar. In addition, we asked the investigators to identify relevant sources of literature that could be in the form of: websites, newsletters (online or print), reports (online or print), annual reports, research or quality assurance reports, any persons that had sampled hidden refugee and asylum seeker populations, and any another relevant contact person. Free text searching was implemented using the same search terms as in the main search. Grey literature were subjected to the same inclusion criteria described above.
Papers identified as eligible for analysis were read and key information extracted by research assistants. This included the study focus (mental health or health), design, study setting, target sample size and descriptors of the study’s target population. Other extracted data consisted of information from all stages of a research study where representativeness may be threatened: key sources of potential selection bias such as the development or defining of a sampling frame; random (or non-random) selection components; recruitment and sampling methods/considerations/techniques; the barriers to participation in health research; strategies implemented to improve participation; and response rates and attrition.
Only findings reported in the original publications or publications using the same study data were used for extraction. Authors were not contacted for additional information.
Meetings with the research assistants occurred regularly, and any discordance during the search, extraction and assessment tasks was resolved by a consensus panel, which included the research assistant(s) and two senior authors (FS & JE).
The searches identified a total of 20 publications for inclusion. A summary of the search strategy is shown in Fig. 1, and details of the separate main and grey searches are available in Additional files 2 and 3: Appendices B and C, respectively. As per the eligibility criteria, all 20 had achieved a community representative sample of a hidden group, where the level of evidence for a representative sample had been rated by the reviewers as medium or higher [18, 19, 23–40]. Seventeen had specifically sampled RAS and three involved another hidden group [18, 19, 38]; see Table 2. All 20 publications described sampling techniques for accessing hidden groups for participation in health-related research (see Table 2).
Main search and separate grey search
In the main search, 1857 records were initially identified in electronic databases and another 29 were recovered from reference checks, see Fig. 1 and Additional file 2: Appendix B. After removing duplicates, 893 records underwent abstract/title screening, of which 238 progressed to full-text examination for eligibility. Reasons for failing the full-text examination are indicated in Additional file 2 Appendix B, and only 36 progressed to undergo assessment by the reviewers for having a representative sample(s). A further 24 were excluded because these studies failed the criteria for having medium to high level evidence of representative samples. Therefore, there were 12 included publications identified in the main search.
In the grey search, 92 records were initially identified and underwent abstract/title screening, of which 52 progressed to full-text examination for eligibility (see Additional file 3: Appendix C). Reasons for failing the full-text examination are indicated in Additional file 3: Appendix C, and 21 progressed to undergo assessment by the reviewers for a representative sample(s). A further 13 were excluded because these studies failed the criteria for having medium-high level evidence of representative samples. Therefore, the grey search identified eight publications.
Of the 20 publications included in this review, the main search identified 12 papers (60.0%) and the grey literature search identified another eight papers (40.0%). Half of the publications identified from the grey search were peer-reviewed publications (4/8, 50.0%), see Tables 2 and 3. Overall, this review includes 16 peer-reviewed publications [18, 19, 23–27, 31, 34, 36, 38–40], one non-peer reviewed protocol report , and three governmental reports [28, 29, 37]; see Table 2. The latter four reports were identified during the grey literature search.
The peer-reviewed publication by Ao et al.  is shown together with the non-peer reviewed publication by Cochran et al.  in Table 2, because the former was only identified after the latter publication was found during the grey search; both describe the same study and Ao et al.  was not identified in the main search.
Evidence level of a community representative sample
Only four studies (20.0%, 4/20) were rated as having ‘high’ level of evidence for achieving a community representative sample for a hidden group; therefore these four studies were judged to have ‘low risk’ of selection bias, see Tables 1 and 3. These high quality representative samples were from a Canadian study consisting of 340 Ethiopian refugee migrants , and three large-scale Australian studies with RAS groups [30, 37, 39]. Two of these four studies were identified in the grey search only and are non-peer reviewed articles [30, 37].
Another six studies with refugees were graded as having medium-high evidence of a representative sample [23, 28, 29, 33, 36, 40] and the remaining ten studies were rated as demonstrating medium evidence; seven of these had sampled RAS [24–27, 31, 34, 35] and the remaining three had sampled other hidden groups [18, 19, 38].
Probability (random) sampling procedures were used in 50.0% (10/20) of studies [23–28, 32, 33, 39]; see Table 3. Three studies did not use any random strategies but instead attempted to invite all eligible participants from within the defined sample frame; two of these were rated high for a representative sample and were large studies involving Australian refugees [30, 37]; and the third rated medium was a Swiss study that had sampled consecutively from a national register of adult asylum seekers . Another study used systematic sampling of every nth name from a sampling frame , which is not strictly ‘random’ sampling. Yet another study applied multistage stratified and quota sampling, but because the authors did not specify if there was a random component in the sampling, it remains undetermined whether probability method(s) were employed . The remaining six studies used only non-probability sampling methods [18, 19, 31, 35, 38, 40].
Networked-based sampling techniques were described in nine (45.0%) of the 20 reviewed studies; six included snowballing (30.0%) [25, 27, 31, 32, 35, 40], four purposive/convenience sampling (20.0%) [18, 23, 24, 40], two used respondent driven sampling (RDS; 10.0%) [19, 38], and one utilized online sampling via social media platforms such as Facebook . Four studies applied some type of probability (random) sampling methods and supplemented this with non-probability sampling, such as snowballing or convenience sampling [23–25, 27]; see Table 3.
Other design issues
Table 3 summarizes the sampling frame, sampling methods and other sampling considerations reported in the 20 included studies. More than half of the studies (65.0%, 13/20) expressed an a priori aim to approximate a representative sample [19, 23, 28, 30, 31, 33, 35–40]. Table 4 summarizes further information on the 20 studies focusing on possible barriers and identified threats to representative samples. Long recruitment periods of between 12 and 25 months were noted to facilitate recruitment from hidden groups in four studies [24, 27, 32, 40]. Weighting methods can be used to adjust the obtained sample to be representative of the target population, and were reported in five studies [25, 30, 37–39].
The reviewed studies demonstrate that it is possible to achieve a representative sample in RAS groups using either (or both) probability or non-probability sampling techniques, if the following requirements are met: a) engaging the target group, and b) key research design considerations. Both of these elements are discussed in reference to examples from the 20 studies in this review. We will also discuss issues reported in these 20 studies regarding the barriers to representative sampling, and suggest strategies for overcoming these barriers.
Engagement with the target group
Engagement with the target group was universally identified as necessary for creating a representative community sample of hidden groups, including RAS. Engagement strategies included developing culturally responsive translated materials [23–26, 28, 30–35, 37, 39, 40], ongoing active engagement with target community members and leaders [24, 32, 35, 40], field-workers who spoke the language [23, 30, 32, 33, 35, 37, 39, 40] or were members of the target community [24, 32, 35], recruitment and site visits after hours and weekends [24, 40], and conducting the research at multiple sites to address travel limitations [27, 38].
Key research design considerations
All reviewed studies identified research design considerations essential in developing representative community samples of RAS. More than half of the studies (60.0%) reported an a priori aim to recruit a representative sample [19, 23, 28, 30, 31, 33, 35–40]. This aim clearly articulated the study intent and guided the study design.
A second key study design consideration was the establishment (or identification) of a reliable sampling frame for the hidden group, necessary for both representative sampling and to assess sample representativeness. Overall, 15 (75.0%) of the studies reported a sampling frame [23–31, 33–37, 39] and nine of these had access to government resettlement databases/registries [23, 26, 27, 29, 30, 33, 34, 36, 37]. Given that governments in major countries of resettlement maintain resettlement records, when used in conjunction with ethical and transparent recruitment methods, a reliable sample frame can be developed in collaboration with government bodies.
In the absence of a readily available sampling frame for the target population, some studies reported creative methods to construct a suitable frame. These methods fell into two types of frames: 1) creating lists of names and details for every member of the target group [24, 25, 32], and 2) obtaining non-identifiable data describing demographics and areas of residence [18, 35, 38, 40]. The first frame had the advantage of allowing a random sample to be drawn from the list [24, 25, 32]. Both frames enabled the representativeness of the sample to be confirmed. A good example of the first approach is a Canadian study rated in this review as having high level evidence for a representative sample that had created a comprehensive sample frame by identifying 4854 households with at least one Ethiopian refugee resident . This resource intensive study included methods to identify and confirm potential Ethiopian names from telephone books. It also described the importance of developing strong community networks with the target group to facilitate participation. The result of this 25-month study stage was a list of almost all Ethiopian refugees residing in the city of Toronto. An example of the second approach was a study involving 1165 Somalian and Oromo refugees in the United States of America (USA) in which sample demographics were compared with available demographics from public records of school enrolments, birth statistics and state resettlement records  to determine representativeness.
In some research, the lack of representativeness in a sample is addressed by statistical techniques such as weighting , a conventional design feature used in survey research involving a statistical analysis plan. Weighting methods were reported in five studies [25, 30, 37–39]. One Australian study  weighted responses by both demographic characteristics of the underlying population taken from governmental settlement records, and by sampling rates, which differed between strata in the sample. Used together, these weighting techniques produced a high-quality dataset, broadly representative of the sample frame targeted in the study .
Recording the number and characteristics of people who refuse to participate in research is an important form of data collection for all study types [20, 43, 44], as it can inform the researcher about the degree of non-representativeness in a sample and the potential for selection bias . However, in many studies, only the response rate is reported and even then some response rates may be inaccurate. The majority of reviewed studies (75.0%) reported response rates [23–28, 30, 32–37, 39, 40] ranging between 41%  and 99% . The high response rates reported in three studies, 95% , 97% , and 99% , may cast some doubt about accuracy, because it is uncommon to have nearly perfect rates of recruitment. When collected ethically, information about non-respondents can provide a greater understanding of the overall sample and sample frame. For example, in a large multiphase epidemiologic study of prevalence of exposure to torture in Somalian and Oromo refugees in the USA, records of 35 people who refused participation were collected to assess selection bias .
The use of multiple non-probability sampling methods was shown to be effective in producing representative samples. Although devising and implementing diverse sampling strategies may require additional resources, it appeared to enhance sample representativeness by facilitating access to diverse social networks within the target group. For example, in the previously mentioned American study of 1165 Somalian and Oromo refugees, participants were recruited by cluster sampling (41%), social networking (21%), snowball (31%) and convenience sampling (7%) . Snowball sampling was often used in studies to reach those not readily accessible, e.g., recently arrived migrants and extremely isolated people [25, 27, 31, 32, 35, 40]. Having multiple and diverse ‘seed’ snowball or linkage starting points was recommended so that people could be accessed from different social networks [19, 35, 38, 40]. In the study of refugee migrants in China, 12 individuals of varied age, gender, occupation, and residential address were recruited as initial ‘seeds’ .
RDS is a non-probability sampling method that was designed for sampling hidden groups [19, 38, 45, 46]. It involves identifying seeds who then recruit usually between 0–3 participants; participation and recruitment are often incentivised. An advantage of RDS is that participant social networks can be mapped, as seeds are given individually coded ‘coupons’, which they then pass onto those that they recruit [19, 45]. The coupons are returned to the researcher when the recruited participant presents to takes part in the research. RDS was intended as a means of generating unbiased population estimates, but samples can vary considerably depending on initial seed selection, resulting in unstable outputs and reduced representativeness . Evidence of this instability was seen in one of the studies that used RDS, where two seed groups were established for the purposes of comparing the influence of different methods of seed selection on recruitment. Results indicated that initial seed selection had the potential to strongly influence the type of participants; where one of the seed groups under investigation tended to recruit participants similar to themselves, the opposite was true of the second seed group . As with all non-probability sampling, RDS has several strengths, it is a cost-effective method of recruitment, particularly from hidden groups. RDS also enables the recording and understanding of participants’ social networks. Although this can assist in analysis, it cannot guarantee representative sample compositions.
When possible, sampling strategies should also include a probability (random) sampling component to promote sample representativeness. Probability sampling procedures were used in almost half (45.0%) of the reviewed studies [23–28]. There were four studies that applied some type of probability sampling, supplemented with non-probability sampling such as snowballing or convenience sampling [23–25, 27]. For example, randomly selecting participants from a large pool of primary care registries was likely to over-estimate the prevalence of mental illness among Somali refugees in the UK, therefore the study included additional community sampling . There may be concern regarding the representativeness of the study sample when both probability and non-probability sampling methods are used . However, the use of only non-probability sampling in hidden groups can potentially draw out certain types of individuals needed for attaining representativeness .
Another strategy to improve sampling, if resources are available, is to invite all eligible participants from within the defined sample frame. Three studies adopted this approach; two of these were large studies involving Australian refugees that were rated high for a representative sample [30, 37]; and the third, rated medium, was a Swiss study that sampled consecutively from a national register of adult asylum seekers .
Long recruitment periods of between 12 and 25 months facilitated inclusion of refugee groups in four studies based in United Kingdom , Germany , Canada , and the USA . The previously mentioned Canadian study that identified 4854 households with at least one member from the target group was resource intensive; sampling took over two years . However, engagement with the target community is a time consuming process. A study of refugee migrants in China reported significant difficulties in obtaining the trust of potential seed participants in a short time, regardless of how the study was presented. Even among persons who initially appeared interested, some failed to attend the study appointment and subsequently dropped out of the study. This resulted in modifying the recruitment strategies and engaging seeds who already had established relationships with the researchers .
Online social network sites are new and potentially extensive sampling frames that can be used to target groups over a wide geographical area; for example, Facebook is a social-network website with more than 1.2 billion active users worldwide . The effectiveness of social networking sites to recruit from hidden populations was examined in one study, which, together with field recruitment, used Facebook and dating websites to generate a sample of 3640 men who have sex with men  The study reported that in addition to being cost effective, Facebook offers particularly powerful new targeting capabilities that researchers may be able to exploit to gain access to hidden groups. This online recruitment method may be applicable to certain RAS sub-groups, for example, technologically literate refugees. However, it is noted that online recruitment methods may not be useful in all RAS groups due to barriers to Internet use, lack of technological literacy, safety concerns, and accessibility issues .
Barriers and limitations to creating a representative sample from a hidden group of refugees and asylum seekers
Barriers and limitations were identified in creating a representative sample from a hidden group of RAS. These included the use of the popular snowballing technique which by design, cannot produce a probability sample of observations (and therefore no weights), since there is no way of determining the number of persons who ‘know’ each person in the sample. Barriers in engaging the target group for research included fear of breach of confidentiality. For example, in the previously discussed study with Somalian and Oromo refugees in the USA, trust between the researchers and community was reported to be important, as participants needed reassurance that their research involvement was confidential and would not jeopardize their public credibility . Postal surveys were also viewed as a barrier to representative samples because many members of RAS groups might not be able or disinclined to respond. For example, in a study of refugees living in Sweden, a 65-question survey on mental health was mailed to 413 households yielding a 63% response rate; this response to a postal survey could be considered acceptable, but the study reported that non-responders in this case were likely to be biased towards those with poor health, therefore limiting representativeness of the mental health results obtained . This same study also commented on another barrier that was addressed in their study design, namely to reduce focus on ethnicity in the survey and cover letter in order to engage refugees who no longer self-identified as refugees . The utility of using census data to identify the target group was recognised by one study as a limitation, as census data is not always accurate for locating small, mobile refugee migrants and illegal migrants . Geographic information systems were used to assess representativeness in yet another study and showed that despite the aim to recruit a diverse sample of migrant workers in China, the majority of participants resided or worked in close proximity to the study sites, therefore limiting the generalizability of the result to populations outside of these areas .
Study limitations and strengths
During the process of conducting this review, we encountered an unanticipated result: half of the publications identified from the grey search were in fact peer-reviewed publications (4/8, 50%) that were not detected in the main search. One was a thesis , which explains why it was not identified in the main search. Another publication by Ao et al.  was only identified via an item describing the same study found in the grey literature , a likely explanation being that the Ao et al.  paper was only published very recently. The remaining two peer-reviewed papers were not identified in the main search, as their titles, abstracts, keywords and Medical Subject Headings did not include key terms used in our search strategy despite describing relevant concepts. This finding provides evidence for the potentially inconsistent indexing of such publications, a previously documented limitation of the literature on RAS . It also raises the possibility that other relevant articles were not identified by this review.
A further limitation was the exclusion of non-English publications. Although the grey literature search strategy was designed to identify relevant research from countries with the highest proportional intakes of RAS groups , including many non-English speaking countries, we were unable to obtain and translate studies in languages other than English due to time and resource constraints. The possibility of language bias can therefore not be ruled out. In addition, the majority of studies included in the review were conducted in high-income countries that actively accept refugees for resettlement and are not likely to be generalizable to low- and middle-income countries where most refugees live . Finally, the inclusion of studies rated as having medium level evidence of obtaining a representative sample may limit conclusive comments regarding the effectiveness of some of the strategies discussed in this review.
The inclusion of study designs other than gold-standard randomized controlled trials was a limitation but also a strength of this review; the descriptive studies, case-studies and studies using non-probability sampling techniques provided insights into ways to increase representative sampling with hidden groups. However, the degree of heterogeneity between studies meant that results could not be combined statistically in a meta-analysis.
A strength of this review was the identification and inclusion of non-peer reviewed but highly relevant governmental reports [28, 29, 37] and a protocol article , found in the grey search. This shows the importance of including grey literature when investigating RAS groups, as governments are often well placed to undertake studies with these populations and the resulting reports are not always published in academic forums. We plan to detail our grey search in a separate ‘how-to’ publication so that other researchers can use this effective technique to search for relevant literature about RAS groups.
This review suggests that representative samples of RAS and other hidden groups residing in the community can be generated, but that generating such samples requires specific efforts, including actively engaging the populations of interest, and incorporating the careful use of non-probability sampling, as well as other design considerations. In summary, key design considerations revealed in the reviewed studies were: an a priori aim to recruit a representative sample; a reliable sampling frame to check sample representativeness; recording of response rates and non-responder characteristics depending on ethical considerations; the requirement for long recruitment periods; the use of multiple non-probability sampling methods, including snowballing to access the most isolated; the use of multiple and diverse seed starting points to access different social networks; the tracking of respondent network recruitment; and, when possible, the inclusion of a probability sampling component. Finally, online social networking sites are providing new forms of sampling frames that potentially enable access to hidden groups across large geographical ranges. We anticipate that the findings from this study will assist researchers aiming to recruit representative samples of RAS groups, and will also encourage more rigorous reporting of future studies so that the representativeness of samples of RAS groups in research can be more readily assessed.
Bonevski B, Randell M, Paul C, Chapman K, Twyman L, Bryant J, et al. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Med Res Methodol. 2014;14:42. doi:10.1186/1471-2288-14-42.
Casa BL, Negi NJ, Hong M. Culturally competent social work research: methodological considerations for research with language minorities. Soc Work. 2012;57(1):1–10. doi:10.1093/sw/swr002.
Dawood M. Sampling rare populations. Nurse Res. 2008;15(4):35–41. doi:10.7748/nr2008.07.15.4.35.c6661.
Vigneswaran D, Quirk J. Quantitative methodological dilemmas in urban refugee research: a case study of Johannesburg. J Refugee Stud. 2013;26(1):110–6. doi:10.1093/jrs/fes035.
Willis G, Smith T, Shariff-Marco S, English N. Overview of the special issue on surveying the hard-to-reach. J Off Stat. 2014;30(2):171–6. doi:10.2478/jos-2014-0011.
Bloch A. Methodological challenges for national and multi-sited comparative survey research. J Refugee Stud. 2007;20(2):230–47. doi:10.1093/jrs/fem002.
Faugier J, Sargeant M. Sampling hard to reach populations. J Adv Nurs. 1997;26(4):790–7. doi:10.1046/j.1365-2648.1997.00371.x.
Fazel M, Wheeler J, Danesh J. Prevalence of serious mental disorder in 7000 refugees resettled in western countries: a systematic review. Lancet. 2005;365(9467):1309–14. doi:10.1016/S0140-6736(05)61027-6.
Ogilvie LD, Burgess-Pinto E, Caufield C. Challenges and approaches to newcomer health research. J Transcult Nurs. 2008;19(1):64–73. doi:10.1177/1043659607309142.
Sadler GR, Lee H, Lim RS, Fullerton J. Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy. Nurs Health Sci. 2010;12(3):369–74. doi:10.1111/j.1442-2018.2010.00541.x.
Maier T, Schmidt M, Mueller J. Mental health and healthcare utilization in adult asylum seekers. Swiss Med Wkly. 2010;140:w13110. doi:10.4414/smw.2010.13110.
McColl H, McKenzie K, Bhui K. Mental healthcare of asylum-seekers and refugees. Adv Psychiatr Treat. 2008;14(6):452–9. doi:10.1192/apt.bp.107.005041.
Murray K, Davidson G, Schweitzer R. Psychological wellbeing of refugees resettling in Australia: a literature review prepared for the Australian psychological society. Melbourne: The Australian Psychological Society; 2008.
Murray KE. Sudanese perspectives on resettlement in Australia. J Pac Rim Psychol. 2010;4(1):30–43.
Enticott JC, Cheng IH, Russell G, Szwarc J, Braitberg G, Peek A, et al. Emergency department mental health presentations by people born in refugee source countries: an epidemiological logistic regression study in a Medicare local region in Australia. Aust J Prim Health. 2014. doi:10.1071/PY13153.
Lavrakas PJ. Encyclopedia of survey research methods. Thousand Oaks: Sage Publications; 2008.
Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. 2011. Available from http://handbook.cochrane.org.
Vial AC, Starks TJ, Parsons JT. Finding and recruiting the highest risk HIV-negative men who have sex with men. AIDS Educ Prev. 2014;26(1):56–67. doi:10.1521/aeap.2014.26.1.56.
Wylie JL, Jolly AM. Understanding recruitment: outcomes associated with alternate methods for seed selection in respondent driven sampling. BMC Med Res Methodol. 2013;13:93. doi:10.1186/1471-2288-13-93.
Patel MX, Doku V, Tennakoon L. Challenges in recruitment of research participants. Adv Psychiatr Treat. 2003;9:229–38. doi:10.1192/apt.9.3.229.
Moher D, Liberati A, Tetzlaff J, Altman DG, Grp P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Phys Ther. 2009;89(9):873–80. doi:10.1371/journal.pmed.1000097.
United Nations High Commissioner for Refugees. UNHCR Global Trends Report - Table 24. Indicators of host country capacity and contributions 2010. https://docs.google.com/spreadsheets/d/1bJR5428WlSjYVCZ5WmSiapDkj9D9nHXGrjstc4KzzWY/edit?hl=en_US&pref=2&pli=1#gid=11. Accessed 2 Jul 2015.
Ao T, Shetty S, Sivilli T, Blanton C, Ellis H, Geltman PL, et al. Suicidal ideation and mental health of Bhutanese refugees in the United States. J Immigr Minor Health. 2016;18(4):1–8. doi:10.1007/s10903-015-0325-7.
Bhui K, Craig T, Mohamud S, Warfa N, Stansfeld SA, Thornicroft G, et al. Mental disorders among Somali refugees: developing culturally appropriate measures and assessing socio-cultural risk factors. Soc Psychiatry Psychiatr Epidemiol. 2006;41(5):400–8. doi:10.1007/s00127-006-0043-5.
Bilsborrow RE, Mena CF, Arguello E. Colombian refugees in Ecuador: sampling schemes, migratory patterns and consequences for migrants. Int J Global Environ Iss. 2011;11(3–4):271–98. doi:10.1504/IJGENVI.2011.044561.
Blight KJ, Ekblad S, Persson J, Ekberg J. Mental health, employment and gender. Cross-sectional evidence in a sample of refugees from Bosnia-Herzegovina living in two Swedish regions. Soc Sci Med. 2006;62(7):1697–709. doi:10.1016/j.socscimed.2005.08.019.
Bogic M, Ajdukovic D, Bremner S, Franciskovic T, Galeazzi GM, Kucukalic A, et al. Factors associated with mental disorders in long-settled war refugees: refugees from the former Yugoslavia in Germany, Italy and the UK. Br J Psychiatry. 2012;200(3):216–23. doi:10.1192/bjp.bp.110.084764.
Citizenship and Immigration Canada. Evaluation of Government Assisted Refugees (GAR) and Resettlement Assistance Program (RAP). 2011. http://www.cic.gc.ca/english/resources/evaluation/gar-rap/index.asp. Accessed 25 Jun 2016.
Commissariat for Refugees. Situations and needs of refugee population in the Republic of Serbia. 2008. http://www.kirs.gov.rs/docs/RefugeeNeedsAsessmentReportSerbia.pdf. Accessed 25 Jun 2016.
De Maio J, Silbert M, Jenkinson R, Smart D. Building a new life in Australia: introducing the longitudinal study of humanitarian migrants. Fam Matters. 2014;94:5–14.
Dunlavy AC. The impact of acculturation, trauma, and post-migration stressors on the mental health of African immigrants and refugees in Sweden. Pittsburgh: University of Pittsburgh; 2010.
Fenta H, Hyman I, Noh S. Mental health service utilization by Ethiopian immigrants and refugees in Toronto. J Nerv Ment Dis. 2006;194(12):925–34. doi:10.1097/01.nmd.0000249109.71776.58.
Gerritsen AA, Bramsen I, Deville W, van Willigen LH, Hovens JE, van der Ploeg HM. Physical and mental health of Afghan, Iranian and Somali asylum seekers and refugees living in the Netherlands. Soc Psychiatry Psychiatr Epidemiol. 2006;41(1):18–26. doi:10.1007/s00127-005-0003-5.
Heeren M, Mueller J, Ehlert U, Schnyder U, Copiery N, Maier T. Mental health of asylum seekers: a cross-sectional study of psychiatric disorders. BMC Psychiatry. 2012;12:114. doi:10.1186/1471-244X-12-114.
Khavarpour F, Rissel C. Mental health status of Iranian migrants in Sydney. Aust N Z J Psychiatry. 1997;31(6):828–34. doi:10.3109/00048679709065508.
Maximova K, Krahn H. Health status of refugees settled in Alberta: changes since arrival. Can J Public Health. 2010;101(4):322–6.
McAuliffe M. Seeking the views of irregular migrants: decision making, drivers and migration journeys. Irregular migration research program occassional paper series. Australia: Department of Immigration and Border Protection; 2013.
Qiu P, Yang Y, Ma X, Wu F, Yuan P, Liu Q, et al. Respondent-driven sampling to recruit in-country migrant workers in China: a methodological assessment. Scand J Public Health. 2012;40(1):92–101. doi:10.1177/1403494811418276.
Silove D, Steel Z, Bauman A, Chey T, McFarlane A. Trauma, PTSD and the longer-term mental health burden amongst Vietnamese refugees: a comparison with the Australian-born population. Soc Psychiatry Psychiatr Epidemiol. 2007;42(6):467–76. doi:10.1007/s00127-007-0194-z.
Spring M, Westermeyer J, Halcon L, Savik K, Robertson C, Johnson DR, et al. Sampling in difficult to access refugee and immigrant communities. J Nerv Ment Dis. 2003;191(12):813–9. doi:10.1097/01.nmd.0000100925.24561.8f.
Cochran J, Geltman PL, Brown C, Anderton S, Montour J, Vargas M et al. Suicide and suicidal ideation among Bhutanese refugees-United States, 2009–2012. In: Morbidity and Mortality Weekly Report. 2013. http://www.cdc.gov/mmwr/preview/mmwrhtml/mm6226a2.htm. Accessed 2 Nov 2015.
Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res. 2011;46(3):399–424. doi:10.1080/00273171.2011.568786.
Altman DG, Moher D, Schulz KF. Improving the reporting of randomised trials: the CONSORT Statement and beyond. Stat Med. 2012;31(25):2985–97. doi:10.1002/sim.5402.
von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9. doi:10.1016/j.ijsu.2014.07.013.
Johnston LG, Sabin K. Sampling hard-to-reach populations with respondent driven sampling. Methodological Innov Online. 2010;5(2):38–48. doi:10.4256/mio.2010.0017.
Keygnaert I, Dialmy A, Manço A, Keygnaert J, Vettenburg N, Roelens K et al. Sexual violence and sub-Saharan migrants in Morocco: a community-based participatory assessment using respondent driven sampling. Global Health. 2014;10(1):32. doi:10.1186/1744-8603-10-32.
Salganik MJ. Variance estimation, design effects, and sample size calculations for respondent-driven sampling. J Urban Health. 2006;83:98–112. doi:10.1007/s11524-006-9106-x.
Mikal JP, Woodfield B. Refugees, post-migration stress, and internet use: a qualitative analysis of intercultural adjustment and internet use among Iraqi and Sudanese refugees to the United States. Qual Health Res. 2015;25(10):1319–33. doi:10.1177/1049732315601089.
Mason E. Forced migration studies: surveying the reference landscape. Libri. 2000;50(4):241–51. doi:10.1515/LIBR.2000.241.
We would like to acknowledge the team of research assistants that provided technical assistance during this review, and these are listed in alphabetic order: Breana Burns, Julian Matthews, Serena Menezes, Anthony Mowbray Sanduni Sankhika and Victor Yang.
This work was funded by a Faculty Strategic Grant from the Faculty of Medicine, Nursing and Health Sciences at Monash University (SGS15-0146).
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
All authors participated in the design of this project and were involved in drafting the manuscript or revising it critically for important intellectual content. JE, FS and GM initiated this review and remained the core authors for the duration including overseeing revisions requested by reviewers. HM provided oversight of construction and interpretation of issues in the transcultural domain. RK provided specialist expertise in transcultural epidemiology. SV, HC and GR provided expertise on understanding transcultural primary mental health and participation in research. JE and FS provided oversight of the review processes and supervised a team of assistants throughout the duration of the project. JE and FS devised the templates used by the assistants to screen titles and abstracts and extract data from eligible papers. JE and/or FS would meet at least weekly with the assistants to discuss and resolve arising queries. JE created the graded study quality assessment system based on key study factors that indicated possible sources of selection bias, adapted from a well know Cochrane method described by Higgins and Green (2011). JE re-ran the peer-reviewed literature search in PubMed and PsycInfo in January 2016 to identify additional relevant studies published in 2015, of which none were identified. JE and KB graded all articles independently and when necessary in collaboration together and consensus reached by discussion. KB extracted the relevant information from the articles identified in the grey search in September 2016. JE and KB checked data extraction for accuracy. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
About this article
Cite this article
Enticott, J.C., Shawyer, F., Vasi, S. et al. A systematic review of studies with a representative sample of refugees and asylum seekers living in the community for participation in mental health research. BMC Med Res Methodol 17, 37 (2017). https://doi.org/10.1186/s12874-017-0312-x
- Asylum seeker
- Stateless person
- Mental disorders
- Hidden population