Dropout in a longitudinal, cohort study of urologic disease in community men
- Naomi M Gades†1Email author,
- Debra J Jacobson†2,
- Michaela E McGree2,
- Michael M Lieber3,
- Rosebud O Roberts1,
- Cynthia J Girman1, 4 and
- Steven J Jacobsen5
© Gades et al; licensee BioMed Central Ltd. 2006
Received: 15 August 2006
Accepted: 14 December 2006
Published: 14 December 2006
Reasons for attrition in studies vary, but may be a major concern in long-term studies if those who drop out differ systematically from those who continue to participate. Factors associated with dropout were evaluated in a twelve-year community-based, prospective cohort study of urologic disease in men.
During 1989–1991, 2,115 randomly selected Caucasian men, ages 40–79 years from Olmsted County, Minnesota were enrolled and followed with questionnaires biennially; 332 men were added in follow-up. A random subset (~25%) received a urologic examination. Baseline characteristics including age, benign prostatic hyperplasia (BPH) symptoms, comorbidities, and socioeconomic factors were compared between subjects who did and did not participate after the twelfth year of follow-up.
Of the 2,447 men, 195 died and were excluded; 682 did not participate in 2002. Compared with men in the 40–49 year age group, men ≥ 70 years of age at baseline had a greater relative odds of dropout, 2.65 (95% CI: 1.93, 3.63). In age-adjusted analyses, relative to men without stroke, men who had suffered a stroke had a higher odds of dropout, age-adjusted OR 3.07 (95% CI: 1.49, 6.33). Presence of at least one BPH symptom was not associated with dropout, (age-adjusted OR 1.12 (95% CI: 0.93, 1.36)).
These results provide assurance that dropout was not related to primary study outcomes. However, factors associated with dropout should be taken into account in analyses where they may be potential confounders.
Attrition becomes a major concern in long-term studies if those who drop out differ systematically from those who continue to participate. Common statistical methods used for cohort studies assume there is no difference between subjects who drop out of the study and those who continue to participate. However, if this assumption is false, the validity of the study may be jeopardized and the difference between the observed values and "true" values may increase with study follow-up . This is of particular concern if disease severity influences study participation. Losses in long-term studies may be due to refusals to continue in a study, inability to trace individuals for follow-up, and deaths.
Few studies of urologic conditions, such as benign prostatic hyperplasia (BPH), have examined dropout of study participants over the long term. One prospective cohort study of urinary symptoms followed participants for up to four years, and documented a dropout rate of 29%. Men with the highest levels of lower urinary tract symptom bother and interference at baseline, dropped out by last year of follow-up . Two long-term, double-blind clinical trials which evaluated urologic outcomes, the Medical Therapy of Prostatic Symptoms (MTOPS) trial  and the Proscar Long-term Efficacy and Safety Study (PLESS) , evaluated dropout. In the MTOPS study the rate of loss was 2.5 per 100 patient-years in the placebo group and ranged from 2.2 to 3.3 in the treatment groups . By the end of the PLESS study, 42% of patients had discontinued treatment. However, follow-up data were available on 92% of the randomized patients, but differences in men who dropped out compared with those who continued to participate were not examined . Roehrborn  examined the importance of patient follow-up after discontinuation and concluded that in order to compare the true event rate across clinical trials, discontinuation rates and total patient follow-up for clinical outcomes must be assessed, a recommendation that is applicable to cohort studies as well.
The Olmsted County Study of Urinary Symptoms and Health Status among Men is a community-based, prospective cohort study that offers an ideal opportunity to identify characteristics associated with dropout, and to determine if these factors might bias results from this study.
Study design and population
The Olmsted County Study of Urinary Symptoms and Health Status among Men was initiated in December 1989 and details have been published elsewhere [7–9]. Briefly, this study is a community-based, prospective cohort study of Caucasian men who were 40–79 years of age on January 1, 1990, randomly selected from the Olmsted County, Minnesota population. The goal of the study was to determine the natural history of urinary symptoms and BPH. Potential subjects were identified through the Rochester Epidemiology Project . Community medical records of all the randomly selected men were reviewed for history of prostate cancer, prostatectomy, and other medical conditions which may impede normal voiding function, such as neurologic disease, lower back surgery and urethral stricture. After exclusion for these pre-existing conditions and/or treatments, 3,874 men were asked to join the study and of these 2,115 agreed to participate (55% participation rate).
Individual participants were visited in their homes, signed informed consent documentation, and were asked to complete a questionnaire which documented demographic information, including the participant's age, marital status, education level, and annual salary. The questionnaire also assessed lower urinary tract symptom severity with questions similar to the American Urological Association Symptom Index (AUASI) [11–13]. Urinary bother and interference with daily activities were assessed using a Health-Related Quality of Life (HRQL) questionnaire specific for BPH . Information on the presence of comorbidities and other urologic conditions, including erectile dysfunction (ED), diabetes mellitus, stroke, heart disease, and hypertension were reported on the baseline questionnaire. In addition, men voided into a portable uroflow meter (Dantec 1000) to assess lower urinary function.
A 25% random sub-sample of the study cohort, 475 out of 537 men (88%), agreed to participate in a detailed clinical urologic examination, including serum prostate specific antigen (PSA) level, digital rectal examination, and transrectal sonographic imaging of the prostate to determine prostatic volume. Measurements were made of the anteroposterior, transverse, and superoinferior diameters and prostate volume was estimated using the formula for a prolate ellipsoid, π/6 (transverse × anteroposterior × superoinferior) . All study procedures were approved by the Mayo Foundation Institutional Review Board (IRB).
Men who were diagnosed with prostate cancer, received a radical prostatectomy, or transurethral resection of the prostate (TURP) were not excluded from the cohort, but were excluded from participating in the detailed urologic examination. During the second and third biennial follow-up, men who did not participate in the follow-up were replaced by men randomly selected from the community, after being screened for the exclusion criteria used at baseline (n = 229 and n = 103, respectively). Of the replacement men, randomly selected individuals were added to the clinic sub-sample in the years two and four of follow-up, (n = 100 and n = 58, respectively). Since that time the study has been maintained as a closed cohort. At the twelfth year of follow-up, 682 (30.3% of live enrollees) men had dropped out of the study.
Baseline urologic health status, comorbidities, HRQL, and socioeconomic status were compared among the men who dropped out and those who continued to participate in the twelfth year of follow-up of the study. Men who died between contacts (n = 195) were removed from the analysis, including 46 men in the clinic subset. An alpha of p < 0.05 was considered statistically significant.
Urologic measures were dichotomized according to standard cut points. Urinary symptoms were assessed by the AUASI. Scores ranged from 0 (none) to 35 (severe), with scores from 8 to 19 classified as "moderate" symptoms and scores ≥ 20 classified as "severe" symptoms. These two categories were combined in the analyses, and scores > 7 were considered moderate/severe symptoms . Other urinary outcomes evaluated were peak urinary flow rate (< 12 ml/second) , prostate volume (> 30 cm3) , serum PSA level (> 1.4 ng/mL, upper 25th percentile for this cohort at baseline), and creatinine (> 1.4 ng/dL) . A chi-square test was used to compare the distribution of urologic measures among participants and those who dropped out.
Logistic regression was used to calculate age-adjusted odds ratios (OR) comparing individuals who dropped out and those who continued to participate in the twelfth year of follow-up of the study. A multivariable model was constructed to simultaneously adjust for age, urologic health status, comorbidities, HRQL, and socioeconomic factors in predicting dropout from the study. This model contained the variable with the strongest age-adjusted association from each domain. Forward selection logistic regression verified the final model. All 2-way interactions between variables were also evaluated. To account for multiple comparisons in these models a p-value of 0.005 was used for inclusion-exclusion in the model. All analyses were conducted using SAS, version 8.2 (SAS Institute, Cary, N.C.)
Characteristics of men who participated versus those who dropped out of The Olmsted County Study of Urinary Symptoms and Health Status among Men, Rochester, Minnesota (1989–2003)
Age-adjusted comparisons of relative odds of dropping out of The Olmsted County Study of Urinary Symptoms and Health Status among Men based on baseline measurements, Rochester, Minnesota (1989–2003)
Primary Study Outcomes
Age-adjusted OR* (95% CI*)
AUASI* (> 7 vs. ≤ 7)
1.19 (0.97, 1.45)
Peak Urinary Flow Rate (QMAX) (< 12 vs. ≥ 12 mL/s)
1.18 (0.93, 1.50)
Prostate Volume (> 30 vs. ≤ 30 cm3)
0.98 (0.62, 1.56)
Prostate-Specific Antigen (> 1.4 vs. ≤ 1.4 ng/mL)
0.90 (0.55, 1.46)
Any Benign Prostatic Hyperplasia (Yes vs. No)
1.12 (0.93, 1.36)
Member of Clinic Subset (Yes vs. No)
0.73 (0.59, 0.90)
Diabetes (Yes vs. No)
1.24 (0.77, 1.99)
Stroke (Yes vs. No)
3.07 (1.49, 6.33)
Hypertension (Yes vs. No)
1.23 (0.97, 1.56)
Coronary Disease (Yes vs. No)
1.25 (0.92, 1.68)
Any Comorbidity (Yes vs. No)
1.31 (1.06, 1.61)
Other Urologic Measures
Creatinine (> 1.4 vs. ≤ 1.4 ng/dL)
1.10 (0.45, 2.66)
Erectile Dysfunction (Yes vs. No)
1.69 (1.26, 2.28)
Quality of Life Measures
Satisfied with Urinary Condition (Yes vs. No)
0.65 (0.49, 0.87)
Satisfied with overall health/HRQL* (Yes vs. No)
0.29 (0.19, 0.43)
Visited a doctor more than 4 times in the past 12 months (Yes vs. No)
1.30 (0.96, 1.75)
Interference Score (> 9 vs. ≤ 9)
1.32 (1.07, 1.62)
High School Graduate or More (Yes vs. No)
0.40 (0.29, 0.56)
Married or Living Together (Yes vs. No)
0.81 (0.60, 1.08)
Salary ($15,000–$34,999 vs. ≤ $14,999/year)
0.65 (0.48, 0.90)
Salary (≥ $35,000 vs. ≤ $14,999/year)
0.44 (0.33, 0.60)
Odds of dropping out of The Olmsted County Study of Urinary Symptoms and Health Status among Men before the seventh biennial contact based on the multivariable model baseline measurements, Rochester, Minnesota (1989–2003)
Unadjusted OR* (95% CI*)
Multivariable Adjusted OR (95% CI)
Baseline age 40–49
Baseline age 50–59
0.66 (0.52, 0.83)
0.59 (0.46, 0.74)
Baseline age 60–69
0.78 (0.61, 1.01)
0.56 (0.42, 0.75)
Baseline age 70+
2.65 (1.93, 3.63)
1.49 (1.01, 2.19)
1.86 (1.44, 2.41)
1.52 (1.11, 2.07)
3.99 (2.00, 7.97)
2.39 (1.13, 5.05)
Satisfied with overall health/HRQL*
0.28 (0.18, 0.41)
0.35 (0.23, 0.55)
High School Graduate or more
0.38 (0.28, 0.51)
0.43 (0.30, 0.60)
Member of Clinic Subset
0.79 (0.65, 0.97)
0.74 (0.60, 0.92)
Our results suggest several factors that may help predict which men are more likely to drop out versus continue to participate in a long term study of urologic health. A critical finding however, is that, after adjusting for age, there were no differences in primary study outcomes at baseline between individuals who dropped out and those who continued to participate in the twelfth year of follow-up. Our findings showed that men with ED were more likely to drop out of the study compared to those without ED. It is possible that this condition may have been too frustrating for the men to continue to participate in the study or initially they may have perceived a potential benefit for this condition by participating in the study. Alternatively, as ED is associated with several comorbidities and life-style factors, i.e. smoking, diabetes, cardiovascular disease [19, 20], drop out associated with ED may be related to other health issues.
Our results show that older men with poorer general health status were more likely to drop out of the study. The increased dropout in older men with comorbid conditions is consistent with other studies [21–23], where nonresponse bias in the older subset was a concern. Therefore, additional efforts may be required to encourage older participants and those with co-morbid conditions to continue participation in long-term studies. In addition, this demonstrates the need for assessing the effects of age stratification and age-adjustment when evaluating study outcome variables. Furthermore, comorbidities may be confounded with age, as the risk for comorbid conditions tends to increase with age [21–23].
Health-related quality of life measures also showed that men who suffered more interference from their urologic symptoms were more likely to drop out than men with less interference. This is consistent with a five-year natural history study of BPH in Scotland  in which men who reported having the highest levels of bother and interference at baseline were less likely to participate in the survey at 5 years. It was proposed that one reason for this was that men may adjust to their symptoms over time and adapt their life-style to accommodate symptoms. Alternatively, these men may not have seen any personal benefit to continued participation in the study.
Men with a lower socioeconomic status were also more likely to drop out of the study. Although few urologic studies have documented the effect of education level on study dropout, numerous studies of other disease conditions have documented the potential effect of education level on study attrition. For example, in a longitudinal, multicenter AIDS cohort study examining retention after 9.5 years, lower education level was significantly associated with nonparticipation . In addition, a study examining predictors of inactivation and reasons for participant inactivation during skin cancer chemoprevention found that a low education level was a significant predictor of inactivation . One explanation for this finding is that men with a lower education level may not fully appreciate the benefits of study participation. An approach to this issue may be to offer educational seminars or orientation sessions for study subjects that outline and discuss the benefits participants and society may receive from the study.
In our study, with the approval of the Mayo IRB, we have used a combination of active and passive follow-up to assess characteristics associated with participant dropout. This has allowed us to gauge the probability and degree of study bias due to nonparticipation. This level of follow-up is unique, and may not be feasible for other studies, as few studies have access to medical records for both participants and nonparticipants. However, our study may have greater validity for the target study population, due to our ability to assess not only participants, but also those subjects who have previously dropped out of the study.
There are some limitations to the generalizability of this study. All participants in the Olmsted County Study of Urinary Symptoms and Health Status among Men are Caucasian and were 40–79 years of age at study entry. Therefore, the ability to generalize these findings to other ethnic populations and age groups may be limited. We excluded men who died between contacts. By doing this, we assumed that death was unrelated to our outcome. However, the men who died may have been in poorer health and may have been at greater risk for clinical urologic conditions. We repeated our initial analysis including deaths between contacts as dropouts and the results were similar (data not shown). Moreover, nonparticipation, per se, was not related to primary study outcomes at baseline (after adjusting for age).
Another potential limitation was the 55% participation rate; however, factors associated with participation may not be the same factors associated with dropout. We have previously documented differences in the characteristics of baseline participants and nonparticipants in age, home location, and prior history of urologic conditions . The age-adjusted period prevalence rate for benign prostatic hyperplasia was 9.6% (95% CI: 8.1, 11.0) for full participants, 8.2% (95% CI: 5.8, 10.6) for partial participants and 5.3% (95% CI: 3.6, 6.9) for complete non-responders. These results suggest that initial participation in our study may have been at least partially driven by concerns about urologic health. No differences were found between participants and nonparticipants in other major medical comorbidities. Additionally, longitudinal assessment of the cohort did not show any differences in urologic outcomes identified through medical record abstraction between participants and nonparticipants . Therefore, while it is possible that some of the men who did not initially participate in our study might have been more likely to drop out, an earlier study of these groups did not suggest that initial nonparticipants differed dramatically from participants.
Regardless of these limitations, this study demonstrates that cohort studies may encounter differential attrition associated with poor health status, increasing age, lower education level and socioeconomic status. Even well designed cohort studies and clinical trials will suffer if differential attrition biases results. Thorough planning, special efforts to encourage continued participation and statistical methods can be implemented to minimize dropout and correct for non-response bias. Differential attrition in this study did not bias our primary study outcomes, however future reports from this and other urologic studies which focus on ED will need to address the degree to which dropout might bias those results.
The authors thank the Olmsted County Study personnel for their help in the study and Ms. Sondra Buehler for her assistance in preparation of this manuscript. This project was supported by research grants from the Public Health Service, National Institutes of Health (DK58859, RR00585, and AR30582), and Merck Research Laboratories.
- Szklo M, Nieto FJ: Quality assurance and control. Epidemiology: Beyond the Basics. Edited by: Brown M. 2004, Sudbury, MA, Jones and Bartlett Publishers, Inc.Google Scholar
- Lee AJ, Garraway WM, Simpson RJ, Fisher W, King D: The natural history of untreated lower urinary tract symptoms in middle-aged and elderly men over a period of five years. EurUrol. 1998, 34: 325-332.Google Scholar
- McConnell JD, Roehrborn CG, Bautista OM, Andriole GL, Dixon CM, Kusek JW, Lepor H, McVary KT, Nyberg LM, Clarke HS, Crawford ED, Diokno A, Foley JP, Foster HE, Jacobs SC, Kaplan SA, Kreder KJ, Lieber MM, Lucia MS, Miller GJ, Menon M, Milam DF, Ramsdell JW, Schenkman NS, Slawin KM, Smith JA, Medical Therapy of Prostatic Symptoms Research G: The long-term effect of doxazosin, finasteride, and combination therapy on the clinical progression of benign prostatic hyperplasia. N Engl J Med. 2003, 349: 2387-2398. 10.1056/NEJMoa030656.View ArticlePubMedGoogle Scholar
- Wessells H, Roy J, Bannow J, Grayhack J, Matsumoto AM, Tenover L, Herlihy R, Fitch W, Labasky R, Auerbach S, Parra R, Rajfer J, Culbertson J, Lee M, Bach MA, Waldstreicher J: Incidence and severity of sexual adverse experiences in finasteride and placebo-treated men with benign prostatic hyperplasia. Urology. 2003, 61: 579-584. 10.1016/S0090-4295(02)02401-9.View ArticlePubMedGoogle Scholar
- Roehrborn CG, McConnell JD, Saltzman B, Bergner D, Gray T, Narayan P, Cook TJ, Johnson-Levonas AO, Quezada WA, Waldstreicher J: Storage (irritative) and voiding (obstructive) symptoms as predictors of benign prostatic hyperplasia progression and related outcomes. Eur Urol. 2002, 42: 1-6. 10.1016/S0302-2838(02)00210-5.View ArticlePubMedGoogle Scholar
- Roehrborn CG: Reporting of acute urinary retention in BPH treatment trials: importance of patient follow-up after discontinuation and case definitions. Urology. 2002, 59: 811-815. 10.1016/S0090-4295(02)01649-7.View ArticlePubMedGoogle Scholar
- Chute CG, Panser LA, Girman CJ, Oesterling JE, Guess HA, Jacobsen SJ, Lieber MM: The prevalence of prostatism: a population-based survey of urinary symptoms. J Urol. 1993, 150: 85-89.PubMedGoogle Scholar
- Jacobsen SJ, Guess HA, Panser L, Girman CJ, Chute CG, Oesterling JE, Lieber MM: A population-based study of health care-seeking behavior for treatment of urinary symptoms: The Olmsted County Study of Urinary Symptoms and Health Status Among Men. Arch Fam Med. 1993, 2: 729-735. 10.1001/archfami.2.7.729.View ArticlePubMedGoogle Scholar
- Oesterling JE, Jacobsen SJ, Chute CG, Guess HA, Girman CJ, Panser LA, Lieber MM: Serum prostate-specific antigen in a community-based population of healthy men: Establishment of age-specific reference ranges. JAMA. 1993, 270: 860-864. 10.1001/jama.270.7.860.View ArticlePubMedGoogle Scholar
- Melton LJ: History of the Rochester Epidemiology Project. Mayo Clin Proc. 1996, 71: 266-274.View ArticlePubMedGoogle Scholar
- Barry MJ, Fowler FJJ, O'Leary MP, Bruskewitz RC, Holtgrewe HL, Mebust WK, The, Measurement, Committee, of, the, American, Urological, Association.: Correlation of the American Urological Association Symptom Index with self-administered versions of the Madsen-Iversen, Boyarsky and Maine Medical Assessment Program Symptom Indexes. J Urol. 1992, 148: 1558-1563.PubMedGoogle Scholar
- Cockett AT, Barry MJ, Holtgrewe HL, Sihelnick S, Williams R, McConnell J: Indications for treatment of benign prostatic hyperplasia. The American Urological Association Study. Cancer. 1992, 70(1 Suppl): 280-283. 10.1002/1097-0142(19920701)70:1+<280::AID-CNCR2820701315>3.0.CO;2-E.View ArticleGoogle Scholar
- AUA Practice Guidelines Committee. AUA guideline on management of benign prostatic hyperplasia (2003). Chapter 1: Diagnosis and treatment recommendations. J Urol. 2003, 170: 530-547. 10.1097/01.ju.0000078083.38675.79.Google Scholar
- Epstein RS, Deverka PA, Chute CG, Panser L, Oesterling JE, Lieber MM, Schwartz S, Patrick D: Validation of a new quality of life questionnaire for benign prostatic hyperplasia. JClinEpidemiol. 1992, 45: 1431-1445.Google Scholar
- Terris MK, Stamey TA: Determination of prostate volume by transrectal ultrasound. J Urol. 1991, 145: 984-987.PubMedGoogle Scholar
- Barry MJ, Fowler FJ, O'Leary MP, Bruskewitz RC, Holtgrewe HL, Mebust WK, Cockett ATK, Measurement..Committee.of.the.American.Urological.Association T: The American Urological Association Symptom Index for Benign Prostatic Hyperplasia. J Urol. 1992, 148: 1549-1557.PubMedGoogle Scholar
- Corica FA, Jacobsen SJ, King BF, Bostwick DG, Jacobson DJ, Girman CJ, Guess HA, Lieber MM: Prostatic central zone volume, lower urinary tract symptom severity and peak urinary flow rates in community-dwelling men. Journal of Urology. 1999, 161: 831-834. 10.1016/S0022-5347(01)61784-1.View ArticlePubMedGoogle Scholar
- Rule AD, Larson TS, Bergstralh EJ, Slezak JM, Jacobsen SJ, Cosio FG: Using serum creatinine to estimate glomerular filtration rate: accuracy in good health and in chronic kidney disease. Ann Intern Med. 2004, 141: 929-937.View ArticlePubMedGoogle Scholar
- Gades NM, Nehra A, Jacobson DJ, McGree ME, Girman CJ, Rhodes T, Roberts RO, Lieber MM, Jacobsen SJ: Association between smoking and erectile dysfunction: A population-based study. Am J Epidemiol. 2005, 161: 346-351. 10.1093/aje/kwi052.View ArticlePubMedGoogle Scholar
- Muller A, Mulhall JP: Cardiovascular disease, metabolic syndrome and erectile dysfunction. Curr Opin Urol. 2006, 16: 435-443.View ArticlePubMedGoogle Scholar
- Rockwood K, Stolee P, Robertson D, Shillington ER: Response bias in a health status survey of elderly people. Age Ageing. 1989, 18: 177-182.View ArticlePubMedGoogle Scholar
- Launer LJ, Wind AW, Deeg DJ: Nonresponse pattern and bias in a community-based cross-sectional study of cognitive functioning among the elderly. Am J Epidemiol. 1994, 139: 803-812.PubMedGoogle Scholar
- Taylor JM, Cooper KL, Wei JT, Sarma AV, Raghunathan TE, Heeringa SG: Use of multiple imputation to correct for nonresponse bias in a survey of urologic symptoms among African-American men. Am J Epidemiol. 2002, 156: 774-782. 10.1093/aje/kwf110.View ArticlePubMedGoogle Scholar
- Dudley J, Jin S, Hoover D, Metz S, Thackeray R, Chmiel J: The Multicenter AIDS Cohort Study: retention after 9 1/2 years. Am J Epidemiol. 1995, 142: 323-330.PubMedGoogle Scholar
- Cartmel B, Moon TE, Levine N, Rodney S, Alberts D: Predictors of inactivation and reasons for participant inactivation during a skin cancer chemoprevention study. Cancer Epidemiol Biomarkers Prev. 2000, 9: 999-1002.PubMedGoogle Scholar
- Panser LA, Chute CG, Guess HA, LarsonKeller JJ, Girman CJ, Oesterling JE, Lieber MM, Jacobsen SJ: The natural history of prostatism: the effects of non-response bias. Int J Epidemiol. 1994, 23: 1198-1205.View ArticlePubMedGoogle Scholar
- Jacobsen SJ, Jacobson DJ, Girman CJ, Roberts RO, Rhodes T, Guess HA, Lieber MM: Natural history of prostatism: Risk factors for acute urinary retention. J Urol. 1997, 158: 481-487. 10.1016/S0022-5347(01)64508-7.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/6/58/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.