Title: Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000-2010): are the searches clearly explained, systematic and reproducible?
Abstract: Research Synthesis MethodsVolume 5, Issue 2 p. 116-130 Original Article Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Mary M. Mullins, Corresponding Author Mary M. Mullins Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USA Correspondence to: Mary Mullins, Centers for Disease Control and Prevention, Division of HIV/AIDS Prevention, Prevention Research Branch, 1600 Clifton Rd., Mailstop E-37, Atlanta, Georgia, 30333, USA. E-mail: [email protected]Search for more papers by this authorJulia B. DeLuca, Julia B. DeLuca Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USASearch for more papers by this authorNicole Crepaz, Nicole Crepaz Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USASearch for more papers by this authorCynthia M. Lyles, Cynthia M. Lyles Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USASearch for more papers by this author Mary M. Mullins, Corresponding Author Mary M. Mullins Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USA Correspondence to: Mary Mullins, Centers for Disease Control and Prevention, Division of HIV/AIDS Prevention, Prevention Research Branch, 1600 Clifton Rd., Mailstop E-37, Atlanta, Georgia, 30333, USA. E-mail: [email protected]Search for more papers by this authorJulia B. DeLuca, Julia B. DeLuca Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USASearch for more papers by this authorNicole Crepaz, Nicole Crepaz Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USASearch for more papers by this authorCynthia M. Lyles, Cynthia M. Lyles Division of HIV/AIDS Prevention, CDC, Atlanta, GA, USASearch for more papers by this author First published: 08 October 2013 https://doi.org/10.1002/jrsm.1098Citations: 23 The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinkedInRedditWechat Abstract Systematic reviews are an essential tool for researchers, prevention providers and policy makers who want to remain current with the evidence in the field. Systematic review must adhere to strict standards, as the results can provide a more objective appraisal of evidence for making scientific decisions than traditional narrative reviews. An integral component of a systematic review is the development and execution of a comprehensive systematic search to collect available and relevant information. A number of reporting guidelines have been developed to ensure quality publications of systematic reviews. These guidelines provide the essential elements to include in the review process and report in the final publication for complete transparency. We identified the common elements of reporting guidelines and examined the reporting quality of search methods in HIV behavioral intervention literature. Consistent with the findings from previous evaluations of reporting search methods of systematic reviews in other fields, our review shows a lack of full and transparent reporting within systematic reviews even though a plethora of guidelines exist. This review underscores the need for promoting the completeness of and adherence to transparent systematic search reporting within systematic reviews. Published 2013. This article is a U.S. Government work and is in the public domain in the USA. Supporting Information Filename Description Appendix_A.1_Database_Overview.docWord document, 25 KB Supporting info item Appendix_A.2-CINAHL_Search.docWord document, 28.5 KB Supporting info item Appendix_A.3-Cochrane_Search.docWord document, 21.5 KB Supporting info item Appendix_A.4-EMBASE_Search.docWord document, 27.5 KB Supporting info item Appendix_A.5-MEDLINE_Search.docWord document, 29 KB Supporting info item Appendix_A.6-PsycINFO_Search.docWord document, 27.5 KB Supporting info item Appendix_A.7-Sociological_Abstracts.docWord document, 27 KB Supporting info item Appendix_B-102_included_References.docWord document, 50 KB Supporting info item Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article. References Auerbach JD, Hayes RJ, Kandathil SM. 2006. Overview of effective and promising interventions to prevent HIV infection. World Health Organization Technical Report Series 938: 43–78. Bateganya MH, Abdulwadud OA, Kiene SM. 2007. Home-based HIV voluntary counseling and testing in developing countries. Cochrane Database of Systematic Reviews 4: CD006493. Bateganya MH, Abdulwadud OA, Kiene SM. 2010. Home-based HIV voluntary counseling and testing (VCT) for improving uptake of testing. Cochrane Database of Systematic Reviews 7: CD006493. Berg R. 2009. The effectiveness of behavioural and psychosocial HIV/STI prevention interventions for MSM in Europe a systematic review. Euro Surveillance 14(48): pii: 19430. BMJ Evidence Based Centre. 2012. Clinical evidence: study design search filters. Available at: http://clinicalevidence.bmj.com/x/set/static/ebm/learn/665076.html Booth A. 2006. "Brimful of STARLITE": toward standards for reporting literature searches. Journal of the Medical Library Association 94(4): 421–429. Boynton J, Glanville J, McDaid D, Lefebvre C. 1998. Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. Journal of Information Science 24(3): 137–157. Burton J, Darbes LA, Operario D. 2010. Couples-focused behavioral interventions for prevention of HIV: systematic review of the state of evidence. AIDS and Behavior 14(1): 1–10. Centre for Reviews and Dissemination. 2001. Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews. Report Number 4 ( 2nd edn). Centre for Reviews and Dissemination. 2009. Systematic reviews: CRD's guidance for undertaking reviews in health care. Available at: http://www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf Chalmers I, Haynes B. 1994. Reporting, updating, and correcting systematic reviews of the effects of health care. British Medical Journal 309(6958): 862–865. Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. 2012a. Methodogical standards for the reporting of new Cochrane Intervention Reviews Version 1.1 (17 December 2012). Available at: http://www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR%20Reporting%20standards%201.1_17122012_1.pdf Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. 2012b. Methodological standards for the conduct of new Cochrane Intervention Reviews Version 2.2 (17 December 2012). Available at: http://www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR_conduct_standards%202.2%2017122012.pdf Chen P. 2007. Measures needed to strengthen strategic HIV/AIDS prevention programmes in China. Asia Pacific Journal of Public Health 19(1): 3–7. Coffin P. 2000. Syringe availability as HIV prevention a review of modalities. Journal of Urban Health 77(3): 306–330. Counsell C. 1997. Formulating questions and locating primary studies for inclusion in systematic reviews. Annals of Internal Medicine 127(5): 380–387. CSA ProQuest-Sociological Abstracts [database online]. 2005. [Accessed May 2010; Updated May 2011]. PROQUEST, Ann Arbor, MI. Darbes L, Crepaz N, Lyles C, Kennedy G, Rutherford G. 2008. The efficacy of behavioral interventions in reducing HIV risk behaviors and incident sexually transmitted diseases in heterosexual African Americans. AIDS 22(10): 1177–1194. DeLuca JB, Mullins MM, Lyles CM, Crepaz N, Kay L, Thadiparthi S. 2008. Developing a comprehensive search strategy for evidence-based systematic review. Evidence Based Library and Information Practice 3(1): 3–32. DesJarlais DC, Semaan S. 2005. Interventions to reduce the sexual risk behaviour of injecting drug users. International Journal of Drug Policy 16(Supplement 1): S58–S66. EBSCOhost-CINAHL [database online]. 1981. [Accessed May 2010; Updated 2011]. EBSCO Publishers, Ipswich, MA. Elwy AR, Hart GJ, Hawkes S, Petticrew M. 2002. Effectiveness of interventions to prevent sexually transmitted infections and human immunodeficiency virus in heterosexual men a systematic review. Archives of Internal Medicine 162(16): 1818–1830. EQUATOR Network. 2011. Available at: http://www.equator-network.org/ European Centre for Disease Control and Prevention. 2011. Evidence-based methodologies for public health – How to assess the best available evidence when time is limited and there is a lack of sound evidence. Available at: http://ecdc.europa.eu/en/publications/Publications/1109_TER_evidence_based_methods_for_public_health.pdf Fehrmann P, Thomas J. 2011. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods 2(1): 15–32. Fisher JD, Smith L. 2009. Secondary prevention of HIV infection: the current state of prevention for positives. Current Opinion in HIV and AIDS 4(4): 279–287. Foss AM, Hossain M, Vickerman PT, Watts CH. 2007. A systematic review of published evidence on intervention impact on condom use in sub-Saharan Africa and Asia. Sexually Transmitted Infections 83(7): 510–516. Golder S, Loke Y, McIntosh HM. 2008. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology 61(5): 440–448. Hailey D. 2003. Toward transparency in health technology assessment a checklist for HTA reports. International Journal of Technology Assessment in Health Care 19(1): 1–7. Hammerstrom K, Wade A, Jorgensen A-MK. 2010. Searching for studies a guide for information retrieval for Campbell Systematic Reviews. Campbell Systematic Reviews (Supplement 1). DOI: 10.4073/csrs.2010.1 Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. 1994. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association 1(6): 447–458. Health Technology Assessment. 2007. A checklist for HTA reports (updated 2007). Available at: http://www.inahta.org/HTA/Checklist/ Hemingway P, Brereton N. 2001. What is a systematic review? Available at: http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/syst-review.pdf Herbst JH, Task Force on Community Preventive Services. 2007. Recommendations for use of behavioral interventions to reduce the risk of sexual transmission of HIV among men who have sex with men. American Journal of Preventive Medicine 32(Supplement 4): S36–S37. Herbst JH, Sherba RT, Crepaz N, DeLuca JB, Zohrabyan L, Stall RD, Lyles CM, HIV/AIDS Prevention Research Synthesis Team. 2005. A meta-analytic review of HIV behavioral interventions for reducing sexual risk behavior of men who have sex with men. Journal of Acquired Immune Deficiency Syndromes 39(2): 228–241. Herbst JH, Beeker C, Mathew A, McNally T, Passin WF, Kay LS, Crepaz N, Lyles CM, Briss P, Chattopadhyay S, Johnson RL. 2007a. The effectiveness of individual-, group-, and community-level HIV behavioral risk-reduction interventions for adult men who have sex with men. A systematic review. American Journal of Preventive Medicine 32(Supplement 4): 38–67. Herbst JH, Kay LS, Passin WF, Lyles CM, Crepaz N, Marin BV, HIV/AIDS Prevention Research Synthesis (PRS) Team. 2007b. A systematic review and meta-analysis of behavioral interventions to reduce HIV risk behaviors of Hispanics in the United States and Puerto Rico. AIDS and Behavior 11(1): 25–47. JPT Higgins, S Green (eds.) 2006a. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6 [updated September 2006]. Available at: http://www.cochrane.org/training/cochrane-handbook JPT Higgins, S Green (eds.) 2006b. Locating and selecting studies. In Cochrane Handbook for Systematic Reviews of Interventions 4.2.6 (updated September 2006); Section 5. The Cochrane Collaboration. Available at: http://www.cochrane.org/training/cochrane-handbook Hoffmann O, Boler T, Dick B. 2006. Achieving the global goals on HIV among young people most at risk in developing countries: young sex workers, injecting drug users and men who have sex with men. World Health Organization Technical Report Series 938: 287–315. Hogben M, McNally T, McPheeters M, Hutchinson AB. 2007. The effectiveness of HIV partner counseling and referral services in increasing identification of HIV-positive individuals a systematic review. American Journal of Preventive Medicine 33(2 Supplement 1): S89–S100. Institute of Medicine (IOM). 2011. Finding What Works in Health Care: Standards for Systematic Reviews. The National Academies Press, Washington, D.C. Jankowski TA. 2008. Essential Guide to Becoming an Expert Searcher: Proven Techniques, Strategies, and Tips to Finding Information. Neal -Schuman Publishers Inc, New York. Katcher B. 2006. MEDLINE: A Guide to Effective Searching in PubMed and Other Interfaces ( 2nd edn). Ashbury Press, San Francisco, CA. Lefebvre C, Manheimer E, Glanville J. 2008. Chapter 6: searching for studies. In JPT Higgins, S Green (eds.), Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0 (updated February 2008). The Cochrane Collaboration. Available at: http://www.cochrane.org/training/cochrane-handbook Lefebvre C, Manheimer E, Glanville J. 2009. Chapter 6: searching for studies. In JPT Higgins, S Green (eds.), Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 (updated September 2009). The Cochrane Collaboration. Available at: http://www.cochrane.org/training/cochrane-handbook Lefebvre C, Manheimer E, Glanville J. 2011. Chapter 6: searching for studies. In JPT Higgins, S Green (eds.), Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 (updated March 2011). The Cochrane Collaboration. Available at: http://www.cochrane.org/training/cochrane-handbook Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. 2009. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology 62(10): e1–e34. Lyles CM, Kay LS, Crepaz N, Herbst JH, Passin WF, Kim AS, Rama SM, Thadiparthi S, DeLuca JB, Mullins MM, HIV/AIDS Prevention Research Synthesis Team. 2007. Best-evidence interventions: findings from a systematic review of HIV behavioral interventions for US populations at high risk, 2000–2004. American Journal of Public Health 97(1): 133–143. Manhart LE, Holmes KK. 2005. Randomized controlled trials of individual-level, population-level, and multilevel interventions for preventing sexually transmitted infections: what has worked? Journal of Infectious Diseases 191(Supplement 1): S7–S24. McCoy SI, Kangwende RA, Padian NS. 2010. Behavior change interventions to prevent HIV infection among women living in low and middle income countries a systematic review. AIDS and Behavior 14(3): 469–482. McGowan J, Sampson M. 2005. Systematic reviews need systematic searchers. The Journal of the Medical Library Association 93(1): 74–80. McGowan J, Sampson M, Lefebvre C. 2010. An evidence based checklist for the peer review of electronic search strategies (PRESS EBC). Evidence Based Library and Information Practice 5(1): 149–153. Meader N, Li R, Des J, Pilling S. 2010. Psychosocial interventions for reducing injection and sexual risk behaviour for preventing HIV in drug users. Cochrane Database of Systematic Reviews 1: CD007192. Medical Library Association. 2011. Medical Library Association policy statement: role of expert searching in health science libraries. Available at: http://www.mlanet.org/resources/expert_search/policy_expert_search.html Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. 1999. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 354(9193): 1896–1900. Moher D, Liberati A, Tetzlaff J, Altman DG. 2009. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of Internal Medicine 151(4): 264–269, W64. Moher D, Altman DG, Liberati A, Tetzlaff J. 2011. PRISMA statement. Epidemiology 22(1): 128. National Library of Medicine. 2011. Fact sheet: what's the difference between MEDLINE and PubMed? Available at: http://www.nlm.nih.gov/pubs/factsheets/dif_med_pub.html Niederstadt C, Droste S. 2010. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. International Journal of Technology Assessment in Health Care 26(4): 450–457. OVID-EMBASE [database online]. 1988. [Accessed May 2010; Updated May 2011] Wolters, Kluwer, New York, NY. OVID-MEDLINE [database online]. 1988. [Accessed May 2010; Updated 2011] Wolters Kluwer, New York, NY. OVID-PsycINFO [database online]. 1988. [Accessed May 2010; Updated 2011] Wolters Kluwer, New York, NY. Padian NS, McCoy SI, Balkus JE, Wasserheit JN. 2010. Weighing the gold in the gold standard: challenges in HIV prevention research. AIDS 24(5): 621–635. Patrick TB, Demiris G, Folk LC, Moxley DE, Mitchell JA, Tao D. 2004. Evidence-based retrieval in evidence-based medicine. Journal of the Medical Library Association 92(2): 196–199. Paul-Ebhohimhen VA, Poobalan A, van Teijlingen ER. 2008. A systematic review of school-based sexual health interventions to prevent STI/HIV in sub-Saharan Africa. BMC Public Health 8: 4. Rees R, Kavanagh J, Burchett H, Shepherd J, Brunton G, Harden A, Thomas J, Oliver S, Oakley A. 2004. HIV Health Promotion and Men who have Sex with Men (MSM): A Systematic Review of Research Relevant to the Development and Implementation of Effective and Appropriate Interventions. Eppi Centre, London. Relevo R, Balshem H. 2011. Finding evidence for comparing medical interventions. In Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Agency for Healthcare Research and Quality, Rockville, MD; 1–27. Ross DA. 2010. Behavioural interventions to reduce HIV risk: what works? AIDS 24(Supplement 4): S4–S14. Russak SM, Ortiz DJ, Galvan FH, Bing EG. 2005. Protecting our militaries a systematic literature review of military human immunodeficiency virus/acquired immunodeficiency syndrome prevention programs worldwide. Military Medicine 170(10): 886–897. Sampson M, McGowan J. 2006. Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology 59(10): 1057–1063. Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. 2008a. PRESS: Peer Review of Electronic Search Strategies. Canadian Agency for Drugs and Technologies in Health, Ottawa. Available at: http://www.cadth.ca/media/pdf/477_PRESS-Peer-Review-Electronic-Search-Strategies_tr_e.pdf Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. 2008b. No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology 61(8): 748–754. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. 2009. An evidence-based practice guideline for the peer review of electronic search strategies. Journal of Clinical Epidemiology 62(9): 944–952. Shahmanesh M, Patel V, Mabey D, Cowan F. 2008. Effectiveness of interventions for the prevention of HIV and other sexually transmitted infections in female sex workers in resource poor setting a systematic review. Tropical Medicine and International Health 13(5): 659–679. Shea B, Dube C, Moher D. 2001. Assessing the quality of reports of systematic reviews: the QUOROM statement compared to other tools. In M Egger, GD Smith, DG Altman (eds.), Systematic Reviews in Health Care: Meta-Analysis in Context. BMJ Books, London; 122–139. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. 2007. Development of AMSTAR a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7: 10. Shepherd J, Kavanagh J, Picot J, Cooper K, Harden A, Barnett-Page E, Jones J, Clegg A, Hartwell D, Frampton GK, Price A. 2010. The effectiveness and cost-effectiveness of behavioural interventions for the prevention of sexually transmitted infections in young people aged 13–19 a systematic review and economic evaluation. Health Technology Assessment 14(7): iii–iv. Simera I, Moher D, Hoey J, Schulz KF, Altman DG. 2010. A catalogue of reporting guidelines for health research. European Journal of Clinical Investigation 40(1): 35–53. Sorsdahl K, Ipser JC, Stein DJ. 2009. Interventions for educating traditional healers about STD and HIV medicine. Cochrane Database of Systematic Reviews 4: CD007190. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB. 2000. Meta-analysis of observational studies in epidemiology a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283(15): 2008–2012. Takacs IG, Demetrovics Z. 2009. The efficacy of needle exchange programs in the prevention of HIV and hepatitis infection among injecting drug users. [Hungarian]. Psychiatria Hungarica 24(4): 264–281. The Cochrane Library. 2010. Issue 5. Wiley, Chichester. The Cochrane Library. 2011. Issue 5. Wiley, Chichester. The InterTASC Information Specialists' Sub-Group. 2012. Search filter resource. Available at: http://www.york.ac.uk/inst/crd/intertasc/ The Librarians' Searching Group. 2013. Things to keep in mind when search MEDLINE. Available at: http://www.nyam.org/fellows-members/ebhc/eb_medline2.html Trelle S, Shang A, Nartey L, Cassell JA, Low N. 2007. Improved effectiveness of partner notification for patients with sexually transmitted infections: systematic review. British Medical Journal 334(7589): 354. Underhill K, Operario D, Montgomery P. 2007a. Abstinence-only programs for HIV infection prevention in high-income countries. Cochrane Database of Systematic Reviews 4: CD005421. Underhill K, Operario D, Montgomery P. 2007b. Systematic review of abstinence-plus HIV prevention programs in high-income countries. PLoS Medicine 4(9): e275. Underhill K, Montgomery P, Operario D. 2008. Abstinence-plus programs for HIV infection prevention in high-income countries. Cochrane Database of Systematic Reviews 1: CD007006. Vandenbroucke JP. 2009. STREGA, STROBE, STARD, SQUIRE, MOOSE, PRISMA, GNOSIS, TREND, ORION, COREQ, QUOROM, REMARK… and CONSORT: for whom does the guideline toll? Journal of Clinical Epidemiology 62(6): 594–596. Vidanapathirana J, Abramson MJ, Forbes A, Fairley C. 2005. Mass media interventions for promoting HIV testing. Cochrane Database of Systematic Reviews 3: CD004775. Vijayakumar G, Mabude Z, Smit J, Beksinska M, Lurie M. 2006. A review of female-condom effectiveness: patterns of use and impact on protected sex acts and STI incidence. International Journal of STD and AIDS 17(10): 652–659. White VJ, Glanville JM, Lefebvre C, Sheldon TA. 2001. A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. Journal of Information Science 27(6): 357–370. Wright NM, Walker J. 2006. Homelessness and drug use – a narrative systematic review of interventions to promote sexual health. AIDS Care 18(5): 467–478. Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. 2009. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association 97(1): 21–29. Zhang L, Sampson M, McGowan J. 2006. Reporting of the role of the expert searcher in Cochrane reviews. Evidence Based Library and Information Practice 1(4): 3–16. Citing Literature Volume5, Issue2June 2014Pages 116-130 ReferencesRelatedInformation