Files and links (2)
Published (Version of record)CC BY V4.0, Open Access
Published (Version of record)CC BY V4.0, Open
Journal article
Large expert-curated database for benchmarking document similarity detection in biomedical literature search
Database, Vol.2019, pp.1-66
2019
PMID: 33326193
Metrics
1 Record Views
Abstract
Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.
Please note: The full list of RELISH Consortium members can be found in the published version of the paper. Only a subset is listed in the metadata recorded here.
Details
- Title
- Large expert-curated database for benchmarking document similarity detection in biomedical literature search
- Creators
- Peter Brown - Griffith UniversityRELISH ConsortiumYaoqi Zhou - Griffith University
- Contributors
- RELISH Consortium (Institution)Aik-Choon Tan - College Station Medical CenterMohamed A. El-Esawi - Tanta UniversityThomas Liehr - Friedrich Schiller University JenaOliver Blanck - College Station Medical CenterDouglas P. Gladue - ARS, USDA, Plum Isl Anim Dis Ctr, Greenport, NY 11944 USAGabriel M. F. Almeida - Univ Jyvaskyla, Dept Biol & Environm Sci, Jyvaskyla, FinlandTomislav Cernava - Graz University of TechnologyCarlos O. Sorzano - CSIC, CNB, Natl Biotechnol Ctr, Dept Macromol Struct, Madrid, SpainAndy W. K. Yeung - Univ Hong Kong, Fac Dent, Oral & Maxillofacial Radiol Appl Oral Sci & Commu, Hong Kong, Peoples R ChinaMichael S. Engel - Univ Kansas, Div Entomol, Biodivers Inst, Lawrence, KS 66045 USAArun Richard Chandrasekaran - SUNY Albany, RNA Inst, Albany, NY 12222 USAThilo Muth - Robert Koch InstituteMartin S. Staege - Martin Luther University Halle-WittenbergSwapna V. Daulatabad - Indiana Univ Purdue Univ, IU Sch Informat & Comp, Dept BioHlth Informat, Indianapolis, IN 46202 USADarius Widera - Univ Reading, Sch Pharm Stem Cell Biol & Regenerat Med, Reading, Berks, EnglandJunpeng Zhang - Dali UniversityZhongheng Zhang - Zhejiang Univ, Sch Med, Sir Run Run Shaw Hosp, Dept Emergency, Hangzhou 310016, Zhejiang, Peoples R ChinaAdrian Meule - Univ Hosp Munich LMU, Dept Psychiat & Psychotherapy, Munich, GermanyMarco Cascella - College Station Medical CenterKen Honjo - Univ Tsukuba, Fac Life & Environm Sci, Ibaraki, JapanWilly A. Flegel - NIH, Dept Transfus Med, Bethesda, MD USAOlivier Pourret - UniLaSalleCarl S. Goodyear - University of GlasgowMark J. van Raaij - CSIC, CNB, Natl Biotechnol Ctr, Dept Macromol Struct, Madrid, SpainCong-Cong Yin - Henry Ford Hlth Syst, Dept Immunol, Detroit, MI USAZuzanna Bukowy-Bieryllo - Polish Acad Sci, Inst Human Genet, Poznan, PolandLuca G. Campana - Univ Padua, Dept Surg Oncol & Gastroenterol DISCOG, Padua, ItalyNicholas A. Kurniawan - Eindhoven University of TechnologyDavid Lalaouna - University of StrasbourgFelix J. Huttner - Heidelberg Univ, Dept Gen Visceral & Transplantat Surg, Heidelberg, GermanyBrooke A. Ammerman - Univ Notre Dame, Psychol, Notre Dame, IN 46556 USAFelix Ehret - Supreme Council Of HealthPaul A. Cobine - Auburn Univ, Dept Biol Sci, Auburn, AL 36849 USAEne-Choo Tan - KK Womens & Childrens Hosp, KK Res Ctr, Singapore, SingaporeHyemin Han - University of AlabamaWenfeng Xia - Wellcome TrustChristopher McCrum - Maastricht UniversityRuud P. M. Dings - College Station Medical CenterFrancesco Marinello - Univ Padua, Dept Land Environm Agr & Forestry, Padua, ItalyHenrik Nilsson - Univ Gothenburg, Dept Biol & Environm Sci, Gothenburg, SwedenBrett Nixon - University of NewcastleKonstantinos Voskarides - University of CyprusLong Yang - Shandong Agr Univ, Coll Plant Protect, Agr Big Data Res Ctr, Tai An, Shandong, Peoples R ChinaVincent D. Costa - NIMH, Neuropsychol Lab, Bldg 9, Bethesda, MD 20892 USAJohan Bengtsson-Palme - Univ Wisconsin, Wisconsin Inst Discovery, Madison, WI USAWilliam Bradshaw - Univ Oxford, Struct Genom Consortium, Oxford, EnglandDominik G. Grimm - Weihenstephan Triesdorf Univ Appl Sci, TUM Campus Straubing Biotechnol & Sustainabil, Bioinformat, Straubing, GermanyNitin Kumar - Univ Michigan, Cardiovasc Res, Ann Arbor, MI 48109 USAElvis Martis - Bombay Coll Pharm, Pharmaceut Chem, Mumbai, Maharashtra, IndiaDaniel Prieto - Inst Invest Biol Clemente Estable, Dev Neurobiol, Montevideo, UruguaySandeep C. Sabnis - Institut de Recherche en Génie Civil et MécaniqueSaid E. D. R. Amer - Kafr El Sheikh Univ, Fac Sci, Dept Zool, Kafr Al Sheikh, EgyptAlan W. C. Liew - Griffith Univ, Sch Informat & Commun Technol, Gold Coast, Qld 4222, AustraliaPaul Perco - College Station Medical CenterFarid Rahimi - ACTGiuseppe Riva - Univ Cattolica Sacro Cuore, Appl Technol Neuropsychol Lab, Milan, ItalyChongxing Zhang - College Station Medical CenterHari P. Devkota - Kumamoto UniversityKoichi Ogami - Nagoya City UniversityZarrin Basharat - University of KarachiWalter Fierz - Labormed Zentrum Dr Risch, Vaduz, LiechtensteinRobert Siebers - Univ Otago, Med, Dunedin, New ZealandKok-Hian Tan - College Station Medical CenterKaren A. Boehme - University of FreiburgPeter Brenneisen - Griffith UniversityJames A. L. Brown - Griffith UniversityBrian P. Dalrymple - Univ Western Australia, Inst Agr, Perth, WA, AustraliaDavid J. Harvey - University of OxfordGrace Ng - SingHlth Polyclin, Punggol Polyclin, Singapore, SingaporeSebastiaan Werten - College Station Medical CenterMark Bleackley - La Trobe Univ, La Trobe Inst Mol Sci, Biochem & Genet, Melbourne, Vic, AustraliaZhanwu Dai - University of BordeauxRaman Dhariwal - Agr & Agri Food Canada, Lethbridge Res & Dev Ctr, Lethbridge, AB, CanadaYael Gelfer - George Institute for Global HealthMarcus D. Hartmann - MaxPawel Miotla - College Station Medical CenterRadu Tamaian - ICIT, ICSI Analyt Natl Res & Dev, Ramnicu Valcea, VL, RomaniaPragashnie Govender - Univ KwaZulu Natal, Occupat Therapy, Westville Campus, Durban, South AfricaOliver J. Gurney-Champion - Inst Canc Res, Joint Dept Phys, London, EnglandJoonas H. Kauppila - Karolinska InstitutetXiaolei Zhang - College Station Medical CenterNatalia Echeverria - Univ Republica, Fac Ciencias, Ctr Invest Nucl, Lab Virol Mol, Montevideo, UruguaySanthilal Subhash - Univ Gothenburg, Dept Med Biochem & Cell Biol, Gothenburg, SwedenHannes Sallmon - Charité - Universitätsmedizin BerlinMarco Tofani - Bambino Gesu Pediat Hosp, Dept Neurosci & Neurorehabil, Neurorehabil Unit, Rome, ItalyTaeok Bae - Indiana Univ Sch Med Northwest, Microbiol & Immunol, Gary, IN USAOliver Bosch - University of RegensburgParaic O. Cuiv - Univ Queensland, Diamantina Inst, Brisbane, Qld, AustraliaAntoine Danchin - Institut CochinBarthelemy Diouf - St Jude Childrens Res Hosp, Pharmaceut Sci Dept, 332 N Lauderdale St, Memphis, TN 38105 USATuomas Eerola - Univ Durham, Mus, Durham, EnglandEvangelos Evangelou - University of IoanninaFabian V. Filipp - Ludwig-Maximilians-Universität MünchenHannes Klump - College Station Medical CenterLukasz Kurgan - Virginia Commonwealth UniversitySimon S. Smith - Univ Queensland, Inst Social Sci Res, Brisbane, Qld, AustraliaOlivier Terrier - University of LyonNeil Tuttle - Griffith UniversityDavid B. Ascher - Univ Melbourne, Dept Biochem & Mol Biol, Parkville, Vic, AustraliaEdward Narayan - The University of Queensland
- Publication Details
- Database, Vol.2019, pp.1-66
- Publisher
- Oxford University Press
- Number of pages
- 66
- Grant note
- Griffith University Gowonda HPC Cluster; Queensland Cyber Infrastructure Foundation.
- Identifiers
- 991013269109102368
- Copyright
- © The Author(s) 2019
- Academic Unit
- Faculty of Science and Engineering
- Language
- English
- Resource Type
- Journal article