n2c2 Homepage


  • July 1 — Registration opens
  • July 23 (9AM ET) — Training data release
  • September 10 (9AM ET) — Test data release for concepts and for end-to-end evaluation
  • September 12 (11:59PM ET) — System outputs due for concepts and end-to-end
  • September 13 (9AM ET) — Test data release for relations
  • September 14 (11:59PM ET) — System outputs due for relations
  • September 21 (11:59PM ET) — Abstracts due
  • November 2 — Workshop (with AMIA Fall Symposium)

Task Description

This task aims to answer the question: “Can NLP systems automatically discover drug to adverse event (ADE) relations in clinical narratives?” The task builds on past medication extraction tasks, but examines a broader set of patients, diseases, and relations as compared with previous challenges. The task consists of three subtasks:

  1. Concepts: Identifying drug names, dosages, durations and other entities.
  2. Relations: Identifying relations of drugs with adverse drugs events (ADEs)[1] and other entities given gold standard entities.
  3. End-to-end: Identifying relations of drugs with ADEs and other entities on system predicted entities.

Definitions and background

The World Health Organization (WHO) gives the following definitions:

  • An adverse drug event (ADE) as “an injury resulting from medical intervention related to a drug;”
  • An adverse drug reaction (ADR) as “a response to a drug that is noxious and unintended and occurs at doses normally used in man for the prophylaxis, diagnosis or therapy of disease, or for modification of physiological function[2].”

These well-established definitions for ADE and ADR, which have been used for almost 50 years[3], continue to be used in a variety of recent research studies[4]-[6]. In their publication, Bates et al. state that the ADE definition is preferred because it is both “more comprehensive and clinically significant than the ADR”[7]. For this reason, our aim is to automatically detect ADEs. In addition, we are interested in accurately detecting drug names, dosages, durations, routes and other drug relations.


The data consists of discharge summaries of nearly ~500 discharge summaries drawn from the MIMIC-III (Medical Information Mart for Intensive Care III) clinical care database[1]. These data were annotated by 7 domain experts, including four physician assistant students and three nurses. Both entity tags and attributes were used to indicate the presence of drug and ADE information.

Data for the challenge will be released under a Rules of Conduct and Data Use Agreement. The data use agreement needs to be completed directly with MIMIC.

Sample gold standard files and the annotation guidelines will soon be available for download.

Evaluation Format

The evaluation will be conducted using withheld test data. Participating teams are asked to stop development as soon as they download the test data. Each team is allowed to upload (through this website) up to three system runs for each of the subtasks. System output is to be submitted in the exact format of the ground truth annotations, which will be provided by the organizers.


Participants are asked to submit a 500-word long abstract describing their methodologies. Abstracts may also have a graphical summary of the proposed architecture. The abstract should not exceed 2 pages (1.5pt line spacing, 12pt-font size). The authors of either top performing systems or particularly novel approaches will be invited to present or demonstrate their systems at the workshop. A special issue of a journal is planned following the workshop.

  • Ozlem Uzuner, co-chair, George Mason University
  • Michele Filannino, co-chair, MIT
  • Amber Stubbs, co-chair, Simmons College
  • Kevin Buchan, SUNY at Albany
  • Kahyun Lee, George Mason University
  • Susanne Churchill, Harvard Medical School
  • Isaac Kohane, Harvard Medical School


[1] Johnson AE, Pollard TJ, Shen L, Lehman LW, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG. "MIMIC-III, a freely accessible critical care database," Sci Data. 2016 May 24;3:160035. doi: 10.1038/sdata.2016.35.

[2] WHO. International drug monitoring: the role of national centres. Tech Rep Ser WHO 1972, no 498.

[3] Edwards IR, Aronson JK. "Adverse drug reactions: definitions, diagnosis, and management,” Lancet. 2000 Oct 7;356(9237):1255-9.

[4] Reddy VL, Pasha SJ, Rathinavelu M, and Reddy YP, “Assessment of knowledge, attitude and perception of pharmacovigilance and adverse drug reaction (ADR) reporting among the pharmacy students in south India,” IOSR J Pharm Biol Sci. 2014 February, 9(2):34-43.

[5] Petrovic M, Tangiisuran B, Rajkumar C, van der Cammen T, Onder G, “Predicting the Risk of Adverse Drug Reactions in Older Inpatients: External Validation of the GerontoNet ADR Risk Score Using the CRIME Cohort,” Drugs Aging. 2017 Feb;34(2):135-142. doi: 10.1007/s40266-016-0428-4.

[6] Toklu HZ, Soyalan M, Gültekin O, Özpolat M, Ayd?n MD, Günay AC, Yavuz DO, Akici A, Demirdamar R. "The Knowledge and Attitude of the Healthcare Professionals towards Pharmacovigilance and Adverse Drug Reaction Reporting in Northern Cyprus,” Pharmacovigilance 4:193. doi:10.4172/2329-6887.1000193.

[7] Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, Laffel G, Sweitzer BJ, Shea BF, Hallisey R, et al. "Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group,” JAMA. 1995 Jul 5;274(1):29-34.