Announcement of Data Release and Call for Participation
2014 i2b2/UTHealth Shared-Tasks and Workshop
on Challenges in Natural Language Processing for Clinical Data

Tentative Timeline
Registration: begins March 14, 2014
Training Data Release: 1 May, 2014
Test Data Release: 1 July, 2014
System Outputs Due: 3 July, 2014
Systems Due: 1 September, 2014
Paper Submission: 1 September, 2014
Workshop: November 14, 2014

UPDATE: The 2014 Workshop Registration online form is now available. Please submit the form before November 14, 2014.

There are now two new tracks for this year's i2b2 shared tasks: one on clinical NLP software usability, and another on novel data use for the new corpus. Please click here for more information.

The 2014 i2b2/UTHealth challenge consists of two traditional NLP tracks:

  • Track 1: De-identification: Removing protected health information (PHI) is a critical step in making medical records accessible to more people, yet it is a very difficult and nuanced task. This track addresses the problem of de-identifying medical records over a new set of over 1300 patient records, with surrogate PHI for participants to identify.
  • Track 2: Identifying risk factors for heart disease over time: Medical records for diabetic patients contain information about heart disease risk factors such as high blood pressure and cholesterol levels, obesity, smoking status, etc. This track aims to identify the information that is medically relevant to identifying heart disease risk, and track their progression over sets of longitudinal patient records.

The data for this task is provided by Partners HealthCare. All records have been fully de-identified and manually annotated for risk factors related to diabetes and heart disease risk factors.

Data for the challenge will be released under a Rules of Conduct and Data Use Agreement. Obtaining the data requires completing a registration, which will start March 14, 2014.

For either track, i2b2 will give the participants the option to submit their system software in addition to their system output for evaluation. Teams that submit software will be evaluated separately from teams that submit only system output. These systems will be evaluated for useability. Please see our upcoming software useability evaluation track for guidelines on expectations.

In addition, the i2b2 organizing committee is currently considering hosting a general "i2b2 Software for Clinical NLP" track that allows all past and current i2b2 challenge participants to submit and share with the community any software developed for any i2b2 shared-task.

Evaluation Dates, File Formats, and Evaluation Metrics

The evaluation for the NLP tracks will be conducted using withheld test data. Participating teams are asked to stop development as soon as they download the test data. Each team is allowed to upload (through this website) up to three system runs for each of the tasks. System output is expected in the form of standoff annotations, following the exact format of the ground truth annotations to be provided by the organizers.

Evaluation of submitted software will be evaluated by the program committee on factors such as ease of installation and ease of use.

Participants are asked to submit a short paper describing their system and analyzing their performance. Papers should be in AMIA style and should not exceed five pages. Authors of top performing systems and of particularly novel approaches will be invited to present or demo their systems at the workshop. Submitted software can be presented at the final workshop in the form of a poster or live demo.

UPDATE: The 2014 Workshop Registration online form is now available. Please submit the form before November 14, 2014.

We are pleased to announce two new tracks, added in addition to the NLP tracks 1 (de-identification) and 2 (heart disease risk factors) for the 2014 shared task and workshop.

The 2014 i2b2/UTHealth Challenge now introduces two additional tracks:

  • Track 3: Software Usability Assessment: This is a new track introduced this year for testing the usability of software. This track is meant to evaluate the i2b2 challenge software for how easily users learn and use the software to achieve their goals. The Software Usability Assessment Track is open to all current and prior i2b2 challenge participants who have developed systems on any of the i2b2 datasets since 2006. To participate in this track, the participants are expected to provide the name(s) and link(s) of their system(s). After evaluation, the comments of the evaluators will be provided to the participants, which should be useful for improving the usability of the system.
  • Track 4: Novel Data Use: The data released for this 2014 i2b2 challenge Tracks 1 (de-identification) and 2 (heart disease risk factors) are unique among publicly available clinical data sets in that they represent longitudinal data selected by an MD for the purpose of identifying risk factors in a diabetic population. However, these data can be used to answer other questions on these patients. This Track is for participants who want to build on their existing systems or the systems developed for Tracks 1 and 2, in order to answer new questions with these data. Some example questions include (but are not limited to): Are the medications having the desired effect? Is the patient responding to treatment for their hypertension? Is the patient responding to treatment with their lipids? Is the patient experiencing an adverse effect from their medications? Are some risk factors are more highly correlated with CAD than others?

Participants are encouraged to define their own questions to ask of these data.

In addition, participants of this track can contribute visualizations on NLP output for Tracks 1 and 2, or provide novel de-identification methods. Note that this track uses data from Tracks 1 and 2 and will follow the same data release timeline as those tracks.

The data for the 2014 i2b2 Challenge is provided by Partners HealthCare. All records have been fully de-identified and manually annotated for risk factors related to diabetes and heart disease risk factors.

Data for the challenge is released under a Rules of Conduct and Data Use Agreement. Obtaining the data requires completing a registration, which started March 14, 2014. Training data was released on May 1, 2014 and is now available to registered users.

Evaluation Dates, File Formats, and Evaluation Metrics

Evaluation of software submitted to Track 3: Usability evaluation will be performed with a relevant use case for the specific task and using formal usability testing software. Usability evaluators will be students and staff affiliated with the University Of Michigan Center for Managing Chronic Disease who have extensive experience working with patient charts and patients with complex chronic diseases. The evaluation criteria and procedures include:

  1. Quality of end user manual;
  2. Time/effort taken to install the software in a test environment and to execute an install-validation demonstration task;
  3. Time/effort it takes to replicate the procedures that generated the results the authors previously reported (if any);
  4. Time/effort it takes to replicate the procedures on a different corpus;
  5. Quality of coding and coding documentation.

Particpants in Track 3 are asked to email Dr. Anupama E. Gururaj (Anupama.E.Gururaj@uth.tmc.edu) the following information:

  • Participant information (name, affiliation, and email) *
  • Name of the system *
  • URL for downloading the system *
  • Manual(s) - any type of instructions for users to install and/or use the system
  • Documentation - for continued development of the software
  • Source codes - for open source systems

* denotes a required field

Track 4 will be evaluated on the basis of papers to be submitted to the organizing committee. Due to the open-ended nature of this track, there will be no system comparison or ranking. The aim is to demonstrate creative, clinically-relevant and varied uses of the Track 1 and Track 2 data. Therefore, papers describing novel and well-executed systems will be considered for publication in a journal special issue related to this shared task.

Participants in Track 4 are asked to submit a short paper describing their system and analyzing their performance. Papers should be in AMIA style and should not exceed five pages. Authors of top performing systems and of particularly novel approaches will be invited to present or demo their systems at the workshop. Submitted software can be presented at the final workshop in the form of a poster or live demo.

Organizing Committee:
Ozlem Uzuner, co-chair, SUNY at Albany
Amber Stubbs, co-chair, SUNY at Albany
Hua Xu, co-chair, University of Texas, Houston
John Aberdeen, MITRE
Susanne Churchill, Partners Healthcare
Cheryl Clark, MITRE
Dina Demner Fushman, NIH/NLM
Joshua Denny, Vanderbilt University
Bill Hersh, Oregon Health and Science University
Lynette Hirschman, MITRE
Issac Kohane, Partners Healthcare
Vishesh Kumar, Massachusetts General Hospital
Anna Rumshisky, UMass Lowell
Stanley Shaw, Massachusetts General Hospital
Peter Szolovits, MIT
Meliha Yetisgen, University of Washington
Kai Zheng, University of Michigan

Please see the announcements for more information. Questions on the challenge can be addressed to Ozlem Uzuner, n2c2.challenges@gmail.com.