AAPM Thoracic Auto-segmentation Challenge

Organized by MarkGooding - Current server time: Sept. 20, 2017, 4:19 p.m. UTC


Pre-AAPM Challenge
June 19, 2017, midnight UTC


AAPM Live Challenge
Aug. 2, 2017, midnight UTC


Competition Ends


Numerous auto-segmentation methods exist for Organs at Risk in radiotherapy. The overall objective of this auto-segmentation grand challenge is to provide a platform for comparison of various auto-segmentation algorithms when they are used to delineate organs at risk (OARs) from CT images for thoracic patients in radiation treatment planning. The results will provide an indication of the performances achieved by various auto-segmentation algorithms and can be used to guide the selection of these algorithms for clinic use if desirable. The challenge is made up of multiple phases:

Phase 1 will conducted via this website in advance of the AAPM meeting. 12 test images will be provided and results will be submitted online. The 3 top place contestants in this phase will be invited to present at the challenge symposium at AAPM.

Phase 2 will be conducted live at the AAPM. A further 12 test images will be provided for evaluation, and participants will have 2 hours to generate results. Participants need not have participated in Phase 1 to be part of Phase 2.

Symposium Following Phase 2 a symposium will be held at which the results of both previous phases will be presented.

Details of the AAPM symposium

Phase 3 will be an on-going benchmarking conducted via this website. Both test sets from phase 1 and 2 will be included within the on-going assessment.

The Prize

The top 3 participants in the Pre-AAPM challenge will be invited to present at the Challenge Symposium at the AAPM 2017 Annual Meeting.

The top 3 participants in the AAPM live challenge will receive conference registration reimbursement and a certificate of merit from AAPM.

Get Started

  1. Register here to get access
  2. Download the data after approval
  3. Submit your results
  4. Win the Challenge

Important Dates

  • May 19, 2017 Release of training data
  • Jun 19, 2017 Jun 21, 2017 Release of off-site test data
  • Jul 03, 2017 Jul 17, 2017 Off-site test result submission close.
  • Jul 10, 2017 Jul 18, 2017 Off-site test results released.
  • Jul 30 - Aug 03, 2017 AAPM Annual Meeting
    • Aug 02, 2017 Live challenge at AAPM. New test data released.
    • Aug 03, 2017 Segmentation symposium at AAPM. Results announced.
  • Aug 04, 2017 Online leader board available.

New! Live challenge information is now available.

Contouring Guidelines


Standard name: Esophagus

RTOG Atlas description: The esophagus should be contoured from the beginning at the level just below the cricoid to its entrance to the stomach at GE junction. The esophagus will be contoured using mediastinal window/level on CT to correspond to the mucosal, submucosa, and all muscular layers out to the fatty adventitia.

Additional notes: The superior-most slice of the esophagus is the slice below the first slice where the lamina of the cricoid cartilage is visible (+/- 1 slice). The inferior-most slice of the esophagus is the first slice (+/- 1 slice) where the esophagus and stomach are joined, and at least 10 square cm of stomach cross section is visible.


Standard name: Heart

RTOG Atlas description: The heart will be contoured along with the pericardial sac. The superior aspect (or base) will begin at the level of the inferior aspect of the pulmonary artery passing the midline and extend inferiorly to the apex of the heart.

Additional notes: Inferior vena cava is excluded or partly excluded starting at slice where at least half of the circumference is separated from the right atrium.


Standard names: Lung_L, Lung_R

RTOG Atlas description: Both lungs should be contoured using pulmonary windows. The right and left lungs can be contoured separately, but they should be considered as one structure for lung dosimetry. All inflated and collapsed, fibrotic and emphysematic lungs should be contoured, small vessels extending beyond the hilar regions should be included; however, pre GTV, hilars and trachea/main bronchus should not be included in this structure.

Additional notes: Tumor is excluded in most data, but size and extent of excluded region are not guaranteed. Hilar airways and vessels greater than 5 mm (+/- 2 mm) diameter are excluded. Main bronchi are always excluded, secondary bronchi may be included or excluded. Small vessels near hilum are not guaranteed to be excluded. Collapsed lung may be excluded in some scans. Regions of tumor or collapsed lung that are excluded from training and test data will be masked out during evaluation, such that scores are affected by segmentation choices in those regions.

Spinal cord

Standard name: SpinalCord

RTOG Atlas description: The spinal cord will be contoured based on the bony limits of the spinal canal. The spinal cord should be contoured starting at the level just below cricoid (base of skull for apex tumors) and continuing on every CT slice to the bottom of L2. Neuroformanines should not be included.

Additional notes: Spinal cord may be contoured beyond cricoid superiorly, and beyond L2 inferiorly. Contouring to base of skull is not guaranteed for apical tumors.

Evaluation Criteria

Auto-segmented contours will be compared against the manual contours for all test datasets using the following evaluation metrics as implemented in Plastimatch. RTSS will be voxelised to CT resolution for all calculations. Evaluation will be performed in 3D. To prevent uncertainty with the extent to which the Spinal cord and Esophagus should be contoured, submitted contours will be cropped to the extent of the test data. Therefore, you will not be penalised for contouring too great an extent of the structure in the inferior-superior direction, but will for under-segmentation.

Dice Coefficient

This is a measure of relative overlap, where 1 represents perfect agreement and 0 represents no overlap.

Dice calculation

where X and Y are the ground truth and test regions.

Mean surface distance

The directed average Hausdorff measure is the average distance of a point in X to its closest point in Y. That is:

\[ \vec{d}_{H,\mathrm{avg}}(X,Y) = \frac{1}{|X|} \sum_{x \in X} \min_{y \in Y} d (x,y) \]

The (undirected) average Hausdorff measure is the average of the two directed average Hausdorff measures:

\[ d_{H,\mathrm{avg}}(X,Y) = \frac{\vec{d}_{H,\mathrm{avg}}(X,Y) + \vec{d}_{H,\mathrm{avg}}(Y,X)}{2} \]

Hausdorff distance (95% Hausdorff distance)

The directed percent Hausdorff measure, for a percentile r, is the r th percentile distance over all distances from points in X to their closest point in Y. For example, the directed 95% Hausdorff distance is the point in X with distance to its closest point in Y is greater or equal to exactly 95% of the other points in X. In mathematical terms, denoting the r th percentile as Kr, this is given as:

percentile distance

The (undirected) percent Hausdorff measure is defined again with the mean:

Mean hausdorff

Normalisation of the score

Different organs and measures will have different ranges of scores, therefore it is not possible to simply average them to get an overall score. Therefore to be able to normalise the scores with respect to expected values 3 cases have been contoured by multiple observers. The mean score of these observers will be used as a reference score against which submitted contours will be compared. For any organ/measure a perfect value(Dice = 1, AD/HD =0) will be scored 100. A value equivalent to the average inter-observer reference will be given a score of 50. A linear scale will be used to interpolate between these values, and extrapolate beyond them, such that a score of 0 will be given to any result below the reference by more than the perfect score is above the reference.

Score = 50 + (max( (T-R)/(P-R) * 50 ), 0 )

Where T is the test contour measure, P is the perfect measure, and R is the reference measure for that organ/measure.

For example, given a reference Dice of 0.85; a test contour with a Dice of 0.9 against the "ground truth" will score 66.6, where as a test contour with a Dice of 0.72 against the "ground truth" would score 7.

The winners

The normalized scores for all organs, measure and test cases will be averaged (mean) to give a final score. The winner will be the team with the highest final score.

Submitted contours should include all of the the structures found in the training data, named in the same way as the training data. i.e. each case should contain:

  • Esophagus
  • Heart
  • Lung_L
  • Lung_R
  • SpinalCord

The results should be submitted as a single DICOM RTSTRUCT file per test case. Each file should be named according to the patient ID for the case - i.e. LCTSC-Test-S1-101.dcm

Structure files for all 12 test cases should encapsulated into a single zip file. There must be no folder structure within the zip file. No specific naming is required for the zip file. This zip file can then be uploaded via the participate page.

If you have file naming errors, files missing, structure naming errors the website should report these to you, but may result in that submission being scored zero.


Conversion to DICOM RTSTRUCT using open source software

To convert your local format to DICOM-RT, you may use 3D Slicer or CERR.

Below is the instruction of using 3D Slicer to perform the conversion:

1) Load all images. At time of load, click "Labelmap" checkbox for each structure
2) Go to Segmentation module
3) For "Active segmentation", choose "Create new segmentation"
4) For each structure, repeat:
    4a) In "Export/Import segments", choose structure, and import as labelmap
5) Click on one of the segments, and then click "Edit selected"
6) Set the "Master volume" to your CT

7) Go to Data module.
8) On background, right click choose "Create new subject"
9) On subject, right click choose "Create child study"
10) Drag the CT and segmentation node onto the child study
11) Right click on study, choose "Export to DICOM"

12) Enjoy your newly created DICOM-RT file

Live Competition Guidelines

Please submit your off-site test results to enter the live competition. The live competition of this grand challenge will be held in conjunction with the 2017 AAPM annual meeting, which will be held at the Colorado convention center at 700 14th street, Denver, Colorado, USA. Meeting information is available here. Please register for the meeting for the live competition.

The live competition will be held 1:45PM-3:45PM on Wednesday, August 2nd, 2017 in room 612. Please arrive no later than 1:30PM so you have enough time to set up your computer. You will need to bring your own laptop for the live challenge. The live challenge data will be distributed using a USB drive. You will need to complete the segmentation and submit the results within 2 hours. There will be no dedicated internet drop for each laptop. You may use the AAPM Wi-Fi for internet access (about 2Mbps Up/Down). Please be advised that the Wi-Fi connection may not be reliable.

Please use the same submission guidelines for the off-site test. The results should be submitted as a single DICOM RTSTRUCT file per test case. Each file should be named according to the patient ID for the case. We will collect your entry via USB to ensure its successful submission.

The challenge symposium will be held 10:00AM-12:00AM on Thursday, August 3rd, 2017 in room Four Seasons 2. The live competition winners will be announced and invited to a round table discussion following the oral presentations by the top participants in the off-site challenge . The top 3 winners in the live competition will receive conference registration reimbursement and a certificate of merit from AAPM. The customized certificate of merit will be mailed to the winners after the AAPM meeting.

New! Remote Participation in Live Competition

Participants who cannot attend the AAPM meeting are allowed to remotely participate in the live competition, but will not be eligible for awards. The live test data will be made available on the website at the beginning of the live competition (around 1:45PM Mountain Time, August 2nd, 2017). Remote participants should submit their results directly to the online evaluation system. Only one submission is allowed within 24 hours. The scores from remote participants will be presented at the challenge symposium if it is available before the symposium (10:00AM Mountain Time, August 3rd, 2017).

New! Release of Live Competition Data

Please login to your account to learn the options to obtain the live competition data.

  • Anonymous participation is not allowed
  • By entering you give the organizers to publish the results of this study
  • Results will not be linked to participants in publications without express permission of the participant to do so
  • Entry by commercial entities is permitted, but should be disclosed
  • Team participation is allowed, but the team members should have the same affiliation and each team should have no more than 3 persons.

Organizers and Major Contributors

  • Jinzhong Yang (MD Anderson Cancer Center)
  • Greg Sharp (Massachusetts General Hospital)
  • Mark Gooding (Mirada Medical)
  • Harini Veeraraghavan (Memorial Sloan Kettering Cancer Center)
  • Wouter van Elmpt (Maastro Clinic)
  • Samuel Armato III (University of Chicago)
  • Keyvan Farahani (NIH/NCI)
  • Andre Dekker (Maastro Clinic)
  • Tim Lustberg (Maastro Clinic)
  • Justin Kirby (NIH/NCI)
  • Kirk Smith (TCIA)
  • Tracy Nolan (TCIA)
  • Jayashree Kalpathy-Cramer (Harvard University)
  • Artem Mamonov (Harvard University)
  • Andrew Beers (Harvard University)



Training Phase

Start: May 18, 2017, midnight

Description: Participants download the training data with ground truth to train and optimize their algorithms for the segmentation task.

Pre-AAPM Challenge

Start: June 19, 2017, midnight

Description: Participants perform segmentation on the off-site test data and submit their results to compete for the three seats to present their work at Challenge Symposium at the AAPM 2017 Annual Meeting.

AAPM Live Challenge

Start: Aug. 2, 2017, midnight

Description: Participants perform segmentation on the live test data at the AAPM 2017 Annual Meeting to win the Challenge and awards from AAPM.

Competition Ends


You must be logged in to participate in competitions.

Sign In