Competition

Contouring Guidelines

The structures were contoured following the DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG Oncology and RROG consensus guidelines reported in Brouwer et.al Radiotherapy and Oncol. 2015 for the parotids and submandibular glands and

Parotid glands

DICOM structure names: Parotid_L and Parotid_R

The parotid gland is contoured so its bounded cranially by external auditory canal and mastoid process and on the caudal end by the posterior part of submandibular space. In the anterior direction it is drawn bordering the masseter muscle, at the posterior border of the mandibular bone, and the medial and lateral pterygoid muscle. In those cases where visible, the parotid gland includes the anterior extension going over the surface of the masseter muscle alongside the medial border of the mandible. The retromandibular vein is included in the parotid gland contour. The lateral and medial boundaries of the parotid consist of subcutaneous fat of the platysma and the parapharyngeal space, respectively.

Submandibular glands

DICOM structure names: Glnd_Submand_L and Glnd_Submand_R

The submandibular gland contour is bounded by the medial pterygoid muscle. The mylohoid muscle surrounds it in the cranial, anterior, and posterior border. Fatty tissue forms the caudal boundary. The submandibular gland is contoured posterior to the mandibular bone.

Lymph nodes level II

DICOM structure names: LN_Neck_II_L, LN_Neck_II_R

The level II group of lymph nodes are contoured bounded on the cranial side by the caudal edge of lateral process of the C1 vertebra, and in the caudal side by the body of the hyoid bone. Anteriorly it is bounded by the submandibular gland and posterior edge of posterior bellow of digastric muscle. Posterior to it is the posterior edge of the sternocleiomastoid muscle. On the lateral side it is bounded by the deep medial surface of the sternocleiodomastoid and plastina muscle and the parotid gland. Medial boundary of the level II lymph node is marked by the medial edge of the internal carotid artery. In some patients, Level II lymph nodes may include level VIIa and may extend to cranial edge of C1.

Lymph nodes level III

DICOM structure names: LN_Neck_III_L, LN_Neck_III_R

The level III or the middle jugular group of lymph nodes is bounded by the body of hyoid bone cranially and by the cricoid cartilage caudally. It is bounded by the sternocleidomastoid muscles anteriorly, posteriorly and laterally. On its medial edge is the common carotid artery. In some patients, Level III does not always extend to the caudal edge of cricoid cartilage.

Evaluation Criteria

Auto-segmented contours will be compared against the manual contours for all test datasets using the following evaluation metrics as implemented in Plastimatch. RTSS will be voxelised to CT resolution for all calculations. Evaluation will be performed in 3D. To prevent uncertainty with the extent to which the Spinal cord and Esophagus should be contoured, submitted contours will be cropped to the extent of the test data. Therefore, you will not be penalised for contouring too great an extent of the structure in the inferior-superior direction, but will for under-segmentation.

Dice Coefficient

This is a measure of relative overlap, where 1 represents perfect agreement and 0 represents no overlap.

where X and Y are the ground truth and test regions.

Mean surface distance

The directed average Hausdorff measure is the average distance of a point in X to its closest point in Y. That is:

The (undirected) average Hausdorff measure is the average of the two directed average Hausdorff measures:

Hausdorff distance (95% Hausdorff distance)

The directed percent Hausdorff measure, for a percentile r, is the r th percentile distance over all distances from points in X to their closest point in Y. For example, the directed 95% Hausdorff distance is the point in X with distance to its closest point in Y is greater or equal to exactly 95% of the other points in X. In mathematical terms, denoting the r th percentile as Kr, this is given as:

The (undirected) percent Hausdorff measure is defined again with the mean:

Normalisation of the score

Different organs and measures will have different ranges of scores, therefore it is not possible to simply average them to get an overall score. Therefore to be able to normalise the scores with respect to expected values 3 cases have been contoured by multiple observers. The mean score of these observers will be used as a reference score against which submitted contours will be compared. For any organ/measure a perfect value(Dice = 1, AD/HD =0) will be scored 100. A value equivalent to the average inter-observer reference will be given a score of 50. A linear scale will be used to interpolate between these values, and extrapolate beyond them, such that a score of 0 will be given to any result below the reference by more than the perfect score is above the reference.

Score = max ( 50 + ( (T-R)/(P-R) * 50 ), 0 )

Where T is the test contour measure, P is the perfect measure, and R is the reference measure for that organ/measure.

For example, given a reference Dice of 0.85; a test contour with a Dice of 0.9 against the "ground truth" will score 66.6, where as a test contour with a Dice of 0.72 against the "ground truth" would score 7.

The winners

The normalized scores for all organs, measure and test cases will be averaged (mean) to give a final score. The winner will be the team with the highest final score.

Submission Guidelines

Submitted contours should include all of the the structures found in the training data, named in the same way as the training data. i.e. each case should contain:

• Parotid_L
• Parotid_R
• Glnd_Submand_L
• Glnd_Submand_R
• LN_Neck_II_L
• LN_Neck_II_R
• LN_Neck_III_L
• LN_Neck_III_R

The results should be submitted as a single DICOM RTSTRUCT file per test case. Each file should be named according to the patient ID for the case - i.e. RTMAC-TEST-001.dcm

Structure files for all 12 test cases should encapsulated into a single zip file. There must be no folder structure within the zip file. No specific naming is required for the zip file. This zip file can then be uploaded via the participate page.

If you have file naming errors, files missing, structure naming errors the website should report these to you, but may result in that submission being scored zero.

Conversion to DICOM RTSTRUCT using open source software

To convert your local format to DICOM-RT, you may use 3D Slicer or CERR.

Below is the instruction of using 3D Slicer to perform the conversion:

1) Load all images. At time of load, click "Labelmap" checkbox for each structure
2) Go to Segmentation module
3) For "Active segmentation", choose "Create new segmentation"
4) For each structure, repeat:
    4a) In "Export/Import segments", choose structure, and import as labelmap
5) Click on one of the segments, and then click "Edit selected"
6) Set the "Master volume" to your CT

7) Go to Data module.
8) On background, right click choose "Create new subject"
9) On subject, right click choose "Create child study"
10) Drag the CT and segmentation node onto the child study
11) Right click on study, choose "Export to DICOM"

12) Enjoy your newly created DICOM-RT file

Live Competition Guidelines

Please submit your off-site test results to enter the live competition. The live competition of this grand challenge will be held in conjunction with the 2019 AAPM annual meeting, which will be held in San Antonio, Texas, USA. Meeting information is available here. Please register for the meeting for the live competition.

The live challenge will take place on Monday July 15. 2:00PM - 4:00PM, in Room 007A. This is on the River Level.

There will be 12 new test cases to process. The data for these is available in an encrypted zip on the data page. The password will be disclosed at the start of the session (and on the data page). Participants will have until the end of the session to upload their submission.

Result will be announced during the AAPM Grand challenge symposium on Wednesday (4:30-6pm, Stars at Night Ballroom 1)

For those participating remotely, the password is "july2019".

Terms and Conditions

• Anonymous participation is not allowed
• By entering you give the organizers to publish the results of this study
• Results will not be linked to participants in publications without express permission of the participant to do so
• Entry by commercial entities is permitted, but should be disclosed
• Entries from groups associated with the organisers is permitted, but must be conducted independently and disclosed as such.
• Team participation is allowed, but the team members should have the same affiliation and each team should have no more than 3 persons.

Organizers and Major Contributors

• Greg Sharp (Massachusetts General Hospital)
• Jinzhong Yang (MD Anderson Cancer Center)
• Mark Gooding (Mirada Medical)
• Carlos Cardenas (MD Anderson Cancer Center)
• Abdallah Mohamed (MD Anderson Cancer Center)
• Harini Veeraraghavan (Memorial Sloan Kettering Cancer Center)
• Jayashree Kalpathy-Cramer (Harvard University)
• Artem Mamonov (Harvard University)
• Andrew Beers (Harvard University)

Sponsors