Background

Computed tomography (CT) anatomic images of the lungs are routinely utilised in the clinic for the radiological assessment of respiratory diseases and radiotherapy treatment planning. CT-based methods of mapping regional ventilation without exogenous contrast, referred to as ‘CT ventilation imaging’, which are based on image processing of non-contrast lung CT images acquired at multiple inflation levels either during tidal breathing or breath-hold, can transform CT from a purely anatomic modality into one that can image and quantify pulmonary ventilation [1].

This contrast agent free modality has great promise in the diagnosis and therapy of respiratory diseases characterised by regionally impaired lung ventilation. In particular, obstructive lung diseases such as asthma, chronic obstructive pulmonary disease (COPD) and cystic fibrosis exhibit marked increases in ventilation heterogeneity when compared with normal subjects due to airway narrowing or closure. For asthmatics, the impact of regionally specific therapeutic interventions such as bronchial thermoplasty may benefit from the regional and quantitative information yielded by ventilation imaging from treatment planning to the assessment of regional pathophysiology response to treatment [2]. For COPD patients, regional ventilation can help assess the impact of lung volume reduction surgery [3]. For paediatric patients with cystic fibrosis, an early detection of ventilation dysfunction may enable bronchoscopy or physiotherapy for regionally specific mucus clearance to prevent the onset of lung disease irreversibility [2]. Moreover, ventilation imaging can also be applied to lung cancer patients undergoing radiotherapy to spatially assist in preferential sparing of ventilated lung during the treatment planning process [4] and for pre-operative risk stratification of surgical resection candidates [5].

CT ventilation imaging has also become an increasingly pertinent topic for the AAPM community. The number of AAPM abstracts on CT ventilation has increased consistently since 2009, with a dedicated session on the topic in 2016. Since 2014, academic and clinical physicists involved with CT ventilation have held a CT ventilation-focused meeting at AAPM that has been well attended (approx. 25 people at each meeting). There are at least 6 prospective clinical trials on-going or completed, three of which (NCT02528942, NCT02308709, NCT02843568) are using CT ventilation imaging to drive clinical care decisions and guide treatment planning for lung cancer patients.

Although the modality has gained considerable interest, its physiological accuracy has remained a concern. Numerous single institutional attempts have been made to validate the modality against established clinical pulmonary function tests (PFTs) [6,7] or a diverse array of contrast-based ventilation imaging modalities including nuclear imaging [8,9] and, more recently, hyperpolarised gas MRI [10-12]. However, there exists several possible CT acquisition protocols and numerous algorithms to generate the regional ventilation surrogates. As such, there is a need for a comprehensive multi-institutional validation study. To date, only one preliminary multi-institutional study has attempted a cross-modality validation of CT ventilation imaging [13]. Whilst that study demonstrated worldwide interest and the feasibility of running a major grand challenge in CT ventilation imaging validation, there were several notable limitations: (i) no validation was performed against PFTs; (ii) all CT scans were acquired during tidal breathing (4DCT) with no breath-hold data included despite it being shown to exhibit greater physiological accuracy [14]; (iii) despite the potential applications for non-oncological respiratory diseases, human data consisted only of lung cancer patients; (iv) only three imaging datasets were included, two of which were from human subjects with nuclear imaging as the reference modality despite significant improvements in spatial and temporal resolution and aerosol deposition artefacts with hyperpolarised gas ventilation MRI [4]; Accordingly, a thorough multi-institutional validation of CT ventilation imaging via a comprehensive dataset is critical to the translation of this modality into the clinic.

References

[1]          Guerrero T, Sanders K, Noyola-Martinez J, et al. Quantification of regional ventilation from treatment planning CT. Int J Radiat Oncol Biol Phys 2005;62:630-634.

[2]          Thomen RP, Sheshadri A, Quirk JD, et al. Regional ventilation changes in severe asthma after bronchial thermoplasty with (3)He MR imaging and CT. Radiology 2015;274:250-259.

[3]          Kurose T, Okumura Y, Sato S, et al. Functional evaluation of lung by Xe-133 lung ventilation scintigraphy before and after lung volume reduction surgery (LVRS) in patients with pulmonary emphysema. Acta Med Okayama 2004;58:7-15.

[4]          Ireland RH, Tahir BA, Wild JM, et al. Functional Image-guided Radiotherapy Planning for Normal Lung Avoidance. Clin Oncol (R Coll Radiol) 2016;28:695-707.

[5]          Eslick EM, Bailey DL, Harris B, et al. Measurement of preoperative lobar lung function with computed tomography ventilation imaging: progress towards rapid stratification of lung cancer lobectomy patients with abnormal lung function. Eur J Cardiothorac Surg 2016;49:1075-1082.

[6]          Yamamoto T, Kabus S, Lorenz C, et al. Pulmonary ventilation imaging based on 4-dimensional computed tomography: comparison with pulmonary function tests and SPECT ventilation images. Int J Radiat Oncol Biol Phys 2014;90:414-422.

[7]          Brennan D, Schubert L, Diot Q, et al. Clinical validation of 4-dimensional computed tomography ventilation with pulmonary function test data. Int J Radiat Oncol Biol Phys 2015;92:423-429.

[8]          Castillo R, Castillo E, Martinez J, et al. Ventilation from four-dimensional computed tomography: density versus Jacobian methods. Phys Med Biol 2010;55:4661-4685.

[9]          Kipritidis J, Siva S, Hofman MS, et al. Validating and improving CT ventilation imaging by correlating with ventilation 4D-PET/CT using 68Ga-labeled nanoparticles. Med Phys 2014;41:011910.

[10]        Mathew L, Wheatley A, Castillo R, et al. Hyperpolarized (3)He magnetic resonance imaging: comparison with four-dimensional x-ray computed tomography imaging in lung cancer. Acad Radiol 2012;19:1546-1553.

[11]        Tahir BA, Hughes PJC, Robinson SD, et al. Spatial comparison of CT-based surrogates of lung ventilation with hyperpolarized Helium-3 and Xenon-129 gas MRI in patients undergoing radiation therapy. International Journal of Radiation Oncology*Biology*Physics 2018.

[12]        Tahir BA, Van Holsbeke C, Ireland RH, et al. Comparison of CT-based Lobar Ventilation with 3He MR Imaging Ventilation Measurements. Radiology 2016;278:585-592.

[13]        Kipritidis J. TU-H-202-04: The VAMPIRE Challenge: Preliminary Results From a Multi-Institutional Study of CT Ventilation Image Accuracy. Medical Physics 2016;43:3771-3771.

[14]        Eslick EM, Kipritidis J, Gradinscak D, et al. CT ventilation imaging derived from breath hold CT exhibits good regional accuracy with Galligas PET. Radiother Oncol 2018;127:267-273.

Evaluation Criteria

CT ventilation images will be compared against the corresponding reference ventilation images or repeat CT ventilation images for all test datasets using the following evaluation metrics:

• Dice overlap coefficient
• Voxel-wise and ROI-based Spearman correlation

Moreover, the following global metrics will be computed from the CT ventilation images and correlated against PFTs:

• Percent ventilated volume
• Ventilation defect percent
• Coefficient of variation

Challenge Data

Patient data and reference standard

Data consist of PFTs, multi-inflation non-contrast CT (4D or breath-hold) and contrast-based ventilation images (nuclear imaging or hyperpolarised gas MRI) for patients with lung cancer and several non-oncological obstructive respiratory diseases including cystic fibrosis, asthma and COPD. The corresponding reference ventilation images for 50 CT scans will be provided for training. Several repeat CT datasets are also available for reproducibility analysis.

Training data

The corresponding reference ventilation images for 20% of CT scans from each dataset will be provided for training. 

Table 1: Summary of CT and reference ventilation imaging training data included in grand challenge.

Abbreviations: DTPA = diethylenetriamine-pentaacetic acid; SPECT = single photon emission computerized tomography; PET = positron emission tomography; FRC = functional residual capacity; TLC = total lung capacity.

Table 2: Summary of CT and reference ventilation test data included in grand challenge.

Abbreviations: DTPA = diethylenetriamine-pentaacetic acid; SPECT = single photon emission computerized tomography; PET = positron emission tomography; RT = radiation therapy; FRC = functional residual capacity; TLC = total lung capacity.

Table 3: Summary of CT reproducibility imaging data included in grand challenge.

Submission Guidelines

Workflow for each unique submission. Blue boxes indicate actions by participants. Red boxes indicate CTVIE19 lead organizer actions. 

Produce CT ventilation image (CTVI) and deformation vector field (DVF) for each subject:

(i) Directory structure of CTVIE19 Challenge Data

For an overview of the Challenge Data, please see the Challenge Data section.

Data is provided in both NIfTI (.nii.gz)* and MetaImage (.mha) format. For each subject, inspiratory and expiratory CT images and binary lung masks are included. In addition, the training datasets include reference contrast-based ventilation images and binary masks. Examples of the directory structure and naming convention are displayed below for two subjects in the first study in each of the training and testing datasets:

Training data:

--- Training01

|   --- Subject01

|   |   --- exp.mha

|   |   --- insp.mha

|   |   --- mask_exp.mha

|   |   --- mask_insp.mha

|   |   --- ref_mask_vent.mha

|   |   --- ref_vent.mha

|   --- Subject02

|   |   --- exp.mha

|   |   --- insp.mha

|   |   --- mask_exp.mha

|   |   --- mask_insp.mha

|   |   --- ref_mask_vent.mha

|   |   --- ref_vent.mha

Testing data:

--- Study01

|   --- Subject01

|   |   --- exp.mha

|   |   --- insp.mha

|   |   --- mask_exp.mha

|   |   --- mask_insp.mha

|   --- Subject02

|   |   --- exp.mha

|   |   --- insp.mha

|   |   --- mask_exp.mha

|   |   --- mask_insp.mha

(ii) General requirements of CTVI files

You are welcome to use any CTVI algorithm you like. We request that CTVIs are provided with minimal post-processing (in particular, smoothing).

• CTVIs may be provided in NIfTI (.nii.gz)* or MetaImage (.mha) format.
• All CTVIs should be computed on expiratory geometry (i.e. the expiratory image is the fixed image)

*For those participants using Matlab to generate CTVIs and DVFs, please note that Matlab now offers native support for reading and writing NIfTI images.

(iii) General requirements on DVF files (not applicable to submissions based on machine learning and other techniques which do not employ image registration)

Displacements should be computed in mm. For participants who are using rigid and affine stages in addition to a deformable registration stage, we ask that the outputs of these stages are composed into a single DVF. Each DVF should be provided either in NIfTI or MetaImage format. DVF files should be formatted based on one of the following methods:

• As a single 4D image volume of dimensions (X,Y,Z,3) where X,Y,Z are the same size as the input CT exhale phase image and where the 4th dimension specifies the DVF component in the X,Y,Z directions, respectively.

• Alternatively, you can provide a separate 3D volume for each of the X,Y,Z DVF components. Each 3D image should have dimensions (X,Y,Z). Please label as e.g. “DVF_x.mha”, “DVF_y.mha”, “DVF_z.mha” etc.

(iv) Recommended folder structure

Please use the same folder structure as the CTVIE19 challenge dataset with the addition of an initial folder to specify the submission number. I.e. provide the following:

“SubmissionXX / StudyYY / SubjectZZ / CTVI”

and

“SubmissionXX / StudyYY / SubjectZZ / DVF”

Here, “XX” is a two-digit ID to specify the participant’s unique submission. Note that participants are permitted multiple submissions providing each submission is based on a different ventilation metric and/or registration algorithm; if there is only one submission, please keep this as ‘Submission01’. “YY” is the study ID and “ZZ” is a two-digit subject ID.

(To save bandwidth, please do not include the original images in your upload).

The following is an example directory structure of an upload with MetaImage formatted data for two subjects from Study01 with two submissions:

-- Submission01

--- Study01

|   --- Subject01

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

|   --- Subject02

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

-- Submission02

--- Study01

|   --- Subject01

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

|   --- Subject02

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

Uploading your results

Submission of results files is done by Google Drive. All participants will be provided with a unique username and password to upload results onto an institutional GoogleDrive account. Please note that your submission must be in the format described above.

Pre-submission check

In order to ensure that CTVIs and DVFs are in the correct format, participants are requested to provide their outputs for Subject01 and Subject02 from three test datasets (Studies 1, 6 and 15) as follows:

-- Submission01

--- Study01

|   --- Subject01

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

|   --- Subject02

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

--- Study06

|   --- Subject01

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

|   --- Subject02

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

--- Study15

|   --- Subject01

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

|   --- Subject02

|   |   --- ctvi.mha

|   |   --- dvf_x.mha

|   |   --- dvf_y.mha

|   |   --- dvf_z.mha

Note that this stage is only a preliminary check of the data and outputs provided at this stage will not count towards your ranking.

Final submission

For the final submission, a complete submission is required.

Terms and Conditions

The CTVIE19 challenge is organised in the spirit of cooperative scientific progress. The following rules apply to those who register a team and download the data:

• Anonymous participation is not allowed.
• Entry by commercial entities is permitted, but should be disclosed.
• Once participants submit their outputs to the CTVIE19 challenge organizers, they will be considered fully vested in the challenge, so that their performance results will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers.
• The downloaded datasets or any data derived from these datasets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
• The full data, including reference data associated with the test set cases, is expected to be made publicly available after the publication of the CTVIE19 Challenge. Until the official public release of the data, data downloaded from this site may only be used for the purpose of preparing an entry to be submitted for the CTVIE19 challenge. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products.

Organizers and Major Contributors

Bilal Tahir (Lead organizer) (University of Sheffield and Weston Park Cancer Hospital, UK)

John Kipriditis and Enid Eslick (Northern Sydney Cancer Centre Royal Sydney, Australia)

Paul Keall (The University of Sydney, Australia)

Grace Parraga (Robarts Research Institute, Canada)

Alberto Biancardi, Joshua Astley, Michael Walker and Jim Wild (University of Sheffield)

Dante Capaldi (Stanford University School of Medicine, USA)

Doug Hoover, Brian Yaremko, and Stewart Gaede (London Health Sciences Centre, Canada);

Joe Reinhardt and Eric Hoffman (University of Iowa, USA)

Shankar Siva and Michael Hofman (Peter MacCallum Cancer Centre, Australia)

Tokihiro Yamamoto (UC Davis School Medicine, USA);

Yevgeniy Vinogradskiy (University of Colorado Denver, USA)

Matthew Hatton (Weston Park Cancer Hospital, UK)

Edward Castillo (Beaumont Health System, USA)

Richard Castillo (Emory University, USA)

John Bayouth (University of Wisconsin-Madison, USA)

Samuel Armato and the AAPM Working Group on Grand Challenges

Contact

For further information, please contact the lead organizer, Dr Bilal Tahir (b.tahir@sheffield.ac.uk)