About

Introduction

The goal of this challenge is to compare 3D liver tumor segmentation techniques. General tumor segmentation methods tailored for liver tumor segmentation will be accepted for the competition as well. Algorithms will be divided into three different categories for evaluation: fully automatic methods, methods with minimal user interaction and interactive segmentation methods. The performance of the liver tumor segmentation methods will be evaluated using a set of comprehensive measures.

Developers of liver tumor segmentation techniques from both academia and industry are welcomed to join the contest.

Liver data

Liver tumor CT image data for this competition was acquired on one 64-slice and two 40-slice CT scanners using a standard four-phase contrast enhanced imaging protocol with slice thickness of 1mm or 1.5mm and in-plane resolution of 0.6-0.9mm. All data is stored in Meta format containing an ASCII readable header and a separate raw image data file. This format is ITK compatible. Full documentation is available here.

Liver training and testing data

Data used for this workshop is composed of 30 liver tumors and represents a range of patients and pathology. Data has been randomized into three groups: 10 training tumor images, 10 testing images for the qualifying and 10 for the contest. The downloadable archive consists of the training images (including reference segmentations) and the first 10 testing images (without segmentations).

All the tumors were manually segmented by an experienced radiologist, and confirmed by another radiologist. The munual segmentation is used as reference for evaluation purposes.

Format of submission for segmentation results

In order to compare all segmentation methods at an equal level, whether they use voxel-level segmentations, or sub-voxel surface based segmentations, all methods need to submit a voxel level segmentation. This is also due to the fact that the manual segmentations, which are used as a quasi gold-standard to compare against, are on a voxel-level. Thus, you have to submit a binary volume of the same size and spacing as the corresponding gray-level data, in which the background is labeled as zero (all other values are treated as object). The file has to be stored in RAW format (e.g. MHD from ITK) with 8 bits per voxel. A value of zero is interpreted as background, all other values are treated as object.

In order to automatize the analysis on our end, we need you to store one file for segmentation of each tumor(i.e. segmentation of two tumors in one patient needs to be saved in different files). In case you are using a surface-based segmentation method, you could use ITK's TriangleMeshToBinaryImageFilter for conversion to a volume.

Evaluation measures

For each test case, a reference segmentation is available, called 'reference' here. The segmentation results, called 'segmentation' here, are evaluated by assigning a score to each test case. The maximum score is 100, and will only be obtained when the segmentation is exactly the same as the reference. The total score of a method is obtained by averaging the scores of all test cases.

The score of each test case itself is the average of five scores, each also scaled from 0 to 100. The five scores are obtained from five different evaluation measures:

Volumetric overlap. This is the number of voxels in the intersection of segmentation and reference, divided by the number of voxels in the union of segmentation and reference. This value is 1 for a perfect segmentation and has 0 as the lowest possible value, when there is no overlap at all between segmentation and reference.
Relative absolute volume difference, in percent. The total volume of the segmentation is divided by the total volume of the reference. From this number 1 is subtracted, the absolute value is taken and the result is multiplied by 100. This value is 0 for a perfect segmentation and larger than zero otherwise. Note that the perfect value of 0 can also be obtained for a non-perfect segmentation, as long as the volume of that segmentation is equal to the volume of the reference.
Average symmetric absolute surface distance, in millimeters. The border voxels of segmentation and reference are determined. These are defined as those voxels in the object that have at least one neighbour (from the 26 nearest neighbours) that does not belong to the object. For each voxel in these sets, the closest voxel in the other set is determined (using Euclidean distance and real world distances, so taking intoaccount the generally different resolutions in the different scan directions). All these distances are stored, for border voxels from both reference and segmentation. The average of all these distances gives the averages symmetric absolute surface distance. This value is 0 for a perfect segmentation.
Symmetric RMS surface distance, in millimeters. This measures is similar to the previous measure, but stores the squared distances between the two sets of border voxels. After averaging the squared values, the root is extracted and gives the symmetric RMS surface distance. This value is 0 for a perfect segmentation.
Maximum symmetric absolute surface distance, in millimeters. This measure is similar to the previous two, but only the maximum of all voxel distances is taken instead of the average. This value is 0 for a perfect segmentation.

Scoring system

For each metric, a score of 100 is assigned to the perfect segmentation.

Volumetric overlap error [%] 0
Relative absolute volume difference [%] 0
Average symmetric surface distance [mm] 0
RMS symmetric surface distance [mm] 0
Maximum symmetric surface distance [mm] 0

For each metric, a reference value from segmentation performed by independent users is assigned with a score of 90, which is shown below

Volumetric overlap error [%] 12.94
Relative absolute volume difference [%] 9.64
Average symmetric surface distance [mm] 0.40
RMS symmetric surface distance [mm] 0.72
Maximum symmetric surface distance [mm] 4.0

Score of each metric for a segmentation can be obtained using linear interpolation or extrapolation between the two points specified above. Note that 0 is the minimum score one segmentation will get, and there is no negative score.

The content of the LTS08 website is copyrighted © Wiro Niessen, Theo van Walsum, Coert Metz, Michiel Schaap, Martin Styner, Simon K. Warfield, Xiang Deng, Tobias Heimann, and Bram van Ginneken.