The Liver Tumor Segmentation Benchmark (LiTS)

Patrick Bilic; Patrick Christ; Hongwei Bran Li; Eugene Vorontsov; Avi Ben-Cohen; Georgios Kaissis; Adi Szeskin; Colin Jacobs; Gabriel Efrain Humpire Mamani; Gabriel Chartrand; Fabian Lohöfer; Julian Walter Holch; Wieland Sommer; Felix Hofmann; Alexandre Hostettler; Naama Lev-Cohain; Michal Drozdzal; Michal Marianne Amitai; Refael Vivanti; Jacob Sosna; Ivan Ezhov; Anjany Sekuboyina; Fernando Navarro; Florian Kofler; Johannes C. Paetzold; Suprosanna Shit; Xiaobin Hu; Jana Lipková; Markus Rempfler; Marie Piraud; Jan Kirschke; Benedikt Wiestler; Zhiheng Zhang; Christian Hülsemeyer; Marcel Beetz; Florian Ettlinger; Michela Antonelli; Woong Bae; Míriam Bellver; Lei Bi; Hao Chen; Grzegorz Chlebus; Erik B. Dam; Qi Dou; Chi Wing Fu; Bogdan Georgescu; Xavier Giró-i-Nieto; Felix Gruen; Xu Han; Pheng Ann Heng; Jürgen Hesser; Jan Hendrik Moltz; Christian Igel; Fabian Isensee; Paul Jäger; Fucang Jia; Krishna Chaitanya Kaluva; Mahendra Khened; Ildoo Kim; Jae Hun Kim; Sungwoong Kim; Simon Kohl; Tomasz Konopczynski; Avinash Kori; Ganapathy Krishnamurthi; Fan Li; Hongchao Li; Junbo Li; Xiaomeng Li; John Lowengrub; Jun Ma; Klaus Maier-Hein; Kevis Kokitsi Maninis; Hans Meine; Dorit Merhof; Akshay Pai; Mathias Perslev; Jens Petersen; Jordi Pont-Tuset; Jin Qi; Xiaojuan Qi; Oliver Rippel; Karsten Roth; Ignacio Sarasua; Andrea Schenk; Zengming Shen; Jordi Torres; Christian Wachinger; Chunliang Wang; Leon Weninger; Jianrong Wu; Daguang Xu; Xiaoping Yang; Simon Chun Ho Yu; Yading Yuan; Miao Yue; Liping Zhang; Jorge Cardoso; Spyridon Bakas; Rickmer Braren; Volker Heinemann; Christopher Pal; An Tang; Samuel Kadoury; Luc Soler; Bram van Ginneken; Hayit Greenspan; Leo Joskowicz; Bjoern Menze

doi:10.1016/j.media.2022.102680

The Liver Tumor Segmentation Benchmark (LiTS)

Patrick Bilic, Patrick Christ, Hongwei Bran Li^*, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, Fabian Lohöfer, Julian Walter Holch, Wieland Sommer, Felix Hofmann, Alexandre Hostettler, Naama Lev-Cohain, Michal Drozdzal, Michal Marianne Amitai, Refael Vivanti, Jacob SosnaIvan Ezhov, Anjany Sekuboyina, Fernando Navarro, Florian Kofler, Johannes C. Paetzold, Suprosanna Shit, Xiaobin Hu, Jana Lipková, Markus Rempfler, Marie Piraud, Jan Kirschke, Benedikt Wiestler, Zhiheng Zhang, Christian Hülsemeyer, Marcel Beetz, Florian Ettlinger, Michela Antonelli, Woong Bae, Míriam Bellver, Lei Bi, Hao Chen, Grzegorz Chlebus, Erik B. Dam, Qi Dou, Chi Wing Fu, Bogdan Georgescu, Xavier Giró-i-Nieto, Felix Gruen, Xu Han, Pheng Ann Heng, Jürgen Hesser, Jan Hendrik Moltz, Christian Igel, Fabian Isensee, Paul Jäger, Fucang Jia, Krishna Chaitanya Kaluva, Mahendra Khened, Ildoo Kim, Jae Hun Kim, Sungwoong Kim, Simon Kohl, Tomasz Konopczynski, Avinash Kori, Ganapathy Krishnamurthi, Fan Li, Hongchao Li, Junbo Li, Xiaomeng Li, John Lowengrub, Jun Ma, Klaus Maier-Hein, Kevis Kokitsi Maninis, Hans Meine, Dorit Merhof, Akshay Pai, Mathias Perslev, Jens Petersen, Jordi Pont-Tuset, Jin Qi, Xiaojuan Qi, Oliver Rippel, Karsten Roth, Ignacio Sarasua, Andrea Schenk, Zengming Shen, Jordi Torres, Christian Wachinger, Chunliang Wang, Leon Weninger, Jianrong Wu, Daguang Xu, Xiaoping Yang, Simon Chun Ho Yu, Yading Yuan, Miao Yue, Liping Zhang, Jorge Cardoso, Spyridon Bakas, Rickmer Braren, Volker Heinemann, Christopher Pal, An Tang, Samuel Kadoury, Luc Soler, Bram van Ginneken, Hayit Greenspan, Leo Joskowicz, Bjoern Menze

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

271 Scopus citations

Abstract

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

Original language	English
Article number	102680
Journal	Medical Image Analysis
Volume	84
DOIs	https://doi.org/10.1016/j.media.2022.102680
State	Published - Feb 2023

Funding

Funders	Funder number
Helmut-Horten-Foundation
Fonds de Recherche du Québec - Santé	34939
Fonds de Recherche du Québec - Santé
Deutsche Forschungsgemeinschaft
Universität Zürich
International Graduate School of Science and Engineering	GSC 81
International Graduate School of Science and Engineering
Fondation de l'Association des radiologistes du Québec

Keywords

Benchmark
CT
Deep learning
Liver
Liver tumor
Segmentation

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1016/j.media.2022.102680

Cite this

Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., Lohöfer, F., Holch, J. W., Sommer, W., Hofmann, F., Hostettler, A., Lev-Cohain, N., Drozdzal, M., Amitai, M. M., Vivanti, R., ... Menze, B. (2023). The Liver Tumor Segmentation Benchmark (LiTS). Medical Image Analysis, 84, Article 102680. https://doi.org/10.1016/j.media.2022.102680

@article{08c2f7c5602043058ac1cb5d2417df6e,

title = "The Liver Tumor Segmentation Benchmark (LiTS)",

abstract = "In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.",

keywords = "Benchmark, CT, Deep learning, Liver, Liver tumor, Segmentation",

author = "Patrick Bilic and Patrick Christ and Li, {Hongwei Bran} and Eugene Vorontsov and Avi Ben-Cohen and Georgios Kaissis and Adi Szeskin and Colin Jacobs and Mamani, {Gabriel Efrain Humpire} and Gabriel Chartrand and Fabian Loh{\"o}fer and Holch, {Julian Walter} and Wieland Sommer and Felix Hofmann and Alexandre Hostettler and Naama Lev-Cohain and Michal Drozdzal and Amitai, {Michal Marianne} and Refael Vivanti and Jacob Sosna and Ivan Ezhov and Anjany Sekuboyina and Fernando Navarro and Florian Kofler and Paetzold, {Johannes C.} and Suprosanna Shit and Xiaobin Hu and Jana Lipkov{\'a} and Markus Rempfler and Marie Piraud and Jan Kirschke and Benedikt Wiestler and Zhiheng Zhang and Christian H{\"u}lsemeyer and Marcel Beetz and Florian Ettlinger and Michela Antonelli and Woong Bae and M{\'i}riam Bellver and Lei Bi and Hao Chen and Grzegorz Chlebus and Dam, {Erik B.} and Qi Dou and Fu, {Chi Wing} and Bogdan Georgescu and Xavier Gir{\'o}-i-Nieto and Felix Gruen and Xu Han and Heng, {Pheng Ann} and J{\"u}rgen Hesser and Moltz, {Jan Hendrik} and Christian Igel and Fabian Isensee and Paul J{\"a}ger and Fucang Jia and Kaluva, {Krishna Chaitanya} and Mahendra Khened and Ildoo Kim and Kim, {Jae Hun} and Sungwoong Kim and Simon Kohl and Tomasz Konopczynski and Avinash Kori and Ganapathy Krishnamurthi and Fan Li and Hongchao Li and Junbo Li and Xiaomeng Li and John Lowengrub and Jun Ma and Klaus Maier-Hein and Maninis, {Kevis Kokitsi} and Hans Meine and Dorit Merhof and Akshay Pai and Mathias Perslev and Jens Petersen and Jordi Pont-Tuset and Jin Qi and Xiaojuan Qi and Oliver Rippel and Karsten Roth and Ignacio Sarasua and Andrea Schenk and Zengming Shen and Jordi Torres and Christian Wachinger and Chunliang Wang and Leon Weninger and Jianrong Wu and Daguang Xu and Xiaoping Yang and Yu, {Simon Chun Ho} and Yading Yuan and Miao Yue and Liping Zhang and Jorge Cardoso and Spyridon Bakas and Rickmer Braren and Volker Heinemann and Christopher Pal and An Tang and Samuel Kadoury and Luc Soler and {van Ginneken}, Bram and Hayit Greenspan and Leo Joskowicz and Bjoern Menze",

note = "Publisher Copyright: {\textcopyright} 2022 The Author(s)",

year = "2023",

month = feb,

doi = "10.1016/j.media.2022.102680",

language = "אנגלית",

volume = "84",

journal = "Medical Image Analysis",

issn = "1361-8415",

publisher = "Elsevier",

}

Bilic, P, Christ, P, Li, HB, Vorontsov, E, Ben-Cohen, A, Kaissis, G, Szeskin, A, Jacobs, C, Mamani, GEH, Chartrand, G, Lohöfer, F, Holch, JW, Sommer, W, Hofmann, F, Hostettler, A, Lev-Cohain, N, Drozdzal, M, Amitai, MM, Vivanti, R, Sosna, J, Ezhov, I, Sekuboyina, A, Navarro, F, Kofler, F, Paetzold, JC, Shit, S, Hu, X, Lipková, J, Rempfler, M, Piraud, M, Kirschke, J, Wiestler, B, Zhang, Z, Hülsemeyer, C, Beetz, M, Ettlinger, F, Antonelli, M, Bae, W, Bellver, M, Bi, L, Chen, H, Chlebus, G, Dam, EB, Dou, Q, Fu, CW, Georgescu, B, Giró-i-Nieto, X, Gruen, F, Han, X, Heng, PA, Hesser, J, Moltz, JH, Igel, C, Isensee, F, Jäger, P, Jia, F, Kaluva, KC, Khened, M, Kim, I, Kim, JH, Kim, S, Kohl, S, Konopczynski, T, Kori, A, Krishnamurthi, G, Li, F, Li, H, Li, J, Li, X, Lowengrub, J, Ma, J, Maier-Hein, K, Maninis, KK, Meine, H, Merhof, D, Pai, A, Perslev, M, Petersen, J, Pont-Tuset, J, Qi, J, Qi, X, Rippel, O, Roth, K, Sarasua, I, Schenk, A, Shen, Z, Torres, J, Wachinger, C, Wang, C, Weninger, L, Wu, J, Xu, D, Yang, X, Yu, SCH, Yuan, Y, Yue, M, Zhang, L, Cardoso, J, Bakas, S, Braren, R, Heinemann, V, Pal, C, Tang, A, Kadoury, S, Soler, L, van Ginneken, B, Greenspan, H, Joskowicz, L & Menze, B 2023, 'The Liver Tumor Segmentation Benchmark (LiTS)', Medical Image Analysis, vol. 84, 102680. https://doi.org/10.1016/j.media.2022.102680

TY - JOUR

T1 - The Liver Tumor Segmentation Benchmark (LiTS)

AU - Bilic, Patrick

AU - Christ, Patrick

AU - Li, Hongwei Bran

AU - Vorontsov, Eugene

AU - Ben-Cohen, Avi

AU - Kaissis, Georgios

AU - Szeskin, Adi

AU - Jacobs, Colin

AU - Mamani, Gabriel Efrain Humpire

AU - Chartrand, Gabriel

AU - Lohöfer, Fabian

AU - Holch, Julian Walter

AU - Sommer, Wieland

AU - Hofmann, Felix

AU - Hostettler, Alexandre

AU - Lev-Cohain, Naama

AU - Drozdzal, Michal

AU - Amitai, Michal Marianne

AU - Vivanti, Refael

AU - Sosna, Jacob

AU - Ezhov, Ivan

AU - Sekuboyina, Anjany

AU - Navarro, Fernando

AU - Kofler, Florian

AU - Paetzold, Johannes C.

AU - Shit, Suprosanna

AU - Hu, Xiaobin

AU - Lipková, Jana

AU - Rempfler, Markus

AU - Piraud, Marie

AU - Kirschke, Jan

AU - Wiestler, Benedikt

AU - Zhang, Zhiheng

AU - Hülsemeyer, Christian

AU - Beetz, Marcel

AU - Ettlinger, Florian

AU - Antonelli, Michela

AU - Bae, Woong

AU - Bellver, Míriam

AU - Bi, Lei

AU - Chen, Hao

AU - Chlebus, Grzegorz

AU - Dam, Erik B.

AU - Dou, Qi

AU - Fu, Chi Wing

AU - Georgescu, Bogdan

AU - Giró-i-Nieto, Xavier

AU - Gruen, Felix

AU - Han, Xu

AU - Heng, Pheng Ann

AU - Hesser, Jürgen

AU - Moltz, Jan Hendrik

AU - Igel, Christian

AU - Isensee, Fabian

AU - Jäger, Paul

AU - Jia, Fucang

AU - Kaluva, Krishna Chaitanya

AU - Khened, Mahendra

AU - Kim, Ildoo

AU - Kim, Jae Hun

AU - Kim, Sungwoong

AU - Kohl, Simon

AU - Konopczynski, Tomasz

AU - Kori, Avinash

AU - Krishnamurthi, Ganapathy

AU - Li, Fan

AU - Li, Hongchao

AU - Li, Junbo

AU - Li, Xiaomeng

AU - Lowengrub, John

AU - Ma, Jun

AU - Maier-Hein, Klaus

AU - Maninis, Kevis Kokitsi

AU - Meine, Hans

AU - Merhof, Dorit

AU - Pai, Akshay

AU - Perslev, Mathias

AU - Petersen, Jens

AU - Pont-Tuset, Jordi

AU - Qi, Jin

AU - Qi, Xiaojuan

AU - Rippel, Oliver

AU - Roth, Karsten

AU - Sarasua, Ignacio

AU - Schenk, Andrea

AU - Shen, Zengming

AU - Torres, Jordi

AU - Wachinger, Christian

AU - Wang, Chunliang

AU - Weninger, Leon

AU - Wu, Jianrong

AU - Xu, Daguang

AU - Yang, Xiaoping

AU - Yu, Simon Chun Ho

AU - Yuan, Yading

AU - Yue, Miao

AU - Zhang, Liping

AU - Cardoso, Jorge

AU - Bakas, Spyridon

AU - Braren, Rickmer

AU - Heinemann, Volker

AU - Pal, Christopher

AU - Tang, An

AU - Kadoury, Samuel

AU - Soler, Luc

AU - van Ginneken, Bram

AU - Greenspan, Hayit

AU - Joskowicz, Leo

AU - Menze, Bjoern

PY - 2023/2

Y1 - 2023/2

N2 - In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

AB - In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

KW - Benchmark

KW - CT

KW - Deep learning

KW - Liver

KW - Liver tumor

KW - Segmentation

UR - http://www.scopus.com/inward/record.url?scp=85143743210&partnerID=8YFLogxK

U2 - 10.1016/j.media.2022.102680

DO - 10.1016/j.media.2022.102680

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

C2 - 36481607

AN - SCOPUS:85143743210

SN - 1361-8415

VL - 84

JO - Medical Image Analysis

JF - Medical Image Analysis

M1 - 102680

ER -

The Liver Tumor Segmentation Benchmark (LiTS)

Abstract

Funding

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this