Clinical Science
Application of standardized manual labeling on identification of retinopathy of prematurity images in deep learning
Ji Wang, Guihua Zhang, Jianwei Lin, Jie Ji, Kunliang Qiu, Mingzhi Zhang
Published 2019-08-10
Cite as Chin J Exp Ophthalmol, 2019, 37(8): 653-657. DOI: 10.3760/cma.j.issn.2095-0160.2019.08.013
Abstract
ObjectiveTo evaluate the application of the standard manual labeling on identification of retinopathy of prematurity (ROP) images in deep learning.
MethodsAccording to the International Classification of ROP, different periods of ROP were classified into stage disease and plus disease in this study.From Joint Shantou International Eye Center from August 2009 to July 2018, a total of 1 464 labeled fundus retinal photographs were divided randomly by stratified sampling into 3 groups: stage disease group(subgroup 1∶173, subgroup 2∶117) was used to train for labeling stage disease, whereas plus disease group(subgroup 1∶163, subgroup 2∶116) was used to train for labeling plus disease, and consistent labels group consisted of 895 consistent labeled images on both disease.Graders consisted of senior experts, 3 senior ophthalmologists and 2 interns, and received training for classification and labeling on ROP fundus images.The results were compared among the doctors and doctors with deep learning, and the agreement between non-experts doctors and the reference standards, and deep learning and the reference standards were tested.
ResultsAfter the first training, the overall agreement rate of the senior ophthalmologist group and the intern group were lower than 90% for both two disease labeling.After two to three times of training, in image of consistent labels group, overall agreement rates of senior ophthalmologists and intern doctor's were 98.99% (Kappa=0.979), 99.22% (Kappa=0.984) on stage disease, and 97.43% (Kappa=0.914), 98.11% (Kappa=0.935) on plus disease, respectively.The agreement on stage disease using deep learning based on human-machine combination was 94.08%, Kappa value was 0.880, which achieved good degree.
ConclusionsStandardized manual labeling can improve the intelligentization of deep learning on identification of ROP images, and be considered as an innovative method of homogenization and standardized training for doctors in ophthalmology.
Key words:
Artificial intelligence; Standardized label; Deep learning; Retinopathy of prematurity; Stage disease; Plus disease
Contributor Information
Ji Wang
Joint Shantou International Eye Center, Shantou University &
the Chinese University of Hong Kong, Shantou 515041, China
Guihua Zhang
Joint Shantou International Eye Center, Shantou University &
the Chinese University of Hong Kong, Shantou 515041, China
Jianwei Lin
Joint Shantou International Eye Center, Shantou University &
the Chinese University of Hong Kong, Shantou 515041, China
Jie Ji
Network &
Information Center, Shantou University, Shantou 515063, China
Kunliang Qiu
Joint Shantou International Eye Center, Shantou University &
the Chinese University of Hong Kong, Shantou 515041, China
Mingzhi Zhang
Joint Shantou International Eye Center, Shantou University &
the Chinese University of Hong Kong, Shantou 515041, China
