Clinical Science
Clinical evaluation of artificial intelligence system based on fundus photograph in diabetic retinopathy screening
Meng Li, Gengyuan Wang, Honghui Xia, Xiaoying Tang, Ziqing Feng, Yongyu Yao, Yijin Huang, Wei Fan, Zhe Yuan, Jin Yuan
Published 2019-08-10
Cite as Chin J Exp Ophthalmol, 2019, 37(8): 663-668. DOI: 10.3760/cma.j.issn.2095-0160.2019.08.015
Abstract
ObjectiveTo study the efficiency and accuracy of artificial intelligence (AI) system based on fundus photograph in diabetic retinopathy(DR)screening, and evaluate the clinical application value of AI system.
MethodsA diagnostic trial was adopted.Total of 13 683 color fundus photos were collected in Zhaoqing Gaoyao People's Hospital from March, 2017 to November, 2018.The AI system for DR (ZOC-DR-V1) was established, based on transfer learning + NASNet algorithm, by training 4 465 precisely labeled fundus images (2 510 normal, and 1 955 with any stage of DR). One thousand confirmed fundus images (300 normal and 700 with any stage of DR), diagnosed by AI (AI group) and doctors (3 ophthalmologist doctors and 3 endocrinologist doctors) (doctor group), respectively.Ophthalmologist group and endocrinologist group were both composed of primary, intermediate and senior physicians.The mean reading time of each image and the total time of 1 000 images were recorded.The accuracy and efficiency of AI system and doctor groups were compared.The reading process was divided into two stages.The diagnostic coincidence rate and the average reading time of each group between the two parts were calculated and compared.This study protocol was approved by Ethic Committee of Zhongshan Ophthalmic Center, Sun Yat-sen University (No.2017KYPJ104).
ResultsAfter training, the diagnostic coincidence rate of AI system (ZOC-DR-V1) in test set was 94.7%, AUC was 0.994.In this "man-machine to war" , the diagnostic coincidence rate of primary, intermediate and senior endocrinologist was 94.0%, 91.4% and 93.4%; the diagnostic coincidence rate of primary, intermediate and senior ophthalmologist was 92.7%, 94.4% and 95.6%; the diagnostic coincidence rate of AI system was 95.2%.There was no difference in the diagnostic coincidence rate between AI system and senior ophthalmologist (P=0.749). The mean reading time of each image of primary, intermediate and senior endocrinologists was (4.63±1.87), (3.74±3.47) and (5.71±3.47) seconds, and the total time of 1 000 images of primary, intermediate and senior endocrinologists was 1.29, 1.04 and 1.58 hours; the mean reading time of each image of primary, intermediate and senior ophthalmologists was (7.25±6.58), (5.18±5.01) and (5.18±3.47) seconds, and the total time of 1 000 images of primary, intermediate and senior endocrinologists was 2.02, 1.44 and 1.44 hours; the mean and total time of AI system was (1.62±0.67) seconds and 0.45 hours, and the reading time of AI system was significantly shorter than that of the doctor groups (all at P=0.000). The diagnostic coincidence rates between previous and posterior part of primary endocrinologist, primary and intermediate ophthalmologist were significantly different (χ2=11.986, 6.517, 10.896; all at P<0.05), and the mean reading time in the posterior part was significantly shorter than that in the previous part of intermediate and senior endocrinologist and primary ophthalmologist (t=4.175, 8.189, 5.160; all at P<0.01). While the reading time of AI system remained stable throughout the process(χ2=3.151, P=0.103; t=0.038, P=0.970).
ConclusionsThe ophthalmic AI system based on fundus images has a good diagnostic efficiency, and its diagnostic coincidence rate can compare with senior ophthalmologist, providing a new method and platform for large-scale DR screening.
Key words:
Artificial intelligence; Diabetic retinopathy/diagnosis; Screening; Reading
Contributor Information
Meng Li
Zhongshan Ophthalmic Center, Sun Yat-sen University, State Key Laboratory of Ophthalmology, Guangzhou 510060, China
Gengyuan Wang
Zhongshan Ophthalmic Center, Sun Yat-sen University, State Key Laboratory of Ophthalmology, Guangzhou 510060, China
Honghui Xia
Department of Ophthalmology, Zhaoqing Gaoyao People's Hospital, Zhaoqing 526040, China
Xiaoying Tang
Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Ziqing Feng
Department of Ophthalmology, Zhaoqing Gaoyao People's Hospital, Zhaoqing 526040, China
Yongyu Yao
Department of Ophthalmology, Zhaoqing Gaoyao People's Hospital, Zhaoqing 526040, China
Yijin Huang
Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Wei Fan
Department of Ophthalmology, Zhaoqing Gaoyao People's Hospital, Zhaoqing 526040, China
Zhe Yuan
Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Jin Yuan
Zhongshan Ophthalmic Center, Sun Yat-sen University, State Key Laboratory of Ophthalmology, Guangzhou 510060, China