With the pandemic of the new coronavirus (SARS-CoV-2), health has never been so high and has never been so necessary for years. There has also been a lot of talk about Artificial Intelligence (AI) to combat COVID-19, mainly to speed up the screening of patients and relieve medical staff. However, would the IAs be prepared for this fire test?
One of the first studies to analyze the impact of a deep learning tool in real clinical settings is underway by Google Health in the middle of the pandemic. Still in the public health area, but far from COVID-19, the first analyzes reveal that even the most accurate AIs can make the situation worse (more than help), if they are not adapted to the work environment.
- HACKMED | How AI is transforming Brazilian medicine
Currently, the existing rules for implementing AI in clinical settings follow the standards for FDA clearance in the United States or CE marking in Europe, which focus on accuracy. That's why Google Health researcher Emma Beede argues, "We have to understand how AI tools will work for people in context – especially in healthcare – before they are widely implemented."
Thai test
Google's first opportunity to test its AI in a real-life environment came from Thailand. In the Asian country, an annual target has been set for screening 60% of people with diabetes in search of diabetic retinopathy. This is a complication that occurs when excess glucose, present in the blood, damages the blood vessels in the retina and, if not diagnosed early, can lead to blindness.
However, Thailand has about 4.5 million patients, but only 200 eye doctors and retinal specialists. This is where the technology developed by Google comes into play, which already has authorization from the CE marking (which also covers the country) and awaits the approval of the American FDA.
In tests of effectiveness in a clinical setting, a group by researcher Beede equipped 11 clinics in Thailand with a deep learning system, trained to detect signs of eye disease in patients with diabetes.
- COVID-19 | IBM offers AI technologies to accelerate treatment discovery
How it works?
Usually, in Thailand, the diagnosis of diabetic retinopathy is made remotely, with nurses who take pictures of patients' eyes and send them to a specialist who evaluates the record, in a process that can take up to 10 weeks. This AI can, in theory, identify signs of diabetic retinopathy from an eye scan with more than 90% accuracy and, in principle, provide a result in less than 10 minutes. For this, the system analyzes the same images, looking for indicators of the diabetic condition, such as blocked blood vessels.
For several months, the team of researchers followed the nurses who performed the eye exams, but did not receive entirely positive feedback. After all, as much as it accelerated the whole process when AI got it right, failures took longer to fix.
That's because this deep learning model has been trained with high quality scans. To ensure the accuracy of the diagnosis, the AI also rejects images below a certain quality limit, which made diagnosis in practice very difficult. Since, in the hospital routine, health professionals scan dozens of patients per hour and take pictures even in poor lighting conditions. For this reason, more than a fifth of the images were rejected.
In the case of patients who have their images rejected, they were forced to make a new "appointment" on another day, which became an inconvenience. In addition, the professionals claimed that some cases that did not have an adequate record (needed to schedule a return), could easily be ruled out as they did not show any visible signs of the disease.
As it was necessary to upload images to the cloud for processing, poor connections and instabilities on the Internet caused delays. "They (patients) have been waiting here since 6 am and, in the first two hours, we were only able to track 10 patients", comments a nurse about the experience of the results that should be faster.
- Brazilian startups develop system with AI that detects fever from a distance
The challenge is just beginning
Now, the Google Health team continues to work with local doctors to design new workflows and improve the tool for everyday life. For example, nurses are being trained to use their own judgment in simpler cases. In addition, they study adjustments to better deal with imperfect images.
That's because when it worked well, researcher Beede realized how AI made people who were good at their jobs even better: "There was a nurse who examined 1,000 patients on her own and, with this tool, she is unstoppable." “Patients really didn't care that it was an AI, not a human, reading their images. They were more concerned with what their experience would be ”.
"This is a crucial study for anyone interested in getting their hands dirty and really implementing AI solutions in real-world environments," explains Hamid Tizhoosh, from the University of Waterloo, Canada, who works at AI for medical imaging.