Evidence
Introduction
The clinical validation studies that prove our technology works range from assessing its efficiency at improving the diagnosis and follow-up processes in general to pathology specific clinical trials centred on proving our tool's sensibility and precision regarding that particular condition.
Algorithm development and valitation
Development
Our clinical algorithms are fully developed by us with expert dermatologists as collaborators helping with the clinical knowlegde and real-life application. With over 10 years of experience in the field of AI development for medical applications, our team has actively participated in more than 30 prestigious medical congresses and authored over 13 scientific papers, showcasing our commitment to advancing the intersection of artificial intelligence and healthcare.
To achieve state-of-the-art performance, we employ cutting-edge techniques when training Convolutional Neural Networks (CNNs), including visual transformers. Our deep learning algorithm training adheres to best practices, incorporating data-specific augmentations, one-cycle learning, and an active learning strategy. The latter involves periodic annotation and review of algorithm errors, enabling us to retrain the algorithm with corrected data to continually enhance its performance.
Furthermore, we utilize multi-task networks, encouraging the CNN to learn multiple relevant factors of the problem at hand. For instance, we train the algorithm to predict the body zone, recognizing that certain pathologies may only manifest in specific regions. This approach prevents the algorithm from making incorrect predictions, such as identifying acne on a knee.
In the realm of computer vision challenges like segmentation and object detection, we rely on architectures such as U-Nets or Yolov8 (as of the document's creation) to address these specific tasks effectively.
Central to our work is our proprietary dataset, which comprises over 1 million images encompassing more than 232 distinct skin pathologies. We train our algorithms using this comprehensive dataset, augmenting the data with the techniques mentioned earlier. One of our highly effective techniques involves manual cropping of areas of interest, performed by specialists for a significant portion of the images in our dataset. This meticulous process ensures that the algorithm focuses on the relevant regions within each image, optimizing its performance and accuracy.
Validation
Our algorithms undergo comprehensive validation across multiple stages to ensure their reliability and efficacy. The validation process begins with a retrospective clinical validation utilizing diverse datasets, followed by a prospective clinical study conducted at single or multiple medical centers.
During the initial stage, we meticulously select the most pertinent metrics based on the specific algorithm under evaluation. Here are a few illustrative examples:
- Diagnostic Support: For each of the 232 pathologies, we measure the top1, top3, and top5 accuracy of the algorithm, as well as sensitivity and specificity. Research has indicated that the top5 information is particularly valuable for healthcare professionals in enhancing their diagnostic capabilities, while sensitivity and specificity values aid in identifying pathologies that outperform or underperform the model's average performance. This approach is supported by a study published in Nature.
- Severity Assessment: When evaluating severity, we consider three distinct factors: surface area, number of lesions, and visual sign intensity. Given the unique nature of each task, the metrics employed differ accordingly. We utilize two types of metrics: those that best explain the neural network's performance and those that elucidate the output in terms of the scoring system. For instance, in the case of hive counting in urticaria, we evaluate the algorithm's ability to accurately locate hives using mean Average Precision (mAP) metrics for object detection, as well as the algorithm's precision in counting overall lesions using mean absolute error (MAE) for regression.
To ensure robust results, we apply these metrics to a static and meticulously reviewed dataset comprised of images sourced from multiple hospitals and diverse sources. This dataset, known as the regulatory test set, offers numerous advantages. Notably, it is validated by multiple specialists, lending greater credibility to the annotations. We strive to incorporate maximum variability in terms of illumination, perspectives, pathology severity, skin types, and other relevant factors.
For the latest metrics of our diagnosis support algorithm, including the pathology list and corresponding metrics, please refer to the following link: Legit.Health Pathology list & metrics.
In the subsequent stage, we immerse the algorithms within authentic clinical scenarios and assess clinical outcomes alongside other pertinent metrics such as operative costs. We have successfully completed seven studies to date, with an additional five studies slated to commence in 2023. These endeavors further enhance our understanding of the algorithms' performance and their real-world impact.
Scientific papers and congress presentations
Automatic SCOring of Atopic Dermatitis Using Deep Learning: A Pilot Study
Oral communication of results of clinical validation | AEDV 2023
Dermatology Image Quality Assessment (DIQA): Artificial intelligence to ensure the clinical utility of images for remote consultations and clinical trials
Automatic International Hidradenitis Suppurativa Severity Score System (AIHS4): A Novel Tool to Assess the Severity of Hidradenitis Suppurativa Using Artificial Intelligence
Doble inteligencia: el futuro seguimiento telemático de los pacientes por Dra Elena Sanchez-Largo
Inteligencia Artificial para la optimización de la derivación de pacientes con patologías cutáneas
Escala ALADIN para medición automática de la gravedad de acne | AEDV 2023
Diego Herrera (Almirall) & Taig MacCarthy (Legit.Health) Digitalising clinical endpoints with AI
Company IP
While deep learning algorithms are not patentable in Europe, there exist intellectual property (IP) protections that impose substantial entry barriers in terms of cost and time.
First and foremost, Legit.Health has already obtained certification as a medical device under the Medical Device Regulation (MDR). This achievement necessitated the completion of clinical trials and a lengthy certification process to register as a class 2a device. This arduous journey typically spans 3 to 4 years and incurs costs exceeding €200,000, taking into account the expenses associated with clinical trials alone, without factoring in personnel costs throughout the entire development and validation phases. Thus, these barriers extend beyond mere financial considerations and also impede the entry of larger industry players.
Secondly, our most valuable IP lies in our dataset and the corresponding annotations. The dataset itself is a critical asset for deep learning algorithms. While the architectures of the models are publicly available, the ownership of a comprehensive and extensive clinical dataset like the one possessed by Legit.Health is not easily attainable. Although hospitals or larger healthcare institutions may possess datasets, they may not possess the immediate capacity to train algorithms and effectively compete with us. Overcoming this challenge involves another significant hurdle: annotations.
Annotations refer to the process of assigning diagnoses to each individual image within the dataset. For training a diagnostic algorithm, it is imperative to have the diagnosis for every single image, a resource that we already possess. Moreover, for severity assessment, we have generated thousands of segmentation masks to detect surfaces and measured the intensities of relevant clinical signs. This level of detailed annotation is not commonly undertaken within healthcare institutions. Generating such information is highly time-consuming and incurs substantial costs, as doctors charge upwards of €200 per hour for their expertise.