Clinical Evidence
Introduction
The clinical validation studies that prove our technology works range from assessing its efficiency at improving the diagnosis and follow-up processes in general to pathology specific clinical trials centred on proving our tool's sensitivity and precision regarding that particular condition.
Core AI capabilities
Legit.Health operates 59 validated AI models across 8 clinical categories: diagnosis support, severity quantification, wound assessment, wound staging, surface quantification, lesion detection, phenotype classification, and operational (DIQA). See the complete AI algorithms catalogue →
Algorithm development and validation
Development
Our clinical algorithms are fully developed by us with expert dermatologists as collaborators helping with the clinical knowledge and real-life application. With over 10 years of experience in the field of AI development for medical applications, our team has actively participated in more than 30 prestigious medical congresses and authored 9 peer-reviewed papers, showcasing our commitment to advancing the intersection of artificial intelligence and healthcare.
To achieve state-of-the-art performance, we employ cutting-edge techniques when training Convolutional Neural Networks (CNNs), including visual transformers. Our deep learning algorithm training adheres to best practices, incorporating data-specific augmentations, one-cycle learning, and an active learning strategy. The latter involves periodic annotation and review of algorithm errors, enabling us to retrain the algorithm with corrected data to continually enhance its performance.
Furthermore, we utilize multi-task networks, encouraging the CNN to learn multiple relevant factors of the problem at hand. For instance, we train the algorithm to predict the body zone, recognizing that certain pathologies may only manifest in specific regions. This approach prevents the algorithm from making incorrect predictions, such as identifying acne on a knee.
In the realm of computer vision challenges like segmentation and object detection, we rely on architectures such as U-Nets or Yolov8 (as of the document's creation) to address these specific tasks effectively.
Central to our work is our proprietary dataset, which comprises over 1 million images encompassing more than 232 distinct skin pathologies in the training set (the platform covers 300+ conditions in production). We train our algorithms using this comprehensive dataset, augmenting the data with the techniques mentioned earlier. One of our highly effective techniques involves manual cropping of areas of interest, performed by specialists for a significant portion of the images in our dataset. This meticulous process ensures that the algorithm focuses on the relevant regions within each image, optimizing its performance and accuracy.
Validation
Our algorithms undergo comprehensive validation across multiple stages to ensure their reliability and efficacy. The validation process begins with a retrospective clinical validation utilizing diverse datasets, followed by a prospective clinical study conducted at single or multiple medical centers.
During the initial stage, we meticulously select the most pertinent metrics based on the specific algorithm under evaluation. Here are a few illustrative examples:
- Diagnostic Support: Across the full pathology set, we measure the top1, top3, and top5 accuracy of the algorithm, and we report sensitivity and specificity for the pathologies with sufficient sample size. Research has indicated that the top5 information is particularly valuable for healthcare professionals in enhancing their diagnostic capabilities, while sensitivity and specificity values aid in identifying pathologies that outperform or underperform the model's average performance. This approach is supported by a study published in Nature.
- Severity Assessment: When evaluating severity, we consider three distinct factors: surface area, number of lesions, and visual sign intensity. Given the unique nature of each task, the metrics employed differ accordingly. We utilize two types of metrics: those that best explain the neural network's performance and those that elucidate the output in terms of the scoring system. For instance, in the case of hive counting in urticaria, we evaluate the algorithm's ability to accurately locate hives using mean Average Precision (mAP) metrics for object detection, as well as the algorithm's precision in counting overall lesions using mean absolute error (MAE) for regression.
To ensure robust results, we apply these metrics to a static and meticulously reviewed dataset comprised of images sourced from multiple hospitals and diverse sources. This dataset, known as the regulatory test set, offers numerous advantages. Notably, it is validated by multiple specialists, lending greater credibility to the annotations. We strive to incorporate maximum variability in terms of illumination, perspectives, pathology severity, skin types, and other relevant factors.
The full pathology list and per-algorithm metrics are available in due diligence.
In the subsequent stage, we immerse the algorithms within authentic clinical scenarios and assess clinical outcomes alongside other pertinent metrics such as operative costs. To date, we have completed multiple prospective clinical studies in single and multiple medical centres, and our dataset spans 18 countries and over 1 million annotated images.
Peer-reviewed publications
9 peer-reviewed publications in international journals validating Legit.Health's algorithms:
Scientific paper roadmap 2026-2027
Beyond the 9 peer-reviewed publications already in print, Legit.Health has 7 additional papers in active pipeline across submission and drafting stages, each validating a distinct algorithm and pathology.
In pipeline (under review or drafting)
At 7 forthcoming publications over 18 months, Legit.Health produces peer-reviewed clinical evidence at a rate that outpaces most early-stage clinical AI companies. Each paper validates a distinct algorithm + pathology, opening a new addressable use case for pharma trials, hospital deployments, and insurance triage. This is the clinical evidence engine that competitors cannot match without 3-5 years of catch-up investment.
Automatic SCOring of Atopic Dermatitis Using Deep Learning: A Pilot Study
Oral communication of results of clinical validation | AEDV 2023
Dermatology Image Quality Assessment (DIQA): Artificial intelligence to ensure the clinical utility of images for remote consultations and clinical trials
Automatic International Hidradenitis Suppurativa Severity Score System (AIHS4): A Novel Tool to Assess the Severity of Hidradenitis Suppurativa Using Artificial Intelligence
Doble inteligencia: el futuro seguimiento telemático de los pacientes por Dra Elena Sanchez-Largo
Inteligencia Artificial para la optimización de la derivación de pacientes con patologías cutáneas
Escala ALADIN para medición automática de la gravedad de acne | AEDV 2023
Diego Herrera (Almirall) & Taig MacCarthy (Legit.Health) Digitalising clinical endpoints with AI
Company IP
While deep learning algorithms are not patentable in Europe, there exist intellectual property (IP) protections that impose substantial entry barriers in terms of cost and time.
First and foremost, Legit.Health has already obtained certification as a medical device (CE mark under the Medical Device Directive, Class I, since 2020) and is currently in transition to the Medical Device Regulation (MDR) as a class IIb device under Rule 11 with BSI. This achievement necessitated the completion of clinical trials and a lengthy certification process. This arduous journey typically spans 3 to 4 years and incurs costs exceeding €200,000, taking into account the expenses associated with clinical trials alone, without factoring in personnel costs throughout the entire development and validation phases. Thus, these barriers extend beyond mere financial considerations and also impede the entry of larger industry players.
Secondly, our most valuable IP lies in our dataset and the corresponding annotations. The dataset itself is a critical asset for deep learning algorithms. While the architectures of the models are publicly available, the ownership of a comprehensive and extensive clinical dataset like the one possessed by Legit.Health is not easily attainable. Although hospitals or larger healthcare institutions may possess datasets, they may not possess the immediate capacity to train algorithms and effectively compete with us. Overcoming this challenge involves another significant hurdle: annotations.
Annotations refer to the process of assigning diagnoses to each individual image within the dataset. For training a diagnostic algorithm, it is imperative to have the diagnosis for every single image, a resource that we already possess. Moreover, for severity assessment, we have generated thousands of segmentation masks to detect surfaces and measured the intensities of relevant clinical signs. This level of detailed annotation is not commonly undertaken within healthcare institutions. Generating such information is highly time-consuming and incurs substantial costs, as doctors charge upwards of €200 per hour for their expertise.