When confronted with 200 biopsy-confirmed diagnostic challenges, leading general-purpose large language models achieved an accuracy of only 42%. In contrast, Inception v3—an architecture released more than a decade ago—scored 62%. Across 30 difficult melanoma cases, the vintage algorithm caught nearly twice as many malignancies as the cutting-edge models. This cross-generational blind benchmark delivers a sobering reality check to assumptions that general-purpose frontier models will effortlessly sweep specialized clinical domains.
The winning model underwent merely 20 minutes of fine-tuning and holds just 24 million parameters. Under dermoscopy, deep pigment networks and atypical microvascular patterns depend on specialized convolutional receptive fields engineered to capture localized spatial geometry. Massive web-scale image-text datasets cannot substitute for focused vertical training on microscopic pathology. Facing early-stage invasive malignancies, generalist multimodal models even committed fatal diagnostic errors, recommending that cancer patients be safely discharged.
Tested on 200 Biopsy-Verified Lesions: Frontier LLMs Score Just 40%
In October 2026, a rigorous cross-generational blind trial unfolded in London. Dr. Magnus Lynch, a Consultant Dermatologist at Guy’s Hospital, assembled 200 real-world lesion images verified by histopathology for a head-to-head showdown. On one side stood modern multi-billion-parameter generalist models: GPT-6 Astra and Claude Fable 5.1. On the other was Inception v3, an architecture originally open-sourced by Google in 2015.
The test dataset was curated from MILK10k, an open benchmark published in 2025 by the International Skin Imaging Collaboration (ISIC). Every case had been confirmed via histological biopsy. Half of the cohort (100 lesions) represented high-risk conditions mandating clinical intervention: 30 melanomas, 25 basal cell carcinomas (BCC), 25 squamous cell carcinomas (SCC), and 20 actinic keratoses (AKIEC). The remaining 100 cases comprised benign controls: 50 benign melanocytic nevi and 50 seborrheic keratoses.
The six-class blind classification results were stark. The decade-old Inception v3 correctly categorized 125 cases, achieving 62% accuracy. The frontier generalist models lagged substantially: GPT-6 Astra got 84 correct, while Claude Fable 5.1 reached 85—both hovering around 42%. When evaluation criteria were broadened to top-2 accuracy, the vintage CNN identified 173 cases correctly, compared to just 134 and 138 for the LLMs.
In the critical subset of 30 invasive melanomas—the deadliest form of skin cancer—the older model caught 13 cases on initial diagnosis. The two generalist models managed only 8 and 11, respectively. Medical image interpretation demands deep specialization in microscopic pixel patterns. While frontier LLMs ingest astronomical volumes of multimodal internet data, they fundamentally lack the dedicated visual priors required for subtle dermatopathology.
Calling Invasive Melanoma a Benign Wart: The Frequency Trap of Generalist Models
The failures of the generalist LLMs exhibited pronounced systematic biases. When evaluating ambiguous skin lesions, the models repeatedly fell back on high-prevalence diagnostic labels encountered during pretraining. Over 200 test inferences, GPT-6 Astra predicted basal cell carcinoma 75 times—even though only 25 BCC cases existed in the entire dataset. Among those 50 false-positive BCC calls, it misdiagnosed and effectively buried 14 deadly melanomas.
Claude Fable 5.1 suffered a similar distortion, labeling 53 lesions as actinic keratosis when the ground truth held only 20. This heuristic tendency to force difficult presentations into dominant categories caused dangerous malignancies to slip through unnoticed.
Figure: Clinical macroscopic photograph of an invasive melanoma on the ear. With a Breslow thickness under 0.8 mm, the lesion easily mimics a benign seborrheic keratosis to the naked eye. Both frontier LLMs misdiagnosed the lesion as benign and advised discharge. Source: MILK10k / Lynch Lab (CC BY-NC)
Figure: High-resolution dermoscopic image of the identical ear lesion. Beneath the surface, atypical pigment networks and irregular polymorphous vessels are clearly discernible. Guided by these structural features, the decade-old CNN correctly flagged the malignancy. Source: MILK10k / Lynch Lab (CC BY-NC)
A particularly stark misdiagnosis involved an ear lesion. An early invasive melanoma measured under 0.8 mm in Breslow thickness. Because of its crusted surface and irregular pigmentation, both general-purpose models classified it as a benign seborrheic keratosis. Worse still, their generated clinical summaries explicitly reassured the patient that no further follow-up was needed. Only the decade-old CNN, reading subsurface structures under polarized dermoscopy, sounded the alarm on malignancy.
Figure: An amelanotic invasive melanoma largely devoid of dark pigment. Without classic melanin deposition, GPT-6 Astra misclassified the lesion as basal cell carcinoma with an astonishing 95% confidence score. Source: MILK10k / Lynch Lab (CC BY-NC)
Another amelanotic melanoma exposed similar vulnerabilities. Devoid of characteristic dark pigmentation, the lesion fooled GPT-6 Astra, which declared it a basal cell carcinoma with 95% confidence. Multimodal LLMs struggle to isolate fine-grained vascular arborization and subtle pigment network disruptions. In language models, high confidence often merely reflects smoothed probability distributions from text generation rather than genuine histopathologic discernment.
Fine-Tuned in 20 Minutes: Why Older Architectures Win on Domain Priors
The specialized model that clinched the top score required no extraordinary compute. Running on Google’s 2015 Inception v3 architecture, Dr. Lynch fine-tuned the network on a single NVIDIA H100 GPU in approximately 20 minutes.
The training corpus drew from the ISIC open repository, encompassing 68,011 clinical and dermoscopic images. To preserve strict scientific integrity, the benchmark enforced complete data hygiene: all 200 evaluation lesions and any associated images from the same patients were scrupulously purged prior to fine-tuning.
Inception v3 succeeded because it was optimized solely around specialized dermoscopic priors. Dermoscopy relies on polarized light to bypass the stratum corneum, revealing underlying pigment lattices and capillary beds within the papillary dermis. Fixed convolutional receptive fields excel at locking onto these micro-architectural geometries. Conversely, generalist vision-language models dilute their parameter capacity across vast oceans of mundane web imagery, washing out subtle pathologic cues.
Furthermore, lightweight specialized models offer dramatic advantages for on-device deployment. With only 24 million parameters, Inception v3 compiles cleanly into client-side web browsers via WebGPU. Patients and clinics can process high-resolution dermatological imagery locally on a smartphone without dispatching private medical records to remote cloud servers—guaranteeing sub-second latency while safeguarding sensitive patient data.
Shift to Patient Inquiries: Models Retreat into Blanket Disclaimer Mode
Faced with dismal diagnostic accuracy in formal classifications, the general-purpose models exhibited a starkly different behavioral profile when prompted informally. Dr. Lynch restructured the evaluation prompt: eliminating the six diagnostic options, he presented only clinical images alongside a everyday patient query: “Do I need to see a doctor about this?”
When evaluated on an additional set of 100 biopsy-proven melanomas, both frontier LLMs retreated into extreme risk-aversion. GPT-6 Astra advised seeking medical attention in 99 out of 100 cases, while Claude Fable 5.1 recommended clinical evaluation for 98.
This intense caution is an artifact of alignment and safety guardrails. Pretrained on strict liability-mitigation protocols, frontier LLMs instinctively generate cautious referral templates whenever skin abnormalities or health questions arise. While valuable as a defensive triage strategy, such blanket disclaimers provide zero differential diagnostic insight for clinical workflows.
The moment an evaluation demands rigorous disease classification, the apparent diagnostic competence of generalist models collapses. Defensive boilerplates cannot substitute for granular discriminatory power; conservative conversational alignment creates an illusion of clinical safety that masks an absence of specialized visual grounding.
Contextualizing the Benchmarks: Specialized Models Remain Essential
While these findings are revealing, Dr. Lynch proactively underscored the structural boundaries of the study. First, sample size remains modest: the cohort included 30 melanomas, widening statistical confidence intervals and relying on single-pass inference. Second, case difficulty was extraordinarily high. All 200 lesions were selected from biopsy-confirmed referrals that challenged experienced clinicians on visual inspection alone. While Inception v3 historically achieves an AUROC of 0.96 on standard ISIC datasets, its performance dropped to 0.86 on this difficult cohort.
Moreover, the MILK10k dataset has been publicly accessible since 2025; multimodal LLMs may have already encountered these images during pretraining. That frontier models still scored only 42% despite potential data contamination underscores the deficit in domain-specific visual representation. Dr. Lynch also noted that Inception v3 represents a decade-old baseline; contemporary, regulatory-approved commercial dermatology systems perform far better. Ironically, the benchmark’s statistical scripts and analysis draft were assembled with assistance from Anthropic’s AI coding tools.
Dermatological diagnostic precision cannot thrive without domain-specific visual priors. Generalist frontier models will continue to excel at patient communication, clinical summarization, and initial intake routing. But at the critical frontier of life-and-death diagnostic screening, compact models trained on specialized clinical data remain an indispensable line of defense.
References:
- Lynch Lab Report: LLMs Fall Short of Old CNN in Dermoscopy
- Hacker News Community Technical Discussion
