🏥 Whisper Medical Fine-Tuning Pipeline
MedTranslate ASR — Fine-tune Whisper Large v3 Turbo on medical speech for deployment via WhisperKit on iOS.
Pipeline Overview
- Generate sentences — ICD-10, RxNorm, clinical templates (~20K sentences, CPU)
- Synthesize audio — edge-tts multi-voice TTS (~2-3 hrs, CPU)
- Build HF dataset — package + push to Hub (CPU)
- Fine-tune — LoRA on A10G (~4-6 hrs, switch to A10G hardware first)
- Evaluate — WER + medical entity recall
- Convert — whisperkittools → Core ML → push to Hub
Step 1-3: Data Generation
Run on CPU hardware (free).
Step 4: LoRA Fine-Tuning
⚠️ Switch Space hardware to A10G small before running.
Start with whisper-small to validate, then run whisper-large-v3-turbo.
Base Model
Monitor training:
- TensorBoard: Check the
whisper-medical-ft/runs/directory in your Space files - HF Hub: Checkpoints push automatically every 500 steps
Step 5: Evaluate Fine-Tuned Model
Step 6: Convert to Core ML
Run on CPU hardware (conversion is CPU-bound, free tier).
Ensure FT_MODEL_REPO and COREML_REPO Space secrets are set.
After conversion, update STTEngine.swift in your iOS app:
let config = WhisperKitConfig(
model: "large-v3-turbo",
modelRepo: "yourorg/whisper-medical-v1-coreml"
)