Case Study · Documentation
Skribe.
An AI medical scribe that runs a natural, spoken intake conversation before a doctor's visit - transcribing, questioning, and documenting in real time, then handing the clinician a clean PDF. Built for the AMD Developer Hackathon.
At a glance
Interface
Voice-first web app · 7 languages
Voice loop
Whisper → Llama 3.3 → ElevenLabs
Knowledge
MedQuAD RAG in ChromaDB
Output
Clinician-ready PDF report
Overview
Paperwork, replaced by a conversation
Clinic intake is usually a clipboard of forms filled in a waiting room. Skribe replaces that with a spoken conversation. A patient talks; Skribe listens, asks the right follow-up questions, and writes everything down - producing the structured document a doctor actually needs before the visit begins.
It runs as two conversations. First an intake phase captures the patient's history once and remembers it. Then a symptoms phase documents today's complaint, with every clinical follow-up grounded in a retrieved medical knowledge base so the questions are specific and useful - never a diagnosis, just a thorough record.
Architecture
The real-time voice loop
Client · React voice-first SPA
Mic capture
MediaRecorder
Voice orb
turn-by-turn UI
Audio playback
synthesized reply
Backend · FastAPI (Docker on HF Spaces)
Whisper
speech → text
Groq · Llama 3.3 70B
scribe brain
ElevenLabs
text → speech
ChromaDB + MedQuAD
clinical retrieval
SQLite
users · sessions · intake
One turn = record → transcribe → retrieve → generate → synthesize → play. The loop repeats until the model marks the phase complete, then the report is built.
Per-turn pipeline
Listen → Retrieve → Ask → Speak
Listen
The browser records the patient’s mic audio and posts it to /session/transcribe, where OpenAI Whisper returns a transcript in the patient’s own language.
Retrieve
In the symptoms phase, the transcript is embedded with sentence-transformers and matched against the MedQuAD corpus in ChromaDB. The top-5 clinical Q&A ground the next question.
Ask
Groq Llama 3.3 70B, steered by a strict scribe prompt plus the retrieved context, writes exactly one clinically-directed follow-up. It documents, it never diagnoses.
Speak
ElevenLabs synthesizes the question and it plays back through the voice orb. The loop repeats until every field is collected, then reportlab builds the PDF.
Conversation design
Two conversations, one visit
Intake
Full name, age, current conditions, medications, allergies, and family history. Collected once, saved to the patient’s profile, and pre-loaded on every future visit.
Symptoms
Chief complaint, location, duration, severity on a 1–10 scale, triggers, and associated symptoms. The first question is open-ended; every follow-up is specific and drawn from the knowledge base.
Backend
API surface
POST /session/startOpens a session in a given phase (intake / symptoms) and language.
POST /session/transcribeWhisper speech-to-text for a mic recording.
POST /session/respondAdvances the conversation: LLM + RAG → next question + DONE flag.
POST /session/ttsElevenLabs synthesis (eleven_flash_v2_5) of the question audio.
POST /session/generate-reportParses the collected data and builds the branded PDF.
GET /report/{id}/pdfStreams the finished intake report to the clinician.
/auth · /profileJWT auth (bcrypt) and the reusable patient profile.
GET /rag/statusReports the ChromaDB index build state on startup.
Guardrails
Design decisions
Scribe, not diagnostician
The prompt forbids diagnoses, treatment suggestions, and even naming conditions. Skribe only documents. Every clinical follow-up is derived from retrieved MedQuAD entries, not the model’s guesswork.
Speaks the patient’s language
Whisper detects and transcribes, and the LLM is instructed to respond only in the chosen language across English, Spanish, French, Mandarin, Arabic, Hindi, and Portuguese.
One question at a time
A strict response format (RESPONSE / DONE) keeps the conversation to a single clear question per turn and refuses vague prompts like “tell me more,” so the transcript stays clean and structured.
Memory across visits
Intake data persists in SQLite and is injected as context into the symptoms conversation, so returning patients skip re-entering their history and the scribe already knows them.
Built with