WhatsApp Medical Triage Agent — Advisory System with Clinical Routing

Overview

A production conversational agent deployed on WhatsApp for an orthopedic surgery clinic. It supports patients arriving through the hospital's digital campaigns by providing information about surgical procedures and eligible public healthcare programs, analyzing their diagnostic studies, and routing each case to the appropriate specialist.

The architectural decision that defines the entire system is simple: this is an advisory system, not an autonomous one. The agent never issues a definitive diagnosis. It produces a routing recommendation—for example, "you should be evaluated by the surgeon" or "physical therapy may be sufficient"—while the actual clinical decision is always made by the specialist during the consultation. The physician is the human-in-the-loop by design.

This distinction is fundamental. In a clinical setting, allowing a model to make the final decision is not acceptable. Allowing it to recommend, prioritize, and escalate while a qualified human confirms the outcome is the defensible pattern. The agent surfaces candidates; the physician decides.

Currently running in production, handling approximately 400 messages per day.

Core design decision: the LLM does not control the workflow

The system is built around an explicit 10-state state machine in which every transition is deterministic and auditable. The LLM does not orchestrate the workflow; it performs discrete, bounded tasks within each state.

This provides two properties that are essential in a medical conversational system: traceability—the system always knows the patient's current state and why a transition occurred—and failure containment—an LLM hallucination cannot derail the entire workflow; it can only affect the subtask within the current state, whose output can be validated.

Conversation flow

new
  ↓
ask_study                  ← Does the patient have prior diagnostic studies?
  ↓
categorize_study           ← Study available? → request file | recommend obtaining one
  ↓
[process_study]            ← Claude Vision OCR or text extraction from PDFs/images
  ↓
collect_age                ← Extract the patient's age from the conversation history
  ↓
collect_diagnosis_answers  ← Generate symptom questions dynamically
  ↓                           based on findings from the submitted study
validate_study             ← Check case consistency and completeness
  ↓
routing                    ← Routing recommendation: surgeon / physical therapy
  ↓
gather_data                ← Collect contact details: name, RUT, email, phone
  ↓
send_contact / schedule_appointment

Conversation state is persisted to MongoDB after every transition. If a patient leaves the conversation and returns several hours later, the agent resumes from the exact point where the interaction stopped.

Multimodal ingestion of diagnostic studies

Clinical input is heterogeneous and noisy by nature. The agent supports three input formats:

When a patient sends an image, the agent immediately acknowledges receipt—"Analyzing the information..."—while Claude Vision transcribes and structures the medical content. The resulting text is then passed to the decision layer.

Symptom questions are not static. They are generated dynamically according to the type of study and its findings. For example, a knee MRI triggers a different set of follow-up questions than a hip X-ray.

For potentially surgical cases, the agent provides a preliminary assessment—always emphasizing that the physician will determine the final diagnosis—and schedules an appointment through Google Calendar with an automatically generated Google Meet link.

For non-surgical cases, the patient is referred to physical therapy and receives the appropriate department's contact information.

V1 (production): LLM classification over structured features

The production version uses structured extraction. Instead of returning a free-form verdict, the LLM maps the study, patient age, and symptoms into predefined fields and then produces a routing recommendation using a specialized clinical prompt.

This version has an important limitation that I identified while working on the system, and it became the foundation for its proposed evolution:

An LLM used as a clinical classifier is non-deterministic, and its stated confidence is not reliable. The same input can produce different outputs. More importantly, in a system that depends on a decision threshold, the "confidence" expressed by an LLM is not calibrated. When an LLM says "80% probability," that number does not correspond to an observed 80% success rate. It is probability-shaped text, not a validated probability estimate.

That limitation is tolerable in an advisory system because a human confirms the recommendation. However, to make routing reliable, measurable, and operationally defensible, the decision cannot depend on the model's raw self-reported confidence. That diagnosis of the problem is what motivated V2.

The actual machine learning problem: errors are asymmetric

Before discussing architecture, the central design premise must be explicit because every downstream decision follows from it:

In a routing-based triage system, a false negative is significantly more costly than a false positive.

• Routing a surgical patient → physical therapy is a false negative for the surgical class. This is the expensive failure mode: the patient may never reach the surgeon, and no downstream specialist may correct the recommendation.
• Routing a non-surgical patient → physician is a false positive. It consumes an appointment slot, but the physician can correct the routing decision during the consultation without creating the same level of clinical risk.

This premise leads directly to three design consequences:

1. The north-star metric is not accuracy; it is recall / sensitivity for the surgical class. An overall accuracy of 88% can conceal poor performance on the cases that matter most.
2. The decision threshold should be intentionally biased toward surgical sensitivity, accepting additional false positives in exchange for driving surgical false negatives as close to zero as possible. This is a cost-sensitive learning problem: the asymmetric cost function determines where the threshold should be placed.
3. A clear residual risk remains: patients routed to a physician are reviewed by a human, while patients routed to physical therapy may not receive confirmation from a surgeon. That is where the highest risk remains, and it is the specific failure mode addressed by the abstention layer.

V2 (proposed design): hybrid pipeline with calibrated ML

The proposed evolution does not replace the LLM with a traditional model, nor does it replace machine learning with an LLM. It treats the solution as a compound AI system, assigning each component the task it is best suited to perform.

The question—"Should production use a dedicated ML model or an LLM?"—is a false dichotomy. The stronger architecture uses both, with clearly separated responsibilities:

Diagnostic study + age + symptoms
        ↓
[1] LLM as a FEATURE EXTRACTOR, not the decision-maker
        ↓  Claude Vision + reasoning → fixed-schema JSON:
        ↓  meniscal tear grade, presence of effusion,
        ↓  Kellgren–Lawrence grade, etc.
        ↓
[2] Small, CALIBRATED supervised model
        ↓  logistic regression / gradient boosting
        ↓  structured features → probability estimate
        ↓
[3] CALIBRATION layer
        ↓  Platt scaling / isotonic regression / temperature scaling
        ↓  probability becomes empirically interpretable
        ↓
[4] ABSTENTION / defer-to-human layer
        ↓  gray-zone cases are not forced into a class
        ↓  they are escalated to the physician, never defaulted to physical therapy
        ↓
Routing decision

Why each component exists:

• LLM as extractor, not decision-maker. The difficult part of this domain is converting heterogeneous, noisy clinical text and images into clean features. LLMs are highly effective at this task. V1 already solved structured extraction, so the natural next step is not to rewrite that layer; it is to remove final decision authority from the LLM and assign it to a model whose outputs can be calibrated and evaluated.

• Calibrated supervised model for routing. A dedicated classifier produces a probability that can be calibrated and validated through a reliability curve. An LLM does not provide that property cleanly. This also introduces determinism—the same features produce the same probability—and improves interpretability because the system can show the physician which features contributed to the prediction, increasing clinical trust in the tool.

• Abstention layer (selective prediction / learning to defer). When the model is not sufficiently confident, the system does not force the case into a class. It escalates the case to the physician. This directly addresses the residual risk: an ambiguous case is never sent to physical therapy by default; it is reviewed by a human.

Probability handling is a deliberate product decision: the calibrated probability is an internal control signal, not a patient-facing output. Telling a patient that they have a "73% probability of requiring surgery" is clinically counterproductive: a high number can create unnecessary anxiety, while a lower number can create false reassurance. Internally, the probability is useful for (a) routing and abstention thresholds, (b) prioritizing appointment urgency, and (c) providing decision support to the physician, not to the patient. The patient receives a clear, human-readable recommendation: "we recommend an evaluation with the surgeon" or "physical therapy may be sufficient, and a specialist evaluation remains available if needed."

Cold start: the physician as the source of ground truth

A supervised model requires labeled data. The strength of this design is that the same human-review loop that makes the system safe also generates the data required to improve it. Every consultation produces a pair such as (agent_recommendation, specialist_confirmation). This becomes labeled data generated through normal operations rather than through a separate annotation process.

Bootstrapping strategy:

1. V1 (LLM + rules) supports production routing from day one.
2. Each physician confirmation is stored as a label.
3. Once enough data has accumulated, V2 is trained and calibrated.
4. Continuous monitoring compares recommendations against confirmed outcomes to detect degradation.

Metrics I would instrument—and why

Rather than relying on global accuracy—which can hide the most consequential error—the evaluation framework is built around asymmetric costs:

The purpose of this evaluation design is not to showcase a single headline number. It is to demonstrate which measurements are required for a clinical routing classifier and why these metrics—not aggregate accuracy—represent the system's actual risk profile.

Domain context

The agent includes domain knowledge about national public healthcare programs relevant to knee surgery, such as reduced-copayment coverage for total knee replacement. When a patient meets the eligibility criteria, the agent provides specific information about costs, coverage, and requirements. Patient registration also includes RUT validation.

Automated re-engagement

If a patient abandons the workflow before completing triage, the agent uses an LLM to determine whether re-engagement is appropriate and, when applicable, sends a personalized follow-up message based on the interrupted conversation context.

Multi-provider fallback chain

LLM_CHAIN = openai (gpt-4o) → groq (llama-3.3-70b) → anthropic (claude-3.7-sonnet) → deepseek

Each provider is assigned to tasks that match its operational strengths:

• OpenAI: clinical reasoning, structured extraction, and confirmation steps.
• Groq: low-latency binary or categorical decisions.
• Anthropic Claude: multimodal analysis of medical images.

Results and operational impact

• Physicians receive pre-triaged patients with diagnostic studies already processed and age and symptoms summarized before the consultation.
• The system removes the need to manually filter non-surgical patients from the physician's schedule.
• It provides 24/7 patient support without requiring additional staff.
• The workflow remains advisory from end to end: clinical authority always belongs to the specialist.

Intentional technical debt: I designed the ML evaluation and monitoring pipeline described in V2 and proposed it to the team. At the time, the business prioritized patient-facing features, so continuous monitoring remained identified and documented technical debt with a clear implementation plan. This was a reasonable trade-off in a startup environment, and the path to productionizing it is well defined.

Technical stack

• LLMs for extraction and reasoning: OpenAI GPT-4o · Groq Llama 3.3 70B · Anthropic Claude 3.7 Sonnet · DeepSeek as fallback
• Vision: Anthropic Claude Vision for OCR and interpretation of medical studies
• V2 design: calibrated supervised model—logistic regression or gradient boosting—with calibration and abstention layers
• Document processing: pdfminer, python-docx, openpyxl
• Messaging platform: WhatsApp Business API
• Backend: Python with conversation state persisted in MongoDB
• Scheduling: Google Calendar API + Google Meet
• Notifications: push · email through AWS Lambda · WhatsApp
• Patient registration: Google Sheets API