KlinikeBench

KlinikeBench

333 long-horizon clinical consultations for LLM agents. In each task the agent plays a clinician: it interviews an LLM-simulated patient, orders tests and looks things up through a 140-tool clinical action surface, and records a diagnosis. Everything happens through the clinic CLI inside the task container:

clinic brief                       # patient id, specialty, turn budget
clinic tools                       # tool schemas
clinic say "When did the pain start?"
clinic call order_lab_test '{"patient_id": "P...", "test_name": "CBC"}'
clinic call record_diagnosis '{"patient_id": "P...", "diagnosis": "...", "confidence": "high", "reasoning": "..."}'

Scoring

The verifier scores four gates; reward = 1 only if all pass (pass@1 in the paper):

| gate | passes when | |---|---| | diagnosis | the recorded diagnosis matches the reference (or an accepted synonym) | | required tools | every required clinical tool was called | | parameterised actions | no forbidden tool or parameter violations | | must-ask | an LLM judge confirms every must-ask history topic was covered |

/logs/verifier/reward.json also reports each gate separately, plus episode length.

Budgets: 10 patient turns and 30 agent steps (slightly more for tool-heavy tasks), 40 min agent timeout.

Requirements

The simulated patient (in the task container) and the must-ask judge (in the verifier) call an OpenAI-compatible chat-completions endpoint. The paper's protocol uses gpt-5.4-mini for both, which is the default:

export OPENAI_API_KEY=...          # used for the patient and the judge
harbor run -d klinikebench/klinikebench -a terminus-2 -m <provider/model>

Override per role with CAA_PATIENT_BASE_URL / CAA_PATIENT_MODEL / CAA_PATIENT_API_KEY and CAA_JUDGE_BASE_URL / CAA_JUDGE_MODEL / CAA_JUDGE_API_KEY. CAA_JUDGE_ENABLED=0 switches the must-ask gate to a rule-based scorer (no judge calls; numbers are then not comparable to the paper). Changing the patient or judge model also changes scores; report them alongside results.

Keeping the key out of the agent's reach. Harbor passes [environment.env] into the container, so the agent's shell can see the patient key. For leaderboard-grade runs, run a small OpenAI-compatible proxy on the host that adds the key, point the patient at it and leave the key unset:

export CAA_PATIENT_BASE_URL=http://172.17.0.1:<port>/v1   # docker0 address on Linux; host.docker.internal on macOS
export CAA_PATIENT_API_KEY= OPENAI_API_KEY=               # the proxy holds the key; the judge can use its own CAA_JUDGE_API_KEY

Each episode makes at most one patient call per patient turn (10 max) and one judge call per must-ask topic.

Other switches

• CAA_REWARD_MODE = outcome (default: the benchmark score), diagnosis (diagnosis gate only) or multigate

((2·diagnosis + required_tools + parameterised_actions + must_ask) / 5). Only changes the scalar reward; every gate is always logged. Used for RL training.

• CAA_MAX_STEPS_OVERRIDE = an integer agent-step cap, or none to remove it (step-cap ablation). Empty by default.

Splits

klinikebench/klinikebench contains all 333 tasks. The fine-tuning experiments use a 283 / 50 train / test split (42 of the 50 test diagnoses never occur in training); the task-id lists are in splits/ of the training code repository (https://github.com/Zehui127/klinikebench-train). To run only the 50 test tasks:

harbor run -d klinikebench/klinikebench $(sed 's|^|-i klinikebench/|' splits/heldout_task_ids.txt) -a terminus-2 -m <provider/model>

Notes

• Cases are built from curated disease knowledge sources (the hpoa, primekg and mendeley tags in task ids);

the patient is always an LLM simulation.

• The container needs outbound network access (for the patient simulator).
• Research use only. This is not a medical device, and agents that score well here are not safe for clinical use.

Changelog

• 1.1 — fixes a bug in 1.0 where the task daemon failed to load the clinical database, so the lookup tools fell

back to per-task data (scores mostly moved on the required-tools gate). Adds CAA_REWARD_MODE and CAA_MAX_STEPS_OVERRIDE; their defaults don't change scoring. Case content is unchanged.

• 1.0 — initial release.

Citation

If you use KlinikeBench, please cite the KlinikeBench paper (arXiv:2609.38480):

@misc{fang2026klinikebench,
  title         = {KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy},
  author        = {Fang, Xueting and Li, Zehui and Yang, Yang and Giovino, Camilla and Patel, Shubh K. and
                   Prajapati, Shailly and Subasri, Vallijah and Shan, Caihua},
  year          = {2026},
  eprint        = {2609.38480},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2609.38480}
}