LetTokenBurn
Sign in
Auto86%Harbor: pendingKnowledge & ReasoningAgentic / Tool Use

SDDBench

Ship Design Decision Benchmark

A benchmark with 5,000 records and 43,496 typed questions evaluating natural-language-driven constrained ship-form editing decisions.

PaperRepo

How to run

ltb · sddbench
$ npm i -g lettokenburn-cli
$ ltb bench info sddbench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

7 sourced entries
HarnessSourceConfidenceLink
Claude Opus 592.9N/AIndependentmediumOct 7, 2026arxiv.org
DeepSeek V4.1 Flash92.8N/AIndependentmediumOct 7, 2026arxiv.org
GPT-5.6 Sol92.4N/AIndependentmediumOct 7, 2026arxiv.org
Jev91.0N/AIndependentmediumOct 7, 2026arxiv.org
Qwen3 Max (2025-09-23)75.6N/AIndependentmediumOct 7, 2026arxiv.org
Gemma367.0N/AIndependentmediumOct 7, 2026arxiv.org
Llama3.263.9N/AIndependentmediumOct 7, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

SDDBench is a subset of the Ship Design Decision Dataset containing 5,000 cleaned records. It features 43,496 typed questions designed to evaluate models on translating natural language requests into discrete ship design actions.

Evaluation metrics include question accuracy, FFD exact match, and calibration scores. The Chip model achieved 95.90% question accuracy and 99.32% FFD exact match on this benchmark.