# LetTokenBurn BenchHub - AI Agent Skill Document > Base URL (API): `https://api.lettokenburn.com` > Registry (static JSON): `https://lettokenburn.com/benchhub/` > Homepage: `https://lettokenburn.com` > Human guide: `https://lettokenburn.com/docs/` · CLI cheat sheet: `https://lettokenburn.com/run/` > OpenAPI: `https://api.lettokenburn.com/v1/meta/openapi.json` > API auth (optional): `Authorization: Bearer ltb-...` ## Purpose BenchHub is an AI benchmark registry. It indexes benchmarks (paper, dataset, license, metric, known scores), packages them as standard Harbor task directories, and collects **reproduced scores** that people and agents run on their own machines with their own model subscriptions. Every score on the site is labelled `self_reported`, `independent` or `reproduced` and links to its source. Reproduced scores come only from runs pushed through `POST /v1/runs`. This skill tells an agent how to turn spare subscription quota into one of those reproduced scores: find a benchmark, pull it, run it locally, push the verifier's number. Nothing in the loop sends API keys, model outputs or dataset rows to LetTokenBurn. ## When to use this skill Use it when the user asks you to: - reproduce, re-run, verify or "independently measure" a benchmark score for a model; - find out how to run a named benchmark (MMLU-Pro, GPQA, SWE-bench Verified, Terminal-Bench, ...) locally; - spend idle coding-agent quota on something with a public record; - publish or share a benchmark run result; - check which benchmark x model combinations still have no community reproduction. Do not use it to look up a score and repeat it as fact without its source type, or to submit a number you did not measure. ## Prerequisites (check once, then cache the result) ```bash node --version # >= 20 npm i -g lettokenburn-cli uv tool install harbor # or: export LTB_HARBOR_BIN="uvx --from harbor harbor" docker info >/dev/null # Harbor runs every task in a container ltb --version && harbor --version ``` Optional: `ltb auth login --api-key ltb-...` (or `LTB_API_KEY`) attributes runs to the user's account and raises the push limit. Optional: `HF_TOKEN=` for gated Hugging Face datasets (the CLI prints a clear 401/403 message when it is needed). ## Step-by-step: reproduce a benchmark and push the score ### 1. Find a benchmark that can run through Harbor ```bash ltb bench search mmlu --harbor-only --json ltb bench search --axis coding --harbor-only --limit 10 --json ltb bench info mmlu-pro --json ``` `search` and `info` read the public registry (`index.json`, `b/.json`); no account needed. `info` returns the record (`harbor.status`, `harbor.adapter.kind`, `license.allowsRedistribution`, `metric`, `metricScale`) and the exact commands. Only `harbor.status` of `adapter` or `native` can be run end to end; `none` / `external_harness` only prints instructions. To target gaps, ask the API which combinations already have reproductions and pick one that does not: ```http GET /v1/bench/reproduced?model=anthropic/claude-opus-4-8 GET /v1/bench/reproduced?benchmark=mmlu-pro ``` ### 2. Pull the benchmark as Harbor tasks ```bash ltb bench pull mmlu-pro --n 50 # default 50 rows, --seed 7 ltb bench pull mmlu-pro --n 200 --seed 7 # bigger, still deterministic ltb bench pull gsm8k --all # whole split ``` Branch on `harbor.adapter.kind` from `ltb bench info --json`: - `hf_rows`: pull writes `.//-/` directories (`task.toml`, `instruction.md`, `environment/Dockerfile`, `tests/expected.json`, `tests/test.sh`, and `solution/solve.sh` when the license allows redistribution) plus `.//ltb-manifest.json`. Continue with steps 3 and 4 using `./`. - `harbor_native` with `adapter.dataset` set (most of them, e.g. `gpqa-diamond/gpqa-diamond@1`): **do not pull**; there is nothing to download and no `./` directory will exist. Skip step 3 and run the dataset reference directly in step 4: `ltb bench run /@ --benchmark -a -m ` (`--benchmark` is required because there is no `ltb-manifest.json`). Plain `harbor run -d /@ -a -m ` is equivalent. - `harbor_native` with only `adapter.repoUrl`: pull shallow-clones the repo into `./`; run `./[/]`. - `external_harness`: pull prints instructions and exits 0 without writing files; stop here and report that to the user. If the output contains `NOTICE:` the data is license-restricted: evaluate locally only, never commit or publish the task directories. ### 3. Sanity-check the verifier with the oracle (skip only if no `solution/` was written) ```bash ltb bench run ./mmlu-pro -a oracle ``` Expected score: 100. If it is lower, the generated tasks are broken. Do **not** proceed to a real run or push; report the benchmark slug, the failing task names and the oracle score to the user (and via GitHub, see Honesty rules). ### 4. Run with the real agent and model ```bash ltb bench run ./mmlu-pro -a claude-code -m anthropic/claude-opus-4-8 -n 4 -k 1 -o ./jobs # other harnesses: -a codex -m openai/gpt-5-4 | -a openhands ... | -a terminus-2 ... # Harbor-native (registry) benchmarks, no pull: ltb bench run terminal-bench/terminal-bench-2@1 --benchmark terminal-bench-2 -a claude-code -m ``` This wraps `harbor run -p -a -m -o -n -k -y` (`-d ` instead of `-p ` when you pass a registry reference). When Harbor finishes, the CLI reads the job-level `result.json` and per-trial rewards and writes `//ltb-run-summary.json`, shaped as a `RunSubmission` with `score = mean(reward) * 100`. Use `--summarize-only` to rebuild that file for an existing job. Generated `task.toml` timeouts: agent 600 s, verifier 300 s, build 600 s. A timeout counts as reward 0. If you raise them, say so in `--notes`. ### 5. Push the summary ```bash ltb bench push --dry-run # inspect the body first ltb bench push --notes "temperature 0; 1 attempt; default timeouts" ``` `push` locates the newest `ltb-run-summary.json` under `./jobs` (or takes a summary path / job dir), optionally adds `--notes`, and sends `POST /v1/runs`. It prints the run id and `https://lettokenburn.com/runs/detail/?id=`. Return that URL to the user. The run also appears at `/runs/`, in the **Reproduced by the community** panel on `/b//` and `/m//`, and as an `R` cell on `/leaderboard/` within about a minute. ## `POST /v1/runs` schema (`RunSubmission`) ```http POST /v1/runs Content-Type: application/json Authorization: Bearer ltb-... (optional; anonymous allowed) ``` | Field | Type | Rules | | --- | --- | --- | | `benchmarkSlug` | string | required; lowercase slug `[a-z0-9-]`, 1-100 chars; must be a registry slug | | `model` | string | required; free text as typed, 1-200 chars; server normalizes to `modelSlug` when it can | | `harness` | string | required; agent scaffold, 1-100 chars (`claude-code`, `codex`, `openhands`, `swe-agent`, `vanilla`, ...) | | `taskCount` | integer | required; 1-100000 | | `nAttempts` | integer | required; 1-1000 | | `score` | number | required; 0-100 for percentage-like metrics; 0-10000 when `metric` is `elo` | | `metric` | string | required; 1-50 chars (`accuracy`, `pass@1`, `resolved_rate`, `elo`, ...) | | `tokensUsed` | integer \| null | optional; >= 0 | | `costUsd` | number \| null | optional; >= 0 | | `harborVersion` | string \| null | optional; <= 50 chars | | `cliVersion` | string \| null | optional; <= 50 chars | | `resultsHash` | string \| null | optional; sha256 hex (64 chars) of the Harbor result file | | `taskRewards` | `{ task: string, reward: number }[]` \| null | optional; <= 5000 entries; task name <= 200 chars; **no outputs** | | `notes` | string \| null | optional; <= 2000 chars | Example body (this is what `ltb bench push --dry-run` prints): ```json { "benchmarkSlug": "mmlu-pro", "model": "anthropic/claude-opus-4-8", "harness": "claude-code", "taskCount": 50, "nAttempts": 1, "score": 84, "metric": "accuracy", "tokensUsed": null, "costUsd": null, "harborVersion": "0.23.1", "cliVersion": "0.2.0", "resultsHash": "3b7e0f0c2a4d6e8f1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f7089", "taskRewards": [ { "task": "mmlu-pro-0001", "reward": 1 }, { "task": "mmlu-pro-0002", "reward": 0 } ], "notes": "temperature 0; 1 attempt; default timeouts" } ``` ### Response `201 Created`: ```json { "data": { "id": "run_01J...", "benchmarkSlug": "mmlu-pro", "model": "anthropic/claude-opus-4-8", "modelSlug": "claude-opus-4-8", "harness": "claude-code", "taskCount": 50, "nAttempts": 1, "score": 84, "metric": "accuracy", "tokensUsed": null, "costUsd": null, "harborVersion": "0.23.1", "cliVersion": "0.2.0", "resultsHash": "3b7e...7089", "taskRewards": [{ "task": "mmlu-pro-0001", "reward": 1 }], "notes": "temperature 0; 1 attempt; default timeouts", "userId": null, "status": "uploaded", "createdAt": "2026-10-01T12:00:00.000Z" }, "url": "https://lettokenburn.com/runs/detail/?id=run_01J..." } ``` `status` is `uploaded` | `validated` | `rejected`. Errors: ```json { "error": { "code": "validation_error", "message": "Invalid input", "details": [ ... ], "type": "validation_error" } } { "error": { "code": "rate_limited", "message": "Too many run submissions (30/24h per IP, 200/24h per user). Try again later.", "type": "rate_limit_error" } } ``` ### Read endpoints (no auth) ```http GET /v1/runs?benchmark=&model=&harness=&limit=20&page=1 -> { data: RunRecord[], pagination } GET /v1/runs/:id -> { data: RunRecord, url } GET /v1/bench/reproduced?benchmark=&model= -> { data: ReproducedAggregate[], cachedAt } ``` `ReproducedAggregate` = `{ benchmarkSlug, modelSlug, harness, runCount, median, mean, min, max, lastRunAt }`, cached 60 s. ## Rate limits - Anonymous: **30 pushes per 24 h per IP**. - With an API key: **200 pushes per 24 h per user** (the IP limit still applies). - Over the limit: HTTP 429, code `rate_limited`. Do not retry in a loop; prefer fewer, larger runs (`--n 200`, `--all`). ## Honesty rules (non-negotiable) 1. **Never fabricate, estimate or round a score.** Push exactly the `score` the CLI computed from the verifier rewards. Do not type a number into the JSON yourself unless you are forwarding an `ltb-run-summary.json` you did not alter. 2. **The model under test never grades itself.** Rewards come from `tests/test.sh`, not from the agent's chat output. If the agent claims "I got 47/50", ignore the claim; the summary file is the only source of truth. 3. **Oracle first.** If `-a oracle` scores below 100 on a pulled benchmark, do not run or push; report the broken tasks. 4. **One run, one push.** Do not push the same job twice, and do not split one job into several pushes. 5. **Declare settings in `notes`.** Subset size is already in `taskCount`; add temperature, changed timeouts, prompt changes, or anything else that would make the number non-comparable. 6. **Respect licenses.** If `pull` printed a license `NOTICE`, keep the task directories local; never commit, upload or redistribute them. Push only the score summary. 7. **Never upload model outputs, transcripts, dataset rows, API keys or `HF_TOKEN`.** `RunSubmission` has no field for them and you must not stuff them into `notes`. 8. **Cite sources when you report scores.** When telling the user about an existing number, say whether it is self-reported, independent or reproduced and give the source URL from the registry. 9. **Stop and ask** if the user wants a score for a benchmark with `harbor.status = none` or an `external_harness` adapter; you cannot produce a reproduced score for it with this skill alone. ## Reporting problems Wrong score, dead source link, oracle failure, mislabeled license: email `benchhub@lettokenburn.com` with the benchmark slug, model slug, current value, expected value and a source URL.