LetTokenBurn
Sign in
Verified86%Harbor: pendingCoding

APPS

Automated Programming Progress Standard

Can Transformers crack the coding interview? We collected 10,000 programming problems to find out. GPT-3 does not perform well, but new models like GPT-Neo are starting to be able to solve introductory coding challenges.

PaperRepoDatasetLeaderboardHomepage

How to run

ltb · apps
$ npm i -g lettokenburn-cli
$ ltb bench pull apps

External harness. pip install inspect-evals && inspect eval inspect_evals/apps --model <provider/model> https://github.com/UKGovernmentBEIS/inspect_evals

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

4 sourced entries
HarnessSourceConfidenceLink
DeepSeek-Coder-Instruct 6.7B33.8N/AIndependentmediumJan 25, 2024arxiv.org
code-davinci-002 175B31.9N/AIndependentmediumJul 21, 2022arxiv.org
WizardCoder 15B26.3N/AIndependentmediumOct 13, 2023arxiv.org
GPT-Neo 2.7B3.9N/AIndependentmediumMay 20, 2021arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

APPS

Dataset Description

APPS is a benchmark for code generation with 10000 problems. It can be used to evaluate the ability of language models to generate code from natural language specifications.

Dataset Structure

We also offer an apps_mini subset, which includes 1500 questions divided proportionally of introductory, interview, and competition categories, with a ratio of 1:1:1(500 questions each).

How to Use

You can also filter the dataset based on difficulty level: introductory, interview and competition. Just pass a list of difficulty levels to the filter. For example, if you want the most challenging questions, you need to select the competition level:

Evaluation results

Please refer to Table 3 of code llama for original results if needed.

Citation