Repo Year in Review Limited info

karpathy/autoresearch

Lets an AI agent autonomously edit a single-GPU nanochat training file, run 5-minute experiments, and keep improvements measured by val_bpb.

  • 97.6k GitHub stars
  • Python
View on GitHub (opens in a new tab) Setup guide
curl -LsSf https://astral.sh/uv/install.sh | sh
karpathy/autoresearch preview image

What it is

autoresearch is a small repo that gives an AI agent a simplified single-GPU implementation of nanochat and lets it experiment autonomously, typically overnight. The agent edits train.py, trains for a fixed 5 minutes, checks whether val_bpb improved, keeps or discards the change, and repeats. The human steers the process by editing program.md, a Markdown instruction file, rather than the Python files.

Who it's for

  • Researchers who want to experiment with autonomous AI-agent-driven LLM research loops
  • Developers with a single NVIDIA GPU who want to iterate on a 'research org' via program.md
  • People adapting the setup to smaller platforms through forks or by tuning the documented hyperparameters

Requirements

Requirements

  • A single NVIDIA GPU (tested on H100)
  • Python 3.10+
  • uv
  • An AI coding agent such as Claude or Codex to run the experiments

Setup

  1. Install uv

    Install the uv project manager if you don't already have it.

    bash
    curl -LsSf https://astral.sh/uv/install.sh | sh

  2. Install dependencies

    Sync the project dependencies with uv.

    bash
    uv sync

  3. Download data and train tokenizer

    One-time step, about 2 minutes.

    bash
    uv run prepare.py

  4. Run a single training experiment

    Manually run one experiment (about 5 minutes) to verify the setup works.

    bash
    uv run train.py

Examples

Kick off the agent

Prompt
prompt
Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.

Expected output: The README's suggested prompt to give your Claude/Codex agent inside the repo (with all permissions disabled), so it reads program.md and starts.

Verify the setup manually

bash
bash
uv sync
uv run prepare.py
uv run train.py

What it does: Installs dependencies, prepares data and the tokenizer, and runs one ~5-minute training experiment before going into autonomous mode.

Pros & cons

Pros

  • Pro:Fixed 5-minute wall-clock budget makes experiments directly comparable regardless of what the agent changes, giving roughly 12 experiments per hour
  • Pro:Small scope: the agent edits only train.py, which keeps diffs reviewable
  • Pro:Self-contained with no distributed training or complex configs: one GPU, one file, one metric
  • Pro:val_bpb is vocab-size-independent, so architectural changes are fairly compared

Cons

  • Con:Requires a single NVIDIA GPU; CPU, MPS and other platforms are not supported in this repo (only via forks)
  • Con:Results are not comparable to those from other people's compute platforms because of the fixed time budget
  • Con:The default program.md is only a bare-bones baseline, so getting better research progress requires iterating on it yourself

Images

karpathy/autoresearch preview image
karpathy/autoresearch preview image · karpathy/autoresearch on GitHub
karpathy/autoresearch preview image