What is build-nanogpt?

karpathy/build-nanogpt — explained in plain English

Analysis updated 2026-07-03 · repo last pushed 2024-08-13

★ 5,305PythonAudience · developerComplexity · 3/5StaleSetup · moderate

Mindmap

mindmap
  root((build-nanogpt))
    What it does
      Build GPT-2 from scratch
      Train on real text data
      Generate coherent text
    Learning Format
      YouTube video lecture
      Incremental git commits
      Hands-on coding
    Output
      124M parameter model
      Text generation
      Cloud GPU in 1 hour
    Audience
      AI learners
      Curious developers
      Students
    Tech Stack
      Python
      PyTorch
      Cloud GPU

mindmap root((build-nanogpt)) What it does Build GPT-2 from scratch Train on real text data Generate coherent text Learning Format YouTube video lecture Incremental git commits Hands-on coding Output 124M parameter model Text generation Cloud GPU in 1 hour Audience AI learners Curious developers Students Tech Stack Python PyTorch Cloud GPU

Click or tap to explore — scroll the page freely

What do people build with it?

USE CASE 1

Work through the tutorial step by step to build your own GPT-2 model and understand how language models actually work under the hood.

USE CASE 2

Train the 124M parameter model on a rented cloud GPU in about an hour for roughly $10 to see a working AI text generator you built yourself.

USE CASE 3

Use the incremental git commit structure to pause the video at any step, inspect the code changes, and resume with full context.

USE CASE 4

Experiment with the trained model by giving it prompts and observing how text generation quality improves as training progresses.

What is it built with?

PythonPyTorch

How does it compare?

	karpathy/build-nanogpt	apple/coremltools	facebookresearch/sapiens
Stars	5,305	5,271	5,393
Language	Python	Python	Python
Last pushed	2024-08-13	—	2026-05-26
Maintenance	Stale	—	Maintained
Setup difficulty	moderate	moderate	hard
Complexity	3/5	3/5	4/5
Audience	developer	developer	researcher

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · moderate Time to first run · 1h+

Full training requires a rented cloud GPU (~$10 for 1 hour), local CPU runs are possible but very slow.

No license information is mentioned in the explanation.

Copy-paste prompts

Prompt 1

I'm following the build-nanogpt tutorial and I'm at the attention mechanism step. Explain how the scaled dot-product attention works in the context of this GPT-2 implementation and show me the forward pass code.

Prompt 2

Help me rent and configure a cloud GPU (like a Lambda Labs A10 or RunPod) to train the 124M parameter nanogpt model in about an hour for under $10.

Prompt 3

I've trained my nanogpt model and want to generate text from it. Write Python code that loads the checkpoint and generates 200 tokens starting from a custom prompt.

Prompt 4

Explain the difference between the text generation in this nanogpt repo and a chatbot like ChatGPT, why doesn't this model follow instructions?

Prompt 5

I want to modify the build-nanogpt training loop to log loss to Weights & Biases so I can track training progress in a dashboard. Show me how to add the WandB integration.

Frequently asked questions