catalog / ports

nanochat (Python)

Karpathy's nanochat — tokenizer, pretraining, fine-tuning, RL and inference.

65 capabilities 2 sections extracted from nanochat (Python) source: D:\Dev\Lab\NanoChat.Upstream

Build against it

# give your agent the catalog — no account, no install
claude mcp add provespec -- npx -y provespec-mcp

# then, from any agent session
spec_get(slug: "nanochat-python")                → the capability tree
spec_checklist(slug: "nanochat-python")          → the questions to answer
spec_grade(slug: "nanochat-python", grades: [...])  → parity % + the gap list

# or just fetch it
curl https://provespec.com/catalog/nanochat-python/spec.json

Grade every capability below as yes · partial · no · na, with a file reference as evidence. Whatever you leave ungraded counts as missing — the gap list is the work queue.

Or just paste this to your agent

Use the ProveSpec MCP server. Fetch the "nanochat-python" spec with
spec_get, then walk spec_checklist and grade this repository against it —
one verdict per capability, each with a file reference as the note.
Call spec_grade with the results and give me the gap list as the plan.

If the server is not registered, add it first:
  claude mcp add provespec -- npx -y provespec-mcp

The specification

AI 57

Tokenizer
  • ✓ BPE tokenizer training rustbpe nanochat/tokenizer.py, scripts/tok_train.py
  • ✓ GPT-4-style split pattern regex in nanochat/tokenizer.py, \p{N}{1,2} variant
  • ✓ tiktoken interop merges -> tiktoken.Encoding for fast encode/decode
  • ✓ Chat special tokens 9 specials bos, user/assistant, python/output start-end
  • ✓ Conversation render + loss mask render_conversation: assistant supervised, tool output not
  • ✓ Tokenizer persistence pickle ~/.cache/nanochat/tokenizer round-trip
  • ✓ Compression metrics scripts/tok_eval.py vs GPT-2/GPT-4 tokenizers
Data pipeline
  • ✓ ClimbMix shard downloader nanochat/dataset.py: parallel, retries, atomic rename
  • ✓ In-app Parquet ingestion row-group streaming, rank-strided
  • ✓ BOS best-fit packing loader nanochat/dataloader.py, ~100% row utilization
  • ✓ Resumable loader state {pq_idx, rg_idx, epoch} resume dict
  • ✓ SFT conversation packing chat_sft.py best-fit packing, pad-never-crop
  • – Upstream RNG reproduction is the reference implementation
Model & kernels
  • ✓ GPT transformer (RoPE, QK-norm, relu2) nanochat/gpt.py, logit softcap 15, untied embeddings
  • ✓ Grouped-query attention separate n_kv_head
  • ✓ Sliding-window attention SSSL window_pattern tiled per layer
  • ✓ Value embeddings + gates ResFormer-style, sigmoid gate
  • ✓ Smear + backout prev-token gate; mid-layer residual backout
  • ✓ Optimized attention kernels FA3 Flash Attention 3 with SDPA fallback (flash_attention.py)
  • ✓ Backward pass autograd PyTorch autograd
Pretraining
  • ✓ Base training loop scripts/base_train.py: grad-accum, compile, MFU/ETA
  • ✓ Muon + AdamW optimizer Polar Express, NorMuon, MuonEq, cautious decay (optim.py)
  • ✓ LR/momentum/WD schedules warmup-constant-warmdown; momentum 0.85-0.97-0.90
  • ✓ Scaling-law autotuning horizon from param-data ratio; B ~ D^0.383
  • ✓ Gradient accumulation base_train.py
  • ✓ Resume mid-run model + per-rank optimizer shard + loader position
  • ✓ In-training evals & samples val bpb, CORE, greedy samples, wandb
SFT
  • ✓ SFT training flow scripts/chat_sft.py
  • ✓ Data mixture (SmolTalk+MMLU+GSM8K) 460K + aux x3 + GSM8K x4, deterministic shuffle
  • ✓ Progress-driven LR schedule keyed to dataset progress 0-1
  • ✓ Optimizer warm-start from base loads pretrain momentum buffers per rank
Reinforcement learning
  • ✓ GRPO rollouts & advantages chat_rl.py: mean-subtracted advantage, DAPO-style norm
  • ✓ RL training loop on-policy REINFORCE on GSM8K, periodic checkpoints
  • ✓ pass@k evaluation all-reduced pass@1..k during RL
Evaluation
  • ✓ CORE benchmark 22 tasks core_eval.py: MC, schema, LM scoring
  • ✓ Bits-per-byte metric loss_eval.py, vocab-size invariant
  • ✓ ChatCORE aggregate ARC-E/C, MMLU, GSM8K, HumanEval, baseline-centered
  • ✓ GSM8K evaluation tasks/gsm8k.py, tool-call parsing
  • ✓ HumanEval execution sandboxed subprocess: rlimit, scrubbed env (execution.py)
  • ✓ MMLU/ARC categorical eval letter-logit restriction (chat_eval.py)
  • ✓ Inference benchmark infer_bench.py: TTFT/TPOT/MBU/MFU sweep
  • ✗ Durable eval history & integrity CSV results only; no integrity gates or run history
Inference engine
  • ✓ KV-cache incremental decode engine.py KVCache, FA3 layout
  • ✓ Prefill-once, clone-for-samples batch-1 prefill copied into n-row decode cache
  • ✓ Streaming generation token-column streaming with sampled/forced masks
  • ✓ Temperature/top-k sampling seeded generator, 0=argmax
  • ✓ Calculator tool-use loop python_start/end state machine + safe eval
  • ✓ Multi-turn chat session chat_cli.py token accumulation
  • ✓ CPU fallback inference device autodetect cuda/mps/cpu
Infrastructure
  • ✓ Checkpoint save/load + resume checkpoint_manager.py, ZeRO-2 optimizer shards
  • ✓ Model lineage registry (base/sft/rl) load_model source registry
  • ✗ Model deploy/rollback registry
  • ✗ Artifact integrity (hashes)
  • – Cross-implementation checkpoint import is the source of checkpoints
  • ✓ Distributed multi-GPU training custom reduce_scatter/all_gather in optimizer, no DDP
  • ✓ Mixed precision (bf16/FP8) bf16 autodetect + ~150-line FP8 path (fp8.py)
  • ✗ Config provenance tracking flags logged but not chosen-vs-derived tracked

UI 8

  • ✓ Chat CLI scripts/chat_cli.py REPL + one-shot mode
  • ✗ Web chat UI ui/ folder empty in this revision
  • ✗ MCP server
  • ✗ Desktop observatory app
  • ✗ Architecture visualizers
  • ✗ Interactive tokenizer explorer visualize_tokenization ANSI dump only
  • ✗ Training workflow UI shell scripts (runs/speedrun.sh)
  • ✓ End-to-end speedrun script runs/speedrun.sh: 8xH100 ~1.5h pipeline
Download spec.json All 53 specs