Runtime 36
Tokenizer and data · Tokenizer encoding contracts
-
✓
Ordered ordinary batch encoding
Encode an ordered batch into independent token arrays without changing input order.
-
✓
Named prepend and append encoding
Resolve named special tokens and add them at text boundaries.
-
✓
Raw integer prepend and append IDs
Accept already-resolved token IDs at text boundaries.
-
✓
Named and raw boundaries on batch input
Apply named or integer boundary tokens to every item in a batch.
-
✓
Raw single-token bytes
Return the exact byte sequence represented by one token ID.
-
✓
Supplementary-plane Unicode encoding
Round-trip Unicode text whose code points require surrogate pairs in UTF-16 runtimes.
Tokenizer and data · Training data pipeline
-
✓
Tokenizer training and official data acquisition
Train the tokenizer from streamed ClimbMix text and acquire official parquet shards safely.
-
✓
Best-fit packing and resumable distributed loader
Create BOS-aligned packed rows with rank slicing and resumable parquet position state.
Pretraining · Model architecture and optimization
-
✓
Model architecture and CPU optimization
Build the NanoChat GPT and train it with AdamW and Muon-compatible optimization.
Pretraining · Training policy
-
✓
Compute-optimal depth scaling and schedules
Scale width, heads, tokens, batches, learning rates, decay, and accumulation from depth.
Pretraining · Accelerated execution
-
✓
CUDA, dtype, compilation, and FP8
Run base training with CUDA-aware dtype selection, compilation, scaling, and optional FP8.
Supervised fine-tuning · Conversation training
-
✓
Conversation render, shifted loss mask, and masked scoring
Render conversations and optimize only the intended assistant tokens with shifted masks.
-
✓
Best-fit multi-conversation row packing
Pack multiple rendered conversations into fixed SFT rows while preserving BOS alignment and masks.
-
✓
Upstream SmolTalk and task mixture ordering
Train from the ordered SmolTalk, ARC, GSM8K, and identity mixture.
Supervised fine-tuning · Task dataset semantics
-
✓
Finite task start, stop, and step slicing
Expose bounded logical views over task datasets with validated slice semantics.
-
✓
Ordered task dataset sequences
Concatenate tasks into an ordered curriculum without losing per-task indexing.
-
✓
Immutable HubDataset row views
Return independent Python row objects from an immutable Arrow-backed logical view.
Reinforcement learning · Rollout policy optimization
-
✓
GSM8K rollout policy optimization
Optimize sampled GSM8K completions using exact-answer rewards and policy gradients.
Checkpoints and resume · Checkpoint lifecycle
-
✓
Atomic save, load, resume, and latest-native recovery
Persist model, optimizer, metadata, loader, and loop state and recover a selected or latest checkpoint.
-
✓
Task-specialization warm start
Initialize SFT from a base model and optionally restore optimizer momentum while retaining SFT rates.
-
✓
Real upstream checkpoint and tokenizer interchange
Load and save the native model, optimizer, metadata, and tokenizer files used by upstream scripts.
Inference, chat, and tool use · Generation and chat
-
✓
KV-cached generation and sampling
Prefill once, clone cached attention state, and sample reproducible continuations with limits.
-
✓
Token-native chat
Represent and extend the conversation as native tokens with explicit terminal boundaries.
-
✓
Headless chat CLI
Run interactive or prompt-mode terminal chat over a loaded checkpoint.
Inference, chat, and tool use · Tool execution
-
✓
Safe calculator evaluator
Evaluate an allowlisted subset of arithmetic and string-count expressions with a timeout.
-
✓
Arbitrary Python execution
Run generated Python in a constrained subprocess with resource and environment controls.
Evaluation and benchmarks · Quality evaluation
-
✓
BPB, CORE, GSM8K, ARC, and MMLU evaluations
Compute validation bits-per-byte and the core base-model task suite.
-
✓
ChatCORE orchestration and aggregation
Run and aggregate generative GSM8K and HumanEval with categorical ARC and MMLU chat evaluations.
-
✓
Multiple-choice prompt formatting
Render categorical task choices into the canonical conversation prompt and answer contract.
-
✓
HumanEval
Render HumanEval prompts and score generated Python completions through the constrained executor.
Evaluation and benchmarks · Inference performance benchmark
-
✓
Inference benchmark matrix
Measure prefill, decode, batch, prompt, weight, and VRAM characteristics.
Distributed runtime and scripts · Distributed training and data
-
✓
DDP training and coordinated overflow and checkpoint state
Coordinate accumulation, overflow decisions, metrics, and checkpoint participation across ranks.
-
✓
Parquet row-group partition and resume planning
Assign row groups by rank and resume from a recorded parquet file, row group, and epoch.
Distributed runtime and scripts · Attention runtime
-
✓
Flash Attention 3 and SDPA fallback
Use FA3 where available and a behaviorally tested SDPA fallback otherwise.
Distributed runtime and scripts · Pipeline automation and baseline
-
✓
Official pipeline and run scripts
Orchestrate tokenizer, pretraining, SFT, RL, evaluation, and chat workflows.
-
✓
Canonical headless test baseline
Run the upstream tokenizer, inference, task, optimizer, execution, and attention fallback test modules.