Zguoxu/tiny-transformer-from-scratch
Resource
Can I trust it?
No red flags in our checks.
Stars from real accounts No data
We haven't checked its stars for fakes yet.
Active OK
Last change 14 days ago.
Licence OK
MIT: you can use it for free, also in a commercial product. Keep the licence notice.
Quick checks from public data. They help, but they are no guarantee.
What it is
A small Transformer implemented from scratch in PyTorch without nn.Transformer or nn.MultiheadAttention, covering embeddings, positional encoding, multi-head attention, encoder, decoder, training and greedy generation. It includes an English-to-Chinese translation demo trained…
Who it's for
Learners with basic PyTorch knowledge who want to study the computations inside a Transformer for machine translation
How to use it
Install
python -m pip install -r requirements.txtFrom its README. Check it there before running it.
Ask your AI to set it up
Paste this into Claude, ChatGPT or your coding assistant.
Help me install and set up Zguoxu/tiny-transformer-from-scratch in my project. Follow its README: https://github.com/Zguoxu/tiny-transformer-from-scratch. Explain each step and check it is safe before running anything.
Star curve
Get alerts for repos like this
We'll email you when something in Models & inference starts rising. Free, once a week, only what matches.
What arrives: one email on Mondays with the repos that match, never more. See last week's. Privacy
For maintainers
Add this badge to your README. It shows the repo's place on the radar, updates every hour and links back to this page.
[](https://repoalerts.com/repo/Zguoxu/tiny-transformer-from-scratch?ref=badge)Similar repos
- aa1143/NamiGPT
A Chinese-language step-by-step tutorial that builds GPT-2 from scratch in PyTorch without using existing Transformer modules. It walks through tokenization, embeddings, attention, and blocks using a
- prabeakarve/microgpt
A dependency-free GPT implementation in pure NumPy, with a byte-level tokenizer, multi-head causal self-attention, hand-written backward pass, AdamW training with warmup and cosine schedule, and top-k
- Interactive-Active-AI/NextTransformer
A research framework and benchmark for comparing Transformer architectures under a fixed training budget. It trains recurrent-depth, sparse and linear attention, residual-stream variants and hybrids o
- YeonwooSung/llm-architecture-from-scratch
A collection of from-scratch PyTorch implementations of modern LLM architectures, including GPT-2 XL, Llama variants, DeepSeek V3, Qwen3, and Gemma 3. A comparison table lists each model's normalizati
- SheepSeb/tinyworldformer
A Rust project that runs two small neural networks on a Raspberry Pi Pico 2 W: a looped transformer world model that generates game frames and rewards, and an MLP actor that plays inside the generated
- datawhalechina/zero-to-sglang
A free, bilingual (English and Chinese) tutorial course on LLM inference. It covers inference concepts such as KV Cache, prefill and decode, then guides readers through building a mini-sglang step by