Train an LLM on an iGPU: nanochat (Karpathy) on a plain integrated GPU
nanochat is Andrej Karpathy’s project that shows, in a single readable repository, every step of a “ChatGPT”: tokenizer, pretraining, fine-tuning (SFT) and a chat interface. It is designed for 8 NVIDIA H100 GPUs, but it also offers a reduced configuration for CPU.
We ran it entirely on an iGPU: the Intel UHD 770, the GPU built into the processor (a desktop Core i5), with no dedicated graphics card and no CUDA. All of it with PyTorch XPU, the native support for Intel GPUs in PyTorch, then with the iGPU and the CPU computing together.
Results in short
- A plain iGPU is enough to train an LLM from scratch: installation, data, tokenizer, pretraining (33 min), SFT (about 6 min) and first answers in under an hour, on the VM’s integrated GPU, with no graphics card.
- 1.33 s per pretraining step, or about 6,150 tokens/s, for a small 4-layer model. On the same work, the GPU is about 1.3 times faster than the 6 vCPUs.
- iGPU and CPU together: by giving 4 rows out of 16 in each batch to the CPU, pretraining goes from 1.33 to 1.13 s per step (15% less time), for almost the same model quality (validation 1.55 vs 1.52). Giving the CPU more work actually slows things down.
- The resulting model plays the assistant and speaks English, but stays incoherent: that is down to the model size (37 million parameters, mostly embedding tables, 39 min of training), not the platform.
- It took 4 adaptations for the Intel GPU and 4 fixes to nanochat for a small configuration. Everything is in a one-command script, at the end of the article.
Why an iGPU?
An iGPU (integrated GPU) is the graphics part that sits inside the processor, on the same chip as the CPU cores. It is usually reserved for the display; here, it is used for compute.
- It shares the server’s RAM instead of having its own video memory. No 8 or 12 GB VRAM limit, and no costly copies between CPU and GPU: this is what makes the iGPU + CPU training below efficient. Our model only used 1.5 GB.
- PyTorch supports it natively (XPU build),
torch.compileincluded: the same code as with CUDA, give or take a few lines. - It costs a fraction of a dedicated GPU: our iGPU Pro plan is €44.99 per month excl. VAT, while a dedicated GPU in the cloud often rents for €300 to €500 per month.
- It is much less powerful (under 1 TFLOPS): it does not replace a dedicated GPU for training a large model. It is enough for learning, experimenting and running small models.
Test setup
| Item | Value |
|---|---|
| Virtual machine | iGPU Pro plan: 6 vCPU, 24 GB RAM, Debian 13, kernel 7.3 |
| CPU / GPU | Intel Core i5-13500, integrated Intel UHD 770 GPU (shared with other VMs) |
| Software | nanochat (commit 92d63d4), PyTorch 2.9.1 XPU build, torch.compile with Triton XPU |
| Model | d4: 4 layers, width 256, 512-token context, 32,768-token vocabulary |
| Pretraining | 1,500 steps of 8,192 tokens (12.3 million tokens) |
| SFT | 220 steps of 8,192 tokens, a mix of conversations, MMLU and GSM8K |
This is nanochat’s “CPU” configuration, scaled down to fit in under an hour. The real
nanochat (speedrun.sh, a 20-layer model) takes about 4 h on 8 H100s: that is three orders
of magnitude more compute, out of reach for an integrated GPU.
1. The numbers
Raw compute
| 2048×2048 matrix multiplication | TFLOPS |
|---|---|
| fp32 | 0.71 |
| bf16 | 0.73 |
The GPU is shared: when another VM is using it, we measured as low as 0.27 TFLOPS. This GPU has no dedicated matrix units, so bf16 is no faster than fp32, but it uses half the memory.
Training and chat
| Stage | Duration | Detail |
|---|---|---|
| Installation (packages, nanochat, PyTorch XPU) | ~1 min | |
| Data (8 files) + tokenizer | 50 s | on the CPU |
| 1st pretraining iteration | 83 s | GPU kernel compilation, once only |
| Pretraining, 1,500 steps | 33 min | 1.33 s/step, ~6,150 tokens/s, 1.5 GB of GPU memory |
| SFT, 220 steps | ~6 min | ~1.5 s/step |
| 1st chat answer | 3 min | compilation |
| Following answers | 13 to 15 s | long answers, up to the token limit |
GPU vs CPU, same work
| SFT, steps of 16,384 tokens | s/step | tokens/s |
|---|---|---|
| Intel UHD 770 GPU | 2.64 | 6,200 |
| 6 vCPU (i5-13500) | 3.0 to 3.5 | 4,630 |
The learning curve
| Start | End | |
|---|---|---|
| Pretraining, validation bits per byte | 3.20 | 1.52 |
| SFT, validation bits per byte | 1.35 | 1.10 |
Lower is better: it means the model predicts text more accurately. Both curves go down steadily, with no incidents.
2. iGPU and CPU together
The GPU computes while the 6 vCPUs wait. So we made both work at the same time (patch nanochat_hybrid_patch.py):
- Each batch of 16 rows is split in two: one part goes to the GPU, the other to a copy of the model in CPU memory. Both compute in parallel.
- The gradients of both parts are added together, weighted by the number of rows. This is mathematically the same computation as a full batch on a single processor.
- The optimizer updates the model on the GPU, then the weights are copied back to the CPU copy. On an integrated GPU, the GPU and CPU share the same memory: this copy costs about 0.1 s per step.
Everything runs in fp32: this server’s CPU has no fast bf16 instructions, and on this GPU fp32 is as fast as bf16.
How much work to give the CPU?
| GPU + CPU rows (CPU threads) | s/step | Compared with GPU only in fp32 (1.40 s) |
|---|---|---|
| 16 + 0 (GPU only) | 1.40 | baseline |
| 12 + 4 (3 or 4 threads) | 1.16 | −17% |
| 10 + 6 (5 threads) | 1.32 | −6% |
| 10 + 6 (4 threads) | 1.41 | no gain |
| 8 + 8 (4 threads) | 1.43 | slower |
Counter-intuitive: giving more rows to the CPU slows everything down. Our measurements show that the CPU always finishes before the GPU, but that the GPU part gets slower while the CPU is computing: the cores doing the computation are no longer available to drive the GPU (launching its kernels, handling automatic differentiation). The best setting therefore leaves some headroom: 4 rows out of 16 and 3 compute threads on 6 vCPUs.
The full run, GPU only vs GPU + CPU
| Pretraining, 1,500 steps | GPU only (bf16) | GPU + CPU (fp32) |
|---|---|---|
| Time per step | 1.33 s | 1.13 s |
| Tokens per second | ~6,150 | ~7,200 |
| Total duration | 33 min | 28 min |
| Final validation (bits per byte) | 1.52 | 1.55 |
| SFT then chat | identical | identical |
The final validation is slightly worse in the hybrid run (1.55 vs 1.52). We have not isolated the cause yet: the switch to fp32, or an effect of splitting the work between GPU and CPU.
Two limits to keep in mind: the gain depends on the number of vCPUs (with 2 vCPUs, there is nothing to share), and we only applied this method to pretraining, the longest stage. SFT runs on the GPU alone.
3. What the model answers
What is the capital of France? The city of France has a fundamental concept in Japan, which is a city of France’s Italian history and country. […] a fundamental concept in Paris and the way to showcase the world’s most fundamental and modern aspects of the history…
Hi! Who are you? I’m glad you’ve got a few minutes of time. We’re passionate about the concept of “A”…
The model has learned the shape of an assistant answer and some word associations (France, Paris), but not how to reason: with 4 layers and 39 minutes of compute, that is the expected result. The base model, before SFT, just continued the text without answering.
4. What we had to change
For the Intel GPU (PyTorch XPU)
- Recognizing the GPU: nanochat only knows
cuda,mpsandcpu. We addxpu(detection, synchronization, memory). - No double-precision (fp64) compute on this GPU: one line of the rotary embeddings
computation went through fp64 and crashed (
UR error: 45). We rewrite it in fp32. - Level Zero headers:
torch.compilecompiles its GPU kernels with Triton XPU, which needs thelibze-devpackage. - 512 work-item limit per group: sorting the 32,768 vocabulary tokens (
topk) exceeds this limit. We do it on the CPU; the cost is negligible.
For nanochat in a small configuration
These four problems also show up on CPU: they come from the reduced configuration (512-token context), not from the GPU.
- The SFT loader gets stuck: it only takes conversations that fit entirely in 513 tokens.
Longer ones stay in its queue until it fills up, after which it only produces padding: the
model stops learning, drifts and ends up at
nan. We truncate conversations to 513 tokens. - A
nanloss when a batch contains no answer tokens (0/0 division). We divide by at least 1. - A learning rate too high for such a small model: we use 20% of the default value for SFT, and we stop at 220 steps (beyond that, the model collapsed suddenly, always at the same step).
- The chat loaded an old model:
chat_clipicks the checkpoint with the highest step number, not the most recent one. We pass it-s 219.
5. Run nanochat on your iGPU, step by step
Everything is done over SSH on the VM (Debian 13, recent kernel). Allow one hour, 40 minutes of which is compute.
All the code (script, Docker, patches) is on GitHub: LinuxGPU-VPS/examples/nanochat-xpu.
Quick option: a single command
curl -fsSLO https://raw.githubusercontent.com/LinuxGPU-VPS/examples/main/nanochat-xpu/run.sh
sudo bash run.sh # iGPU only
sudo HYBRID_CPU_ROWS=4 bash run.sh # iGPU + CPUThe script runs exactly the steps below, times each one in
~/nanochat-logs/timeline.txt and stops at the first error.
Docker option
The same pipeline in a container, with nothing to install on the machine except Docker:
git clone https://github.com/LinuxGPU-VPS/examples
cd examples/nanochat-xpu
docker compose up --build # iGPU only
HYBRID_CPU_ROWS=4 docker compose up --build # iGPU + CPU
docker compose run --rm nanochat /opt/linuxgpu/train.sh chat # then chat with the modelThe image (Ubuntu 24.04) includes the Intel GPU driver (Level Zero), PyTorch XPU, nanochat and the patches. The container reaches the iGPU through /dev/dri. Data and models go to ./data, and the time of each step to ./logs/timeline.txt.
1. Check that the iGPU is visible
ls /dev/dri # card0 and renderD128: renderD128 is the iGPUOn our VMs, the gpu-check command also shows “Intel(R) UHD Graphics 770” and tests OpenCL,
Vulkan and video encoding.
2. Install the system packages
sudo apt-get install -y git curl build-essential libze-devlibze-dev contains the Level Zero headers, the low-level interface of Intel GPUs. Without
them, torch.compile fails when it compiles its kernels for the iGPU (error
level_zero/ze_api.h: No such file or directory).
3. Install nanochat and PyTorch for Intel GPUs (XPU)
git clone https://github.com/karpathy/nanochat && cd nanochat
git checkout 92d63d4 # the tested version
curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv && uv sync --extra cpu # nanochat dependencies (with CPU PyTorch)
VIRTUAL_ENV=$PWD/.venv uv pip install --reinstall torch==2.9.1 \
--index-url https://download.pytorch.org/whl/xpu # replaces CPU PyTorch with the XPU build
.venv/bin/python -c "import torch; print(torch.xpu.is_available(), torch.xpu.get_device_name(0))"
# True Intel(R) UHD Graphics 770xpu is the name PyTorch gives to Intel GPUs, like cuda for NVIDIA. The XPU build brings
everything you need (SYCL runtime, Triton XPU for torch.compile): no proprietary driver to
install. Do not run uv sync again afterwards, it would put the CPU version back.
4. Apply the patches for the iGPU
curl -fsSLO https://raw.githubusercontent.com/LinuxGPU-VPS/examples/main/nanochat-xpu/nanochat_xpu_patch.py
python3 nanochat_xpu_patch.py .The patch changes a few lines of nanochat. The three changes the iGPU cannot do without:
# nanochat/common.py: pick the iGPU when it is present
elif torch.xpu.is_available():
device_type = "xpu"
# nanochat/gpt.py: the iGPU has no fp64, so we compute the frequencies in fp32
inv_freq = torch.exp(channel_range * (-math.log(base) / head_dim))
# nanochat/engine.py: sorting the 32,768 tokens exceeds the iGPU limit (512 work-items)
vals, idx = torch.topk(logits.float().cpu(), k, dim=-1)It also adds the nanochat fixes described above (SFT loader, nan loss, non-finite
gradients).
5. Set up the environment
export NANOCHAT_BASE_DIR=$HOME/.cache/nanochat # data, tokenizer and models
export NANOCHAT_DTYPE=bfloat16 # otherwise nanochat computes in fp32 outside CUDA
export OverrideDefaultFP64Settings=1 # fp64 emulation by the Intel driver,
export IGC_EnableDPEmulation=1 # in case an operation needs it
P=.venv/bin/python6. Download the data and train the tokenizer (CPU, ~1 min)
$P -m nanochat.dataset -n 8
$P -m scripts.tok_train --max-chars=20000000007. Pretrain the model on the iGPU (~33 min)
$P -m scripts.base_train --device-type=xpu --depth=4 --head-dim=64 --window-pattern=L \
--max-seq-len=512 --device-batch-size=16 --total-batch-size=8192 \
--eval-every=500 --eval-tokens=131072 --core-metric-every=-1 --sample-every=500 \
--num-iterations=1500 --run=dummy--device-type=xpu sends the compute to the iGPU. The first step takes about 80 s: PyTorch
compiles its kernels for the iGPU, only once. During training, nvtop (installed on our VMs)
shows the iGPU load.
To make the iGPU and the CPU work together (15% less time), also apply the second patch and run the same command with three variables:
curl -fsSLO https://raw.githubusercontent.com/LinuxGPU-VPS/examples/main/nanochat-xpu/nanochat_hybrid_patch.py
python3 nanochat_hybrid_patch.py .
NANOCHAT_DTYPE=float32 NANOCHAT_CPU_ROWS=4 NANOCHAT_CPU_THREADS=3 \
$P -m scripts.base_train --device-type=xpu ... # same options as above8. Fine-tuning (SFT), then chat (~6 min)
NANOCHAT_CLIP=1.0 $P -m scripts.chat_sft --device-type=xpu --eval-every=50 \
--eval-tokens=65536 --chatcore-every=-1 --num-iterations=220 --init-lr-frac=0.2 --run=dummy
$P -m scripts.chat_cli --device-type=xpu -s 219 -p "What is the capital of France?"
$P -m scripts.chat_cli --device-type=xpu -s 219 # conversation mode-s 219 picks the model from the SFT (the last step). Without this option, chat_cli takes the
model with the highest step number, which may come from an earlier run.
Takeaways
- An iGPU really can train a small language model, with no dedicated graphics card: native
PyTorch,
torch.compile, no CUDA, and the CPU can help it out (15% less time). - It is the right playground for learning how an LLM works end to end, changing the code and seeing the effect within minutes, not for training a useful production model.
- To serve an already trained model, see instead our guides on Hugging Face models on an iGPU and inference with OpenVINO.
Try it
All our VMs have an Intel UHD 770 iGPU usable by PyTorch XPU, OpenCL and Vulkan out of the box, from €14.99 per month excl. VAT. This test ran on the iGPU Pro plan (6 vCPU, 24 GB RAM, 240 GB SSD, the largest share of the iGPU), at €44.99 per month excl. VAT: the right choice for training, thanks to the vCPUs that help the iGPU. See the iGPU plans.