Linux GPU

Blog

Run Hugging Face Models on an Integrated GPU (iGPU) — Ollama & llama.cpp

Pull open models from Hugging Face and run them on an integrated GPU (iGPU) via Ollama and llama.cpp (Vulkan): small LLMs like Gemma, Phi, Qwen. Private, cheap inference from €14.99/month.

· Linux GPU

Run Hugging Face Models on an Integrated GPU (iGPU) — Ollama & llama.cpp

Hugging Face is the largest hub of open AI models — and you don’t need a €500/month graphics card to use them. Small, quantized models run comfortably on an integrated GPU (iGPU / APU) exposed through Vulkan — from just €14.99/month.

What runs well on an iGPU

An integrated GPU isn’t made to train 70B models. But for inference with small models, it’s plenty:

  • Small LLMs — Gemma (Google’s open, Gemini-style family), Llama 3.2, Phi-3, Qwen2.5 — in quantized GGUF (Q4/Q5).
  • Chatbots, summarization, RAG, extraction, classification on your own private server.
  • Vision & embedding models via OpenVINO / ONNX Runtime.

Rule of thumb: pick models up to ~7B parameters, quantized. They fit in the iGPU’s memory and generate a few tokens per second — enough for assistants, batch jobs and back-office automation.

Option 1 — Ollama (the easy way)

Ollama pulls models (many mirrored from Hugging Face) and runs them in one line:

curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma3:4b

You can also run any GGUF straight from a Hugging Face repo:

ollama run hf.co/unsloth/gemma-3-4b-it-GGUF:Q4_K_M

Option 2 — llama.cpp with the Vulkan backend

llama.cpp downloads GGUF models directly from Hugging Face with -hf and uses the iGPU through Vulkan:

llama-cli -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_M \
  -ngl 99 -p "Explain quantization in 3 bullet points"

-ngl 99 offloads the model’s layers to the integrated GPU.

Vision & embeddings — OpenVINO

For Hugging Face vision, OCR or embedding models, convert them with Optimum Intel and target the iGPU with OpenVINO for low-latency inference. Same server, same price.

Why run them yourself

  • Cost — no per-token API bill, no €500/month GPU. One fixed price.
  • Privacy — your prompts and data never leave your server (dedicated IPv6, root SSH, LUKS).
  • Control — pin a model version, run offline, scale with k3s / Kubernetes.

FAQ

Can I run Gemini or GPT locally? No — those are proprietary APIs. But their open cousins (Gemma, Llama, Phi, Qwen) run great on an iGPU.

How fast is it? A 3–4B quantized model generates a handful of tokens per second on the UHD 770 — fine for assistants and batch work, not high-throughput serving.

Which models should I start with? gemma3:4b, qwen2.5:3b or phi3:mini — all available as GGUF on Hugging Face.

Conclusion

Hugging Face + an integrated GPU is the cheapest way to run small open models privately. Pull a GGUF, run it with Ollama or llama.cpp on Vulkan, and ship — from €14.99/month.

Related reading