Run small AI models on an integrated GPU

Small local LLMs (Gemma, Phi, Qwen)

Run small open models — Gemma (the open, Gemini-style family), Llama, Phi, Qwen — right on the iGPU: chatbot, summarization, RAG, extraction. Pull any GGUF from Hugging Face with Ollama or llama.cpp (Vulkan backend). Perfect for simple, private inference.

# Ollama — pull & run a small model in one line
ollama run gemma3:4b

# …or llama.cpp with a Hugging Face GGUF, on the iGPU (Vulkan)
llama-cli -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_M \
  -ngl 99 -p "Summarize this ticket in 3 bullets"

OpenVINO on Intel iGPU

OpenVINO natively targets the Intel iGPU: detection, classification, OCR and segmentation run with good latency without a dedicated GPU. ONNX Runtime works too.

Example

Target the integrated GPU for inference:

from openvino.runtime import Core
core = Core()
print(core.available_devices)  # ['CPU', 'GPU']
model = core.compile_model('model.xml', 'GPU')

Production

Expose an inference API, containerize it and scale out with k3s / Kubernetes. Root SSH, IPv6 included, LUKS.