Local AI Models on a Jetson Nano
# why run models locally
Local models mean no API costs, no rate limits, and your data never leaves your machine. Quantized small models (1–3B parameters) now run well on modest ARM boards — a Jetson Nano, a Raspberry Pi 5, or any old laptop is enough to start.
# jetson nano — first boot
The Jetson Nano is a small ARM board with a 128-core Maxwell GPU and 4 GB of RAM. Setup: flash the JetPack SD-card image (from NVIDIA developer downloads) with balenaEtcher, insert, and boot.
sudo apt update && sudo apt upgrade bring the fresh image up to date uname -a confirm you are on the aarch64 (ARM64) kernel free -h check available memory — you have 4 GB shared between CPU and GPU sudo pip3 install jetson-stats install jtop, the Jetson equivalent of top for CPU/GPU/temps jtop live dashboard: cores, GPU load, temperatures, power mode # tuning the board
sudo nvpmodel -m 0 switch to MAXN power mode — all cores, full clocks (use a 5V/4A barrel-jack supply) sudo jetson_clocks pin clocks to maximum for consistent inference speed sudo fallocate -l 4G /swapfile models need headroom — create a 4 GB swap file sudo chmod 600 /swapfile && sudo mkswap /swapfile set permissions and format the swap file sudo swapon /swapfile enable it (add to /etc/fstab to persist across reboots) swapon -s verify swap is active tegrastats NVIDIA's built-in utilization monitor — watch RAM, GPU and temps while a model runs # running your first model
curl -fsSL https://ollama.com/install.sh | sh install Ollama — it ships ARM64 builds that work on Jetson ollama run tinyllama pull and chat with TinyLlama (1.1B) — comfortable in 4 GB ollama run gemma3:1b small models in the 1–2B range are the sweet spot for the Nano ollama ps show loaded models and how much memory they use ollama list list downloaded models # llama.cpp — the lightweight alternative
git clone https://github.com/ggml-org/llama.cpp plain C/C++ inference, no runtime dependencies cmake -B build && cmake --build build -j4 build on-device (add CUDA flags to use the Nano GPU) ./build/bin/llama-cli -m model.Q4_K_M.gguf -p "hello" run a 4-bit quantized GGUF model from the command line Rule of thumb for 4 GB: use Q4 quantized models under ~2B parameters. If a model swaps heavily, drop one size down — a fast small model beats a crawling big one.