Jump to the TL;DR if you just want the summary of setting it all up minus the story.

Why host a model?

The world of AI coding is moving at lightspeed in 2026. Nearly every week I learn about some cool new tooling that people are using to build software with AI assistance. It's an exciting time to be a software engineer, but an uncertain one as well. To satisfy my own curiosity (and to ensure my skills and knowledge stay relevant), I have been investing a lot of my personal time in experimenting with AI development. Since this is outside of my day-job, I use my own personal licenses. But the deeper I go, the more I run into usage limits.

I like using the latest frontier class models for the hard stuff. I do not love spending their rate limits on "rename this variable" and "write the obvious boilerplate." So I decided to host a large language model (LLM) on my own hardware, free and off of rate limits. As an aside, I also like the idea of my data being kept local, and not shipped off to some major AI corporation on every prompt. But the main motivation is to soak up the easy work with the local model and leave the frontier credits for the work that actually needs a frontier model.

And that's what this post is about. This is the first part of a series where I will explore what it takes to host your own LLM, make it securely available from anywhere, evaluate its usefulness, and the cool things you can do with your own self-hosted AI model. Today's post is about step #1: setting up hardware to host an LLM.

Sourcing hardware during the RAMpocalypse

2026 is a brutal market for PC components. There's a massive AI-driven shortage of DRAM and GDDR which has made prices skyrocket. As of this post, a 32 GB DDR4 kit that used to cost about $50-$70 is sitting at roughly $200. Market analysts say relief isn't expected until at least 2027. Rather than get fleeced in a seller's market, I decided to just use hardware that I had lying around.

Enter my gaming PC. I built it back in 2020 (during the pandemic restrictions) when I had all the free time in the world for things like gaming. I haven't used it very much since becoming a dad in 2022, so it's mostly just been collecting dust in my basement.

The Specs

My gaming PC originally had a GTX 1080 carried over from an even earlier build. Around 3 years ago I swapped it for a Radeon RX 6750 XT 12GB that I impulse bought on a good sale. The RX 6750 XT is a fine mid-level card for 1080p and some 1440p gaming today. However, it is generally not a card that you'll see recommended for running LLMs locally.

That said, it's not a bad card for it either (with some caveats I will get to later). For inference, the most important factor is VRAM, because that's what determines what size of model you can hold. The 12 GB in my RX 6750 XT is enough to hold a real coding model, with extra room for context. I didn't set out to choose AMD over NVIDIA here. It's just what I had.

The rest of the box is nothing special. The CPU is a genuinely dated Ryzen 5 2600, which is fine because it doesn't need to do much during GPU inference. For RAM it has 16 GB of DDR4, which is a bit tight since I also want to run orchestration processes on this machine. But given the sheer redonkulousness of RAM prices in 2026, it's fine. If I hit a wall because of it, I'll source a used 32 GB kit to swap it with.

I did a fresh install of Ubuntu 26.04 since it's free, lean, fits the tooling, and I plan on mostly using it headless via SSH anyway.

Full spec table for the curious:

The unsupported card problem

Here is where the caveat of my GPU comes into play. AMD's compute stack is called ROCm. This is their answer to NVIDIA's CUDA. There is a hardcoded list of supported GPU targets for ROCm, and your card has to be on it. My RX 6750 XT reports its chip target as gfx1031, which is not supported. The very similar chip in the 6800 and 6900 series cards is gfx1030, and that one is supported. gfx1031, the 6700 series, is not. Same architecture, one digit off, left off the list.

Note: ROCm's official supported-GPU list changes between releases, and gfx1031 support has been a moving target. Best to check for yourself since things may have changed since the time of this posting.

The usual workaround

There is a very simple workaround that a lot of people have reported success with on Linux. It basically tricks ROCm into thinking your card is supported. You install the GPU drivers and ROCm runtime as usual, add your user account to the video and render groups, and then spoof the GFX version to 10.3.0 like so:

export HSA_OVERRIDE_GFX_VERSION=10.3.0

Vulkan: the alternative

Vulkan is a graphics and compute API that these cards support natively with no ROCm needed. Ollama, which is what I use to serve models, runs inference on the Vulkan backend. The main downside is it's inference-only, meaning you can't fine-tune or train models. Since I'm only trying to serve an existing model for now, it's a non-issue for me.

Comparison: ROCm workaround vs. Vulkan (RX 6750 XT / gfx1031)

Note: It's not that Vulkan is inherently incapable of training/tuning models. It's just that the typical training stack targets CUDA/ROCm.

Verdict

For my use case (mostly-headless inference box), Vulkan wins. And I'm giving up little for it: the old assumption that ROCm is meaningfully faster has largely eroded, and on RDNA2 in particular (the generation the Vulkan backend was first tuned on) the two are close enough that speed isn't the deciding factor.

The goal of this project is to make AI-assisted workflows easier for me. I want to spend my time building cool things, not debugging why the latest update broke my hacky workaround. Vulkan just works, and keeps working across driver updates, which is what I want for an always-on server.

Setup

I started with a clean slate. A fresh install of Ubuntu 26.04 LTS Desktop. I went with desktop because I do have a monitor attached (set up at a desk in my basement), and will occasionally work from it directly using IDEs and such. But most of the time it will be connected to and used headless via my MacBook Air.

Step 1: The Vulkan graphics stack

Update the system and install the Vulkan userspace drivers and a couple of diagnostic tools:

sudo apt update && sudo apt full-upgrade -y
sudo apt install -y mesa-vulkan-drivers vulkan-tools radeontop

Confirm the card is visible to Vulkan:

vulkaninfo | grep -i deviceName
# should print "AMD Radeon RX 6750 XT" (mine shows RADV NAVI22)

radeontop is the tool we'll use later to prove the GPU is actually doing the math, so it's worth installing now.

Step 2: Install Ollama and pull a model

Ollama is the easiest way to serve a local model. Install it with one line:

curl -fsSL https://ollama.com/install.sh | sh

On this hardware Ollama installs as a systemd service. On first run, it resolves the Vulkan backend, drops the unsupported ROCm device, and loads onto the discrete RX 6750 XT with ~11.6 GiB of usable VRAM.

Note: When Ollama starts, it will show a warning that it dropped the ROCm device. This is expected on a gfx1031 card. It realized that the GPU doesn't support ROCm and fell back to Vulkan.

Picking a model for 12GB of VRAM

Now that Ollama is set up, it's time to choose a model for it to serve. This is where the VRAM budgeting gets real.

My VRAM limitation rules out the current flagships: qwen3-coder:30b (~19GB), Codestral 22B (~13GB), and anything 24B+ dense. What's left is still a strong field of 7–16B coders.

I went with qwen2.5-coder:14b because this box is intended to be the worker tier of a two-tier setup: easy tasks run locally, hard ones escalate to externally hosted models. I wanted the fastest, most reliable, tool-capable code specialist that fits. If I need a higher-reasoning model, I will use an externally hosted 'frontier-class' model.

The closest competitor for my use was deepseek-coder-v2:16b. I may yet try it out, benchmark it, and see how it compares in my daily workflow, since it has the advantage in speed and a much larger context than the rest.

Pulling

With the model chosen, pulling it is one line:

ollama pull qwen2.5-coder:14b-instruct-q4_K_M

Quick smoke test:

ollama run qwen2.5-coder:14b-instruct-q4_K_M "write a python function that reverses a string"

If it answers, the model runs. However, that does not prove it's using the GPU. That part is next.

Step 3: Prove it's actually on the GPU (not silently on the CPU)

It's possible for Ollama to discover your GPU, load the weights into VRAM, and still run the compute on CPU. The logs will look normal, VRAM will show occupied, and the tokens will trickle while your CPU melts.

To verify the GPU compute, I used radeontop. Open it in one terminal like so:

radeontop

Then, in a second terminal, fire off a prompt long enough to sustain load for several seconds:

ollama run qwen2.5-coder:14b-instruct-q4_K_M \
  "write a detailed, heavily commented python implementation of a red-black tree with insert, delete, and search"

Now watch radeontop while tokens are streaming. What you're looking for:

The signal is the transition: idle → active → idle, synced to token output.

Note: Watch Graphics pipe and Shader Clock, not VRAM. The weights can be in VRAM while the math runs on the CPU. You have to watch during streaming. The compute drops to zero as soon as generation ends.

On the VRAM numbers: Ollama reports usable VRAM in GiB (it logged ~11.6 GiB available), while radeontop reads out in GB using its own accounting. The two don't line up exactly.

On my box, Graphics pipe hit ~93% and Shader Clock ~90% during generation, then fell to 0% and ~0.11% after. That confirms GPU inference over Vulkan. If your compute stays flat while tokens are streaming, that indicates CPU fallback, so you'll need to do some debugging.

A tmux tip: split the terminal (Ctrl-b %), run radeontop on the left and your ollama run on the right, and you can literally watch the GPU spike as it "thinks." It's oddly satisfying.

Step 4: The context-window decision (8K, on purpose)

By default, Ollama picks a context length for you, but I wanted to set it explicitly, because context costs VRAM. A bigger context window means a bigger KV cache sitting in memory alongside the weights. On a 12 GB card, that budget is tight, so I capped the default served context at 8K, which gives ~10.3 GB resident with headroom to spare (even though Qwen2.5-Coder's native context is 32K).

I set it with a systemd drop-in, so I'm not editing Ollama's packaged unit file:

sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
EOF

sudo systemctl daemon-reload
sudo systemctl restart ollama

Verify it took:

systemctl show ollama -p Environment --no-pager   # echoes OLLAMA_CONTEXT_LENGTH=8192

8K is a fine default for most coding prompts. If a specific task needs more, you don't have to change the service. You can override context per request:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:14b-instruct-q4_K_M",
  "prompt": "...",
  "stream": false,
  "options": { "num_ctx": 16384, "temperature": 0.2 }
}'

Raise num_ctx for the one long-context call that needs it; don't leave it high globally or you'll pay the VRAM tax on every request.

A couple of other handy per-request knobs:

• temperature 0.1–0.3 for deterministic code, higher for brainstorming
• keep_alive pass -1 to pin the model resident when you're about to hammer it (by default it unloads after 5 idle minutes and the next request eats a few-second reload).

Ollama also exposes an OpenAI-compatible API at http://localhost:11434/v1 (e.g. /v1/chat/completions). Any tool compatible with "OpenAI API, custom base URL" can point at it locally with a dummy key. The server is stateless. It takes the full message history on every call, so your client holds the conversation.

Step 5: zram, and a packaging gotcha

This box has 16 GB of RAM, and the DDR4 shortage of 2026 makes adding more expensive. The model itself lives in VRAM, so system RAM isn't the hot path, but the OS plus whatever I stack on top can spike. I'm planning on setting up some orchestration and automation for unattended workflows, so zram is cheap insurance.

If you don't know, zram is a compressed swap device that lives in RAM, giving you meaningfully more usable memory at a small CPU cost.

On Ubuntu 26.04 the right tool is systemd-zram-generator (not the older zram-tools / zram-config packages, which use different config paths and will just confuse you):

sudo apt install -y systemd-zram-generator

sudo tee /etc/systemd/zram-generator.conf >/dev/null <<'EOF'
[zram0]
zram-size = min(ram, 8192)
compression-algorithm = zstd
EOF

sudo tee /etc/sysctl.d/99-zram.conf >/dev/null <<'EOF'
vm.swappiness = 180
EOF
sudo sysctl --system >/dev/null

zram-size = min(ram, 8192) is evaluated in MiB, so on a 16 GB machine that's an 8 GB compressed device.

vm.swappiness = 180 looks absurd if you remember the old "100 is the max" rule, but kernels since 5.8 cap at 200. Since zram swap is RAM-fast, biasing the kernel toward evicting pages into compressed RAM is cheap and frees space for page cache. 150 is a good option too. The difference is noise.

The gotcha

Here's what confused me. The package ships a vendor default that activates a 4 GB / lzo-rle device at install time, before my config file existed. A plain systemctl start on an already-active zram device does nothing. It keeps the install-time parameters and silently ignores your config.

Symptom: you set up 8 GB / zstd, run zramctl, and see 4G / lzo-rle instead. Your config appears to do nothing.

Fix: use restart, not start. It tears the device down and rebuilds it from your /etc config:

sudo systemctl daemon-reload
sudo systemctl restart [email protected]

Verify the end state:

zramctl            # want: /dev/zram0  zstd  8G  [SWAP]
swapon --show      # zram0 at priority 100; disk swapfile as the low-priority tail
cat /proc/sys/vm/swappiness   # 180

One more caveat: zram is not an OOM guarantee. A hard enough runaway process blows past RAM + zram and still hits the OOM killer. The on-disk swapfile (priority -1) is the tail that catches overflow; zram (priority 100) fills first. For a box that's going to run unattended workloads, the layering is worth having.

What I ended up with

A machine that:

• runs Qwen2.5-Coder 14B on a 12 GB AMD GPU over Vulkan (no ROCm, no override hacks), with GPU compute verified, not assumed;
• serves it on both the native Ollama API and an OpenAI-compatible endpoint, so any local tool or editor plugin can point at it;
• caps context at a VRAM-sane 8K by default with per-request overrides for the rare long task;
• has zram as a RAM-pressure cushion for whatever I stack on top.

And the whole thing runs on hardware I already owned, for the cost of electricity.

TL;DR

The cliffnotes on setting up your own self-hosted LLM server:

1. Install an LTS Linux (Ubuntu 24.04/26.04) with a recent kernel for your GPU.
2. Check your GPU's compute support. AMD: is your gfx target on ROCm's list? If not (e.g. RX 6700-series gfx1031), plan on Vulkan: sudo apt install mesa-vulkan-drivers vulkan-tools radeontop.
3. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh.
4. Pull a model that fits your VRAM. 12 GB → a 14B at Q4_K_M (~10 GB). Don't try to cram a 30B into 12 GB.
5. Verify GPU compute with radeontop during generation. Watch Graphics pipe / Shader Clock, not VRAM. Discovery ≠ compute.
6. Set the context ceiling with a systemd drop-in (OLLAMA_CONTEXT_LENGTH=8192) to keep the KV cache inside your VRAM budget; override per request with num_ctx when needed.
7. Set up zram with systemd-zram-generator, and remember to restart (not start) the setup service so your config isn't shadowed by the vendor default.

Next steps

Now I have my very own local-tier coding model, running for free (minus the cost of electricity) on a GPU that I had lying around in a computer that lives in my basement.

What I am building towards is a coding agent that can do the easy work locally for free, allowing me to save my frontier model usage limits for harder work. Eventually, I would like to have a fully automated workflow that I can leave unattended (with guardrails in place) while I am doing other things. This box is the first step in reaching that.

There are a few more tasks that need to be done, with more posts to come documenting my process and results.

Reaching it from anywhere, privately

This post covers setting up a local AI server that only serves on localhost. Next I will cover getting to it from my laptop or my phone, from anywhere, over an encrypted mesh network with nothing exposed to the public internet. That's the next post.

Is it actually any good? The uncomfortable question.

Can I really trust a quantized 14B running on a consumer card with real work? I put it through the standard coding benchmarks, and the results were good, but surprising, and also alarming in a way that turned out to be the benchmark's fault, not the model's. I am working on a full write-up, and may even try out some other models for comparison.

The agent loop itself

The whole point of doing all this. I'm still experimenting with the many tools available for orchestrating multiple models, and running feedback loops so they can keep running even when I am not attending to them. Maybe I'll even end up writing some scripts of my own to do this. There are a few side projects that I have in the works to test everything out with, so expect future write-ups on what I discover.

Final notes

For now, I am happy to have my own free model running on a mid-level card that I didn't have to pay 2026 prices for. You don't need the latest, most expensive hardware to run a real and useful model at home. All you need is something with enough VRAM. It doesn't even have to be on AMD's supported list.

Go and check out what your idle GPU could be doing instead.