Installing a MaluOS GPU server

Before you start

You need an x86-64 machine with an NVIDIA RTX card and MaluOS Server on it. If the server is not installed yet, follow Setting up your first server first and come back here at its step 7.

	Minimum	Recommended
Machine	x86-64 with an NVIDIA RTX card	a server or a dedicated desktop; a laptop shares only when idle
GPU memory	8 GB	24 GB or more; the models you can serve are decided by VRAM
System memory	8 GB	16 GB or more
Disk	60 GB	models are 2 to 40 GB each; plan for the set you will keep
Network	a stable name or address on the LAN	wired; port 11435 reachable from the desks
Driver	the 595 branch (580 as the alternative for older cards)

Not this guide. An NVIDIA DGX Spark is arm64 and does not run MaluOS; see NVIDIA ARM. An Apple Silicon Mac is covered in M-series Mac. An RTX card in an x86-64 box is the MaluOS case.

The driverCanonical-signed kernel modules, so Secure Boot needs no MOK enrolment and no DKMS build

Local modelsOllama under the gateway, with tiers your callers ask for by name

The node APIstatus, health, a queue with cancel and priority, runs, models and keys on port 11435

Paired deskseach with its own key, each call ledgered on the desk and on the host

1Install the NVIDIA component

The driver is a MaluOS component, kept off the install medium because of its size. It brings the signed modules for the generic kernel, the 595 driver with 580 as the alternative, nvidia-smi and nvtop. Ollama carries its own CUDA runtime, so no toolkit is needed.

sudo apt install maluos-nvidia
sudo reboot

After the reboot:

maluctl gpu status
nvidia-smi

What you should see: the card with its driver version, memory, temperature and power, and a line naming the chassis and the sharing rule it implies (a server shares always). If the GPU is missing, the module did not load; dmesg | grep -i nvidia says why, and with Secure Boot on make sure the signed linux-modules-nvidia-595-generic package was what apt installed rather than the DKMS alternative.

2Install Ollama and pull models

The gateway installs a pinned, checksummed Ollama release as a localhost-only service under its own user. Pull models that fit the card: as a rule of thumb a model needs a little more VRAM than its download size at the default quantisation.

sudo maluctl gateway ollama install
sudo maluctl gateway ollama pull llama3.2:3b          # fast tier default
sudo maluctl gateway ollama pull llama3.1:8b          # standard tier default
sudo maluctl gateway ollama pull qwen2.5:14b          # strong tier default; pick what your VRAM allows
sudo maluctl gateway ollama pull nomic-embed-text     # embed tier
sudo maluctl gateway ollama status

What you should see: the models on disk, and the GPU in use when you test: maluctl gateway chat "hello" --model standard answers in seconds and nvidia-smi shows the Ollama process holding VRAM.

The defaults for the tiers (fast, standard, strong, embed) are seeds; replace them with what the card runs well, and keep the busiest model resident so the first call of the day is not a load:

sudo maluctl gateway alias set strong --tier strong --backend ollama-local --model qwen3:32b --position 0
sudo maluctl gateway ollama warm qwen3:32b
sudo maluctl gateway catalog

3Turn sharing on

Sharing is a switch on the serving host, enforced on the serving side: with it off, an inference call from the network is refused as not_shared while the node API keeps answering status. A server's chassis rule is always; a dedicated desktop opts in; a laptop opts in and shares only while idle.

maluctl gateway share status
sudo maluctl gateway share on

4Issue a key for each client

Every desk, tool or manager that will call this host gets its own inference key. A key is a principal on this host, key:<name>, with a policy of its own: the class it runs in, whether it may reach this host's cloud rungs, which tiers it may ask for, how many calls at once. The token is shown once; hand it to the desk's owner over a channel you trust.

sudo maluctl gateway key issue --name ed-desk --class interactive --cloud-allowed no
sudo maluctl gateway key issue --name build-box --class batch --tiers standard,embed --max-concurrent 2
sudo maluctl gateway key list

A node key (--kind node) is for a manager that reads this host's queue and runs rather than for inference; you do not need one yet.

5Expose the gateway on the network

The gateway serves its Unix socket only until you expose it. Exposing opens the TCP listener on port 11435, bound to the address you choose, and refuses to start without at least one key.

sudo maluctl gateway expose --bind 0.0.0.0
maluctl gateway status

If the host runs a firewall, allow 11435 from the LAN only, for example sudo ufw allow from 192.168.1.0/24 to any port 11435 proto tcp. From a desk, curl http://gpu-1.lan:11435/node/v1/health answers.

6Pair a desk

On each desk, as its owner, add this host as a backend of kind remote. The key goes into the desk's vault for the gateway's own use, never into a file, and the host becomes one more rung of the desk's ladder.

sudo maluctl gateway backend add gpu-1 http://gpu-1.lan:11435 --key mg_… --models qwen3:32b,llama3.1:8b
sudo maluctl gateway alias set strong --tier strong --backend gpu-1 --model qwen3:32b --position 0
maluctl gateway backends
maluctl gateway chat "Say hello from the GPU server." --model strong
maluctl gateway routes

What you should see: the backend healthy with its models, an answer, and a route row naming gpu-1. The desk's callers change nothing: the call goes to the desk's socket, the scheduler picks the rung by policy, class and expected wait, and the call is ledgered twice, on the desk under the caller and on this host under the key, joined by their ids.

Three rules worth knowing. A LAN node is governed by the policy field remote_allowed, separately from the cloud. The sensitivity gate applies as for the cloud, since data leaves the desk. And there are no chains: a call that arrived from another gateway is never forwarded on.

7Operate it

maluctl gateway node status            # this node: lanes, GPUs, loaded models, keys seen
maluctl gateway node queue             # what is waiting, with cancel ID and priority ID CLASS
maluctl gateway node runs --limit 20   # recent runs with their timings and tokens
maluctl gateway usage --since 7d --by principal
maluctl gateway alerts
sudo maluctl gateway key revoke ed-desk

On a MaluOS desktop the same picture is in the System Monitor: the GPUs tab for the local card, the MaluOS Network tab for this node, its keys and the paired nodes, and the Queue tab with cancel and priority.

If something is wrong

A desk's call is refused with not_shared. Sharing is off on the host, or a laptop host is not idle. sudo maluctl gateway share on.
expose refuses to start. Issue a key first (step 4); the listener will not open without one.
The backend is unhealthy on the desk. The port is not reachable: firewall, or the host was exposed on the wrong address. Test with curl http://HOST:11435/node/v1/health from the desk.
model_missing on a run. The desk asked for a model that is not on the host. Pull it on the host, or fix the alias on the desk.
Slow answers, VRAM full. One generation at a time per backend is the default (max_concurrent: 1); the queue is doing its job. A second model being loaded evicts the first; keep one resident with warm and give the other tiers smaller models.
Still stuck. Open an issue at github.com/maludb/maluos/issues.

What next

Pairing by hand is today's way. The estate's agent directory and the Sovereign LLM Manager will take over key issuing and discovery across nodes; a GPU server set up like this joins them without being reinstalled.
Join the estate. sudo maluctl estate join URL --code CODE --kind inference makes this host a node that reports its GPUs and health to the office.
Add a cloud rung on the host for callers whose key allows it, so a long queue overflows to the cloud under your policy rather than theirs.

← All guides Agent Skill (SKILL.md) Shared GPUs, in the repository The inference node API