Tutorial

Restore from an RDMA file share

Put a snapshot on a file share your GPU hosts mount over RDMA, once, and every host restores from it at the speed of the fabric instead of the internet. On a fast shared file system, three 8×H100 replicas of GLM-5.3-Flash served in under 60 seconds, all reading one share.

You need two or more GPU hosts prepared as in Install, with the same GPUs and driver branch, InfiniBand or RoCE NICs, and an NFS server that exports over RDMA: a VAST, NetApp, DDN or Weka cluster, or a Linux server with nfsd listening on RDMA. The example uses glm53-flash on 8×H100; any model in the catalog works the same way.

Mount the share over RDMA

On every GPU host, load the RDMA transport for NFS and mount the export on port 20049, the NFS-over-RDMA port:

bash

$ sudo modprobe rpcrdma
$ sudo mkdir -p /mnt/warmctl
$ sudo mount -t nfs -o proto=rdma,port=20049,rsize=1048576,wsize=1048576 nfs.cluster.internal:/warmctl /mnt/warmctl

These are the stock Linux client options. Storage vendors often add their own, such as more connections per mount or multipath across NICs; use what your vendor recommends. Check that the mount really uses RDMA and not TCP:

bash

$ nfsstat -m | grep -A1 /mnt/warmctl

The flags line should contain proto=rdma. If it says proto=tcp, the RDMA mount failed and the kernel fell back; check dmesg for rpcrdma errors.

Measure what the share delivers

A restore copies the whole snapshot from the share, so its read speed sets the copy time. Write a test file and read it back with parallel streams, the way warmctl reads:

bash

$ fio --name=share --directory=/mnt/warmctl --rw=read --bs=16M --size=8G --numjobs=16 --direct=1 --group_reporting

Note the READ: bw= figure. At 10 GiB/s a 404 GiB snapshot copies in about 40 seconds; at 3 GiB/s, a typical network disk, it takes over two minutes.

Copy the snapshot onto the share

On one host, mirror the model from the registry into the share:

bash

$ warmctl mirror glm53-flash /mnt/warmctl

mirror copies the platforms this host can restore, as an OCI image layout under /mnt/warmctl/glm53-flash. Add --all-platforms to copy every GPU and driver variant, or use --catalog to copy the whole catalog. Running it again later copies only what changed.

Point every host at the share

On every GPU host:

/etc/warmctl/config.yaml

cache: /mnt/warmctl

up and pull now look in the share first and go to the registry only for what is missing. A snapshot a host has to download from the registry is written into the share on the way, so the next host finds it there. In Kubernetes, set WARMCTL_CACHE=/mnt/warmctl in the pod instead.

Restore on another host

On a second host, start the model by name:

bash

$ warmctl up glm53-flash

up copies the snapshot from the share to local disk in 64 MiB pieces, 64 at a time, checks each piece's digest, and restores the engine from there. When the API answers, the snapshot is on local disk and the model serves:

bash

$ warmctl images
$ curl -s localhost:8000/v1/models

Start the same model on more hosts at once: each reads the share in parallel, as far as its NICs and the storage allow.

The restore always runs from local disk; the share is only the source of the copy. A snapshot restored straight from a virtiofs share took 63 s instead of 14 s, so copying first at full speed and restoring locally is faster end to end.

What you did

Mounted an NFS share over RDMA and measured its read speed.
Mirrored a snapshot onto it once with warmctl mirror.
Made every host use it as a cache, and restored from it by name.

Next: Mirror and cache snapshots for registry mirrors and pre-pulling, or run it in Kubernetes.