Run Qwen3.8-27B in 8-12 GB With GSQ-RCO Non-Uniform GGUF Quantizations

The Deep Algorithms and Systems Lab (DASLab) at IST Austria just published something interesting for anyone running local LLMs: non-uniform GGUF quantizations of Qwen3.8-27B built with two of their own methods, GSQ and RCO. Instead of applying a single quantization type to the whole model, these files assign a different bit-width to every tensor, and they still run as completely standard GGUF files in llama.cpp, Ollama and LM Studio.

The headline numbers: the smallest file is 8.4 GB at 2.5 bits per weight and already beats the BF16 baseline on zero-shot tasks. The 11.8 GB variant at 3.5 bpw is effectively lossless on reasoning benchmarks, at less than a fifth of the BF16 size (53.8 GB).

What is non-uniform (mixed-precision) quantization

Normal GGUF quantization is uniform: you pick one type, e.g. Q4_K_M or IQ2_S, and every weight tensor in the model gets that type. Non-uniform quantization works differently. Every tensor gets its own quantization type, chosen per-tensor according to how sensitive that tensor is to quantization error. A model can mix, say, 2-bit tensors where the model is robust with 5-bit tensors where it is not, all inside one standard GGUF file.

The released files combine two methods, both from the DASLab:

• GSQ (Gumbel-Softmax Quantization, paper, code): post-training scalar quantization that jointly learns the per-coordinate grid assignments and the per-group scales via a Gumbel-Softmax relaxation. It closes most of the gap between scalar and vector quantization at 2-3 bits, while staying deployable in standard scalar formats like GGUF.
• RCO (Riemannian Constrained Optimization, paper, code): takes a database of candidate quantizations per tensor and assigns one quantization type to each tensor under a total size budget. The budget constraint is reformulated as a smooth Riemannian manifold in logit space, which allows gradient-based optimization directly on the task loss while enforcing the budget exactly, without constraint-specific hyperparameter tuning.

The pipeline in practice: each weight tensor is first quantized with GSQ at every candidate GGUF type, building a searchable database of tensor variants. RCO then does the budget-constrained search to pick one variant per tensor so the whole file hits the target bit-width, and the selected variants are stitched into a single standard GGUF.

Available files

Four sizes of Qwen3.8-27B, plus a BF16 vision projector (mmproj) for multimodal use that is shared across all quantizations:

Each quantization also ships an optional -mtp build (about 0.35 GB larger) that carries the model’s Multi-Token Prediction head for speculative decoding in llama.cpp. The weights are otherwise identical, so quality is unchanged.

Every file also includes reproducibility artifacts: a tensor-allocation dump listing the exact quantization type assigned to every tensor (with a histogram and the target bit-width), and the importance matrix (1000 chunks of 4096 tokens) used during quantization. You can audit how the file was built without opening the model.

Results

The IQ3_S file at 3.50 bpw is the task-lossless operating point: it matches the BF16 base model exactly on AIME25 (100.00) and LiveCodeBench v6 (85.71) and is within 0.51 points on GPQA-Diamond, at 11.8 GB versus 53.8 GB for BF16 (a 4.6x size reduction). Against Unsloth Dynamic quantizations it leads by 3.33 points on AIME25 and 1.71 on LiveCodeBench while being 0.2 GB smaller.

At the low end the improvement is even bigger: at the same 8.4 GB, IQ2_XS leads Unsloth Dynamic IQ2_S by 10.00 points on AIME25, 8.59 on GPQA-Diamond and 4.57 on LiveCodeBench v6.

The IQ2_S file also posts the best zero-shot average of the table (75.70 vs 74.34 for BF16, a 101.8% recovery), and at 3.00 bpw the model already matches the base on AIME25.

Running it in llama.cpp

The files are standard GGUF, so nothing special is needed. If you already run llama.cpp (on a CUDA server built from source, on an Apple Silicon Mac, or via the new installer), you can use them directly.

Quick start

llama.cpp can pull the file straight from Hugging Face with the -hf flag, using the format repo:file:

# OpenAI-compatible server with web UI (downloads the file automatically)
llama-server -hf ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S -ngl 99 --port 8080

# Interactive chat in the terminal
llama-cli -hf ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S -ngl 99

Or download explicitly with the Hugging Face CLI first:

pip install -U "huggingface_hub[cli]"
hf download ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --local-dir .

llama-cli -m Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf -ngl 99

Pick the file that fits your memory: 8.4 GB for a 16 GB machine with room to spare, 10-12 GB if you want the task-lossless IQ3_S point. As always, -ngl 99 offloads all layers to the GPU.

Speculative decoding with the MTP head

The -mtp variants (e.g. Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf) carry the model’s Multi-Token Prediction head for speculative decoding. Load the -mtp file instead of the base one; quality is unchanged, the model is about 0.35 GB larger.

Vision (multimodal)

Qwen3.8-27B is a vision model, and the repo ships the BF16 vision projector for all quantizations:

hf download ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF mmproj-Qwen3.8-27B-BF16.gguf --local-dir .

llama-mtmd-cli -m Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf \
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf \
  --image photo.jpg -p "Describe this image."

Ollama and LM Studio

ollama run hf.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

In LM Studio just search the repo name and pick a GSQ-RCO-* build from the file list.

Takeaway

Mixed-precision quantization used to be a manual, per-model tuning exercise. With GSQ + RCO it becomes a budget-constrained optimization that runs automatically, and the result is a plain GGUF you can drop into the tooling you already use. If you have a machine with 16-32 GB and want the best quality-per-byte out of a 27B-class model, the IQ3_S or IQ3_XXS file here is probably the best 10-12 GB you can download right now.