Qwen3 8 Flash Next 125B MoE Modell Image © PCMasters.deQwen3 8 Flash Next 125B MoE Modell (Image © PCMasters.de)

Hardware Requirements and Memory Allocation

Running Qwen3.8-Flash-Next requires a significant amount of memory. The minimum requirement for the smallest quantized version is 75 GB of RAM or unified memory. For optimal stability, devices with 96 GB of memory are recommended.

Memory requirements vary depending on the quantization level used. Quantization Level and Total Memory Requirements (RAM + VRAM / Unified):

  • 1-bit: 75 GB
  • 2-bit: 79 GB
  • 3-bit: 90 GB
  • 4-bit: 96–114 GB
  • 5-bit: 163 GB
  • 8-bit: 200 GB
  • BF16: 355 GB

The 1-bit quantized version is approximately 79% smaller than the original BF16 (355 GB) while maintaining an 80% accuracy rate in the top 1%. This is achieved through the use of “Per Layer Embeddings” (PLE) and n-grams, which function as lookup tables. Unlike other layers, these n-gram components are maintained at a quantization level of at least 4 bits to prevent a significant degradation in model accuracy.

Runtime Frameworks and Installation

The model can be deployed in two main ways: Unsloth Desktop and llama.cpp.

Deployment via Unsloth Desktop

Unsloth Desktop offers a cross-platform interface for macOS, Windows, and Linux. It features automatic memory offloading and multi-GPU detection. Beyond simple inference, this framework supports:

  • Access to self-healing tools and integrated web search.
  • Local execution of Python and Bash code.
  • Automatic optimization of inference parameters.
  • Integration with external tools such as Claude Code and MCP.

Installation can be performed via the official desktop application or via the command line using curl for Unix-based systems and PowerShell for Windows.

Deployment with llama.cpp

For users who prefer a lower-level implementation, the model is available in GGUF format. To use it, you’ll need the latest version of llama.cpp, compiled with CUDA support for NVIDIA GPUs or Metal support for Apple Silicon. The process involves downloading the GGUF files from Hugging Face or ModelScope and running them using the llama-cli or llama-server binaries.

Here is an example of how we perform the deployment using four TESLA V100 32 GB accelerators:

CUDA_VISIBLE_DEVICES=0,1,2,3 /home/andy/Downloads/llama.cpp/build/bin/llama-server --model /home/andy/Downloads/llama.cpp/models/qwen3. 8-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --host 0.0.0.0 --port 8080 -ngl 999 --tensor-split 1,1,1,1 --ctx-size 231072 --parallel 1 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --threads 32 --jinja
  • CUDA_VISIBLE_DEVICES=0,1,2,3: All four GPUs are used for this
  • --cache-type-k q8_0: A Q8 KV cache is used
  • --tensor-split 1,1,1,1: The model is distributed equally across all four GPUs; the load is also distributed this way across all four GPUs
  • --host 0.0.0.0: Locally accessible on the network via the server’s local IP address
  • --port 8080: Port 8080 will be exposed
  • --ctx-size 231072: The context size in the KV cache is set to 230k tokens

Performance Optimization via Multi-Token Prediction (MTP)

To increase inference speed, Qwen3.8-Flash-Next supports Multi-Token Prediction (MTP). This technology enables the model to predict multiple tokens simultaneously rather than sequentially, resulting in a speed increase of 1.3 to 1.7 times. On a single RTX 6000 PRO GPU, MTP can boost throughput from a baseline of 100 tokens/s to 170 tokens/s.

On our four TESLA V100s, we achieved mixed results ranging from 300 to 700 tokens/s. Overall, this was sufficient for our coding tasks.

The MTP implementation requires 1 to 2 GB of additional memory. To reduce this overhead, shared MTP modules are available that exclude the embedded tokens and share them with the main model.

MTP Memory Overhead After Quantization:

  • BF16: General MTP (7.77 GB) vs. Shared MTP (5.23 GB)
  • Q8_0: General MTP (4.14 GB) vs. Shared MTP (2.79 GB)
  • Q4_K_M: General MTP (2.79 GB) vs. Shared MTP (1.91 GB)

Operating Parameters and Thinking Modes

Qwen3.8-Flash-Next is a hybrid reasoning model, which means it operates with different settings depending on whether it is in thinking mode or command mode (non-thinking mode). Parameter / Reasoning Mode / Command Mode

Additionally, the model supports a reasoning_effort parameter to control the depth of analysis:

  • xhigh (default): For complex tasks that require a comprehensive analysis.
  • medium: A balance between speed and accuracy.
  • low: Optimized for efficiency and speed.
  • none: Disables the advanced reasoning process.

The model also features a “Preserve Thinking” function that retains the train of thought from previous conversation rounds to ensure higher accuracy during longer interactions.