Shrink a DeepSeek-V4-Flash GGUF model by consolidating its experts without retraining it.
Reproduce a documented expert-pruning experiment on AMD ROCm hardware with a pinned llama.cpp fork.
Study quality-gate thresholds for validating a compressed mixture-of-experts model before publishing results.
| itres-labs/fold-q4 | 0xallam/my-recipe | 0xhassaan/nn-from-scratch | |
|---|---|---|---|
| Stars | 0 | — | 0 |
| Language | Python | Python | Python |
| Last pushed | — | 2022-11-22 | — |
| Maintenance | — | Dormant | — |
| Setup difficulty | hard | moderate | moderate |
| Complexity | 5/5 | 2/5 | 4/5 |
| Audience | researcher | general | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires a pinned llama.cpp fork, ROCm/HIP hardware, a legally obtained DeepSeek-V4-Flash model, and over 400GB of disk space.
FOLD-Q4 is a research tool for shrinking the DeepSeek-V4-Flash language model, which is built from many small specialist sub-networks called experts. Instead of retraining the model, it keeps a smaller set of the real experts, 192 out of the original 256 per layer in the tested setup, and has any expert slot that got dropped reuse whichever kept expert is closest to it. The result is a physically smaller version of the model file that can run on the ROCm and HIP backend used by AMD graphics hardware. The idea builds on a method called ConMoE for consolidating experts, adapted here specifically for DeepSeek-V4-Flash files and a pinned, exact version of the llama.cpp project used to run these models. The author is careful to state what this project does not prove: it does not claim the shrunk model matches the original on standard benchmarks, does not claim it reduces the actual compute per token, and does not claim to have solved graph replay problems for ROCm in general. A file called PAPER.md spells out exactly what the evidence does and does not support, and a VALIDATION.md file tracks which checks were actually run before release. The full workflow is handled by a single script, fold_q4.py, which clones or checks out the required llama.cpp fork, applies a small, documented set of patches, builds the ROCm backend, measures a reference version of the model, selects which experts to keep using calibration data, builds mapping files between the original and the shrunk model, and finally copies over only the kept expert data while checking the result against quality thresholds before writing the final file. Every stage can be resumed, and results are tied to the exact script, patch, and model versions used, so outputs from different versions cannot be mixed together by mistake. Running this requires Linux, Python 3.10 or newer, a C or C++ toolchain, ROCm and HIP tools, a legally obtained copy of the DeepSeek-V4-Flash model in two different quantization formats, and a large amount of disk space, more than 400 gigabytes in the tested case. The README does not state a license.
A research tool that shrinks the DeepSeek-V4-Flash language model by consolidating its expert sub-networks, producing a smaller GGUF file for ROCm hardware.
Mainly Python. The stack also includes Python, llama.cpp, ROCm.
Setup difficulty is rated hard, with roughly 1day+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
Verify against the repo before relying on details.