What is a6b-k-expansion?

This repository is a research log documenting an experiment on a large AI language model called Qwen3.6-35B-A3B. That model is a Mixture-of-Experts model, meaning it has many specialized sub networks called experts, and normally only activates 8 of them for each piece of text it processes. The project's core idea is to force the model to activate 32 experts per token instead of 8, without adding any new weights, which increases the amount of active computation per token. Doing this alone makes the model perform worse on knowledge and math tests, and the project measures exactly how much worse, then tries to repair that loss. The repair method is a targeted fine-tuning approach. The researchers find which experts get used most often once 32 are active, then retrain only small additional pieces attached to those specific experts, while leaving the routing logic and everything else frozen and unchanged. These additional pieces can be switched off to return to the original model at any time. Across three rounds of this fine-tuning, using increasingly varied training data, the accuracy gap compared to the original 8 expert setup shrinks until it becomes a statistical tie on the tests used, though the underlying numbers are still slightly negative rather than fully recovered. The README also reports a side benefit on a coding test, where the fine-tuned version outperforms both the original model and the untouched 32 expert version, while producing shorter answers, but warns that pushing for shorter answers too aggressively can hurt performance elsewhere. All results are described as measured through paired comparisons on fixed test sets rather than estimated. The repository includes several self contained HTML report pages with charts explaining the architecture change and the test results, along with a written methodology document. Since GitHub does not render HTML files directly, viewing them requires either enabling GitHub Pages for the repository or downloading the files and opening them in a browser. The work is explicitly ongoing, with additional training and evaluation steps still in progress rather than finished.