Reproduce baseline faithful-calibration results for existing language models
Train a model using reinforcement learning with metacognitive feedback
Select high-value training examples using metacognitive data selection instead of random sampling
Evaluate how honestly a model expresses uncertainty about its own answers
| yale-nlp/rlmf | alexrosbach/replibook | arlandaren/proagents | |
|---|---|---|---|
| Stars | 25 | 25 | 25 |
| Language | Python | Python | Python |
| Setup difficulty | hard | easy | easy |
| Complexity | 5/5 | 2/5 | 1/5 |
| Audience | researcher | ops devops | vibe coder |
Figures from each repo's GitHub metadata at analysis time.
Requires a conda environment, several paid API keys (Google, OpenAI), a HuggingFace token, and a manually downloaded dataset before experiments will run.
RLMF is the code for a research project from Yale about teaching large language models to be more honest about how confident they actually are in their own answers. The core idea is that a model which can accurately judge whether its own answer is likely correct is in a better position to improve, similar to how a person who is good at noticing their own mistakes tends to learn faster. The repository implements two related methods. The first, reinforcement learning with metacognitive feedback, adjusts how the model is trained during preference optimization by rewarding it not just for good answers but for accurately judging the quality of its own answers. The second, metacognitive data selection, uses that same self judging ability to pick out which training examples are most valuable to learn from, which the authors report works better than simply picking training examples at random or through standard active learning approaches. According to the README, this approach improves on standard reinforcement learning by as much as 63 percent on their measured tasks, while also making models better at expressing honest limits on their own capability, described as faithful calibration, across a range of different models and tasks. Setting the project up involves creating a Python conda environment, installing the listed dependencies, and downloading a small language model for a supporting library. Running the experiments requires API keys for Google and OpenAI models as well as a HuggingFace access token, and optionally a Weights and Biases account for tracking training runs. One dataset used in the baseline experiments has to be downloaded separately and placed into a specific folder with a renamed filename. The repository is organized around a src folder containing training code, evaluation metrics for measuring how faithfully a model expresses uncertainty, prompt templates used for evaluation, and scripts for running experiments on open source models. A linked academic paper accompanies the code. The full README is longer than what was shown.
Research code from Yale that trains language models to more honestly judge their own confidence, using reinforcement learning based on the accuracy of the model's self-assessments.
Mainly Python. The stack also includes Python, Conda, HuggingFace.
The README does not state a license for this project.
Setup difficulty is rated hard, with roughly 1h+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
Verify against the repo before relying on details.