whatisgithub

What is yams?

oli-26/yams — explained in plain English

Analysis updated 2026-05-18

0PythonAudience · researcherComplexity · 3/5Setup · easy

In one sentence

A research project testing exactly when giving an AI model external memory helps, based on whether it can already verify the answer itself.

Mindmap

mindmap
  root((YAMS))
    What it does
      Tests when memory helps
      Verify gap concept
      Model comparison study
    Tech stack
      Python
      MCP server
    Use cases
      AI memory research
      Benchmark reproduction
      Coding agent memory
    Findings
      Small models gain most
      Large models unaffected
      Checkability matters
    Audience
      Researchers
      AI practitioners

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

What do people build with it?

USE CASE 1

Study when adding external memory to a language model actually improves accuracy

USE CASE 2

Reproduce the verify-gap benchmark results with the included dependency-free scripts

USE CASE 3

Try the MCP server that gives a coding agent validated memory recall and pushback

USE CASE 4

Read the full technical report as a case study in evaluating AI memory systems

What is it built with?

PythonMCP

How does it compare?

oli-26/yams0xallam/my-recipe0xhassaan/nn-from-scratch
Stars00
LanguagePythonPythonPython
Last pushed2022-11-22
MaintenanceDormant
Setup difficultyeasymoderatemoderate
Complexity3/52/54/5
Audienceresearchergeneraldeveloper

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · easy Time to first run · 30min

The core benchmarks are pure Python 3.10+ stdlib with no install, but the full experiment harness needs a paid API key and is not included.

So what is it?

YAMS, short for Yet Another Memory System, is a research project that studies when giving a language model access to an external memory actually improves its answers, and when it makes no difference at all. The author's central finding is that memory only matters when there is a gap between what a model can work out or check for itself from the evidence in front of it, and what the memory claims. When that gap is open, a correct memory helps and a wrong one hurts. When the model could already verify the answer on its own, the memory has no effect either way. The repository backs this claim with experiments run across three model sizes, comparing answers with no memory against answers using a validated rule stored in memory. Smaller models showed a large accuracy gain from good memory, a mid-sized model showed a smaller and less certain gain, and the largest model performed identically with or without memory because it could already verify the answers itself. A separate test showed the same single model rejecting a wrong memory when it could check it, but being misled by that same wrong memory when it could not, showing the effect comes from whether an answer is checkable rather than from which model is used. The full technical report is included as a PDF, and the repository provides several small, dependency-free Python scripts in a benchmarks folder that regenerate the test questions and reproduce the core results. These require Python 3.10 or later and nothing else to run. The heavier system that produced some of the original results, built with FastAPI and SQLite, is not included, but its raw output data is provided separately for reference. There is also a working example of the memory system in use: an MCP server in the mcp folder that plugs a validated memory into a coding agent, letting the agent recall stored rules and check its own actions against them. The author is upfront that this is a single person's exploratory study with a small number of trials, not a finalized or peer-reviewed result, and lists the study's limitations directly in the README.

Copy-paste prompts

Prompt 1
Explain the verify-gap concept from the YAMS report in plain language
Prompt 2
Help me run the quad_bench.py and trust_test.py benchmarks from this repo
Prompt 3
Walk me through how the MCP server in this repo gives an agent validated memory
Prompt 4
Summarize the headline results table comparing no-memory vs validated-rule accuracy
Prompt 5
Explain the caveats and limitations the author lists for this study

Frequently asked questions

What is yams?

A research project testing exactly when giving an AI model external memory helps, based on whether it can already verify the answer itself.

What language is yams written in?

Mainly Python. The stack also includes Python, MCP.

How hard is yams to set up?

Setup difficulty is rated easy, with roughly 30min to a first successful run.

Who is yams for?

Mainly researcher.

Open on GitHub → Ask about another repo

This repo across BitVibe Labs

Verify against the repo before relying on details.