What is memsyco-bench?

MemSyco-Bench is a research benchmark for testing how well language models and their memory systems handle remembered user preferences, especially in cases where blindly following a memory would lead to a worse answer. The core problem it studies is a kind of digital people pleasing, where a model favors something it remembers about a user even when that memory is outdated, contradicted by better evidence, or simply does not apply to the situation at hand. The benchmark is built around five task types. Personalized Memory Use checks whether a model actually applies a relevant remembered preference when it should. Valid Memory Selection checks whether a model correctly follows the user's most recent preference instead of an older, outdated one. Memory-Evidence Conflict checks whether a model can override a remembered preference when stronger outside evidence points elsewhere. Contextual Scope Control checks whether a model applies a preference only where it actually fits, rather than everywhere. Objective Fact Judgment checks whether a model still gives a factually correct answer even when a remembered preference favors a familiar but wrong one. Together these five tasks cover 1,550 evaluation samples with a shared evaluation pipeline for generating answers, judging them, and analyzing results. The project compares nine different setups side by side, including having no memory at all, using the full raw conversation history, and several named memory systems such as MemZero, A-MEM, LightMem, MemoryBank, NaiveRAG, MemGPT, and Supermemory. This lets researchers see how each memory approach performs across the five preference-handling tasks. To use it, you set up a Conda Python environment, install the core requirements, and optionally install extra dependencies for the full set of memory baselines. You then configure the necessary API keys and run a provided shell script, with a PowerShell wrapper available for Windows users, either running the full nine-setting evaluation suite or a smaller example limited to specific tasks and methods. Results, cached completions, and logs are written to a local output folder that is not included in the repository. The project accompanies an academic paper and provides a public leaderboard, and asks that users cite the work if they find it useful. The full README is longer than what was shown.