What is pl-ragas-analyzer?

pl-ragas-analyzer is a small research project that studies how well a RAG system, meaning a retrieval-augmented generation setup that looks up documents before answering a question, can be evaluated rather than trying to build a great RAG system itself. The RAG part is intentionally simple: it uses plain cosine similarity over embeddings with no vector database at all. The real subject is measuring the quality of that retrieval and the answers it produces, using a tool called Ragas with Claude acting as the judge that scores each answer. The documents behind this project are recent bills from the Brazilian Chamber of Deputies, in Portuguese, submitted in June and July of 2026. Because these bills are newer than the training data of any language model, a correct answer can only come from the retrieval step finding the right document, not from the model already knowing the answer from memory. The data is pulled from a free, key-free government API. The project fetches around 100 bills, builds an embedding index using a local Ollama model, and then answers questions by finding the most relevant bills and asking Claude to respond in Portuguese based on what was retrieved. A separate evaluation script runs 16 hand-written questions through six Ragas metrics, including faithfulness, answer relevancy, and whether retrieval returned the right documents versus irrelevant noise, plus one custom metric checking whether an answer cites the bill's official number. Several ablation experiments are included too, such as running with no retrieval at all, changing how many documents are retrieved, injecting irrelevant documents on purpose, and combining exact number matching with vector search for bill numbers that plain embeddings cannot understand. The generator and judge can also be swapped to run entirely on local models through Ollama instead of the Anthropic API, which removes API cost and lets someone compare how well a local judge agrees with Claude's scoring. Setup requires Python 3.11, a running Ollama instance with an embedding model pulled, and an Anthropic API key for the default configuration, since running the full Claude-judged evaluation costs roughly one to two dollars per run. The README notes a few compatibility issues with specific Ragas and library versions but does not state a license for the project.