Python Framework for RAG Output Evaluation

I developed a RAG evaluation framework to measure and continuously improve the quality of an AI-powered Knowledge Assistant used by CAD engineers at NXP Semiconductors.

As we regularly introduced improvements to our RAG pipeline, including hybrid search, document chunking, context-window expansion, and reranking, we faced an important challenge: how could we objectively determine whether a change actually improved the chatbot's performance?

Manual testing and user feedback were valuable, but they weren't enough to consistently measure improvements or detect regressions. To address this, I worked on building a representative Q&A benchmark covering real engineering questions, technical troubleshooting scenarios, and complex requests requiring information from multiple documents.

The evaluation dataset combines real historical support incidents, AI-generated question variations, and questions written by internal application experts. This allows us to calculate retrieval-quality scores, compare different RAG configurations, and make development decisions based on measurable results rather than subjective impressions.

Project Context - Measuring RAG Quality

I work within the Design Engineering Support team at NXP Semiconductors, where we developed a RAG-powered Knowledge Assistant to help CAD engineers find technical documentation, troubleshoot issues, and use internal engineering tools more efficiently.

The chatbot retrieves information from internal FAQ websites, technical documentation, and knowledge articles before generating answers using LLMs. As the application evolved, we regularly introduced improvements to its retrieval pipeline:

• Document chunking strategies
• Semantic and keyword-based hybrid search
• Context-window expansion
• Document reranking
• Improvements to answer generation

However, we quickly encountered a problem. After implementing a change, it was difficult to determine whether the chatbot had actually become better. A modification could improve answers for certain questions while making others worse. Testing a few examples manually wasn't enough to understand the overall impact. We needed a repeatable evaluation process that could measure retrieval quality, compare different implementations, and identify regressions.

Building a Representative Q&A Benchmark

I worked on developing an evaluation dataset covering the different types of questions engineers might ask in their daily work. The objective was to create a benchmark representative of our internal knowledge base, including all ingested tools and most of the available documentation. We identified several categories of questions to evaluate different retrieval capabilities :

• Single-Document Questions Questions where all the information required to answer is available in a single document. These scenarios help evaluate whether the retrieval system can identify and return the correct source.
• Cross-Document Questions Questions requiring information from multiple documents. For example, a user might need to combine configuration instructions from one application with requirements documented in another. These scenarios evaluate whether the system can retrieve complementary information from different sources.
• Simple and Explicit Questions Straightforward requests where the answer is explicitly documented. These questions establish a baseline for evaluating the system's ability to retrieve clear, factual information.
• Complex Reasoning and Information Composition More advanced requests requiring information to be combined across different technical subjects. For example: "How can I develop a Python script that integrates Tool A with Tool B?" The documentation may describe each tool separately, but the assistant needs to retrieve relevant information about both to construct a useful answer.
• Unanswerable and Near-Match Questions We also included questions for which the knowledge base does not contain a valid answer. Some scenarios contain no relevant information, while others are intentionally more challenging: similar documentation exists, but it doesn't actually apply to the user's situation.

These cases are particularly important because a RAG system should not treat loosely related information as sufficient evidence.

Improving Benchmark Quality with Real-World Questions

One of the most interesting lessons came from our initial approach to generating evaluation questions. At first, we used AI to generate questions directly from the documentation. This was convenient because it allowed us to quickly produce a large number of questions with known source documents.

However, we discovered an important limitation. AI-generated questions often used terminology and sentence structures that were too similar to the original documentation. This made them relatively easy for the retrieval system to answer, producing results that weren't necessarily representative of real user interactions.

Real engineers don't always use the terminology found in technical documentation. They describe symptoms, use abbreviations, omit context, or explain problems in their own words. To make the benchmark more realistic, we introduced three complementary approaches.

• Using Historical ServiceNow Incidents We collected real questions written by engineers in historical support incidents. These provided authentic examples of how users describe technical problems. We then checked whether the information required to answer each question was available in our knowledge base, ensuring that questions intended to be answerable were suitable for evaluation.
• Generating Multiple Question Variations For each question, we used AI to generate five alternative formulations with different communication styles. This helped introduce more linguistic diversity, including variations in wording, technical precision, and how users describe their needs. The objective was to ensure that retrieval quality did not depend too heavily on the exact phrasing of a question.
• Involving Internal Application Experts We also asked internal tool teams to contribute and review questions. Their expertise helped us include realistic technical scenarios and validate whether the benchmark reflected actual engineering needs.

By combining these three approaches, we created a more challenging and representative evaluation dataset.

Supporting Continuous Improvement

The evaluation framework became an important part of how we approached RAG development. Combined with demonstrations every two weeks and direct feedback from CAD engineers, it provided a more structured way to improve the assistant.

User feedback helped us identify problems and new requirements, while benchmark results helped us measure the impact of technical changes. This brought together two complementary perspectives:

• Qualitative feedback: Understanding whether the assistant is genuinely useful to engineers.
• Quantitative evaluation: Measuring whether retrieval performance improves consistently across a representative dataset.

I was the only developer on this project. Meaning I've done everything including :

• Developing a Python framework to handle the QnA testplans, complexity, and quality scoring,
• Generating questions through AI based on documents and common incidents.

• Developed a repeatable benchmarking approach that produces retrieval-quality scores, enabling objective comparisons between different implementations.
• Enabled regression detection when changes to retrieval strategies,
• Created a reusable quality-assurance foundation allowing to easily evolve the question sets.