Introduction

Retrieval-Augmented Generation (RAG) is an important approach for building AI applications that combine information retrieval with large language models. Instead of relying entirely on the knowledge learned during training, a RAG system retrieves relevant information from a knowledge source and uses it to provide context for a response.

However, building a RAG pipeline is only part of the challenge. We also need a reliable way to determine whether the retrieval component is returning useful information. This is where AI evaluation becomes essential.

As part of my internship work at Valentius Kryptix, I developed an AI Evaluation Framework focused on benchmarking and analysing RAG retrieval quality using Python. The project explores how evaluation metrics, controlled experiments, and reranking techniques can help us understand and improve retrieval behaviour.

Project Overview

The objective of this project was to build a reusable evaluation framework that measures retrieval performance against a benchmark dataset.

The framework includes benchmark validation, retrieval integration, metric calculation, automated report generation, and a cross-encoder reranking experiment. I created a benchmark containing 25 test cases covering topics represented in the project’s knowledge sources.

Each labelled test case includes a question and relevant chunk identifiers. This makes it possible to compare retrieved results against expected relevant content rather than relying only on subjective inspection.

The implementation uses Python and includes automated tests to validate important parts of the evaluation workflow.

Key Evaluation Metrics

To analyse retrieval quality, I implemented several metrics that measure different aspects of the results.

1. Precision@K

Precision@K measures the proportion of the top K retrieved chunks that are relevant.

A higher value indicates that a larger proportion of the retrieved context is relevant. This is particularly useful when we want to limit unnecessary information passed to a language model.

2. Recall@K

Recall@K measures how many of the relevant chunks were retrieved within the top K results.

Recall helps identify whether the retrieval system is missing useful information. Increasing the number of retrieved chunks can improve coverage, but it can also introduce less relevant content.

3. Mean Reciprocal Rank (MRR)

Mean Reciprocal Rank evaluates how highly the first relevant result appears in the ranked retrieval list.

It is useful when the position of the first relevant chunk matters. Returning a relevant result at rank one is generally preferable to returning it much further down the list.

4. Source Hit Rate and Duplicate Rate

Source hit rate measures whether retrieval results include content from an expected relevant source. Duplicate rate helps identify repeated source results within the retrieved list.

Together, these metrics provide a broader view of retrieval behaviour than any single metric can provide.

Experiment 1: Comparing Different Top-K Values

One of the key experiments compared retrieval configurations with K values of 3, 5, and 10 using the 25-case benchmark.

The measured results were:

Metric	K = 3	K = 5	K = 10
Precision@K	0.440	0.264	0.136
Recall@K	0.980	0.980	1.000
MRR	0.980	0.980	0.980

The results demonstrate an important trade-off. Retrieving fewer chunks produced higher precision, while retrieving ten chunks achieved slightly higher recall.

In this experiment, K = 3 was a promising configuration when prioritising concise context, whereas K = 10 achieved the highest measured recall. The appropriate setting depends on the requirements of the application rather than on a universal rule.

Experiment 2: Cross-Encoder Reranking

I also implemented an experiment using the cross-encoder/ms-marco-MiniLM-L-6-v2 model.

The workflow retrieves an initial set of ten candidate chunks, scores each question–chunk pair with the cross-encoder, and reranks the candidates before evaluating the final top three results.

The direct Top-3 baseline achieved Precision@3 of 0.44, Recall@3 of 0.98, and MRR of 0.98. The reranking experiment achieved Precision@3 of 0.44, Recall@3 of 0.98, and MRR of 1.00.

The results show that reranking improved the measured mean reciprocal rank without changing aggregate precision or recall in this experiment. This illustrates why it is important to evaluate multiple metrics: an improvement in ranking quality does not necessarily mean that every metric will improve.

These findings apply to the current benchmark and configuration. Further testing with additional questions and independently reviewed relevance labels would be needed before drawing broader conclusions.

Testing and Reliability

The project includes automated tests for benchmark validation, retrieval adapter behaviour, metric calculations, report handling, and evaluation logic.

The latest recorded test run completed with 33 passing tests. These tests help catch implementation errors and make the framework more reliable as it evolves.

The repository also includes the benchmark data, source documents, documentation, and experiment implementation so that the project can be inspected and extended.

Key Learnings

This project reinforced three important lessons.

First, evaluation metrics should be selected according to the problem being measured. Precision, recall, and ranking quality describe different aspects of retrieval performance.

Second, controlled experiments make trade-offs visible. Comparing different Top-K configurations provides more useful evidence than selecting a value based only on intuition.

Third, reranking should be evaluated empirically. A more sophisticated retrieval pipeline does not automatically improve every metric, so comparisons should use the same benchmark and clearly documented evaluation settings.

Conclusion

Building this AI Evaluation Framework gave me practical experience with RAG retrieval evaluation, benchmark design, information retrieval metrics, Python testing, and cross-encoder reranking.

The project demonstrates how a structured evaluation workflow can turn retrieval behaviour into measurable results and support more informed engineering decisions.

I look forward to extending this work with broader benchmarks, further retrieval experiments, and additional evaluation methods.

Explore the project on GitHub: https://github.com/Vedant-1724/ai-evaluation-framework