Valentius Kryptix

By Samruddhi Shirgave, Intern at Valentius Kryptix

Large language models are impressive, but they only know what they saw during training. Ask one about your own report, your company notes or a private document, and it will either admit it doesn’t know or, worse, produce a confident answer with nothing behind it. This is often called a hallucination. During my internship at Valentius Kryptix, I built a Retrieval-Augmented Generation (RAG) system to reduce exactly this problem. In this post I explain what RAG is, how I built it with Python, Chroma and Gemini, what surprised me, and where it still falls short.

The Frustrating Problem: Hallucinations

An LLM generates text that sounds right, whether or not it has the facts. When the facts are missing, it may fill the gap with an answer that is fluent but unsupported. That is a serious risk for anyone who wants to trust a chatbot with real documents. RAG helps reduce this by giving the model the relevant text from your own documents and instructing it to answer only from that context. It does not eliminate the problem, but it gives the model something real to work from.

What Is RAG? A Powerful, Simple Idea

RAG combines two steps. First, a retrieval step searches your own documents for the passages most relevant to a question. Second, a generation step gives those passages to the language model together with the question, so the answer is grounded in your content instead of only in the model’s training data. The model is not retrained; it simply gets the right context at the right time.

The Documents I Used

I ingested two real documents, about 9,000 words in total: my own prompt engineering report, and a detailed set of notes on planning agribusiness data analysis and forecast evaluation. Using real content mattered, because it let me ask questions that only these files can answer.

Step 1: Chunking Without Losing Meaning

Documents are too long to send to a model whole, and single sentences lose meaning. I split the text into chunks of about 2,000 characters (roughly 500 tokens) with 300 characters of overlap between neighbouring chunks. I also preferred to cut at paragraph or sentence boundaries, so a chunk rarely ends in the middle of a thought. This produced 40 chunks. I printed a few of them to check that they read sensibly before moving on.

Step 2: Embeddings and the Vector Database

Each chunk was converted into an embedding, a list of numbers that captures its meaning, using Gemini’s embedding model. I stored the embeddings, the text and the source file name in Chroma, a free vector database that runs locally. I used cosine distance, so chunks with similar meaning end up close together.

Step 3: Retrieval, Where Most Problems Hide

When a user asks a question, the question is embedded the same way and Chroma returns the top 4 closest chunks. Before connecting any generation, I printed the retrieved chunks to confirm they were actually relevant. This was one of the most useful habits of the project, because retrieval is where most RAG problems hide.

Step 4: Grounded Generation

The retrieved chunks and the question are placed into a prompt that tells the model to answer using only the given context and to say it doesn’t know if the answer is missing. This instruction helps reduce made-up answers.

The Test: Plain LLM vs RAG

I asked: “In my prompt engineering report, what was the schema match rate for V1?”

Without RAG, the model replied that it has no access to my files and cannot know the answer. With RAG, the top three retrieved chunks came from my report, one of them containing the exact result, and the model answered “1/6”, which matches the report.

[Insert screenshot here: the with and without RAG outputs from ask.py. Make sure no API key is visible.]

This is one test, so it shows that the pipeline works rather than proving it is perfect.

Surprising Lessons I Learned

1. Chunking decisions shape everything. Too large and the chunk carries noise, too small and it loses context. Overlap and sensible boundaries helped.
2. Always inspect retrieval before generation. In my test, the fourth retrieved chunk came from an unrelated part of the notes. The model still answered correctly, but it showed that top-k retrieval is not perfect.
3. Prompt instructions matter. Telling the model to answer only from the context makes its behaviour easier to trust and to test.

Honest Limitations and Next Steps

My system is small: two documents, one run per question, and a fixed top-4 retrieval. Next I would try re-ranking the retrieved chunks, test different chunk sizes, and measure retrieval quality on a larger set of questions. Showing the source file for each answer would also make it easier to verify.

Conclusion

RAG is a practical way to make a language model useful on private or recent information without retraining it. Building the pipeline from scratch, instead of hiding it behind a framework, helped me understand each piece: chunking, embeddings, retrieval and grounded generation.

The full code and the with-vs-without-RAG comparison are on GitHub: https://github.com/samruddhishirgave-droid/rag-chatbotot

Outputs: