Valentius Kryptix

Introduction
Large Language Models (LLMs) present incredible general intelligence and conversational capabilities, but when asked about proprietary internal data, company protocols, or niche domain documentation, standard LLMs tend to have hallucinate confident but inaccurate responses. “How can we get an LLM to reliably answer questions using our private documents without fine-tuning model weights?” This is one of the most critical issues faced in contemporary artificial intelligence and data science. As part of my internship at Valentius Kryptix, I worked on a practically motivated AI engineering task to build a Grounded Retrieval-Augmented Generation (RAG) System from scratch in Python to build a system which would retrieve relevant passages from local documents and inject these into the prompt context of an LLM, ensuring strict adherence to proprietary non-parametric knowledge. Additionally to the end of learning how to interpret how these core components operate, rather than merely relying on a third-party framework wrapper to implicitly manage the workflow, I built the pipeline from scratch to better understand how these components work under the hood:

• Ingesting custom multiple-page text and markdown documents
• Implementing sliding-window token chunking
• Storing dense vector embeddings in a local vector database (Chroma DB)
• Performing top-k Cosine vector similarity retrieval
• Engineering grounded system prompts using the Google Gemini API (gemini-2.5-flash)
• Benchmarking ungrounded baseline LLM calls against our grounded RAG system.

What is Retrieval-Augmented Generation (RAG)?
Retrieval-Augmented Generation (RAG) is an architectural pattern in Generative AI that allows an LLM to fetch relevant context from an external data source before generating any response.

Parametric vs. Non-Parametric Memory
In order to understand why RAG is necessary, let’s discuss the difference between two distinct types of memory systems:

• Parametric Memory : Knowledge baked directly into the neural network weights of an LLM during initial training. It is static, hard to update, and prone to hallucinations on specialized knowledge.
• Non-Parametric Memory : External context retrieved from dynamic databases and injected into the model’s prompt at runtime.
  By grounding answers in non-parametric memory, we can ensure that if a question couldn’t be answered using the provided document context, the model will explicitly refuse to answer rather than fabricating details.

Pipeline Architecture & Implementation
Here is how the end-to-end pipeline was constructed over the course of 6 key stages.

1. Document Ingestion & Recursive Token Chunking
   Document chunking divides large texts into smaller, semantically consistent blocks. Splitting text strictly by character count or single sentences results in fragmented context. To ensure semantic integrity, I employed a sliding-window chunking strategy using tiktoken targeting 500 tokens per chunk with a 75 token overlap (~15%). This overlap ensures key entities split over sentence boundaries are not lost during vector mapping.
2. Local Vector Database Storage (Chroma DB)
   Each text chunk must be converted into a continuous high-dimensional vector representation. I integrated Chroma DB, a local persistent vector database, utilizing the all-MiniLM-L6-v2 Sentence-Transformer model. Chroma DB organizes these vectors into Hierarchical Navigable Small World (HNSW) graph indices, enabling Approximate Nearest Neighbor (ANN) search with O(log N)query speed.
3. Vector Context Retrieval
   When processing user queries, the system converts the question into an embedded vector and computes Cosine distance against stored document vectors in Chroma DB to return the top-k most relevant context chunks.
4. Grounded LLM Generation with Gemini API
   The retrieved context passages are formatted into a strict system prompt and passed to the Google Gemini API (gemini-2.5-flash). Setting temperature=0.0 forces deterministic token decoding, minimizing stochastic variation and maximizing adherence to context boundaries.

Baseline vs. Grounded RAG Comparison
In order to verify that the system effectively grounds answers in custom documents rather than relying on general pre-training data, I evaluated the pipeline using a side-by-side comparison of the following:

Benchmark Execution Output:
When tested on proprietary document content, the plaintext LLM generated broad parametric statements, whereas the grounded RAG pipeline produced exact definitions matching local context documents.

What Did This Experiment Teach Me?
Building this project gave me several really critical engineering lessons which have helped inform my understanding of Generative AI workflows:
1. Preprocessing Dictates Retrieval Quality: Naive chunking destroys document relationships. Fine-tuning chunk boundaries and managing token overlaps directly impacts lretrieval accuracy.
2. Context Precision Over Context Volume: Passing too many context chunks can confuse an LLM. Retrieving top-3 dense passages consistently yields more accurate 0answers than passing 20 unranked passages.
3. Deterministic Guardrails Are Critical: Setting temperature=0.0 and explicitly directing the model to acknowledge missing information is essential for preventing non- parametric hallucinations.

Conclusion
Building a production RAG system requires more than simply calling an API. It requires clear data ingestion, dense vector search, and strict context grounding. By integrating local vector indexing using Chroma DB and the Google Gemini API, this project demonstrates how proprietary document sets can be queried accurately without retraining or relying on hallucinations. The complete codebase, including the ingestion logic, chunking algorithms, Chroma storage setup, and comparison benchmarks are publicly available on GitHub.

GitHub Repository:https://github.com/pasiketansaig5genai-blip/Rag-System