Building a Document-Based RAG System with Sentence Transformers and ChromaDB

Introduction

Large language models can generate useful answers, but they may not know the information contained in a user’s own documents. Retrieval-Augmented Generation (RAG) addresses this limitation by retrieving relevant information from a document collection and providing it to a language model as context.

In this project, I built a document-based RAG system that combines document processing, semantic search, vector storage, retrieval, and language-model generation. The goal was to create an application that answers questions using information from a specific collection of documents.

Understanding Retrieval-Augmented Generation

RAG combines two processes: retrieval and generation. First, the system searches a document collection for passages relevant to a user’s question. Next, the retrieved passages are included in the prompt sent to a language model, which generates an answer using that context.

This is useful when a user needs answers grounded in a particular knowledge base instead of relying only on a model’s general training data. It also allows the document collection to be updated without retraining the language model.

Document Loading and Text Chunking

The first stage of the pipeline loads documents and extracts their text. This project supports formats such as PDF and Markdown. The extracted text is divided into smaller chunks so that retrieval can identify relevant passages instead of processing an entire document for every question.

The project uses approximately 220 tokens per chunk with a 45-token overlap. Overlap preserves context between neighboring chunks. The chunk size was selected with the embedding model’s input-length limit in mind, helping reduce the risk of truncating longer passages.

Chunking is an important design decision. Large chunks can contain unrelated information, while very small chunks may lose useful context. The right balance depends on the source documents, the embedding model, and the questions the system needs to answer.

Generating Embeddings with Sentence Transformers

After chunking, each passage is converted into a numerical representation called an embedding. The project uses Sentence Transformers with the all-MiniLM-L6-v2 model.

Embeddings represent semantic information in a vector space. This allows the system to find text based on meaning rather than relying only on exact keyword matches. The all-MiniLM-L6-v2 model produces 384-dimensional embeddings.

For example, a question can retrieve a relevant passage even when the question uses different words from the original document, provided their meanings are sufficiently similar.

Storing Embeddings in ChromaDB

The project uses ChromaDB as its vector database. It stores the embeddings alongside the associated text and metadata, such as source filenames and page information.

A persistent vector database allows the application to reuse stored embeddings rather than processing every document from scratch each time. When documents change, the ingestion pipeline can update the stored collection.

Metadata is also important because it helps the application identify the origin of retrieved passages and provide source references with generated answers.

Retrieving Relevant Information

When a user submits a question, the retrieval component searches the vector database for relevant chunks. These chunks are selected according to their similarity to the query.

The retrieved information becomes the context for the generation stage. Retrieval quality is important: if the selected passages are irrelevant or incomplete, the language model may not have enough evidence to answer accurately.

Generating Context-Grounded Answers

The application combines the user’s question with the retrieved passages and sends the resulting prompt to a language model. The model uses the supplied information to generate a response.

The system can display source information to help users identify which documents contributed to the answer. If the document collection does not contain enough information, the application should avoid presenting unsupported details as facts.

Testing and Evaluation

Testing helps ensure that the components work together correctly. The project includes tests for document loading, chunking, configuration, generation, ingestion, and retrieval.

A useful evaluation compares a RAG response with a plain language-model response that receives no retrieved document context. When a question asks about a detail found only in the local documents, the RAG system can retrieve the relevant passage, while the plain model may not have enough information to answer.

This comparison demonstrates the value of grounding generation in a document collection. The final result depends not only on the language model but also on the quality of the source documents and retrieved passages.

Challenges and Lessons Learned

Building the system highlighted the importance of document extraction, choosing an appropriate chunk size, generating consistent embeddings, and storing metadata correctly. Testing the individual components also made it easier to identify issues in the pipeline.

Another important lesson is that generating a fluent answer does not automatically make it correct. Relevant retrieval, source attribution, and handling missing information are all important for a dependable document-question-answering system.

Conclusion

This RAG project combines document loading, text chunking, Sentence Transformers, ChromaDB, semantic retrieval, and language-model generation in a practical application.

It provided hands-on experience with the major stages of a retrieval pipeline and demonstrated how a language model can answer questions using a dedicated document collection. The same approach can support applications such as technical-document assistants, internal knowledge search, and document-based question answering.

Project repository: https://github.com/kurellynandu-007/rag-system