APP-RAG: local document assistant

01 Problem and context

Querying a corpus of technical PDFs requires manual searching; keyword search fails with natural language questions, and generic LLMs hallucinate content not found in the material.

Question answering over closed document sets: PDF collections someone needs to query in plain language without the model filling in what the text does not say. The use case is material that cannot leave the machine (internal, technical or confidential documents), which rules out third-party APIs and forces the whole loop, from embedding to generation, to run on local models.

02 Solution

RAG pipeline built with LangChain (LCEL): PDF loading with PyPDF, recursive chunking (1000/200), FastEmbed embeddings (BAAI/bge-small-en-v1.5), and top-5 retrieval from ChromaDB. Generation runs on local LLMs via Ollama (llama3:8b, mistral, phi) using a prompt that grounds responses strictly in the retrieved context, cites source pages, and explicitly flags when information is not present in the documents.

1. 01Ingestion: PDFs loaded with PyPDFLoader, keeping each fragment's page number so it can be cited later.
2. 02Chunking: recursive splitting into 1,000-character chunks with 200 characters of overlap, so an idea cut in half still appears intact in the neighbouring chunk.
3. 03Indexing: embeddings generated locally with FastEmbed (BAAI/bge-small-en-v1.5) and persisted to ChromaDB on a mapped volume.
4. 04Retrieval: the retriever returns the five closest chunks to the query, formatted with their source page.
5. 05Constrained generation: a LangChain Expression Language chain assembles the prompt from the retrieved context and forces the model to answer from that alone.
6. 06Runtime control: model (mistral, llama3, phi) and temperature picked from the interface, defaulting to 0.2 to bias answers towards factuality.
7. 07Packaging: Docker Compose brings up Ollama and the application as separate containers with persistent volumes.

03 How it's built

Conversational assistant based on RAG (Retrieval-Augmented Generation) architecture to query a corpus of technical PDFs in natural language. It indexes documents using local embeddings (FastEmbed) in ChromaDB and answers using local LLMs served by Ollama, with a LangChain chain that restricts responses to the retrieved context. The entire solution (Streamlit app and Ollama server) is orchestrated with Docker Compose, featuring model and vector database persistence.

04 Ethical decisions

• ✦The prompt forces the model to abstain: if the answer is not in the retrieved context it has to say so, rather than papering over the gap with outside knowledge.
• ✦Every chunk reaches the model with its page number, so any claim can be checked against the original document.
• ✦Everything runs locally on Ollama: indexed documents never leave the machine.
• ✦Default temperature is set to 0.2, favouring factuality over fluency.

05 Highlights

• ✦End-to-end RAG pipeline built with LangChain (LCEL): PyPDF, recursive chunking (1000/200), FastEmbed embeddings, and top-5 retrieval from ChromaDB.
• ✦100% local and private execution: LLMs served via Ollama in a Docker container, with no external API dependencies.
• ✦Docker Compose orchestration of two services (Ollama and Streamlit app) with a dedicated network and persistent volumes.
• ✦Streamlit chat interface with dynamic model selection (llama3:8b, mistral, phi) and temperature control.
• ✦RAG prompt constrained to the retrieved context, with source page citations and explicit responses when information is not found in the documents.

06 Limitations

There is no dataset of its own. The system indexes whatever PDFs are dropped into the folder, so answer quality depends entirely on the corpus loaded. No formal retrieval evaluation was run, so the project reports no figures: the anti-hallucination constraint is a design choice, not a measured result. Two known limitations: chunking is fixed and was never tuned per document type, and the embedding model is English-optimised while the queries are in Spanish.