Skip to content
FA Ferdous Azad

Build a Retrieval‑Augmented Generation (RAG) App with Open‑Source LLMs

Learn how to create a full‑stack Retrieval‑Augmented Generation app using open‑source LLMs, LangChain, ChromaDB, and FastAPI in a step‑by‑step tutorial.

Tutorials ragopen-source-llmtutorialpythonlangchainfastapi
Ferdous Azad posted
Contents
  1. Intro
  2. Prerequisites
  3. Step 1: Create a clean Python environment
  4. Step 2: Install core dependencies
  5. Step 3: Download the open‑source LLM
  6. Step 4: Prepare the document corpus and create embeddings
  7. Step 5: Build the LangChain RAG pipeline
  8. Step 6: Expose the RAG service with FastAPI
  9. Architecture diagram
  10. Common pitfalls
  11. Next steps
  12. FAQ
Raw

#Intro

In this tutorial we’ll build a simple Retrieval‑Augmented Generation (RAG) web service that answers user questions over a custom document collection using an open‑source LLM (Mistral‑7B‑Instruct). By the end you’ll have a FastAPI endpoint that:

  • Retrieves the most relevant chunks with ChromaDB.
  • Sends the retrieved context to the LLM via llama‑cpp.
  • Returns a concise answer to the caller.

#Prerequisites

  • Basic familiarity with Python 3.9+ and virtual environments.
  • A GPU with at least 8 GB VRAM (or CPU‑only fallback).
  • 10 GB of free disk space for model weights and embeddings.
  • Git installed.

Note

The steps work on Linux, macOS, and Windows Subsystem for Linux (WSL). Adjust paths accordingly.

#Step 1Create a clean Python environment

Bash
# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

#Step 2Install core dependencies

Bash
pip install --upgrade pip
pip install langchain==0.1.5 \
            chromadb==0.4.5 \
            fastapi==0.110.0 \
            uvicorn[standard]==0.27.0 \
            sentence-transformers==2.2.2 \
            huggingface-hub==0.23.0 \
            llama-cpp-python==0.2.68

Tip

llama-cpp-python provides a thin Python wrapper around the llama.cpp inference engine, which works on both CPU and GPU.

#Step 3Download the open‑source LLM

We’ll use the 7‑b instruction‑tuned Mistral model hosted on Hugging Face.

Bash
# Create a folder for model files
mkdir -p models/mistral-7b-instruct
cd models/mistral-7b-instruct

# Use huggingface-cli to download the ggml‑quantized file
pip install huggingface_hub
python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="TheBloke/Mistral-7B-Instruct-v0.1-GGML",
    revision="ggmlv3-q4_0",
    local_dir=".",
    allow_patterns=["*.bin"]
)
PY

The resulting ggml-model-q4_0.bin file is ~4 GB and ready for llama‑cpp.

#Step 4Prepare the document corpus and create embeddings

For the demo we’ll index a small set of markdown files located in data/.

Python
# embed_and_store.py
import os
from pathlib import Path

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
import chromadb
from chromadb.utils import embedding_functions

# 1️⃣ Load raw documents
docs_path = Path("data")
raw_texts = []
for file in docs_path.rglob("*.md"):
    raw_texts.append(file.read_text(encoding="utf-8"))

# 2️⃣ Split into chunks (≈ 500 tokens)
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", " ", ""],
)
chunks = splitter.split_text("\n".join(raw_texts))

# 3️⃣ Create embeddings with a sentence‑transformer model
embed_model = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
embedding_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

# 4️⃣ Initialise Chroma (persisted in ./chroma_db)
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    name="rag_collection",
    embedding_function=embedding_fn,
)

# 5️⃣ Add chunks to the vector store
ids = [f"id-{i}" for i in range(len(chunks))]
metadatas = [{"source": str(docs_path)} for _ in chunks]
collection.add(
    ids=ids,
    documents=chunks,
    metadatas=metadatas,
)
print(f"Indexed {len(chunks)} chunks.")

Run the script:

Bash
python embed_and_store.py

#Step 5Build the LangChain RAG pipeline

Python
# rag_chain.py
import os
from typing import List

from langchain.vectorstores import Chroma
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.llms import LlamaCpp
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# Initialize the vector store (same collection as Step 4)
embedding = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vector_store = Chroma(
    collection_name="rag_collection",
    embedding_function=embedding,
    persist_directory="./chroma_db",
)

# Load the LLM via llama‑cpp
llm = LlamaCpp(
    model_path="models/mistral-7b-instruct/ggml-model-q4_0.bin",
    n_ctx=2048,
    n_threads=os.cpu_count(),
    temperature=0.2,
    max_tokens=256,
)

# Prompt that forces the model to cite sources
prompt = PromptTemplate(
    input_variables=["context", "question"],
    template=(
        "You are a helpful assistant. Use only the provided context to answer the question.\n"
        "If you cannot answer, say you don't know.\n\n"
        "Context:\n{context}\n\nQuestion: {question}\nAnswer:"
    ),
)

# RetrievalQA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vector_store.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=True,
    chain_type_kwargs={"prompt": prompt},
)

def answer(question: str) -> dict:
    """Return answer and source docs."""
    result = qa_chain({"query": question})
    return {
        "answer": result["result"],
        "sources": [doc.metadata for doc in result["source_documents"]],
    }

#Step 6Expose the RAG service with FastAPI

Python
# main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

from rag_chain import answer

app = FastAPI(title="Open‑Source RAG Service")

class QueryRequest(BaseModel):
    question: str

class QueryResponse(BaseModel):
    answer: str
    sources: list

@app.post("/query", response_model=QueryResponse)
def query(req: QueryRequest):
    if not req.question.strip():
        raise HTTPException(status_code=400, detail="Question cannot be empty")
    result = answer(req.question)
    return QueryResponse(answer=result["answer"], sources=result["sources"])

Run the API:

Bash
uvicorn main:app --host 0.0.0.0 --port 8000

Visit http://localhost:8000/docs for an interactive Swagger UI.

#Architecture diagram

flowchart TD
    A["User (HTTP POST)"] --> B["FastAPI endpoint"]
    B --> C["RAG chain (LangChain)"]
    C --> D["Vector store (ChromaDB)"]
    D --> E["Embedding model (MiniLM)"]
    C --> F["LLM (llama‑cpp + Mistral‑7B)"]
    F --> G["Generated answer"]
    G --> B

#Common pitfalls

Symptom Likely cause Fix
OSError: libcuda.so not found Running GPU‑optimized llama‑cpp on a machine without CUDA drivers. Install the correct NVIDIA driver or add n_gpu_layers=0 to force CPU mode.
Retrieval returns unrelated chunks Text splitter created overly large chunks, causing loss of context. Reduce chunk_size to 300–400 tokens and increase chunk_overlap.
API latency > 5 s Model loaded on CPU with large n_ctx. Use a GPU, or switch to a smaller quantization (q8_0) if GPU memory is limited.
UnicodeDecodeError while reading markdown Files contain non‑UTF‑8 encoding. Open files with encoding="utf-8", errors="ignore" or convert to UTF‑8.

#Next steps

  1. Scale the vector store – switch to a distributed solution like Milvus or Pinecone for millions of documents.
  2. Add streaming – use FastAPI WebSockets and llama_cpp streaming callbacks to send partial tokens.
  3. Secure the endpoint – integrate OAuth2 or API keys, and rate‑limit with slowapi.
  4. Experiment with prompting – try the “refine” chain type or a custom RAG prompt that includes citations.

#FAQ

Q: Can I replace Mistral‑7B with another open‑source model?
A: Yes. Any GGML‑compatible model (e.g., Llama‑2‑7B, Gemma‑2B) works by changing the model_path and, if needed, adjusting n_ctx to match the model’s context window.

Q: Do I need a GPU to run this in production?
A: A GPU dramatically reduces latency, but the CPU‑only mode is functional for low‑traffic prototypes. For production, a single A100 or RTX 4090 is a common baseline.

Q: How do I keep the embeddings up‑to‑date when the source documents change?
A: Re‑run embed_and_store.py with the overwrite=True flag on the Chroma collection, or use Chroma’s incremental add/update methods to modify only changed chunks.

Was this useful?

Post Share Y Hacker News

Get new posts by email

One short email a week with what I wrote. No spam, unsubscribe in one click.

Drafted with AI assistance from the sources above, then reviewed and published by Ferdous Azad. Spotted an error? Let me know.

Comments

No comments yet. Start the conversation.

Leave a comment

First-time comments are reviewed before they appear. Be kind and on topic.

More in Tutorials

View all