Building a Retrieval-Augmented Generation (RAG) Application with MCP, Gemini, and ChromaDB in Node.js
Building a Retrieval-Augmented Generation (RAG) Application with MCP, Gemini, and ChromaDB in Node.js
Introduction
Large Language Models (LLMs) such as Gemini, GPT, and Claude are powerful, but they have one important limitation—they only know what they were trained on and the information provided in the prompt.
Suppose your company has an internal document containing leave policies, HR rules, medical policies, or confidential project documentation. If you ask an LLM a question about those documents, it may not know the answer because that information was never part of its training data.
This is where Retrieval-Augmented Generation (RAG) becomes useful.
Instead of training a new model, RAG retrieves the most relevant information from your own documents and supplies it to the LLM before generating the final answer.
In this project, we build a complete RAG application using:
Node.js
Google Gemini API
ChromaDB (Vector Database) npm install chromadb
Model Context Protocol (MCP)
By the end of this article, you will understand how these technologies work together to build an AI assistant capable of answering questions from private company documents.
Technologies Used
Node.js
Google Gemini 2.5 Flash
Gemini Embedding Model
ChromaDB
npm install chromadb
Docker
docker run -d --name chromadb -p 8000:8000 chromadb/chroma
Model Context Protocol (MCP)
JavaScript (ES Modules)
What is RAG?
RAG stands for Retrieval-Augmented Generation.
Instead of asking the LLM directly, we first search our documents for the most relevant information.
The retrieved information is then supplied to the LLM, allowing it to generate a more accurate answer.
The overall workflow is:
User Question
↓
Convert Question into Embedding
↓
Search Vector Database
↓
Retrieve Most Similar Chunks
↓
Send Retrieved Context + Question to LLM
↓
Generate Final Answer
Why Not Simply Ask Gemini?
Imagine asking:
How many annual leaves do employees receive?
Gemini has no knowledge of your company's internal leave policy.
However, if we first retrieve:
Employees are entitled to 20 annual leaves.
and provide this as context, Gemini can answer correctly.
This is the core idea behind RAG.
Understanding Embeddings
Computers cannot understand natural language directly.
Therefore, every sentence is converted into a high-dimensional vector called an embedding.
Example:
Vacation
↓
[0.0021, -0.0043, 0.0135, ...]
Leave
↓
[0.0018, -0.0039, 0.0128, ...]
Although the numbers look random, semantically similar sentences produce vectors that are close together.
The Gemini Embedding Model returns vectors containing 3072 dimensions.
Measuring Similarity
After converting text into vectors, we need a way to compare them.
The most common method is Cosine Similarity.
The cosine similarity value ranges from:
1 → identical meaning
0 → unrelated
-1 → opposite
This allows us to find the document chunk that is closest to the user's question.
Why Do We Chunk Documents?
Large documents cannot be embedded efficiently as one huge block.
Instead, we split them into smaller sections.
Example:
Company Leave Policy
Employees are entitled to 20 annual leaves.
Employees may work from home twice a week.
Medical leave requires a doctor's certificate.
Each chunk receives its own embedding.
Introducing ChromaDB
Storing thousands of vectors inside arrays is impractical.
Instead, we use a Vector Database.
We selected ChromaDB because it is:
Open source
Lightweight
Easy to use
Docker friendly
Ideal for learning RAG
Each record stored inside ChromaDB contains:
Document ID
Original Text
Embedding Vector
Optional Metadata
Loading Documents into ChromaDB
The document loading process performs these steps:
Read the document.
Split into chunks.
Generate embeddings.
Store text and embeddings inside ChromaDB.
This is an offline process.
It runs only when documents change.
Searching ChromaDB
When a user asks:
How many vacation days do employees get?
The application:
Generates an embedding for the question.
Searches ChromaDB.
Finds the nearest vectors.
Returns the matching document.
Example:
Retrieved Context
Employees are entitled to 20 annual leaves.
This retrieval happens in milliseconds even with thousands of documents.
Introducing Model Context Protocol (MCP)
Model Context Protocol (MCP) is a standard way for LLMs to communicate with external tools.
Instead of giving the LLM direct access to databases or APIs, MCP exposes functionality as tools.
Example tools:
Search Documents
Get Current Time
Weather
Database Lookup
Calendar
Email
The LLM decides which tool should be used.
MCP Architecture
Application
↓
MCP Client
↓
MCP Server
↓
Registered Tool
↓
Vector Database
↓
Gemini
↓
Final Answer
The application never directly searches the vector database.
Instead, it calls an MCP tool.
Our MCP Tools
We implemented two tools.
getCurrentTime
Returns the system time.
searchDocuments
Accepts a user query.
Generates an embedding.
Searches ChromaDB.
Returns the most relevant document chunk.
Complete RAG Flow
Our application performs the following sequence.
Step 1
User asks:
How many vacation days do employees get?
↓
Step 2
Gemini decides to call:
searchDocuments
↓
Step 3
The MCP server receives the query.
↓
Step 4
The query is converted into an embedding.
↓
Step 5
ChromaDB performs vector similarity search.
↓
Step 6
The best matching chunk is returned.
↓
Employees are entitled to 20 annual leaves.
↓
Step 7
Gemini receives:
Question
Retrieved Context
↓
Step 8
Gemini generates the final response.
Employees are entitled to 20 vacation days annually.
Why Use ChromaDB Instead of Arrays?
Initially we stored embeddings inside a JavaScript array.
While this worked for learning, it has serious limitations.
Arrays:
Lost when the application stops.
Slow for large datasets.
Cannot efficiently search millions of vectors.
ChromaDB provides:
Persistent storage.
Fast similarity search.
Metadata support.
Scalable indexing.
Project Structure
project/
├── app.js
├── server.js
├── load-documents.js
├── search-documents.js
├── company-policy.txt
├── package.json
└── .env
Lessons Learned
While building this project, I learned that:
Embeddings represent semantic meaning rather than keywords.
Cosine similarity measures how close two vectors are.
Vector databases store embeddings efficiently.
RAG is retrieval plus generation—not model training.
MCP separates AI reasoning from tool execution.
ChromaDB can replace manual vector arrays.
Docker makes running infrastructure extremely simple.
Conclusion
Building this project helped me understand that modern AI applications are much more than simply calling an LLM API.
A production-quality AI assistant combines several components:
LLM
Embedding Model
Vector Database
Retrieval Pipeline
MCP Tools
Prompt Engineering
Together, these components form a scalable Retrieval-Augmented Generation system capable of answering questions using private company knowledge.
Rather than memorizing concepts, implementing each step—from embeddings and cosine similarity to ChromaDB integration and MCP tools—provides a much deeper understanding of how real-world AI systems are built.
This project serves as a strong foundation for the next stage of learning: Python, LangChain, AI Agents, and production-grade RAG architectures.
Comments
Post a Comment