Retrieval-Augmented Generation (RAG) has become the industry-standard pattern for building AI applications that reason over private data, internal documentation, logs, and enterprise knowledge bases.
Many RAG architectures introduce a dedicated vector database alongside a traditional database. While that approach works well, it also increases operational complexity.
An alternative is to use Elasticsearch as both the primary document store and the vector store.
In this guide, we'll build a production-oriented RAG architecture using:
- Elasticsearch for document and vector storage
- LangChain for orchestration
- Ollama for local embedding generation
The result is a unified architecture that supports semantic search, metadata filtering, and enterprise-scale retrieval.
Why Use Elasticsearch for RAG?
A common RAG architecture looks like this:
Application
│
├── PostgreSQL (metadata)
│
└── Vector Database
(embeddings)
While functional, this introduces:
- Additional infrastructure
- Synchronization challenges
- Multiple query paths
- Increased operational overhead
Elasticsearch allows both structured documents and vector embeddings to live together.
Application
│
▼
Elasticsearch
├── Documents
├── Metadata
└── Vectors
This unified model offers several advantages.
Unified Storage
Store:
- Raw text
- Metadata
- Vector embeddings
inside a single system.
No duplication or synchronization between databases is required.
Advanced Filtering
Elasticsearch excels at structured querying.
You can combine:
- Semantic search
- Keyword search
- User permissions
- Categories
- Timestamps
within the same query.
Scalability
Elasticsearch is designed for:
- Horizontal scaling
- High availability
- Distributed indexing
- Enterprise workloads
This makes it a strong fit for production RAG deployments.
Hybrid Search
One of Elasticsearch's strongest features is the ability to combine:
- Traditional keyword search (BM25)
- Dense vector similarity search
This often produces better retrieval quality than vector search alone.
High-Level Architecture
The workflow follows a straightforward path from raw documents to retrieved context.
Data Source
│
▼
Embedding Model
(Ollama)
│
▼
Elasticsearch
(Vector Store)
│
▼
Retriever
│
▼
Relevant Documents
│
▼
LLM
Each layer has a distinct responsibility.
Data Source
Documents can originate from:
- JSON files
- PDFs
- Databases
- Internal APIs
- Knowledge bases
For this example, we'll use structured JSON data.
Embedding Layer
The embedding model converts text into dense vector representations.
These vectors capture semantic meaning and allow similarity comparisons.
Elasticsearch Vector Store
Elasticsearch stores:
- Text content
- Metadata
- Vector embeddings
and performs similarity searches across indexed vectors.
Query Layer
When a user submits a query:
- The query is embedded.
- Elasticsearch performs vector similarity search.
- Optional metadata filters are applied.
- Relevant documents are returned.
Technical Stack
The implementation uses:
- Python
- LangChain
- Elasticsearch
- Ollama
The embedding model runs locally, avoiding external API costs.
Configuring Ollama Embeddings
We'll use the qwen3-embedding:4b model.
from langchain_community.embeddings import OllamaEmbeddings
embeddings = OllamaEmbeddings(
base_url="http://localhost:11434",
model="qwen3-embedding:4b"
)
Benefits of local embeddings include:
- No API fees
- Improved privacy
- Reduced latency
- Full control over infrastructure
Designing the RAG Class
A clean implementation encapsulates storage and retrieval logic inside a dedicated class.
from typing import List
from langchain_elasticsearch import ElasticsearchStore
class TermmtrixRag:
def __init__(self):
self.vector_store = self.load_es_store()
self.docs: List = []
self.doc_ids: List = []
def load_es_store(self):
return ElasticsearchStore(
index_name="termtrix",
embedding=embeddings,
es_url="http://localhost:9200/",
es_password="your_password",
es_user="elastic"
)
This class becomes the central interface for ingestion and retrieval.
Document Ingestion and Embedding
The next step is indexing documents.
Each document is converted into a LangChain Document object.
This preserves metadata alongside the text.
from langchain_core.documents import Document
Example Ingestion Logic
def ingest_data(self, data_list):
for item in data_list:
self.docs.append(
Document(
page_content=item["text"],
metadata=item["metadata"]
)
)
self.doc_ids.append(item["id"])
self.vector_store.add_documents(
documents=self.docs,
ids=self.doc_ids
)
Once inserted:
- Embeddings are generated
- Vectors are stored
- Metadata remains attached
- Documents become searchable
Why This Ingestion Pattern Works
There are several practical advantages.
One-Time Embedding Cost
Embeddings are generated once during indexing.
Retrieval becomes significantly cheaper because vectors already exist.
Index Once
│
▼
Generate Embeddings
│
▼
Store Vectors
Metadata Persistence
Metadata remains attached to every document.
Examples include:
{
"source": "tech_knowledge_base",
"author": "Engineering",
"created_at": "2025-01-01"
}
This becomes extremely valuable during retrieval.
Semantic Search and Filtering
Once documents are indexed, users can search using natural language.
Instead of matching exact keywords:
"How do I reset my API key?"
the retriever finds semantically related content even if those exact words don't appear.
Filtered Similarity Search
In production systems, unrestricted vector search is rarely sufficient.
You often need constraints such as:
- Department
- Tenant
- User permissions
- Document type
- Date ranges
Elasticsearch makes this straightforward.
def similarity_search(self, query):
results = (
self.vector_store.similarity_search_with_score(
query=query,
k=1,
filter=[
{
"term": {
"metadata.source":
"tech_knowledge_base"
}
}
],
)
)
for doc, score in results:
print(
f"* [Score={score:3f}] "
f"{doc.page_content} "
f"[{doc.metadata}]"
)
This ensures retrieval remains both relevant and secure.
Why Metadata Filtering Matters
Imagine a multi-tenant SaaS platform.
Without filtering:
Tenant A
│
▼
Could retrieve
Tenant B documents ❌
With metadata filters:
Tenant A
│
▼
Only Tenant A data ✅
This is one of the most important requirements for production RAG systems.
Practical Use Cases
This Elasticsearch-based architecture works particularly well for enterprise workloads.
Log Analysis
Store logs with metadata such as:
- Service name
- Environment
- Timestamp
Then perform semantic searches for:
Why are checkout requests failing?
instead of manually searching error codes.
Internal Knowledge Bases
Build assistants that retrieve information from:
- Engineering documentation
- HR policies
- Legal guidelines
- Customer support content
while respecting access controls.
Documentation Search
Retrieve the most relevant version of documentation based on:
- Product version
- Release date
- Category
before generating responses.
Limitations and Considerations
While Elasticsearch is a strong choice, there are tradeoffs.
Memory Usage
Vector search can become memory-intensive.
Factors include:
- Number of documents
- Embedding dimensions
- Index settings
- Query volume
Plan cluster sizing accordingly.
Re-Indexing Requirements
Embeddings are model-dependent.
If you switch from:
qwen3-embedding:4b
to another model:
nomic-embed-text
all documents must be re-embedded and re-indexed.
The old vectors are no longer comparable to the new ones.
Retrieval Quality Depends on Chunking
Even the best vector database cannot compensate for poor chunking.
Pay attention to:
- Chunk size
- Overlap strategy
- Metadata design
These choices directly impact retrieval quality.
Taking It Further: Adding an LLM
At this point, Elasticsearch acts as an intelligent retrieval layer.
The next step is connecting retrieval to a language model.
User Question
│
▼
Retriever
│
▼
Relevant Documents
│
▼
LLM
│
▼
Final Answer
This transforms semantic search into a fully conversational assistant.
Conclusion
Using Elasticsearch as both a document store and vector store simplifies RAG architectures while preserving enterprise-grade capabilities.
Instead of maintaining separate systems for:
- Documents
- Metadata
- Embeddings
everything lives in a single platform.
Combined with:
- LangChain for orchestration
- Ollama for local embeddings
- Elasticsearch for retrieval
you can build scalable, privacy-focused RAG applications that are ready for real-world production workloads.
Whether you're creating internal assistants, documentation search tools, or log-analysis systems, this architecture provides a powerful foundation.
Key Takeaways
✅ Elasticsearch can serve as both a document database and vector store.
✅ Metadata filtering is essential for production-grade RAG.
✅ Local embedding models through Ollama provide privacy and cost advantages.
✅ LangChain simplifies indexing and retrieval workflows.
✅ Hybrid search combines keyword and semantic retrieval for improved results.
✅ Re-indexing is required whenever embedding models change.
✅ A well-designed retrieval layer is the foundation of an effective AI assistant.