Arx Industries

Production RAG: Building Enterprise-Scale AI with Advanced Retrieval

Published on by James at Arx

Three months ago, we introduced our on-premise RAG MVP—a secure, local AI platform designed for organizations that refuse to compromise on data sovereignty. That system proved the concept: you can harness cutting-edge AI capabilities without sending a single byte to external cloud services.

Today, we're taking it several steps further.

What started as a three-user proof-of-concept has evolved into a production-grade enterprise platform. We've integrated advanced retrieval techniques, implemented sophisticated query optimization, deployed multi-model architecture, and built enterprise-scale infrastructure. This isn't just an incremental improvement—it's a fundamental transformation in capability and reliability.

From MVP to Production: What Changed

The original MVP demonstrated that local RAG was viable. This production system proves it's practical for real organizational deployment.

1. Advanced Retrieval Architecture

  • Hybrid Search: Combines dense vector embeddings with sparse BM25 keyword search using Reciprocal Rank Fusion
  • Neural Reranking: Cross-encoder models (ms-marco-MiniLM) for 15-30% accuracy improvements
  • Query Optimization: Automatic query expansion, decomposition, and intelligent routing
  • Contextual Compression: Reduces LLM context usage by 60-70% while maintaining relevance

2. Enterprise-Grade Vector Database

  • Weaviate Cluster: Multi-node distributed architecture supporting millions of documents
  • HNSW Indexing: Sub-100ms query latency at scale
  • Product Quantization: 30x memory reduction maintaining 95%+ accuracy
  • Real-Time Sync: Automatic incremental indexing

3. Multi-Model Inference System

  • Primary Models: Llama 3 70B and Mistral 8x7B with load balancing
  • vLLM Serving: 20x+ throughput improvements via continuous batching
  • 4-bit Quantization: GPTQ/AWQ maintaining 99%+ accuracy
  • Specialized Models: Task-specific routing (code, summarization, analysis)

4. Production Infrastructure

  • 50+ Concurrent Users: Organization-wide deployment capability
  • High Availability: Redundant components with automatic failover
  • GPU Clusters: NVIDIA A100/H100 with tensor parallelism
  • Distributed Caching: 95% latency reduction for repeated queries

Hybrid Search: The Accuracy Game-Changer

Pure vector search excels at semantic understanding but misses exact matches. Keyword search catches specific terms but misses concepts. Hybrid search delivers both.

How It Works

  1. Dense Retrieval: E5-large-v2 embeddings search vector database semantically
  2. Sparse Retrieval: BM25 algorithm searches inverted index for keywords
  3. Score Fusion: Reciprocal Rank Fusion combines rankings
  4. Reranking: Cross-encoder evaluates top candidates in context
"Hybrid search with reranking delivers 15-30% accuracy improvements over vector-only approaches—critical for technical documentation and compliance queries."

Query Optimization Pipeline

Real users ask complex, multi-part questions. Our system handles this sophistication automatically.

Query Decomposition

Complex questions like "Compare Q4 2024 security incidents to Q4 2023 and summarize remediation strategies" decompose into:

  1. Retrieve Q4 2024 security incidents
  2. Retrieve Q4 2023 security incidents
  3. Retrieve remediation documentation
  4. Synthesize comparison and summary

Intent Classification & Routing

  • Factual Queries: High precision hybrid search
  • Exploratory Questions: Broader semantic search
  • Code Queries: Syntax-aware retrieval
  • Time-Sensitive: Recency-weighted search

Technical Architecture

Production RAG Pipeline

graph TB subgraph "User Layer" USER[User Query] end subgraph "Processing" INTENT[Intent Classifier] OPT[Query Optimizer] end subgraph "Retrieval" DENSE[Dense: E5-large] SPARSE[Sparse: BM25] FUSION[RRF Fusion] RERANK[Reranker] end subgraph "Storage" WEAVIATE[(Weaviate
HNSW+PQ)] INDEX[(Elasticsearch)] end subgraph "Generation" VLLM[vLLM] LLAMA[Llama 3 70B] MISTRAL[Mistral 8x7B] end USER --> INTENT INTENT --> OPT OPT --> DENSE OPT --> SPARSE DENSE --> WEAVIATE SPARSE --> INDEX WEAVIATE --> FUSION INDEX --> FUSION FUSION --> RERANK RERANK --> VLLM VLLM --> LLAMA VLLM --> MISTRAL LLAMA --> USER MISTRAL --> USER style USER fill:#E94560,stroke:#16213E,stroke-width:3px,color:#fff style WEAVIATE fill:#533483,stroke:#16213E,stroke-width:3px,color:#fff style LLAMA fill:#16213E,stroke:#533483,stroke-width:3px,color:#fff

Hybrid Search with Reranking

sequenceDiagram participant U as User participant Q as Optimizer participant D as Dense participant S as Sparse participant R as Reranker participant L as LLM U->>Q: Complex Query Q->>Q: Expand & Decompose par Parallel Retrieval Q->>D: Vector Search D-->>R: Top 50 Semantic and Q->>S: Keyword Search S-->>R: Top 50 Keywords end R->>R: RRF Fusion R->>R: Cross-Encoder Scoring R-->>L: Top 5 Documents L->>L: Generate Response L-->>U: Answer + Citations

Performance Metrics

Retrieval Accuracy

  • Precision@5: 87% (up from 64%)
  • Recall@20: 92% (up from 78%)
  • MRR: 0.83 (up from 0.67)

Query Performance

  • End-to-End: 1.8s average
  • Retrieval: 180ms for hybrid + reranking
  • Generation: 85 tokens/second
  • Cache Hit Rate: 42%

Scale

  • Concurrent Users: 50+ simultaneous
  • Knowledge Base: 2.5M documents indexed
  • GPU Utilization: 78% average
  • Cost per Query: $0.03 vs $0.25+ cloud

Technical Specifications

LLM Infrastructure

  • Models: Llama 3 70B (GPTQ 4-bit), Mistral 8x7B (AWQ 4-bit)
  • Serving: vLLM with continuous batching
  • Hardware: 4x A100 80GB or 2x H100 per instance

Vector & Search

  • Vector Store: Weaviate 1.26+ cluster
  • Index: HNSW with Product Quantization
  • Keyword Search: Elasticsearch 8.x BM25
  • Embeddings: E5-large-v2 (1024D)
  • Reranker: ms-marco-MiniLM cross-encoder

Infrastructure

  • Cache: Redis 7.x cluster
  • Load Balancer: HAProxy with health checks
  • Monitoring: Prometheus + Grafana
  • Logging: ELK stack for audit trails

Security & Compliance

graph TB subgraph Perimeter["🔒 SECURE PERIMETER"] subgraph App["APPLICATION"] UI[Web Interface] AUTH[RBAC/SSO] end subgraph Data["DATA"] VDB[(Weaviate)] DOCS[(Documents)] end subgraph AI["AI LAYER"] LLM[Local Models] EMB[Embeddings] end subgraph Security["SECURITY"] ENCRYPT[AES-256] AUDIT[Audit Logs] end end subgraph Internet["❌ NO INTERNET ACCESS"] CLOUD[Cloud APIs] end UI --> AUTH AUTH --> LLM LLM --> VDB ENCRYPT -.Protects.-> VDB ENCRYPT -.Protects.-> DOCS AUDIT -.Monitors.-> UI Internet -.X BLOCKED X.-> Perimeter style CLOUD fill:#ff4444,stroke:#cc0000,stroke-width:3px style LLM fill:#16213E,stroke:#533483,stroke-width:3px,color:#fff

Compliance Features

  • Data Residency: 100% on-premise processing
  • Access Control: RBAC with SSO integration
  • Encryption: AES-256 at rest, TLS 1.3 in transit
  • Audit Trail: Complete query and access logging
  • Multi-Tenancy: Isolated namespaces per department

Real-World Impact

This production system transforms how organizations leverage their knowledge:

  • C-Suite Intelligence: Instant access to insights across all organizational data
  • Technical Teams: Code search, documentation lookup, debugging assistance
  • Compliance: Policy queries, regulatory research, audit preparation
  • Operations: Incident analysis, runbook access, troubleshooting guides

Deployment Options

graph LR subgraph Standard["STANDARD"] S1[Single Cluster
4x A100
50 Users] end subgraph Enterprise["ENTERPRISE"] E1[Multi-Cluster
8x A100
200+ Users] end subgraph HA["HIGH AVAILABILITY"] H1[Geo-Distributed
12x H100
Mission Critical] end style S1 fill:#533483,stroke:#16213E,stroke-width:3px,color:#fff style E1 fill:#16213E,stroke:#533483,stroke-width:3px,color:#fff style H1 fill:#E94560,stroke:#16213E,stroke-width:3px,color:#fff

Cost Analysis: 3-Year TCO

graph LR subgraph Cloud["CLOUD AI"] C1[Year 1: $180k] C2[Year 2: $225k] C3[Year 3: $270k] CT[Total: $675k
+ Data Risk] C1-->C2-->C3-->CT end subgraph OnPrem["ON-PREMISE"] O1[Year 1: $150k] O2[Year 2: $25k] O3[Year 3: $25k] OT[Total: $200k
+ Full Control] O1-->O2-->O3-->OT end CT-.vs.->OT style CT fill:#ff4444,stroke:#cc0000,stroke-width:3px,color:#fff style OT fill:#00aa00,stroke:#008800,stroke-width:3px,color:#fff

What's Next

We continue pushing the boundaries of what's possible with local AI:

  • GraphRAG integration for relationship-aware retrieval
  • Agentic workflows with tool use and multi-step reasoning
  • Fine-tuning pipeline for domain-specific optimization
  • Multi-modal support (images, diagrams, audio)
  • Active learning for continuous improvement

The future of enterprise AI is local, secure, and under your control.

Technical Documentation

Comprehensive implementation guides and architecture documentation:

➡️ Download Architecture Guide (PDF) - Coming Soon ⬅️

Complete technical architecture including deployment patterns, optimization strategies, and scaling guidelines.

➡️ Download Hybrid Search Guide (PDF) - Coming Soon ⬅️

Step-by-step implementation of hybrid search, reranking, and query optimization techniques.

Questions about production RAG deployment? Connect via our contact page or follow updates through the newsletter.