Overview
As Full Stack Engineer on a team of 2, I built a production-ready RAG (Retrieval-Augmented Generation) AI assistant that businesses can embed into their websites. The system ingests custom knowledge bases and provides contextually accurate responses while supporting multiple LLM providers for flexibility and cost optimization.
The Challenge
Create an intelligent, context-aware AI assistant that can be embedded into any website with domain-specific knowledge.
- Generic chatbots couldn't answer domain-specific questions accurately
- Existing solutions required significant technical expertise to integrate
- No easy way to switch between AI providers based on cost/performance needs
- Real-time streaming responses were essential for good UX but complex to implement
Technical Approach
- Implemented vector embeddings using OpenAI's ada-002 for semantic search
- Built chunking pipeline with overlap to maintain context across document segments
- Created abstraction layer supporting OpenAI GPT-4 and Google Gemini interchangeably
- Designed embeddable widget using Shadow DOM for style isolation
Key Decisions
PostgreSQL with pgvector over dedicated vector DB
Reduced infrastructure complexity while maintaining acceptable performance for our scale
Server-Sent Events for streaming
Better browser compatibility than WebSockets for unidirectional real-time data
Admin panel for knowledge management
Non-technical users needed to update FAQs and documentation without developer involvement
Results
- 80% reduction in support ticket volume for pilot customers
- Sub-200ms retrieval latency for knowledge base queries
- Seamless integration requiring only a script tag to embed
- Cost flexibility allowing 40% reduction by switching models for simple queries
Lessons Learned
- Chunk size and overlap significantly impact retrieval quality - requires experimentation
- Prompt engineering is as important as the retrieval system itself
- Users expect instant responses - streaming is not optional for chat interfaces