01 / The Challenge
Building a high-throughput, multi-tenant AI SaaS platform serving chatbot workflows across various tenants without cross-tenant data leakage or performance bottlenecks.
As the document corpus grew to over 5 million vectors, the standard cosine similarity search began to bottleneck. The goal was to reduce retrieval time to under 500ms while maintaining, or even improving, context accuracy for the LLM.
This required a complete re-architecture of the ingestion pipeline and the implementation of a hybrid search strategy.
02 / Architecture
The new architecture decoupled ingestion from processing, utilizing Redis queues to handle spikes in document uploads without blocking the main application thread.
03 / The Solution
Implementing an event-driven architecture using Apache Kafka, Debezium CDC, and Transactional Outbox combined with a hybrid search strategy.
async def query_vector_store(query: str, filters: dict = None) -> List[Document]:
# Generate sparse vector for hybrid search
sparse_vec = splade_encoder.encode_documents([query])[0]
# Execute parallel search execution
results = await pinecone_index.query( vector=dense_embedding, sparse_vector=sparse_vec, top_k=10, include_metadata=True, filter=filters)
return [_parse_match(m) for m in results.matches]04 / Outcome
The system now handles concurrent user queries with a p99 latency of 420ms. The introduction of hybrid search (keyword + semantic) improved retrieval accuracy scores by 14% on the internal benchmark dataset.