Case Study
VoiceDesk AI
Multi-tenant SaaS that turns a business’s support docs into a 24/7 AI voice agent
01Problem
Small businesses miss support calls around the clock, and generic voice bots hallucinate because they do not actually know the business. VoiceDesk gives a business a live AI phone agent grounded specifically in its own uploaded documents — answering customer calls 24/7 from real knowledge, not guesses.
02Constraints
- A live phone call is unforgiving: the whole retrieve-think-speak loop has to feel conversational, not laggy.
- Multi-tenant by design — every business’s documents and answers must stay strictly isolated from every other’s.
- Ingestion had to handle messy real-world files (PDF, docx, csv, txt) without dragging in multi-GB ML models the app does not need.
- Self-directed build — the product, the full backend and frontend, and the whole 14-service deployment were all mine.
03Approach
- Built the voice pipeline as a chain engineered around a sub-700ms turn-latency target: LiveKit Agent → Silero VAD → Deepgram STT (Nova-3) → Groq LLM (llama-3.3-70b) → Kokoro TTS.
- Made retrieval tenant-aware from the ground up: every vector query filters by both tenant_id and agent_id.
- Built async document ingestion: MinIO → text extraction (pdfminer.six / unstructured) → LlamaIndex chunking → Ollama embeddings (nomic-embed-text) → Qdrant upsert with per-tenant payload filters, run through Celery.
The scope decisions were about protecting the core promise — trustworthy, grounded answers on a live call — and cutting everything that did not serve it.
- Enforced monthly minute quotas before answering, so an org stops at its plan instead of silently running over.
- Made escalation a deliberate, bounded feature: a trigger phrase ends the call with a set message — explicitly no live transfer or hold queue, which would have added telephony complexity the MVP did not need.
- Kept ingestion deliberately lean — skipping unstructured’s heavy [pdf] extra (a multi-GB torch/onnxruntime download) because the app never needed it.
04Architecture
A FastAPI / PostgreSQL core with Redis, a per-tenant-isolated Qdrant vector store, MinIO object storage, Celery workers, and a Next.js 14 PWA — 14 services orchestrated with Docker Compose behind Nginx.
- ▸FastAPI + PostgreSQL — API and system of record
- ▸Qdrant — per-tenant isolated vector search (tenant_id + agent_id filters)
- ▸Redis — caching and coordination
- ▸MinIO — document object storage; Celery — async ingestion pipeline
- ▸LiveKit + Deepgram + Groq + Kokoro — the real-time voice pipeline
- ▸Next.js 14 PWA — tenant dashboard; 14-service Docker Compose stack behind Nginx
05Decisions & Tradeoffs
Moved RAG retrieval and TTS synthesis in-process inside the voice-agent worker.
↳Eliminates network hops during a live call for faster turn-taking, at the cost of a heavier, more tightly-coupled worker process.
Tightened VAD and endpointing defaults below the SDK’s stock values.
↳Faster turn-taking and less dead air, at the risk of clipping a slow speaker — a tuning tradeoff.
Enforced tenant isolation at the data layer, backed by a dedicated cross-tenant-access test.
↳More rigor per query and per test, but the isolation guarantee is verified rather than assumed — the README calls it the suite’s most important security test.
Documented both self-managed and Railway deploy paths (with an OpenAI-embeddings fallback where there is no host for Ollama).
↳More docs to maintain, but the stack is deployable by someone without the exact local setup.
06Outcome
Shipped
- A working multi-tenant voice-agent SaaS: upload docs, get a live AI agent that answers calls grounded in them.
- Verified tenant isolation (cross-tenant access test), monthly minute-quota enforcement, and configurable escalation.
- Async document ingestion across PDF / docx / csv / txt into per-tenant vector search.
- Production-hardening documented in depth: secret rotation, CORS / public-URL config, LiveKit NAT traversal, TLS termination, and a backup strategy across three stateful volumes.