If you’re an established SaaS adding AI to a product that already has customers, the architecture choices you make in the next quarter will shape the next two years.
Most SaaS companies aren’t asking whether to add AI anymore. They’re asking how to add it without breaking the product that their customers already pay for. That’s a different conversation, and it’s the one we have most often now.
The good news is that the patterns have stabilized. The first generation of AI features – single-prompt LLM wrappers, basic chat interfaces, a “summarize this” button bolted onto an existing page – taught the industry what doesn’t scale and what does. What’s working in production today looks meaningfully different from what was working two years ago, and the gap between SaaS companies that have absorbed those lessons and the ones that haven’t is becoming visible to customers.
Here’s what we see working when an existing SaaS adds AI to its stack.
What the architecture looks like now
The default pattern is a dedicated AI service that sits beside your main application rather than inside it. Your existing API communicates with this service via a standard internal contract. The service handles model calls, retrieval, tool use, prompt templates, caching, and observability. None of that lives in your core business logic.
There are good reasons for this beyond cleanliness. AI components change faster than the rest of your codebase. You’ll swap models, change prompts, add retrieval, introduce agent loops, and change vendors when pricing shifts. Keeping all of that behind a stable internal interface means the rest of your team doesn’t have to worry about it.
Inside the service, the patterns we use repeatedly:
A model abstraction layer like LiteLLM or Portkey, so you’re not married to one provider. Frontier model pricing and capability shift every quarter, and a hard dependency on a single vendor is a strategic liability.
A retrieval layer is used when answers need to be grounded in customer data. For most SaaS use cases, this means pgvector or a managed vector store (Pinecone, Weaviate, Turbopuffer), populated from your existing database via an indexing pipeline. The retrieval design matters more than the embedding model – what you index, how you chunk it, how you scope per-tenant, what metadata you attach.
Structured outputs and function calling for anything that has to integrate back with your product. JSON schema validation on every response. If the model can return free-form prose where structured data was expected, it will eventually.
An orchestration framework – LangGraph for anything with branching or state, plain LangChain or direct SDK calls for simple sequential flows. The choice isn’t aesthetic. Pick the wrong one, and you’ll be fighting the framework six weeks in.
MCP (Model Context Protocol) is used when AI features need to interact with the customer’s other tools. This is one of the bigger shifts of the last year. Instead of writing brittle, bespoke integrations for every third-party system your customers care about, you expose tools through MCP and let the agent decide which to call. It changes the build-vs-integrate math significantly.
Streaming responses from day one. The UX of a non-streamed AI response feels broken to customers now. This is a frontend-and-backend decision that’s painful to retrofit.
Observability built in – LangSmith, Langfuse, or OpenLLMetry. You can’t debug AI features with traditional logs. You need traces of the actual prompts, retrievals, tool calls, and intermediate decisions. If you ship without this, you’ll be flying blind when customer reports come in, and they will.
The data question SaaS founders consistently underestimate
The architecture above is the easy part. The harder questions are about your data.
If you’re multi-tenant, your retrieval layer has to enforce tenant isolation at the index level, not just at the application layer. One leaked cross-tenant result is a serious incident. We’ve seen teams discover this the hard way three months into a deployment.
If you serve enterprise customers, you’ll get the question about where their data goes. Some will be fine with the answer “we send it to OpenAI under their zero-retention agreement.” Many will not. Plan for at least one of: a self-hosted model option, Azure OpenAI with regional deployment, AWS Bedrock with a managed model, or bring-your-own-key support where the customer plugs in their own model endpoint. None of these is hard to add up front. All of them are extremely hard to retrofit if you didn’t plan for them.
If you’re in a regulated industry, audit trails on every AI decision aren’t optional. Which model, which prompt version, what retrieval context, what the customer saw, what they did with it. That’s what the observability tooling is actually for.
Where AI actually moves the needle in SaaS
Not every part of your product benefits from AI. The patterns we see working consistently in SaaS:
Search and discovery. Semantic search over the customer’s content nearly always beats keyword search, and the implementation is a weekend of work for a meaningful uplift.
Summarization of anything long. Call transcripts, document threads, support tickets, and activity histories. This is the safest and most trusted AI feature in most products.
Drafting and rewriting. The model produces a starting point, and the user edits. The user stays in control, the trust model is intact, and the productivity gain is real.
Classification, triage, and tagging. Assigning categories, priorities, sentiment, and routing. Cheap, fast, and often replacing a manual or rule-based workflow.
Workflow automation in the gaps. The place AI is most underused in SaaS is the connective tissue between existing features. Pulling structured data from an attachment that a customer just uploaded. Prefilling fields based on context elsewhere in the system. Suggesting the next action based on what similar customers did.
Where AI doesn’t pay off in SaaS, in our experience: anywhere accuracy has to be 100% with no human checkpoint; anywhere your latency budget is under 200ms and your value depends on hitting it cheaply; anywhere the data sensitivity is high, and you don’t have a governance story ready.
Build versus buy, at the right layer
The instinct of many SaaS teams is to either go too low (try to train their own model) or too high (wrap a chat interface around an external API and call it a product). The right layer for most companies is in the middle.
Don’t build the foundation model. Don’t try to compete with the orchestration frameworks. Do build the prompts and agent logic that encode what’s specific about your product and your customers. Do build the evaluation harness that knows what “good” looks like in your domain, because no off-the-shelf eval suite will. The evaluation harness is the most underrated investment we see – it’s what lets you ship changes without breaking production behavior.
Where this leaves things
If you’re a SaaS company that’s shipped a first wave of AI features and the value isn’t quite where you expected, the issue is usually architectural, not the model. The model is rarely the bottleneck. The retrieval design, the agent orchestration, the eval coverage, the data tenancy story – those are where most projects underperform.
If you’d like a second set of eyes on what you’ve shipped or what you’re about to ship, that’s the conversation we like having. Book a twenty-minute call!
