“Self-hosted RAG” gets said a lot and specified rarely. If you are responsible for standing up an AI knowledge system inside your own perimeter — for reasons of data residency, compliance, or simple sovereignty — you need more than a diagram with an arrow labelled “magic.” This is a reference architecture for a private, secure retrieval-augmented generation platform, drawn from how Swifbit is built.
We will walk the whole stack: connectors, the ingestion pipeline, storage, retrieval, the model layer, and the security model that has to wrap all of it. Wherever there is a decision to make, we will name the trade-off.
Design goals
Before components, principles. A credible self-hosted RAG platform should satisfy five goals at once:
- Data stays put. Documents, embeddings, and prompts never leave the customer’s environment.
- Model independence. The same platform runs on a local model or a hosted endpoint, swappable without a rewrite.
- Permission-aware retrieval. Access control is enforced during search, not painted on afterwards.
- Operability. Sync health, failures, and quality signals are observable by the people who run it.
- Portability. It deploys on any VM or container platform, without a proprietary cloud dependency.
The high-level shape
At the coarsest level, the system is four planes: a connector plane that pulls documents in, an ingestion plane that turns them into searchable vectors, a retrieval and generation plane that answers questions, and a control plane for identity, permissions, and audit. Keeping these separated is what lets each scale and fail independently.
Connectors ─▶ Ingestion ─▶ Vector store ─▶ Retrieval ─▶ Model
(isolated) (chunk + (pgvector) (permission (local or
embed) filtered) hosted)
└──────────────── Control plane: authn, RBAC, audit ───────────┘The connector plane
Connectors are where most of the operational risk lives, because they hold credentials to other systems. The key architectural choice is isolation: each connector should run in its own process with its own scoped credentials, so a bug or a compromised integration cannot reach the rest of the platform or the other connectors’ secrets.
Connectors should also be incremental. Re-indexing an entire wiki every sync is wasteful and slow; a good connector tracks modification timestamps or change cursors and only processes what moved. Every sync should record what it did — documents added, updated, skipped, and any errors — so operators can see health at a glance.
Credential handling
Store connector credentials encrypted at rest with authenticated encryption, decrypt them only in the connector process that needs them, and never return them to any client. In Swifbit, secrets are masked in every API response by default — the UI shows that a credential exists without ever shipping it back to the browser.
The ingestion plane
Ingestion is a pipeline: normalise, chunk, embed, index. Each stage has a decision worth making deliberately.
Chunking strategy
Chunk on semantic boundaries — headings, paragraphs, list items — rather than blindly every N characters, and keep a small overlap so a passage that straddles a boundary is not lost. Attach metadata to every chunk: source document, section, timestamps, and the access scope it inherits. That metadata is what makes permission-aware retrieval and precise citations possible downstream.
Embeddings
The embedding model is a swappable dependency, and it should be treated as one. Record which model and version produced each vector, because changing embedding models means re-indexing — mixing vector spaces silently degrades retrieval. For a fully private deployment, run the embedding model locally; for higher throughput you can point at a hosted embedding endpoint you control.
Storage: why pgvector
You can bolt on a dedicated vector database, but for most self-hosted deployments a single PostgreSQL instance with the pgvector extension is the pragmatic choice. It keeps your documents, metadata, and vectors in one transactional store you already know how to back up, secure, and operate — no second system to run, and no eventual-consistency gaps between your metadata and your index.
| Concern | Dedicated vector DB | PostgreSQL + pgvector |
|---|---|---|
| Operational surface | Extra system to run | One database you already know |
| Transactional metadata | Separate store, sync risk | Same transaction as vectors |
| Backup / DR | Bespoke | Standard Postgres tooling |
| Scale ceiling | Very high | High — ample for most orgs |
Retrieval and generation
Retrieval takes the embedded query, runs a nearest-neighbour search, and — before anything else — filters candidates against the requesting user’s permissions. Only the surviving chunks are assembled into the prompt. This ordering matters: filtering after generation risks leaking restricted content into an answer even if you strip the citation.
Generation then runs under explicit constraints — answer only from context, cite sources, mark unverified facts. The output carries citations and a confidence signal derived from match strength. If retrieval returns nothing above a threshold, the honest response is “I don’t have that,” not a fabrication.
The model layer
Model independence is a first-class requirement, not a nice-to-have. The platform should talk to the model through a thin, uniform interface so that the same deployment can run against:
- a fully local model via Ollama, for maximum privacy and zero egress;
- an OpenAI-compatible endpoint hosted inside your own network;
- a managed endpoint such as Vertex AI when you have a contractual data boundary you trust.
Because the interface is uniform, switching providers is a configuration change, not a migration. That protects you from vendor lock-in and lets you match the model to the sensitivity of the workload.
The control plane: security and observability
Everything above is wrapped by identity, role-based access control, and an audit trail. Authentication gates the system; RBAC decides who can administer what; team-scoped source access decides whose knowledge each person can retrieve. The audit trail records questions, drafts, syncs, edits, and configuration changes — the raw material for accountability and compliance evidence.
A retrieval system without a permission model is a data-leak generator with excellent UX. Build the access model first.
Deployment
The whole stack should run from containers on any VM or orchestrator, with configuration through environment variables and secrets injected by your platform. A typical footprint is the application, a PostgreSQL instance with pgvector, and a model runtime — all inside one network boundary you control.
# minimal self-hosted footprint
services:
app: # retrieval, generation, control plane
postgres: # documents + metadata + pgvector index
ollama: # optional local model runtimeHybrid search: why vectors alone are not enough
Pure vector search is excellent at meaning and poor at specifics. Ask for an exact error code, a SKU, or a person’s name and semantic similarity can drift right past the literal match. The fix is hybrid retrieval: run a keyword (lexical) search alongside the vector search and fuse the results, so you get both conceptual recall and exact-match precision.
A common fusion approach is reciprocal rank fusion, which blends the two ranked lists without needing the scores to be on the same scale. The practical upshot: “how do I reset my password” still finds the right article, and “error E-4021” still finds the exact page that mentions it.
Failure modes, and how to design against them
A RAG system does not usually fail with an exception; it fails quietly, by returning a plausible answer built on the wrong context. These are the failure modes worth engineering against from day one.
Stale index
A document changes but its vectors do not, so retrieval serves yesterday’s truth. Defend against it with incremental sync tied to change cursors, and surface last-synced timestamps so operators can see freshness at a glance.
Embedding drift
Someone swaps the embedding model without re-indexing, and new queries are compared against vectors from a different space. Retrieval quality silently craters. Defend against it by stamping every vector with its model and version, and refusing to mix spaces.
Context overflow
Too many chunks are stuffed into the prompt, the important passage falls into the model’s “lost in the middle” blind spot, and the answer ignores it. Defend against it by ranking hard, capping the context budget, and preferring a few high-relevance chunks over many mediocre ones.
Permission leakage
The single most dangerous failure. A restricted document slips into context and its content — or even just its citation — reaches someone who should not see it. Defend against it by filtering at retrieval time, testing with restricted-user fixtures, and treating any leak as a release blocker.
Evaluation and monitoring
You cannot improve what you cannot measure, and “it feels good” is not a metric. Stand up a small evaluation harness before you scale usage.
- A golden question set. Twenty to fifty real questions with known-correct source documents. Re-run it on every change to catch retrieval regressions.
- Retrieval metrics. Track whether the correct source appears in the top-k results (recall@k) and how highly it ranks.
- Groundedness checks. Sample answers and confirm every claim traces to a cited passage.
- Operational signals. Watch low-confidence rates, negative feedback, sync failures, and stale sources — each is a maintenance to-do in disguise.
In Swifbit these signals are first-class: low-confidence answers, thumbs-down feedback, failed syncs, and ageing sources roll up into an admin view, so improving coverage becomes a routine operational loop rather than a research project.
Scaling considerations
Most organisations never approach the ceiling of this architecture, but it is worth knowing where the pressure points are. Embedding throughput at ingestion is usually the first bottleneck — batch it and run it off the request path. The vector index in Postgres benefits from an approximate index (such as HNSW) once you pass a few hundred thousand chunks, trading a sliver of recall for a large latency win. Generation is best decoupled behind a queue so a burst of questions degrades gracefully instead of timing out. None of this requires leaving your own infrastructure.
Putting it together
None of these pieces is exotic on its own. The engineering is in how they fit: isolated connectors, metadata-rich chunks, a single transactional store, hybrid permission-aware retrieval, a swappable model layer, and a control plane that makes the whole thing auditable. Get that composition right and you have a RAG platform you can actually put in front of regulated data.
That is precisely the architecture Swifbit ships. If you would like the deep-dive on any single layer, read the security model or book a technical walkthrough.
