Skip to content

Software Development8 min read

Custom RAG Development Cost: What Drives the Budget

Custom RAG development cost depends less on the AI model than on your documents, permissions and quality bar. Here is what drives the budget and how to scope it.

By NUMVOX team · Published

Short answer: Custom RAG development cost is driven mainly by the number and messiness of your data sources, permission rules, the accuracy you need, the user interface and integrations, and how you evaluate and monitor answers. Running costs come from model usage, embeddings, vector search and hosting. Start with a narrow pilot on one document set to get a reliable estimate.

What a RAG system is, briefly

Retrieval-augmented generation, usually shortened to RAG, is a way of making a large language model answer questions using your own information. When a user asks a question, the system searches your documents for the most relevant passages and gives them to the model along with the question. The model writes an answer based on those passages and can cite where each point came from.

The approach was described in a 2020 research paper and has become the standard way to build assistants over company knowledge, because it avoids retraining a model and lets answers reflect documents as they change.

A production RAG system is more than a model call. It needs pipelines to ingest and update documents, a search index, permission checks, a user interface, logging, evaluation and monitoring. Each of these contributes to cost.

The components that make up custom RAG development cost

Main building blocks of a RAG system
ComponentWhat it involvesWhy it affects cost
Data ingestionConnectors to file stores, wikis, ticketing systems or databases; parsing PDFs, scans and tablesEvery source and format adds work; scanned and inconsistent documents add the most
Chunking and enrichmentSplitting documents into passages, adding titles, dates and metadataGood chunking improves answers but needs testing against your content
Embeddings and indexConverting passages to vectors and storing them in a search indexVolume and update frequency drive build effort and running cost
RetrievalVector, keyword or hybrid search, filtering and re-rankingHigher accuracy targets need more tuning
PermissionsMaking sure users only see answers from documents they may accessComplex permission models are often the hardest part
GenerationPrompts, citations, refusal behaviour and formattingCareful prompt design and guardrails take iteration
Interface and integrationChat UI, embedding in existing tools, single sign-onEach integration point adds development and testing
Evaluation and monitoringTest question sets, quality scoring, feedback capture, loggingEssential for trust, and often underestimated

Factors that raise the budget

  • Many data sources, each with its own connector, format and update schedule.
  • Scanned documents, handwriting or complex tables that need OCR and careful parsing.
  • Document-level or row-level permissions that must be enforced on every answer.
  • High accuracy requirements, for example in legal, financial or medical contexts.
  • Multiple languages in the source documents or the user questions.
  • Integration into several existing tools rather than a single interface.
  • Requirements to keep data in a particular region or on your own infrastructure.
  • Audit trails showing exactly which sources supported each answer.

Factors that keep the budget down

  • Starting with one well-organised document set, such as a product manual or policy library.
  • Clean, digital, text-based documents with consistent structure.
  • A single user group with the same access rights.
  • Using a managed model API and managed vector search rather than self-hosting.
  • A simple web interface or an integration with one existing tool.
  • A clear, agreed list of questions the system should and should not answer.

Build options and their trade-offs

Many organisations start with a framework-based build on a managed model API, then move parts in-house only if volume or regulation justifies it.

Common approaches to building RAG
ApproachBest forTrade-off
Off-the-shelf assistant connected to your filesQuick internal trials with standard sourcesLimited control over retrieval, permissions and answer style
Framework-based custom buildMost business cases needing control over sources and behaviourRequires engineering and evaluation effort
Fully custom with self-hosted modelsStrict data residency or very high volumesHighest build and operating complexity

Running costs after launch

A RAG system has ongoing costs that a traditional application does not. Each question uses model tokens for the retrieved passages, the question and the answer, so cost grows with usage and with how much context you send. Embeddings must be generated when documents are added or changed. Vector search and storage are billed by the provider or need your own infrastructure.

There are also people costs: reviewing feedback, updating prompts, adding sources, maintaining connectors when source systems change, and re-running evaluations when you change models.

Running costs can be controlled by sending only the most relevant passages, caching answers to common questions, choosing a smaller model where quality allows and setting usage limits. Model prices change frequently, so estimates should be based on current provider pricing and your expected question volume.

A phased approach to control cost and risk

The pilot is the most valuable step for budgeting. It shows how messy the data really is, how hard permissions are and what accuracy is achievable, which turns a rough range into a reliable estimate for the rest.

Typical phases for a custom RAG project
PhaseGoalOutput
DiscoveryAgree users, questions, sources, permissions and success measuresScope, test question set, architecture outline
PilotProve answer quality on one source with real usersWorking assistant, evaluation results, cost per question
Production buildHarden security, permissions, monitoring and integrationsLaunch-ready system with runbooks
ExpansionAdd sources, users and features based on evidencePrioritised roadmap

Why evaluation belongs in the budget

A RAG system that sounds confident but is sometimes wrong can do more harm than having no assistant at all. Evaluation is how you find out before users do.

Build a set of real questions with known good answers and sources, ideally written with the people who will use the system. Score the system against it whenever you change chunking, retrieval, prompts or models. Track whether answers are correct, whether they cite the right sources and whether the system declines to answer when it should.

After launch, collect user feedback on each answer and review a sample regularly. Budget for this as an ongoing activity, not a one-off test.

Security and privacy considerations

RAG systems introduce specific risks. Prompt injection hidden in documents can try to change the model's behaviour. Weak permission checks can leak confidential content to users who should not see it. Sensitive data may be sent to a model provider. The OWASP Top 10 for Large Language Model Applications is a useful checklist.

Controls include enforcing permissions at retrieval time, filtering and logging inputs and outputs, reviewing the provider's data retention and training terms, and keeping secrets out of prompts. These are part of a sound build and should be in the estimate from the start.

Costs that are easy to miss

Several costs are routinely left out of early RAG budgets. Document clean-up is the most common: outdated, duplicated and contradictory documents produce poor answers, and someone has to decide which version is correct. That is business work, not engineering, but it takes real time.

Connector maintenance is another. When a source system changes its API or structure, ingestion can break silently, so the system needs monitoring and someone responsible for fixing it.

Model changes also cost time. Providers update and retire models, and each change should be followed by a re-run of the evaluation set and, sometimes, prompt adjustments. Finally, user adoption needs effort: training, clear guidance on what the assistant is for and a visible way to report wrong answers.

Questions to answer before asking for a quote

  • Who will use the system and what questions should it answer?
  • Which sources does it need, in which formats, and how often do they change?
  • Do different users have different access rights?
  • What happens when the answer is wrong, and how accurate does it need to be?
  • Where must data be stored and processed?
  • Which tools should it integrate with, and does it need single sign-on?
  • How many questions do you expect per day at launch and in a year?

How NUMVOX scopes custom RAG projects

We start with a short discovery to agree the users, questions, sources and permissions, and to build a test question set with your team. We then propose a pilot on one source with clear success measures and an estimate of running cost per question. The production phase is quoted after the pilot, when the real complexity is known.

Frequently asked questions

Why is it hard to give a fixed price for RAG development?

Because the biggest cost drivers are hidden in your data and requirements: how messy the documents are, how complex permissions are and how accurate answers must be. These only become clear when real documents are processed and tested. A short discovery and pilot turns uncertainty into evidence, which is why most reliable quotes come after that stage.

Is it cheaper to fine-tune a model instead of building RAG?

Usually not for knowledge questions. Fine-tuning changes how a model behaves but is a poor way to teach it facts that change, and it must be repeated when documents change. RAG lets answers reflect your current documents and cite sources. Fine-tuning can complement RAG for style or format, but it rarely replaces retrieval.

What are the ongoing costs of a RAG system?

Ongoing costs include model usage for each question, embeddings for new or changed documents, vector search and storage, hosting, and monitoring tools. There is also people time for reviewing feedback, maintaining connectors and re-running evaluations. Controlling how much context is sent, caching common answers and choosing an appropriately sized model all help manage these costs.

Can a RAG system respect our existing document permissions?

Yes, but it must be designed in. Permissions should be checked at retrieval time so that only passages a user may see are given to the model. This often means syncing access rules from the source systems and testing them carefully. It is one of the more complex parts of a build and should be scoped explicitly.

Do we need a vector database for RAG?

You need some form of search index, but not necessarily a separate vector database. Many teams use vector search features in databases they already run, such as PostgreSQL with a vector extension, or a managed search service. Hybrid search, combining keyword and vector matching, often gives better results than either alone. Choose based on volume, existing infrastructure and operational skills.

How long does a RAG pilot take?

A focused pilot on one clean document set with one user group can often be completed in a few weeks, including discovery, building, evaluation and user feedback. Pilots with several sources, scanned documents or complex permissions take longer. The pilot should end with measured answer quality and an estimated running cost, not just a demo.

Sources

Planning a project like this?

Send a short brief and the NUMVOX team will reply by email with questions and a suggested next step.