RAG (Retrieval-Augmented Generation) in the ERP Context

E-invoicing in Germany: How to implement the obligation with SAP Business One

Retrieval-Augmented Generation (RAG) is the currently dominant architectural pattern for providing a large language model with current, company-internal knowledge without retraining the model itself. A RAG pipeline fetches relevant text snippets from a vector database with each request and passes them to the LLM along with the user’s query as context. The model then generates the answer based on this context – rather than based on its static training knowledge.

Context

In the ERP environment, RAG is particularly valuable because company knowledge comes from heterogeneous sources: SAP B1 manuals, CoSIT specifications for e-invoicing, DATEV documentation, internal wikis, customer requirement specifications, Coresuite/VFS documentation, blog posts, support cases. A typical pipeline: (1) Source documents are converted into Portions disassembled, (2) mapped into a 768-dimensional vector space using an embedding model (e.g. gemini-embedding-001), (3) stored in a vector database (Supabase with pgvector, Pinecone, Weaviate). At runtime, the user's question is also converted into a vector, and the most similar chunks are retrieved. Cosine similarity retrieved and incorporated as context into the LLM prompt. Advanced RAG variants combine vector search with full-text search (hybrid search), use rerankers, or employ a multi-step approach.Agentic RAGIn products like B1-Helpster, RAG is combined with direct service layer connectivity, enabling the agent to answer conceptual questions as well as query live data.

Demarcation

RAG is not fine-tuning: it doesn't change the model but feeds it contextually. Compared to pure prompt engineering, RAG is dynamic and scales with the size of the knowledge base. Compared to classic search, it provides not just links but a synthesised answer – but carries the risk of the model misinterpreting the sources. Clean citations, traceable source references, and a clear „I don't know" when retrieval is empty are therefore mandatory components of any ERP RAG implementation.