Yes, RAG can work with data in SAP, Salesforce, and relational databases, but structured records need different handling than documents because a customer record or SAP transaction table does not read naturally as prose for embedding. The common approach converts structured rows into text summaries or key-value descriptions before embedding, for example turning a Salesforce opportunity record into a sentence describing the account, stage, and value, so it can be retrieved semantically alongside unstructured documents. A second, often more accurate approach uses the language model as an orchestrator that translates a natural-language question into a structured query, such as SQL against the database or an OData call against SAP, and returns exact figures rather than an approximate text match, which matters when the question needs a precise number rather than a paraphrase. Enterprise integrations to SAP and Salesforce typically go through MCP servers or dedicated APIs that respect existing row-level and object-level permissions, so retrieval never bypasses access controls already enforced in the source system. Combining both approaches, semantic search over descriptive text and structured querying for precise figures, generally produces the most reliable enterprise RAG system. Nanobase AI connects RAG pipelines directly to SAP, Salesforce, and enterprise databases through governed MCP integrations.
Structured data breaks the document-RAG assumption
Standard RAG assumes a corpus of prose that reads naturally once embedded, but a Salesforce opportunity record, a SAP transaction table, or a relational database row is not prose, it is structured fields with a schema behind them. Embedding a raw row as text and retrieving it like a document chunk works for simple lookup questions but performs poorly on aggregation or filtering questions, such as totaling a value across many records, because vector similarity search has no native concept of summing or filtering across rows the way SQL does.
Structured data needs a retrieval pattern matched to the question type, not the same chunk-and-embed approach used for documents.
Two patterns and when each fits
| Pattern | How it works | Best fit |
|---|---|---|
| Row-to-text embedding | Converts each record into a natural-language summary and embeds it like a document chunk | Lookup questions about a specific record, such as "what is the status of this opportunity" |
| Text-to-SQL | Translates the natural-language question into a SQL query executed directly against the database | Aggregation, filtering, or multi-record analytical questions, such as totals, counts, or trends |
| Semantic layer / metrics layer | A defined, pre-vetted set of business metrics the model can query through named functions rather than freeform SQL | Enterprise settings needing governance over which queries and metrics are allowed |
Most enterprise deployments over structured systems need both patterns, routed by question type, rather than a single approach applied to everything.
Building the row-to-text pattern for lookup questions
Converting a Salesforce record or SAP transaction into a coherent text summary, for example turning field values into a sentence describing the opportunity's stage, amount, and owner, makes it embeddable and retrievable the same way a document chunk is. This pattern works well when the question resembles "find the record matching this description" but does not extend to questions requiring computation across many records, since summing values described only in prose is not something similarity search can do.
Row-to-text embedding answers "find this record" questions well and answers "compute this across records" questions poorly, regardless of chunk quality.
Building the text-to-SQL pattern for analytical questions
Text-to-SQL translates a natural-language question into an executable query against the structured system directly, which correctly handles aggregation and filtering but introduces its own risks: a poorly constrained implementation can generate a query the underlying data model does not actually support, or one that runs against tables the user should not access. Constraining the model to a known, vetted schema, applying the same access-control principles used elsewhere in RAG to which tables and columns a query can touch, and validating generated SQL before execution are all necessary guardrails, not optional additions.
Text-to-SQL solves the aggregation problem row embedding cannot, but it needs schema constraints and access validation before a generated query ever runs.
Frequently asked questions
Can one system handle both document RAG and structured-data RAG together?
Yes, this is the common enterprise pattern: a query router determines whether a question is best answered from document retrieval, row-level lookup, or a text-to-SQL query, and directs it to the appropriate pattern rather than forcing every question through one path.
Is text-to-SQL reliable enough for production use?
Modern language models produce reasonably accurate SQL for well-defined schemas, particularly when the schema is documented clearly for the model, but production use still requires query validation and testing against real questions, since an incorrect aggregation can look plausible while being wrong.
Does connecting RAG to SAP or Salesforce require custom integration work?
Typically yes, through each platform's API, OData services for SAP or the Salesforce REST and Bulk APIs, to extract records and keep the row-to-text or SQL access layer current with source data changes.
How do we prevent a text-to-SQL query from accessing unauthorized data?
Constrain the model to a pre-approved, permission-scoped set of tables and columns per user or role, rather than exposing the full schema, and validate that generated queries only touch the approved scope before execution.
How Nanobase AI helps
Nanobase AI builds enterprise integrations that connect RAG systems to SAP, Salesforce, and relational databases through the appropriate pattern, row-level embedding for lookups, governed text-to-SQL for analytical questions, matched to the actual question types a client's users ask. See our enterprise integrations via MCP servers and APIs and the related long context vs RAG discussion for how this compares to stuffing raw records into a large context window.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.