Yes, keeping ChatGPT or another cloud API for general tasks while self-hosting a model for sensitive data is a common and often sensible hybrid strategy, letting an organization get the newest frontier model capability where data sensitivity allows it and full control where it does not. The typical setup routes requests based on data classification, either through user training and policy, where employees are told which tool to use for which task, or through a technical gateway that inspects or tags requests and automatically directs anything touching customer data, financials or source code to the self-hosted model while general writing or research questions go to the cloud API. This avoids the cost and complexity of self-hosting everything while still closing the biggest privacy gap, since the highest-risk data never reaches a third-party vendor. It does require clear policy and, ideally, some technical enforcement rather than relying purely on employee judgment, since shadow AI use tends to creep in if the sanctioned self-hosted tool is noticeably slower or less capable than the cloud option employees are used to. Over time many organizations shift the boundary as their self-hosted model quality improves. Nanobase AI designs these hybrid routing architectures so sensitive workloads move to self-hosted infrastructure without disrupting general AI use.
Getting the best of both without paying for either extreme
Running everything through a cloud API accepts data exposure risk on every task; self-hosting everything means paying the full fixed cost and accepting a capability gap even for low-stakes general tasks. A hybrid approach routes requests by data classification, general writing and research questions go to a cloud API for the newest frontier capability, while anything touching customer data, financials or source code goes to a self-hosted model, closing the biggest privacy gap without the cost of self-hosting every workload.
Two ways to implement the routing decision
Organizations generally choose between two mechanisms for enforcing the classification boundary, differing mainly in how much they depend on individual judgment.
| Approach | How it works | Strength | Weakness |
|---|---|---|---|
| Policy-based routing | Employees are trained on which tool to use for which task type | Low engineering cost, fast to implement | Relies entirely on individual judgment and compliance |
| Technical gateway routing | A gateway inspects or tags requests and automatically directs sensitive ones to the self-hosted model | Enforced consistently, does not depend on individual judgment | Requires building and maintaining the classification and routing logic |
A sample data classification map
A concrete classification map turns an abstract policy into something employees can apply consistently without asking each time.
- Public or general knowledge questions with no company-specific data: cloud API.
- Internal writing, brainstorming and research without customer or financial specifics: cloud API, generally acceptable.
- Any prompt referencing a specific customer, contract or financial figure: self-hosted model, enforced by policy at minimum.
- Source code, credentials or proprietary technical detail: self-hosted model, ideally enforced by a technical gateway.
- Regulated data under HIPAA, GDPR special categories, or similar frameworks: self-hosted model, with no cloud API exception regardless of task type.
Why shadow AI risk does not disappear in a hybrid model
A hybrid setup only works if the self-hosted option is genuinely good enough that employees do not quietly bypass the policy when the sanctioned tool feels slower or less capable, which is the same shadow AI risk that shows up in any self-hosted deployment. If routing depends purely on employee judgment rather than a technical gateway, organizations should expect some leakage at the boundary, particularly under deadline pressure when someone reaches for whichever tool answers fastest rather than whichever tool the policy specifies.
How the boundary shifts over time
Many organizations set the classification boundary conservatively at first, routing more to the self-hosted model than strictly necessary while the internal tool proves itself, then loosen it as trust and self-hosted model quality both improve. Others do the opposite, starting with a narrow self-hosted scope for only the clearest sensitive-data cases and expanding it as usage patterns reveal more categories worth routing internally. Neither direction is wrong; what matters is revisiting the boundary deliberately rather than letting it drift unmanaged.
Frequently asked questions
Is policy-based routing enough, or do we need a technical gateway?
Policy alone can work for smaller organizations with strong AI usage training and a clear, well-communicated policy, but a technical gateway provides consistent enforcement that does not depend on every individual making the right judgment call every time.
Can the same chat interface serve both the cloud API and the self-hosted model?
Yes, many chat interfaces including Open WebUI support multiple backend models, letting users or an automated router select between a cloud API and a self-hosted model from the same familiar interface.
Does a hybrid setup cost less than full self-hosting?
Generally yes, since only a portion of total volume routes to the self-hosted infrastructure, reducing the GPU capacity needed compared to handling all traffic on-premise, while still avoiding full API costs for that same sensitive-data volume.
How do we handle a task that starts general but becomes sensitive mid-conversation?
The safest design routes an entire conversation to whichever tool matches its most sensitive turn, since retroactively protecting only part of a conversation after sensitive data has already reached a cloud API does not undo that exposure.
How Nanobase AI helps
Nanobase AI designs these hybrid routing architectures so sensitive workloads move to self-hosted infrastructure without disrupting general AI use, building either policy frameworks or technical gateways depending on what an organization's risk tolerance and existing tooling call for. This connects to the fundamentals in what is a self-hosted LLM and why companies choose one. Explore /solutions or check /faq for related questions.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.