Most enterprises need both: a small central platform team owning shared infrastructure, security guardrails and an approved tool list, combined with enough departmental autonomy to buy point solutions for needs the central team cannot address quickly, rather than choosing pure centralization or pure department autonomy as an absolute rule. Pure decentralization, every department buying its own AI tools independently, tends to produce duplicated spend, inconsistent data security practices, and integration chaos once departments need to share data or systems, problems that typically surface twelve to eighteen months in rather than immediately at rollout. Pure centralization, where every AI purchase or project must route through one team, tends to create a bottleneck that frustrates business units and pushes them toward shadow AI adoption to route around the central team entirely. A workable middle ground gives the central team ownership of shared infrastructure, security review and an approved tool list, while departments retain budget and authority to select from that list or request exceptions through a fast, lightweight review rather than a lengthy approval process. Revisit this balance annually as the number of AI initiatives and the maturity of the central team both change over time. Nanobase AI often serves as this platform team on a fractional basis for companies not yet large enough to justify a full internal function.
Neither extreme holds up at scale
Pure decentralization, every department buying and building its own AI tools independently, feels fast at first but tends to produce duplicated spend on overlapping tools, inconsistent data security practices across teams, and integration chaos once departments need to share data or systems, problems that typically surface twelve to eighteen months in rather than immediately at rollout. Pure centralization, where every AI purchase or project must route through one team, creates a bottleneck that frustrates business units waiting on a queue, which predictably pushes them toward shadow AI adoption to route around the central team entirely. Most enterprises need a hybrid: a small central team owning shared infrastructure and security guardrails, with departments retaining enough autonomy to move on their own within that framework.
Comparing the three operating models
Both pure extremes have a predictable failure mode, which is exactly why a federated middle ground tends to hold up better over time.
| Model | Speed | Consistency | Main failure mode |
|---|---|---|---|
| Fully decentralized | Fast per department | Low | Duplicated spend, inconsistent security, integration chaos later |
| Fully centralized | Slow | High | Bottleneck pushes departments toward shadow AI |
| Federated (hybrid) | Moderate | Moderate-high | Requires clear division of authority to avoid ambiguity |
What belongs to the central team versus the department
A workable federated model draws a clear line: the central team owns shared infrastructure (GPU capacity, the model and tool abstraction layer, monitoring dashboards), security review, and the approved tool list, while departments retain budget and authority to select from that approved list or request a fast, lightweight exception review rather than routing every decision through a lengthy central approval process. This division prevents both failure modes at once: departments are not blocked waiting on infrastructure decisions the central team should own, and the central team is not relitigating every department's tool choice as long as it falls within approved boundaries.
Signs the balance needs adjusting
- Departments are increasingly requesting exceptions to the approved list, suggesting the list no longer fits actual business needs.
- Shadow AI discovery keeps turning up the same category of unapproved tool across multiple departments, suggesting a gap in what's centrally offered.
- The central team's approval queue is growing faster than its throughput, suggesting either under-resourcing or overly broad centralized scope.
- Departments running similar use cases are building duplicate, incompatible solutions, suggesting the central team should be offering a shared pattern instead.
- Security or compliance incidents keep tracing back to tools acquired outside the approved process, suggesting the fast-exception path is either too slow or too unclear.
Revisiting the model as maturity grows
The right balance for a company running two or three AI use cases looks different from the right balance once twenty use cases are in production across a dozen departments, and treating the operating model as a fixed decision rather than revisiting it annually leads to a structure that no longer matches actual scale. This question overlaps closely with whether a Center of Excellence is needed in the first place, since a CoE is often exactly the vehicle a growing central function takes. Early on, a very light central function suffices; as the number of initiatives and their interdependencies grow, the central team's role in shared infrastructure and consistency typically needs to expand even if that means a somewhat slower approval process than a single department would prefer on its own.
Frequently asked questions
Do small companies need a central AI platform team at all?
Usually not a dedicated function; a single person or a fractional role covering the same responsibilities, an approved tool list, basic security review, is often enough until the company runs enough concurrent AI initiatives to justify more structure.
How big should a central AI platform team be relative to company size?
There is no fixed ratio, since it depends on the number and complexity of AI initiatives rather than headcount alone. Many companies start with a very small team, sometimes a single owner, and grow it as the number of production AI systems increases.
What's the fastest way to know our current model isn't working?
Watch for a rise in either shadow AI incidents (a sign centralization is too restrictive) or duplicated, inconsistent department-level tools (a sign decentralization has gone too far). Both signals point to the same fix: clarify which decisions belong centrally and which stay with departments.
How Nanobase AI helps
Nanobase AI often serves as this platform team on a fractional basis for companies not yet large enough to justify a full internal function, owning the shared infrastructure and approved tool list while leaving departments room to move within that framework.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.