On-Prem AI - are we repeating the cloud migration cycle in reverse?

For the last 15 years, enterprise technology followed a fairly predictable direction: get out of the data centre and move to the cloud. The journey usually looked something like this - on-prem infrastructure, then virtualisation, then cloud migration, then cloud-native, then SaaS and serverless, and finally optimisation.

Organisations spent years moving applications, databases, storage and infrastructure into AWS, Azure and Google Cloud. The arguments were compelling: stop buying servers, remove capacity planning, scale on demand, reduce operational overhead and convert infrastructure from large upfront capital expenditure into consumption-based spending.

Now AI is creating an interesting reversal. We are increasingly hearing: "Should we run AI on-prem?"

At first this sounds like moving backwards. It isn't. The same economic and architectural forces that pushed general-purpose computing into the cloud do not necessarily apply in exactly the same way to AI workloads.

The original cloud migration cycle

The early cloud migration conversation normally started with infrastructure. A company might have had:

• physical servers
• SAN/NAS storage
• VMware clusters
• database servers
• backup infrastructure
• DR sites
• firewalls and load balancers
• dedicated operations teams

The first step was often simply: can we run the same thing somewhere else? That created the famous lift-and-shift migration - applications moved from physical infrastructure into cloud VMs.

Then organisations realised that simply putting a VM in AWS or Azure was not necessarily the end state. The architecture evolved.

The important point is that cloud migration was never just an infrastructure migration. It became an operating-model migration. Teams changed how they handled procurement, networking, security, deployment, observability, disaster recovery, application architecture and cost management.

Eventually another problem appeared. Cloud was incredibly convenient, but convenience could become expensive. Organisations started introducing FinOps, reserved capacity, committed-use discounts, workload optimisation and, in some cases, even cloud repatriation.

AI is beginning a surprisingly similar journey.

The first generation of enterprise AI

For most companies, the easiest way to experiment with generative AI has been through APIs. The architecture is beautifully simple.

No GPUs. No model serving. No CUDA. No inference infrastructure. No model deployment. Just an API call.

For experimentation and many production workloads, this remains extremely attractive. It is the AI equivalent of the early promise of cloud: don't manage the infrastructure, consume the capability.

But as AI moves from experimentation into core enterprise workloads, the questions change. Instead of can we call an AI model?, the conversation becomes:

• What happens when we make tens of millions of calls?
• Can sensitive data leave our environment?
• What latency do we require?
• What happens if the provider changes pricing?
• Can we guarantee model availability?
• Can we control the exact model version?
• Where does the data go?
• What happens when AI becomes part of every business process?

And suddenly running AI infrastructure yourself starts looking less ridiculous.

Enter on-prem AI

The new architecture can look surprisingly familiar.

Instead of buying general-purpose servers, organisations are now evaluating GPU infrastructure. Instead of VMware clusters, the platform might include NVIDIA GPUs, Kubernetes, model-serving frameworks, inference engines, vector databases, model registries, AI gateways, observability, and prompt and evaluation infrastructure.

The infrastructure has changed. The architectural question has not: which workloads should we own, and which workloads should we consume?

Why on-prem AI is becoming attractive

There are several reasons enterprises are considering it.

1. AI compute economics are different

Traditional cloud workloads can be extremely elastic. A web application might scale from ten servers to one hundred servers during a peak period and then shrink again. That makes consumption pricing powerful.

AI inference can behave differently. Imagine a business running a relatively predictable volume of inference continuously:

At high utilisation, renting expensive GPU capacity indefinitely may not always be the most economical model. The calculation begins to resemble the old cloud-versus-own-infrastructure argument.

Low utilisation: cloud wins easily. High utilisation with predictable demand: owning capacity starts becoming interesting.

The exact economics depend heavily on hardware utilisation, power, cooling, staffing, depreciation, model architecture and cloud pricing. But the conversation is happening.

2. Data gravity matters

AI is most valuable when connected to enterprise data. Think about financial records, health information, intellectual property, legal documents, engineering designs, customer records, internal communications and proprietary research.

Sending all of that information to an external model provider introduces additional governance requirements. Running models closer to the data can simplify certain architectures.

Instead of sending the data out across your boundary to reach the model, you can potentially bring the model to the data.

That does not automatically make the system secure. But it changes the security boundary.

3. Data sovereignty

Governments, defence organisations, healthcare providers, financial institutions and regulated industries often have strong requirements around where information can be processed.

On-prem or sovereign AI infrastructure can provide tighter control over model location, data location, logging, network boundaries, access control, encryption and retention. For some workloads, this may be more important than pure infrastructure cost.

4. Predictable latency

API-based AI introduces network latency. For asynchronous workloads, that may be irrelevant. For workloads involving robotics, manufacturing, voice, real-time decision systems, industrial control or edge devices, latency can become much more important.

Placing inference physically closer to the application can make sense.

5. Model control

Using an external model API means the provider owns much of the model lifecycle. Running models yourself gives considerably more control. An organisation can decide:

That can matter when AI becomes part of a deterministic business process.

But we should remember why we moved to cloud

There is a danger here. Enterprise technology has a habit of rediscovering problems we solved ten years ago.

Buying GPUs sounds attractive until you remember what operating infrastructure involves. Suddenly someone needs to manage hardware failures, GPU scheduling, drivers, CUDA versions, networking, storage, cooling, capacity planning, model deployment, autoscaling, security patches, observability and disaster recovery.

You have effectively recreated a specialised data centre. Except now the servers may cost considerably more. And the software stack changes incredibly quickly.

The full AI infrastructure cycle may look familiar

I suspect the enterprise AI journey will eventually look something like this:

The last stage is particularly interesting. Companies will probably stop asking should we use cloud AI or on-prem AI? The better question will become: where should this particular inference request execute?

An AI gateway could make that decision dynamically.

Hybrid AI is probably the end state

This is where the comparison with cloud migration becomes particularly useful. Enterprises eventually learned that infrastructure architecture rarely has to be ideological. Not everything has to be 100% cloud or 100% on-prem.

The same will likely happen with AI.

A business might use a frontier hosted model for extremely complex reasoning. It might use a smaller internal model for high-volume repetitive work such as document classification. Highly sensitive workloads might remain inside the organisation. And edge workloads might run directly on local devices - a camera, a robot, a machine on the factory floor.

One organisation could therefore operate four different AI deployment models at the same time.

The new FinOps will be AI compute optimisation

During the first cloud wave, organisations focused heavily on migration. Later they realised they needed to optimise. AI will probably follow exactly the same path.

Initially: make AI work. Then: make AI reliable. Then: make AI secure. Finally: why is our inference bill $4 million?

At that point architecture becomes an economics problem. For every workload, organisations will run one sum and compare it against the other.

AI architecture will therefore increasingly become a combination of MLOps + Platform Engineering + FinOps.

We are not going back to the data centre

The interesting part is that on-prem AI isn't really a reversal of cloud computing. It is the next stage of workload placement.

Cloud taught the industry an important lesson: infrastructure should exist where it creates the best combination of economics, reliability, security and operational simplicity. AI doesn't invalidate that principle. It reinforces it.

Ten years ago the architecture question was: which applications should we migrate to the cloud?

The next major enterprise architecture question may be: which intelligence should we keep inside the organisation?

And just like the cloud migration era, the answer probably won't be everything or nothing. It will be workload by workload.