↓ Skip to main content

Cloud in the AI Era

·2116 words·10 mins
Eyal Estrin
Author
Eyal Estrin
Author of Cloud Security Handbook & Security for Cloud Native Applications. Cloud Adoption & Cybersecurity expert.

I have been working with public cloud services of the major cloud providers for a decade, and for a while I’ve been asking myself – what has changed from an engineering, architecture, or cybersecurity point of view when organizations began to embrace generative AI for building modern applications.
In this blog post, I will review various domains (or pillars) and try to figure out if the changes are that drastic, or simply a minor change.

Physical Infrastructure
#

The most fundamental component in any architecture (whether cloud-native or AI) is the infrastructure, i.e., compute (to run workloads), storage (to store data temporarily or persistently), and network (for connectivity between components).
We still use standard compute, object storage, and VPCs. However, hardware selection and backplane throughput now dictate performance limits.

What to Offload to Neo-Clouds
#

Neo-clouds specialize strictly in high-density GPU infrastructure without the overhead or costs of traditional hyperscalers’ managed services.

  • Why Offload: Hyperscalers often face GPU quota limitations and charge heavy premiums for high-end accelerators (e.g., NVIDIA H100/H200/GB200 clusters). Neo-clouds offer bare-metal or near-bare-metal GPU clusters with specialized InfiniBand fabrics at 30%–60% lower compute rates.

  • What Remains in Public Cloud: Web apps, API gateways, microservices, and general orchestration engines.

What to Offload to On-Premises Environments
#

On-premises deployment (using platforms like NVIDIA DGX SuperPODs, Dell/HPE GPU servers, and local Kubernetes clusters) makes sense for baseline capacity, extreme latency, and strict data sovereignty.

  • Why Offload: Industrial automation, smart manufacturing, and medical imaging require real-time inference latency (<5ms) that cannot tolerate round-trip travel to a public cloud region. Small Language Models (SLMs) run directly on local edge hardware or private racks.

Bottom line: The shift of cloud to AI workloads is a minor change (evolutionary) from an architecture or engineering point of view. We may have to make some adjustments in terms of selecting the hardware, but it is not considered a major change.

Sustainability and Energy Efficiency
#

Mature organizations and the hyper-scale cloud providers are already prioritizing sustainability and energy efficiency when building or modernizing their data centers around the world. This is not something we can ignore – both in terms of power consumption requirements and the environmental impact.
AI training and high-volume inference generate significant power spikes, requiring explicit region selection and model right-sizing to hit corporate ESG targets.

What to Offload to On-Premises Environments
#

To hit corporate sustainability targets under heavy AI demand, architects must actively route compute to regions offering clean energy and high thermal efficiency.

  • Why Offload: Specialized providers locate GPU data centers directly at stranded energy sources (like flared gas or remote renewables), allowing enterprises to run heavy training runs with lower carbon impact.

Bottom line: The shift of cloud to AI workloads is a major paradigm shift (revolutionary) from an architecture or engineering point of view. AI has turned power grid capacity and thermal limits into non-negotiable architectural constraints rather than routine background operational details.

FinOps and Cost Optimization
#

With the high cost of the public cloud, and now with the demand for expensive hardware and token costs, any organization designing a modern application must embed cost as part of any architecture decision.
The unit of cost is no longer fixed hourly vCPUs; it’s token generation rates, context lengths, and idle GPU capacity. Unmonitored recursive loops can exhaust monthly budgets in hours.

What to Offload to Neo-Clouds
#

Moving terabyte-scale datasets between environments to feed hungry GPU pipelines makes legacy cloud egress fees an immediate architectural liability.

  • Why Offload: Many neo-clouds offer zero-egress or minimal data transfer fees, preventing the severe bandwidth penalties associated with shifting massive dataset checkpoints into public clouds.

What to Offload to On-Premises Environments
#

When inference workloads run continuously around the clock, public cloud pay-per-token pricing models trigger a cost inversion where owning on-premises hardware becomes vastly cheaper than renting cloud GPUs.

  • Why Offload: Cloud AI pricing scales with usage (per-token or per-instance-hour); on-premises hardware amortizes over time. If an enterprise runs steady 24/7 baseline inference workloads, local GPU hardware often achieves a lower total cost of ownership (TCO) within 12–18 months.

Bottom line: The shift of cloud to AI workloads is a major paradigm shift (revolutionary) from an architecture or engineering point of view. AI shifts cloud economics from predictable infrastructure hours to volatile, context-dependent token generation, turning unmonitored execution loops into instant financial liabilities.

Reliability and Resiliency
#

Any mature architecture must take reliability and resiliency under consideration. No production workload would be built without embedding those principles as part of an architecture design.
The core availability patterns (multi-AZ, retry mechanisms) are unchanged, but we now apply them to stateful training jobs and external LLM rate limits.

Bottom line: The shift of cloud to AI workloads is considered a minor change (Incremental/Evolutionary). The underlying architectural patterns like redundant regions, retries, and circuit breakers remain identical; we are simply applying those same foundational rules to training checkpoints and LLM rate limits.

Data and Database Layer
#

As with any other production workload, the data layer must be taken into architectural consideration. Relational and NoSQL platforms remain standard, but native high-dimensional vector search and low-latency RAG pipelines are now fundamental requirements.

What to Offload to On-Premises Environments
#

For highly regulated industries handling sensitive records, keeping data storage and Retrieval-Augmented Generation context pipelines on-premises is essential to guarantee absolute data sovereignty and eliminate cloud privacy risks.

  • Why Offload: Keeping vector stores, embeddings, and fine-tuning datasets inside private on-premises databases keeps sensitive data entirely within the internal security perimeter.

Bottom line: The shift of cloud to AI workloads is considered a mixed shift (Evolutionary Storage, Major Vector Shift). Core storage engines and relational databases simply added features to support vector embeddings, but the real-time context streaming required for RAG pipelines completely transformed how data feeds into applications.

Operational Excellence and Observability
#

Mature organizations look far beyond deployment, monitoring, and maintenance, and embed operational excellence and observability. CPU/Memory charts are insufficient. Platform engineers must now monitor token generation speeds, prompt drift, and output accuracy.

Bottom line: The shift of cloud to AI workloads is considered a major paradigm shift (Revolutionary). Observability has evolved from tracking deterministic system health metrics like CPU and memory to measuring non-deterministic application behaviors like token speeds, hallucination rates, and model drift.

Security, Identity and Application Governance
#

We cannot consider any workload containing customer or business data without thinking about the various security or regulatory topics in every aspect of the workload. It is considered the highest operational risk layer. Security teams shift from securing static HTTP endpoints to governing autonomous, multi-step AI agents and mitigating prompt injection attacks.

What to Offload to On-Premises Environments
#

Running autonomous AI agents and identity guardrails on-premises gives security teams total control over credential delegation, prompt filtering, and governance boundaries without exposing internal logic to third-party cloud endpoints.

  • Why Offload: Regulated industries processing sensitive customer PII or proprietary IP cannot allow prompt payloads or agent execution chains to traverse public networks. Running identity controls, content safety guardrails, and open-weight models on-premises ensures that autonomous agent actions and credential delegation remain entirely within the internal security perimeter.

Bottom line: The shift of cloud to AI workloads is considered a major paradigm shift (Revolutionary). Security teams must shift from defending static code perimeters to governing non-deterministic outputs, autonomous multi-step agents, and prompt payloads entering model context windows.

Summary
#

In this blog post, I reviewed many aspects common to any architect and how they shifted in the context of cloud services when organizations began to develop, build, and maintain AI workloads.
Due to cost, hardware availability, and data sovereignty, organizations should consider designing hybrid architectures that include the use of neo-clouds or on-premises, and this blog post highlights reasons to offload some of the workloads from the public cloud.
As technology evolves, organizations should keep questioning their existing architectures and make architecture and technology decisions accordingly – no “all-or-nothing” decision.

About the Author
#

Eyal Estrin is a cloud and information security architect with more than 25 years in the industry. He is the author of Cloud Security Handbook and Security for Cloud Native Applications.
The views expressed are his own.