How AI Workloads Are Reshaping Data Center Network Architecture

Artificial intelligence workloads are changing the way data center networks are designed. Traditional enterprise data centers were commonly optimized around client-to-server communication, storage access, internet connectivity, and relatively independent application workloads. Large AI clusters operate differently.

Training and large-scale inference distribute computation across many accelerators that must exchange data continuously. As GPU and accelerator counts increase, the network increasingly becomes part of the computing architecture itself. Bandwidth, latency, congestion, traffic balance, network reliability, optical density, power consumption, and cooling must all be considered together.

This shift is driving changes from traditional North-South networking toward high-bandwidth East-West fabrics, from conventional Ethernet configurations toward RDMA-enabled architectures, and from simple hierarchical networks toward highly optimized Clos, rail-based, dual-plane, and multi-plane designs.

1. Why AI Workloads Are Different from Traditional Data Center Traffic

Traditional enterprise applications often generate a mixture of user-to-server, server-to-storage, and application-to-application traffic. Many operations are relatively independent, and a temporary delay on one path does not necessarily affect an entire distributed computation.

AI training is different because many accelerators can participate in the same computation at the same time. Distributed training frequently relies on collective communication, where many GPUs exchange information during operations such as all-reduce, all-gather, reduce-scatter, and broadcast.

This creates large synchronized traffic bursts. A congested or delayed path can therefore affect other participants in the same operation and increase overall job completion time.

2. From North-South to East-West Traffic

One of the most important architectural changes is the increasing importance of East-West traffic.

In an AI cluster, East-West traffic can dominate the compute fabric because accelerators continuously exchange data with other accelerators. This means the internal network must be designed around high-capacity horizontal communication rather than treating inter-server traffic as secondary traffic.

Current AI reference architectures increasingly separate GPU compute fabrics from North-South and management networks. NVIDIA's GB300 NVL72 reference architecture, for example, defines separate GPU Compute East-West, CPU Converged North-South, and out-of-band management fabrics. 

3. AI Networking Becomes Part of the Compute System

In conventional applications, network performance is often considered one infrastructure component among many. In large distributed AI systems, network performance can directly affect accelerator utilization and training efficiency.

When thousands of GPUs participate in the same job, computation and communication become tightly coupled. A network that cannot provide sufficient bandwidth or predictable latency can cause accelerators to wait for data or synchronization.

As a result, AI system architects increasingly evaluate networking together with GPU architecture, memory bandwidth, storage, cooling, power, and software frameworks.

4. Scale-Up and Scale-Out Require Different Network Architectures

AI systems can be divided into multiple networking layers.

Scale-up technologies such as GPU-specific interconnects focus on tightly coupled communication. Scale-out networks must connect a much larger number of independent hosts while maintaining predictable performance. Scale-across extends connectivity beyond the local cluster or data center.

The result is no longer a single network serving every requirement. Modern AI infrastructure can contain multiple purpose-specific fabrics working together.

5. Why Leaf-Spine and Clos Architectures Are Important

Clos-based leaf-spine architectures provide multiple paths between endpoints and can scale horizontally as additional switches and servers are added.

This is particularly useful for AI because many-to-many communication is fundamental to distributed workloads. Instead of relying on a limited number of heavily utilized core paths, a properly designed Clos fabric can provide multiple parallel paths between compute nodes.

AI-scale Ethernet architectures commonly use non-blocking or highly oversubscribed Clos designs depending on the workload and cost model. At very large scales, additional hierarchy can be required, but adding network tiers also increases hop count, latency, cabling, optics, power, and operational complexity.

6. Why AI Networks Use High-Radix Switching

High-radix switches provide a large number of high-speed ports in a single switching system. This allows more endpoints to be connected while reducing the number of switching stages needed for a given cluster size.

Higher radix can therefore influence the entire topology.

However, high-radix switching also increases the importance of optical module density, thermal management, power delivery, and fiber management because a large switch can contain many hundreds of high-speed optical connections.

7. Rail-Optimized Networking for AI Clusters

AI cluster architectures increasingly organize accelerator connections according to rails or consistent accelerator positions. A rail-optimized design connects corresponding accelerator interfaces through separate network paths or planes.

This structure can provide predictable connectivity and improve traffic distribution across multiple fabrics.

In NVIDIA's published NVL72 Ethernet reference architecture, ConnectX-8 800G ports can be broken into 2 × 400G links, with each GPU having dedicated paths to separate network planes. This illustrates how high-speed host interfaces and optical breakout are becoming closely connected to topology design. :contentReference[oaicite:1]{index=1}

8. Why Multi-Plane Networks Are Emerging

As AI clusters grow, increasing the number of tiers in a single fabric can introduce additional latency, jitter, load imbalance, and physical infrastructure.

A multi-plane architecture instead divides the overall network bandwidth into independent fabrics. Each plane can use a relatively shallow topology while multiple planes operate in parallel.

AI-specific networking platforms increasingly use dual-plane or multi-plane approaches to maintain throughput and resiliency as cluster size grows.

9. RDMA Changes Network Design Priorities

Remote Direct Memory Access, or RDMA, allows data to move between application memory regions with much less CPU involvement than traditional networking approaches.

This is particularly useful for AI clusters because accelerator communication generates large amounts of data movement. Reducing unnecessary software processing can help lower communication overhead.

InfiniBand provides native RDMA networking, while Ethernet AI fabrics commonly use RDMA over Converged Ethernet, or RoCE. Both approaches are designed to support high-performance distributed computing, although their networking stacks and operational mechanisms differ.

Current NVIDIA ConnectX-8 platforms support both InfiniBand and Ethernet up to 800Gb/s-class connectivity for AI and hyperscale workloads. :contentReference[oaicite:2]{index=2}

10. Ethernet Is Evolving for AI Workloads

AI workloads do not necessarily require a completely separate networking protocol. Ethernet is being extended with technologies designed to better handle distributed AI traffic.

AI-oriented Ethernet implementations can combine RDMA, congestion management, adaptive routing, telemetry, Quality of Service, and advanced NIC capabilities to provide more predictable behavior under synchronized traffic loads.

Current AI Ethernet platforms are therefore considerably more specialized than a conventional best-effort enterprise Ethernet fabric, even though they continue to use Ethernet as the underlying networking technology.

NVIDIA's current Spectrum-X architecture, for example, uses adaptive routing, congestion-control mechanisms, and NIC-based load balancing specifically to address large-scale AI traffic behavior. :contentReference[oaicite:3]{index=3}

11. Congestion Control Becomes a Core Architectural Function

AI traffic can create synchronized bursts because many accelerators may begin communication at nearly the same time.

This behavior increases the importance of congestion detection and response. Traditional static routing may not distribute traffic efficiently when many large flows compete for the same path.

Modern AI networks can therefore use mechanisms such as:

The objective is not simply to maximize peak link utilization. The network must also minimize congestion-related stalls and keep distributed jobs progressing consistently.

12. Optical Bandwidth Is Scaling with AI Networks

The increasing bandwidth of AI accelerators and switch ASICs is directly affecting optical connectivity.

400G optical links have become an important building block for high-speed AI networks, while 800G is increasingly used for higher-density switch and accelerator connectivity. The next generation of 1.6T interfaces is being developed around even higher electrical and optical lane rates.

As network speeds increase, the optics become increasingly important to the overall architecture rather than simply acting as replaceable physical-layer components.

13. Breakout Optics and Cable Architectures Are Changing

Higher-speed ports frequently need to connect to multiple lower-speed interfaces. Breakout architectures allow one high-speed host interface to be divided into several lower-rate connections.

For example, an 800G host interface can be architected as multiple 400G connections depending on the platform and network topology.

This approach can improve switch-port utilization and support independent network planes, but it also increases the importance of fiber polarity, lane mapping, connector density, cable management, and optical compatibility.

AI network design therefore increasingly treats transceivers, AOCs, DACs, AECs, breakout cables, and patching systems as part of one integrated connectivity architecture.

14. 800G and 1.6T Increase Thermal and Power Constraints

Higher network bandwidth also increases the power density of switches and optical modules.

A large AI switch may contain dozens or hundreds of high-speed optical connections. Even a small increase in per-port power can have a significant impact when multiplied across a complete cluster.

This creates a direct relationship between network architecture and data center cooling.

AI networking can therefore no longer be designed independently from the power and thermal architecture of the facility.

15. Network Architecture Is Becoming More Software-Driven

Hardware capacity alone does not determine AI network performance. Routing, congestion control, traffic scheduling, telemetry, collective communication libraries, and NIC features increasingly influence how effectively the physical fabric is used.

AI networking platforms can use real-time network information to make routing and load-balancing decisions rather than relying entirely on static paths.

At large scale, the network stack can also interact with GPU communication libraries and orchestration systems. This tighter software-hardware integration is intended to reduce communication bottlenecks and improve overall job completion efficiency.

16. Storage Networking Is Also Being Reshaped

AI workloads require large datasets for training and increasingly depend on high-throughput storage systems.

This creates an additional networking path between compute nodes and storage infrastructure. The storage fabric must be capable of supplying data quickly enough to avoid starving accelerators.

Large AI architectures may therefore separate or logically isolate GPU compute traffic from storage, customer, and management traffic while still using common Ethernet infrastructure in some deployments.

NVIDIA's current AI reference architecture explicitly defines storage connectivity and CPU North-South connectivity as separate fabric roles from the GPU Compute East-West network. :contentReference[oaicite:4]{index=4}

17. Management and Infrastructure Traffic Must Be Isolated

Large AI clusters contain far more than GPUs and switches. They also include management controllers, storage systems, DPUs, monitoring systems, orchestration services, and security infrastructure.

Management traffic should therefore not be treated as equivalent to high-bandwidth GPU-to-GPU training traffic.

Dedicated out-of-band management networks, VLAN isolation, logical segmentation, and separate infrastructure services can help maintain operational visibility without competing directly with compute traffic.

Current AI factory reference architectures commonly distinguish compute, converged North-South, storage, customer, and out-of-band management network roles. :contentReference[oaicite:5]{index=5}

18. Reliability Becomes More Important at Cluster Scale

As a cluster grows, the number of physical links, switches, optics, and cables also increases. The probability that some component will require maintenance or experience an issue therefore becomes a system-level consideration.

AI networks need mechanisms for redundancy, path diversity, rapid detection, telemetry, and controlled failure recovery.

Multi-plane networking can provide physical path diversity, while adaptive routing and distributed load balancing can help maintain traffic flow when network conditions change.

The goal is not simply to prevent failures. It is to prevent a localized hardware problem from unnecessarily disrupting a large distributed workload.

19. AI Networking Is Driving Co-Design of Compute and Optical Infrastructure

The evolution of AI networks is increasingly connecting several technology layers that were historically designed more independently.

This co-design approach is one of the defining characteristics of the AI data center. The network is increasingly engineered as part of the complete AI computing platform.

20. The Future of AI Data Center Network Architecture

AI workloads are pushing data center networking toward higher bandwidth, shorter communication paths, greater network parallelism, more intelligent traffic management, and deeper optical integration.

The evolution can be summarized as:

Traditional data center → high-bandwidth East-West networking → RDMA-enabled AI fabric → 400G/800G optical connectivity → multi-plane and rail-optimized architectures → 1.6T and higher-speed optical infrastructure.

Future AI factories are likely to combine increasingly powerful accelerators with high-radix switches, high-speed SuperNICs, advanced Ethernet or InfiniBand fabrics, higher-speed optical interconnects, silicon photonics, and more intelligent traffic management.

For optical interconnect suppliers, this evolution creates demand across multiple layers of the network, including DAC, AEC, AOC, 400G optical transceivers, 800G optical transceivers, 1.6T optical modules, high-density MPO/MTP® connectivity, and longer-reach optical solutions for AI data center interconnect.

21.Q&A