Reasons to 'Flatten' Your Data Center LAN

By Jim Metzler, Ashton, Metzler & Associates on February 16, 2012 9:18 AM

A TechNote on The Next Generation
Jim Metzler
Distinguished Research Fellow and Co-Founder
Webtorials Analyst Division

For more than a decade, the traditional data-center LAN has been built around a three-tier, hub-and-spoke architecture comprising access, distribution and core switches. Just how did this architecture come to be so prevalent, and what performance and cost factors today are driving and enabling IT organizations to consider "flattening" their data center LANs down to two tiers?

Low Port Densities: Three Tiers

The main impetus behind the deployment of the three-tier data center LAN architecture was the low port density of previous generations of data-center LAN switches. For example, it was common for first-generation switches to have as few as 16 or 32 ports. Even a medium-sized data center required many access switches to interconnect its servers, because traffic almost always had to travel between a server connected to one access switch and a server connected to another access switch.

The best way to interconnect these access switches was with another set of switches, referred to as distribution switches. From there, high-end data centers required a large number of access switches to interconnect the distribution switches; this was done with a third set of switches, known as core switches.

There are many ongoing IT initiatives aimed at improving the cost efficiency of the enterprise data center. Among them are server virtualization and a service-oriented architecture (SOA), shared network storage and high-performance computing (HPC). In many cases, these initiatives are placing a premium on the ability of IT organizations to provide highly reliable, very low-latency, high-bandwidth communication among both physical and virtual servers. While the hub-and-spoke topology of the traditional three-tier data center LAN was appropriate for client-to-server communication (sometimes referred to as "north-south" traffic), it became suboptimal when applied to high volumes of server-to-server communication (or "east-west" traffic).

High Port Densities: Two Tiers

One approach to improving server-to-server communication is to flatten the network from three tiers to two: access-layer switches and aggregation/core-layer switches. A two-tier network reduces the number of hops between servers, which in turn lowers latency and potentially improves reliability. The typical two-tier network is also better aligned with server virtualization topologies, where virtual LANs (VLANs) may be extended throughout the data center to support dynamic virtual machine (VM) migration at Layer 2.

Reducing the architecture to two tiers is possible because modular data center switches have moved way beyond their low-port-density predecessors to provide up to 768 non-blocking 10-gigabit Ethernet (GbE) ports or 192 40-GbE ports. Today's high-speed uplinks are often multiple 10-GbE links that leverage link aggregation (LAG). Note that a 40-GbE uplink typically offers performance superior to that of a four-link 10-GbE LAG. This is because the hashing algorithms that load balance traffic across the LAG links can easily yield suboptimal load distribution with a majority of traffic concentrated in a small number of flows.

Most high-performance modular switches already have a switch fabric that provides 100Gbps of bandwidth to each line card, which means that as 40- and 100-GbE line cards become available, they can be installed on - and preserve the enterprise's investment in - existing modular switches. Most vendors of modular switches are currently shipping 40-GbE line cards; 100-GbE line cards will not be widely deployed until late 2012 or 2013.

The significantly higher port densities on the current generation of data-center LAN switches are good news for IT organizations that want to implement a flatter network. More good news: it is also possible to combine multiple data-center LAN switches and have them perform together as if they were a single, very large switch.

However, even though IT organizations can now flatten their data-center LANs, they are not necessarily rushing to do so. The market momentum behind this approach has yet to build.

Email and Social Media Links: Share securely via email | | Tweet

8 Comments

Jim Metzler, Ashton, Metzler & Associates | February 12, 2012 9:24 AM

At the forthcoming Interop conference in Las Vegas, I will co-moderate, along with Mike Fratto of Network Computing, a half-day workshop on data-center LAN design, on Monday, May 7, 2012.

Carlos HP | February 16, 2012 10:14 AM

And what about limiting the broadcast domain? Are you assuming that the performance of the switch is so high that we can forget a huge broadcast domain of sometimes thousands of systems on a single Datacenter?

jan devos IBM | February 17, 2012 1:55 AM

@Carlos.
The flattened DCN architecture consists indeed of DC-wide VLANs, whereas previously, those VLANs were 'localized', typically by terminating them at the distribution switch. Multiple VLANs will continue to exist in the DCN, e.g. for production and acceptance environment, but those now stretch accross the entire DC. So, it is unlikely that a flat DCN will become one single melting-pot of 1000s of systems. Rather, it will superpose a series of DC-wide, functional server VLANs. Neverttheless, this will tend to larger L2 domains as compared to the previous, per-distribution-switch compartmented designs. The increase of MAC addresses per L2 domain due to extending VLANs is also aggravated by intensive virtualization. This effect is addressed by far bigger mac address table spaces in the current gen DC switches. Anyway, you are right that the increasing # of servers (real and virtual) per VLAN exposes server NICs and protocol stacks to an increased rate of 'benign' broadcasts. At the same time, it extends the impact-area of malicious broadcast storms. The latter can today be mitigated by control features built in into the switches.

jan devos IBM | February 17, 2012 2:10 AM

The industry shows full consensus on the need for a new 'fabric based' DC Network Architecture. But at the same time we observe the overwhelming majority of organizations continue to install legacy equipment into a legacy architecture. The most heard reasons for staying with the current, that I am aware of :
(1) We have no need for high 10G port density - teamed 1G server NICs are fine
(2) We have no plans to converge LAN and SAN. We don't want to be an early adopter in this domain
(3) We don't have 1000s or even 100s of VMs. And once we have them in place, they stay where they are (no need for workload mobility)
(4) Changing the architecture is highly disruptive and incurs risks and costs
(5) We don't need the microsecond latency in the switching equipment, as the latency induced by the application layer is some orders of magnitude higher.
(6) The fabric solutions like FabricPath, QFab, VCS or standard Trill are nice, but the current switch or multichassis clustering offered by vendors (VSS, SMLTG, VLAG etc.) fairly well addresses our capacity and high availability needs while eliminating spanning-tree.

Any comments on those reasons of why not to move forward? Any more reasons that you observe?

jan devos IBM | February 17, 2012 2:31 AM

We indeed observe great consensus among the industry regarding the need for a next gen DC Network (Flat, Converged, Dynamic, Virtualized, Low Latency). Surprisingly, the observation is that an overwhelming majority of organizations continues to install traditional switches in traditional desings. I observe the following rasons why organizations don't move forward.
(1) No need for high 10G port density. Teamed 1G server NICs are doing well,
(2) No willingness to converge LAN/SAN : either because (a) there is no SAN (all is NAS), (b) recent investmens in FC SAN, (c) don't want to be early adopter.
(3) Not 1000s, even not 100s of VMs. And once a VM is deployed, it is highly static (no need fori workload mobility).
(4) Fabric switching systems like FabricPath, QFAb, VCS and later Trill are nice, but proprietary virtual chassis and clustering techniques like VSS, VLAG, SMLTG ... pretty well address capacity and availability constraints, while eliminating spanning tree
(5) no need for extremely low latency (microsec) switches. The delay caused by the application layer is several orders of magnitude higher.
(6) The proposed change is radical, disruptive, and incurs risk and cost.
(7) Standards (Trill, QBG, SPB, even DCB) are awaiting maturity.

David CCMC | February 17, 2012 8:34 AM

I've never worked in or built a Data Center as a three-tier - only a campus environment, where it makes sense to consolidate multiple routers to a handfull of high-density Core chassis. Data Centers I've worked in have always been two-tier, with L-2 and L-3 VLANS segmenting traffic across to DR sites or out to the Campus (user) Core.
Lots of vendors are now espousing the need to go to a two-tier Campus model by having all the buildings peer with each other. It can be done, but by having each building with redundant routers, each redundant router talking to three different Core boxes located in different buildings, no single point of failure exists.

Shortest Path Bridging and TRILL are driving Data Center conversions from the Cisco-esque 6500 to 4500 to 3750 design to the flatter design Extreme, Foundry, Force-10 and Nortel/Avaya have always been proponents of - a pair of redundant chassis with Top-of-rack L-2 (some L-3) switches handling the servers directly.

Dave R.

Jim Metzler, Ashton, Metzler & Associates replied to comment from David CCMC | February 20, 2012 6:16 PM

I see things just a little differently. I don't see technologies such as Shortest Path Bridging (SPB) and TRILL as driving flatter data center designs. I see the interest in flatter data center designs being driven by the desire to save money and the desire to reduce latency. I see that technologies such as SPB and TRILL are enablers, as is the deployment of data center LAN switches that have a higher port density.

I do have some concerns with the movement to flatten data center LANs. One concern is that technologies such as SPB and TRILL are still going through the standardization process. Partially because of that, it will be a while before there are a large number of production networks that have implemented these technologies. That creates somewhat of a chicken and egg syndrome. Many IT organizations will not feel comfortable implementing these technologies in their data center LANs until that have proven themselves in production networks. It will be difficult, however, for these technologies to prove themselves in production networks until IT organizations feel comfortable with them.

Another concern that I have with these technologies is the way that the vendors are lining up behind them. While I have no doubt that over time, most data center LAN switch vendors will support all of the relevant standards, that is not how things are likely to play out in the short term. Right now, most of the vendors are backing just one of these protocols. So, if you are interested in SPB, you should be talking to Avaya as they are a major proponent of SPB. Analogously, if you are interested in TRILL, you should be talking to Cisco and Brocade.