BY PATRICK McLAUGHLIN
Communications connectivity protocols Ethernet and InfiniBand are being deployed in today’s most demanding networks, while technologists work behind the scenes to standardize specifications for ever-faster transmission speeds. While InfiniBand has long considered itself a complementary technology to Ethernet, as well as Fibre Channel, among some user groups, the technologies are competitive alternatives.
The competition between the Ethernet and InfiniBand camps is playing itself out in multiple arenas–today’s active high-performance computing (HPC) environments and the research-and-testing laboratories in which the technologies’ futures are taking shape.
Both technologies can run on copper-based as well as optical media and corresponding connectivity.
The InfiniBand Trade Association (IBTA; www.infinibandta.org), founded in 1999, promotes the InfiniBand specification through marketing and public relations activities, and conducts interoperability tests of commercial products that result in a growing integrators’ list that includes hundreds of products divided into cables and devices.
The IBTA has also developed and updates a technology roadmap that, according to the group, shows projected increased market demand for InfiniBand technology that translates to bandwidths nearing 1,000 Gbits/sec in the next three years.
As for today’s computing needs, high-end users are making decisions between Ethernet and InfinBand for speeds more in the neighborhood of 10-Gbits/sec. The IBTA recently publicized a question-and-answer dialogue with Jie Wu, research manager for HPC as part of analyst firm IDC’s technical computing team. As part of that dialogue, the IBTA asked what InfiniBand’s strongest advantages are, and how those advantages can be leveraged to expand the adoption and role of InfiniBand, in competition with 10-Gbit Ethernet.
Wu responded, in part: “Lossless, secured, high bandwidth, and low latency are the most distinct value propositions of InfiniBand today, and those features have advanced performance on many computationally insensitive applications in the technical computing space. IDC sees a trend of increased convergence between technical (HPC) and enterprise data centers and workflows ...
“InfiniBand has a substantially larger footprint in HPC than 10-GbE, but 1-GbE has an HPC footprint on par with InfiniBand, and 10-GbE is working its way through the HPC market. Advances in multicore technology, the proliferation of virtualization in commercial data centers, and the emergence of cloud computing all create new requirements for a more-efficient network infrastructure...As the closest competing technology, 10-GbE is still considered more costly compared to InfiniBand, and the feature set is not as rich and mature as that for InfiniBand. There is also debate on how much it would cost to upgrade from 1-GbE to 10-GbE, all of which make 10-GbE a less-appealing choice for certain industries and applications. At least for now, InfiniBand leads 10-GbE in this battle and enjoys a first-mover advantage.”
Wu concluded, however, by stating, “Ethernet is here to stay. To thrive and grow in this battle, InfiniBand will have to continuously renew and add to its value propositions for the HPC and enterprise data center markets, while building a strong ecosystem with various players involved to help grow the community.”
The IBTA has, for the past decade, made that its mission.
Taking a look at the copper-based options between Ethernet and InfiniBand for 10-Gbit/sec connectivity, the 10GBase-CX4 Ethernet option uses media similar to the twinaxial style that serves InfiniBand. CX4 offers about half the distance of InfiniBand–15 meters compared to 30.
The other copper-based 10-Gigabit Ethernet option, 10GBase-T, operates on twisted-pair wiring and the RJ-45 interface. It has faced significant obstacles in its path to market, but some technologists believe those obstacles are on their way to coming down.
Probably the most significant hurdle to overcome has been power consumption. Arista Networks (www.aristanetworks.com) recently introduced a 10GBase-T switch, one of two to reach market at that time, the other coming from Extreme Networks (www.extremenetworks.com). In a conversation about Arista’s 7100T switch line, the company’s director of marketing, Mark Foss, acknowledges, “At the highest level, 10GBase-T adds 4.5 to 5 watts per port to the power of a switch versus SFP+ versions.”
Controlling power
George Zimmerman, chief technical officer with silicon vendor Solarflare Communications (www.solarflare.com), provides insight into the timeframe in which power consumption will come down within 10GBase-T devices. Solarflare provides physical layer interfaces (PHYs) as well as server controllers for 10-Gigabit Ethernet systems.
“We’re sitting at a really interesting time,” Zimmerman says. “The power consumption of what is being bought off-the-shelf today is at the high end of feasibility. But if you look at the OEMs’ [original equipment manufacturers] designs and what might be termed ‘in the oven,’ you will see a very different story.”
Zimmerman adds, “Today’s monolithic devices, at full performance, use less than 6 watts, and typically in the 5-watts-and-under area,” per port; many devices incorporate power-saving measures, so generally speaking, they consume a maximum of 5 watts per port. “There’s also a dynamic scale that automatically saves power, down to about 4 watts, if less than 100 meters of cable is used. So, the maximum consumption on a short line is about 4 watts. At 4 watts per port, a 24-port switch line-card, which is an industry-class density, produces about 100 watts of power from the PHY. ... Those are devices that today are in active stages of product design and test in major OEMs.”
Ethernet controllers must similarly reduce power consumption, Zimmerman explains. Peripheral Component Interconnect Express (PCIe) cards “can use up to 25 watts,” Zimmerman says. “OEMs say they really want them around 18 watts. With some upgrades in the controller area, we expect to be looking at 10 watts for these add-in cards next year.”
Once the total per-port NIC consumption is as low as 10 watts, users pretty much “stop worrying about counting individual watts on the NIC,” he comments. “As you get under 5 watts for PHYs, you stop counting the PHY silicon consumption in the switch.”
While 10GBase-T builders work steadily to reduce the power their devices consume, many Ethernet observers have already concluded that optical fiber is the logical medium for such high-speed connectivity. Some believe it is the superior medium up to 10-Gbits/sec over distances of 100 meters, and will be the only medium for anything approaching those distances for 40- and 100-Gbit transmission.
“For 40- and 100-Gig, multimode fiber continues to show a significant value proposition,” says Doug Coleman, manager of technology and standards with Corning Cable Systems (www.corningcablesystems.com). “The QSFP [quad small-form factor pluggable] transceiver will support 40-Gig,” he explains.
Within the QSFP will be a 12-fiber MPO connector to which will terminate 12-fiber ribbon cables. The four fibers on each side of the ribbon will be used–four for transmit and four for receive. The four fibers in the middle will be dormant. CS
