Nathan Stratton’s Homepage


by on Jul.30, 2009, under Hardware

Infiniband is an often overlooked technology outside of the supercomputer / clustering space. I think that is a shame given some of the amazing aspects of this technology. Infiniband is a serial connection with a raw full duplex data rate of 2.5 Gbit/s known as 1x single data rate (SDR) mode. In addition to a double data rate (DDR) and a quad data rate (QDR) mode, links can be aggregated in units of 4 or 12 paths yielding up to 120 Gbit/s in 12X QDR mode. In a day where server motherboards are just starting to see 10 Gibt/s ethernet cards, the most common “low speed” infiniband options is 10 Gbit/s 4X SDR cards. Infiniband uses remote direct memory access (RDMA) for data transfer allowing data to be moved between hosts directly without any CPU cycles. All of this happens in about 1/4th the port to port speed of 10 Gbit/s ethernet!

The part I like best about Infiniband is the price, especially the used market. Lets take a look at a common setup on eBay. There are lots of switch options, but I like the TopSpin 120 also know as the Cisco 7000p. This is a 24 port 4X SDR 10 Gbit/s switch that runs for $750 – $1500 depending on the used source. There are even more options for Infiniband cards, I tend to stick with Mellanox chip set based cards and they can be found for as little as $40 for PCI-X and around $125 for PCI express. The only thing that is going to cost you more with Infiniband is the cables, they will run you $20 – $50 each.

Applications that support native infiniband RDMA are going to get the best performance, but with the Infiniband over IP (IPoIB) you can use standard TCP/IP! With IPoIB your infiniband card shows up as a normal interface and you can run DHCP or static IP on it.

Cisco 7000P

Cisco 7000P

Mellanox MHEL-CF128-T

Mellanox MHEL-CF128-T

GORE 4X Infiniband Cable

GORE 4X Infiniband Cable

Be Sociable, Share!

1 Comment for this entry

  • Jonathan Lambert

    Well, after building a few solutions on Mellanox (the major provider of solutions for IB, and the chip manufacturer most people, if not all, use), I can say this: it’s fast. But it’s also buggy, especially on some of the Sun projects (ironically, they’re major supporters) we’ve done.

    But it does provide a really great interconnect fabric, and cost-wise, 10GE and FCoE can’t hold a stick on it for “real world” infrastructure (you know, outside of the Fortune 1000 – those of us who actually have things like “budgets” and actual short “timeframes”).

    The core issues I’ve found are thus:
    Convergence of networks requires a lot. It’s very difficult to architect a reliable Open Source gateway for IB systems: and when you do, doing proper network segmentation isn’t really possible (it’s technically “possible” I guess, but it would require some hardcore low-level work – we couldn’t get vlan management to work reliably), So, bridging in your ip to ib network, while pretty straight forward, leaves you without a lot of vlan extensions and therefore difficult management and security.

    The commercial alternatives for this include a box from Mellanox, some stuff from Cisco, but the absolute best box I’ve come across by far: Absolutely can’t tell you how awesome their stuff is. Bridge IB, FC, and 10G IP in a single box, and their management is fantastic. It’s a lot of money for a startup, but it’s not a lot of money relative to the cost of doing a medium size buildout (40-60 servers @ $500 FC card and $500 for each intel e1000 (just think vmware networks) adds up).

    Anyways, I’ve had a lot of real-world experience with it, and I’m happy we’ve gone down the path. The learning has been expensive, and I’ve come across a LOT of people who are telling me (vmware guys mostly) that FCoE is going to replace it – I think that’s crap. IB has a heck of a lot of advantages, but if they can solve the network management challenges for startups, this stuff could quickly become the STANDARD for federation of middle-of-the-road startup infrastructures.

Leave a Reply


Looking for something?

Use the form below to search the site:

Cool Links!

A few highly recommended links...