Comments on: Infiniband http://www.robotics.net/2009/07/30/infiniband/ Wed, 08 Sep 2010 02:34:23 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Jonathan Lambert http://www.robotics.net/2009/07/30/infiniband/comment-page-1/#comment-84 Jonathan Lambert Sat, 10 Oct 2009 22:11:59 +0000 http://www.robotics.net/?p=186#comment-84 Well, after building a few solutions on Mellanox (the major provider of solutions for IB, and the chip manufacturer most people, if not all, use), I can say this: it's fast. But it's also buggy, especially on some of the Sun projects (ironically, they're major supporters) we've done. But it does provide a really great interconnect fabric, and cost-wise, 10GE and FCoE can't hold a stick on it for "real world" infrastructure (you know, outside of the Fortune 1000 - those of us who actually have things like "budgets" and actual short "timeframes"). The core issues I've found are thus: Convergence of networks requires a lot. It's very difficult to architect a reliable Open Source gateway for IB systems: and when you do, doing proper network segmentation isn't really possible (it's technically "possible" I guess, but it would require some hardcore low-level work - we couldn't get vlan management to work reliably), So, bridging in your ip to ib network, while pretty straight forward, leaves you without a lot of vlan extensions and therefore difficult management and security. The commercial alternatives for this include a box from Mellanox, some stuff from Cisco, but the absolute best box I've come across by far: http://www.xsigo.com/ Absolutely can't tell you how awesome their stuff is. Bridge IB, FC, and 10G IP in a single box, and their management is fantastic. It's a lot of money for a startup, but it's not a lot of money relative to the cost of doing a medium size buildout (40-60 servers @ $500 FC card and $500 for each intel e1000 (just think vmware networks) adds up). Anyways, I've had a lot of real-world experience with it, and I'm happy we've gone down the path. The learning has been expensive, and I've come across a LOT of people who are telling me (vmware guys mostly) that FCoE is going to replace it - I think that's crap. IB has a heck of a lot of advantages, but if they can solve the network management challenges for startups, this stuff could quickly become the STANDARD for federation of middle-of-the-road startup infrastructures. Well, after building a few solutions on Mellanox (the major provider of solutions for IB, and the chip manufacturer most people, if not all, use), I can say this: it’s fast. But it’s also buggy, especially on some of the Sun projects (ironically, they’re major supporters) we’ve done.

But it does provide a really great interconnect fabric, and cost-wise, 10GE and FCoE can’t hold a stick on it for “real world” infrastructure (you know, outside of the Fortune 1000 – those of us who actually have things like “budgets” and actual short “timeframes”).

The core issues I’ve found are thus:
Convergence of networks requires a lot. It’s very difficult to architect a reliable Open Source gateway for IB systems: and when you do, doing proper network segmentation isn’t really possible (it’s technically “possible” I guess, but it would require some hardcore low-level work – we couldn’t get vlan management to work reliably), So, bridging in your ip to ib network, while pretty straight forward, leaves you without a lot of vlan extensions and therefore difficult management and security.

The commercial alternatives for this include a box from Mellanox, some stuff from Cisco, but the absolute best box I’ve come across by far: http://www.xsigo.com/ Absolutely can’t tell you how awesome their stuff is. Bridge IB, FC, and 10G IP in a single box, and their management is fantastic. It’s a lot of money for a startup, but it’s not a lot of money relative to the cost of doing a medium size buildout (40-60 servers @ $500 FC card and $500 for each intel e1000 (just think vmware networks) adds up).

Anyways, I’ve had a lot of real-world experience with it, and I’m happy we’ve gone down the path. The learning has been expensive, and I’ve come across a LOT of people who are telling me (vmware guys mostly) that FCoE is going to replace it – I think that’s crap. IB has a heck of a lot of advantages, but if they can solve the network management challenges for startups, this stuff could quickly become the STANDARD for federation of middle-of-the-road startup infrastructures.

]]>