IPoIB provides standardized IP encapsulation over InfiniBand™ (IBA) fabrics as defined by the IETF Internet Area ipoib working group. The IPoIB project implements this proposed standard as a layer-2 Linux network driver. The primary responsibilities of the driver are performing address resolution to map IPv4 and IPv6 addresses to InfiniBand Unreliable Datagram (UD) address vectors, the management of multicast membership, and the transmission and reception of IPoIB protocol frames. To aide in the portability of this implementation to a wide set of Host Channel Adapters (HCA), the IPoIB project utilizes the services of the Access Layer Infrastructure Drivers project.
In addition to the network driver, the IPoIB project also provides patches to core Linux network administration tools to provide a common and consistent administrative interface as well as patches to a user-mode DHCP client.
High Level Architecture
The IPoIB driver integrates into the Linux network stack as a layer-2 network driver. In general, a network driver is responsible for constructing/deconstructing its own headers, transmitting frames, as well as receiving frames and routing them to the appropriate layer-2 network driver user.
The figure below provides a schematic of how the IPoIB driver fits into the Linux network stack.
In addition to the typical network driver tasks,
the IPoIB driver must also deal with the unique addressing mechanisms of
IBA. Physical link addressing is unique to each type of
interconnect (i.e. Ethernet, ATM, FDDI, etc.).
Therefore, the IP protocol suite defines a physical address
resolution process that is responsible for mapping IP addresses to
physical address. For IPv4
running on Ethernet, this mapping is performed through the Address
Resolution Protocol (ARP). The
IPv6 protocol performs this mapping through a Neighbor
Discovery (ND) protocol. All
of these resolution protocols require broadcast and multicast services.
The IPoIB address resolution protocol uses the same techniques and frame formats used in Ethernet. However, due to the way IBA routes packets through the network, it is cumbersome for one node to tell another the complete address vector required for end-to-end communication. For this reason, the proposed IPoIB standard defines a link hardware address as just the 128 bit port GID and the 24 bit Queue Pair Number (QPN) portion of an address vector. It also includes 8 reserved bits resulting in a pseudo 20 byte hardware address.
A sender is responsible for contacting the Subnet Manager (SM) for complete addressing information to any particular end node based on the source and destination port GIDs contained in the pseudo hardware address. Therefore, full address resolution requires an additional step not typically found in other link architectures. The first step is a broadcast solicitation followed by a unicast reply to exchange GID/QPN information. The next step is to contact the SM, through the Access Layer, to obtain a PathRecord to the destination node. This extra step means that the general ARP and ND implementation of the Linux network stack cannot be used unmodified.
Rather than modifying the existing ARP/ND
implementations or adding a new address resolution protocol module to
the generalized neighbor facility of the Linux network stack, the IPoIB
driver will keep a shadow ARP cache containing the complete IBA address
vector for each pseudo hardware address conveyed to the network
stack. This scheme will result in a small additional delay for the
first outbound frame to a target node as the IPoIB driver contacts the SM
for PathRecord information.
Finally, the IBA places unique considerations on the IPoIB driver for broadcast and multicast operations. In order to participate in broadcast/multicast operations, a node must request membership in the broadcast/multicast group through the SM. The driver will join the broadcast group as part of driver initialization. This is a requirement of the IPoIB protocol as the broadcast group defines the scope of the IPoIB link for a perspective interface. Furthermore, when the Linux network infrastructure requests that a driver set a multicast filter, the IPoIB driver must map the multicast address to a Multicast GID then send a multicast join request to the SM. The net result is a small delay between the request by the network infrastructure and the point where the interface can participate in a multicast group.
Copyright © 2002 Intel Corp.
The IPoIB driver project will provide a Linux-compatible layer-2 network driver implementation that conforms to the proposed IETF IPoIB Encapsulation standard. To facilitate portability to a wide set of HCAs, the driver will be written to the Access Layer interface.
To enable consistent and complete network administration, the IPoIB project will provide patches to core network administration tools and unique IPoIB diagnostic and configuration tools.
Finally, patches for a user-mode IPoIB-aware DHCP client will be provided.
The major goals of the project are:
Provide a Conformant IPoIB Network Driver – Providing a conformant IPoIB implementation will promote interoperability and adoption of InfiniBand technologies in the data center.
Set de facto IPoIB Administrative Interfaces – For InfiniBand technologies to be readily accepted in an IP-centric data center, the management interfaces must be consistent with the IP tools administrators use everyday. Whenever possible, core network administrative tools will be updated to seamlessly comprehend IPoIB interfaces. When new tools are required, this project hopes to set de facto standards for their look and feel.
Open Source – The project intends to perform it's development totally within the context of an open source environment, releasing architecture, design, interface, and code as they become available to collect input from all sources as early as possible
This software is being made available under a choice of one of two licenses. You may choose to be licensed under either the GNU General Public License (GPL) Version 2, June 1991, available at http://www.fsf.org/copyleft/gpl.html, or the Intel BSD + Patent License, further described here.
The deliverables of this project are:
Architecture, Design, and requirements white papers
A complete open source implementation of the IPoIB driver
Patches for network administrative tools and a user-mode DHCP client
Actual sub-project timelines are not yet determined and will be formulated by the participants of the sub-project.
How to Get Involved
The best way to contribute to this effort is to get in touch with the contact for the area you are interested in. New participants are welcomed and encouraged to join in this effort to bring InfiniBand support to the Linux operating system.
The project news is included on the SourceForge site. Click the "News" link at the left under "SourceForge Services".
Last Updated: 05/31/2002 05:02 PM