Overview

IPoIB provides standardized IP encapsulation over InfiniBand™ (IBA) fabrics as defined by the IETF Internet Area ipoib working group.  The IPoIB project implements this proposed standard as a layer-2 Linux network driver.  The primary responsibilities of the driver are performing address resolution to map IPv4 and IPv6 addresses to InfiniBand Unreliable Datagram (UD) address vectors, the management of multicast membership, and the transmission and reception of IPoIB protocol frames.  To aide in the portability of this implementation to a wide set of Host Channel Adapters (HCA), the IPoIB project utilizes the services of the Access Layer Infrastructure Drivers project.

In addition to the network driver, the IPoIB project also provides patches to core Linux network administration tools to provide a common and consistent administrative interface as well as patches to a user-mode DHCP client.

High Level Architecture

The IPoIB driver integrates into the Linux network stack as a layer-2 network driver.  In general, a network driver is responsible for constructing/deconstructing its own headers, transmitting frames, as well as receiving frames and routing them to the appropriate layer-2 network driver user.

The figure below provides a schematic of how the IPoIB driver fits into the Linux network stack.

In addition to the typical network driver tasks, the IPoIB driver must also deal with the unique addressing mechanisms of IBA.  Physical link addressing is unique to each type of interconnect (i.e. Ethernet, ATM, FDDI, etc.).  Therefore, the IP protocol suite defines a physical address resolution process that is responsible for mapping IP addresses to physical address.  For IPv4 running on Ethernet, this mapping is performed through the Address Resolution Protocol (ARP).  The IPv6 protocol performs this mapping through a Neighbor Discovery (ND) protocol.  All of these resolution protocols require broadcast and multicast services.

The IPoIB address resolution protocol uses the same techniques and frame formats used in Ethernet.  However, due to the way IBA routes packets through the network, it is cumbersome for one node to tell another the complete address vector required for end-to-end communication.  For this reason, the proposed IPoIB standard defines a link hardware address as just the 128 bit port GID and the 24 bit Queue Pair Number (QPN) portion of an address vector.  It also includes 8 reserved bits resulting in a pseudo 20 byte hardware address.

A sender is responsible for contacting the Subnet Manager (SM)  for complete addressing information to any particular end node based on the source and destination port GIDs contained in the pseudo hardware address.  Therefore, full address resolution requires an additional step not typically found in other link architectures.  The first step is a broadcast solicitation followed by a unicast reply to exchange GID/QPN information.  The next step is to contact the SM, through the Access Layer, to obtain a PathRecord to the destination node.  This extra step means that the general ARP and ND implementation of the Linux network stack cannot be used unmodified.

Rather than modifying the existing ARP/ND implementations or adding a new address resolution protocol module to the generalized neighbor facility of the Linux network stack, the IPoIB driver will keep a shadow ARP cache containing the complete IBA address vector for each pseudo hardware address conveyed to the network stack.  This scheme will result in a small additional delay for the first outbound frame to a target node as the IPoIB driver contacts the SM for PathRecord information.

Finally, the IBA places unique considerations on the IPoIB driver for broadcast and multicast operations.  In order to participate in broadcast/multicast operations, a node must request membership in the broadcast/multicast group through the SM.  The driver will join the broadcast group as part of driver initialization.  This is a requirement of the IPoIB protocol as the broadcast group defines the scope of the IPoIB link for a perspective interface.  Furthermore, when the Linux network infrastructure requests that a driver set a multicast filter, the IPoIB driver must map the multicast address to a Multicast GID then send a multicast join request to the SM.  The net result is a small delay between the request by the network infrastructure and the point where the interface can participate in a multicast group.

Copyright © 2002 Intel Corp.

Project Definition

The IPoIB driver project will provide a Linux-compatible  layer-2 network driver implementation that conforms to the proposed IETF IPoIB Encapsulation standard.  To facilitate portability to a wide set of HCAs, the driver will be written to the Access Layer interface.

To enable consistent and complete network administration, the IPoIB project will provide patches to core network administration tools and unique IPoIB diagnostic and configuration tools.

Finally, patches for a user-mode IPoIB-aware DHCP client will be provided.

Goals

The major goals of the project are: 

Licensing Details

This software is being made available under a choice of one of two licenses. You may choose to be licensed under either the GNU General Public License (GPL) Version 2, June 1991, available at http://www.fsf.org/copyleft/gpl.html, or the Intel BSD + Patent License, further described here.

Deliverables

The deliverables of this project are: 

Timeline

Actual sub-project timelines are not yet determined and will be formulated by the participants of the sub-project.

How to Get Involved

The best way to contribute to this effort is to get in touch with the contact for the area you are interested in.  New participants are welcomed and encouraged to join in this effort to bring InfiniBand support to the Linux operating system.

Project News

The project news is included on the SourceForge site.  Click the "News" link at the left under "SourceForge Services".

 

Last Updated: 05/31/2002 05:02 PM