Infiniband Architecture
Monty Birge
Introduction
InfiniBand Architecture is the new generation of network
architecture that will potentially replace the Peripheral Component Interconnect
(PCI) bus architecture commonly used in servers today. The bandwidth
capacities of InfiniBand are capable of supporting some of the fastest
communications technologies. InfiniBand emerged from a combination of two
competing architecture standards consisting of Next Generation I/O (NGIO),
which was led by Intel, Microsoft, and Sun, and Future I/O, which was supported
by Compaq, IBM, and Hewlett-Packard. These leading companies, along
with 180 of the top server and networking companies, make up the InfiniBand
Trade Association (ITA), which is dedicated to developing InfiniBand technology.
Problem Definition
System bus, or PCI technology
is beginning to reach its limits in terms of speed. This type of technology
provides a fairly sufficient volume of the Input/Output (I/O) bandwidth
across a range of servers and their nodes, which include PC workstations,
storage devices, printers, etc. However, PCI buses can only support up
to 133 Mbps across all PCI connections, and only 566 Mbps of shared bandwidth
with the 64-bit, 66 MHz buses available in high-end PC servers. This
standard for system throughput is beginning to be overwhelmed by the market’s
demand for larger databases, transaction loads, and greater system scalability
and reliability.
InfiniBand Technology
InfiniBand Architecture
is the technology that can solve this growing I/O problem. “Workstations
and servers used to be speedier than the underlying network infrastructure,
but the infrastructure has surpassed the end devices as it enters the gigabit
threshold. ‘Some say that desktops can’t take advantage of all the bandwidth
because the motherboard is too constrained,’ says Marshall Eisenberg, director
of product marketing at Foundry Networks Inc. ‘InfiniBand has the potential
of turning that around.’”[9] PCI’s data pathway
is shared among a number of devices, while InfiniBand establishes direct
connections between one device and another without having to share the
connection with other devices using a channel-based, switched fabric architecture.
“In addition to high-speed transfer, InfiniBand eliminates the bus bottleneck,
said Jim Pappas of Intel Corp.’s server group and ITA director of marketing.
‘Unlike a bus, where only one thing can be moving over the bus, with a
switched fabric you only use up a pair of threads, one thread to send,
one to receive, out of multiple threads,’ Pappas said. ‘That’s a
huge architectural advantage of moving to fabrics over shared buses.’”
[8]
InfiniBand
Architecture is composed of three categories:
-
InfiniBand Host Channel Adapter (HCA), which connects servers to the InfiniBand
“Fabric;”
-
InfiniBand Target Channel Adapter (TCA), which provides the connection
between storage and/or networking devices within an InfiniBand Switch;
and
-
InfiniBand Switch, which connects servers to remote storage and networking
devices.
Currently,
InfiniBand includes options for channel adapters to contain one, four,
and twelve links for various performance needs, with bi-directional data
rates of 500 Mbps, 2 Gbps, and 6 Gbps, respectively. These connections
can be scaled simply by adding additional links, eliminating the need for
a more expensive bandwidth solution or implementing a totally different
infrastructure.
Business Acceptance
The ITA
firmly believes that the entire computing industry will accept and adopt
this new technology. Although the InfiniBand standard is finished,
products supporting it will not be released until 2001. The initial
adoption of these products, however, will likely take awhile since customers
generally wait for new technologies to mature. Yet, most expect InfiniBand
to prevail in the long run, simply for the fact that it’s more reliable,
offers faster connection speeds, and is supported by Information Technology
giants such as IBM, Intel, Sun, Cisco Systems, Hewlett-Packard, Compaq
and Microsoft. “While widespread adoption of InfiniBand will likely take
years – due in part to recent advances in PCI architecture, known as PCI-X,
as well as the costs associated with switching to new architecture – it
is widely expected that most computing systems eventually will migrate
to the new architecture.” [8] However, there are still
some who will remain hesitant to switch from the technologies that have
worked so well for them for so long. “If InfiniBand is successful,
it doesn’t mean that other forms of networking like Gigabit Ethernet will
be pushed aside.”[9]
Literature Review
This section will discuss
the history of InfiniBand Architecture and some of its competing technologies.
Also, a simple example will be used to help illustrate the idea behind
InfiniBand.
History
On August 31, 1999, Compaq,
Dell, Hewlett-Packard, IBM, Intel, Microsoft, and Sun announced that they
would combine their ideas of the Future I/O (FIO) and Next Generation I/O
(NGIO) input/output architectures into one standard to be used throughout
the computer industry. These companies formed the InfiniBand Trade Association
(ITA), which is based in Portland, Oregon. The ITA’s sole purpose
is to develop this standard into a product that the entire computing industry
will adopt. Since its creation, more that 180 of the world’s leading
technology companies have joined and supported ITA. In October of
2000, ITA completed and released the specifications of version 1.0 of the
InfiniBand standard. Products supporting this standard aren’t expected
to be available until the middle of 2001.
Competing Technologies
Although InfiniBand has
a large list of big-name supporters, it still faces competition.
Developers have created PCI-X, based on the current PCI system bus technology,
to provide relief for the I/O bottlenecks that PCI was starting to create.PCI-X
is capable of supporting a peak bandwidth of 1 Gbps and is able to maintain
backward compatibility with PCI.“PCI-X
provides this capability by increasing the bus frequency of PCI from 66
to 133 MHz and by increasing the efficiency of bus traffic.Storage
and networking devices, such as Ethernet, Ultrawide SCSI, and Fibre Channel
will benefit from this increased bandwidth.” [6]PCI-X
also increases the number of expansion slots to four or more at 66 MHz,
compared to PCI which is only capable of handling one or two slots at 66
MHz, due to strict electrical requirements.PCI-X
is becoming highly preferred by companies in the server market due to its
advances over PCI technology, such as performance, backward compatibility,
and an easy migration from PCI.
Motorola
and other telecommunications hardware makers have backed another new I/O
technology called RapidIO.Other
supporters include Cisco, Nortel Networks and Lucent Technologies.However,
RapidIO will be used mostly within telecommunications equipment, and shows
no indication that it will be used for server interconnect technology.
Another
dilemma that InfiniBand faces is that its most direct cost is with the
new equipment that must be purchased.Although
InfiniBand allows for slow expansion, there are the initial costs that
are needed to install the groundwork for this system.InfiniBand
also requires new system software to be implemented in the operating systems
of the host servers, and that embedded logic be included within the peripherals
and devices that are attached to the network.These
attachments, such as switches and routers, may be products of InfiniBand
technology, or existing equipment may be upgraded in order to be compatible
with the new architecture.Since
InfiniBand is capable of using both packet and connection-oriented systems,
it creates another type of network protocol to deal with.At
a low level, it provides a specification on the type of connection between
devices, but at the same time, it can also work on top of other platforms
such as ATM, Gigabit Ethernet, Fibre Channel, and SONET.Therefore,
a network administrator might have difficulties with network troubleshooting
if there are several layers of network protocols to deal with.
Understanding InfiniBand
Technology
In order to truly understand
InfiniBand Technology and appreciate it, one must be able to understand
the concept behind the popular PCI system bus technology used today.
Imagine a person is asked to pick something up at the grocery store by
their husband or wife.In this case,
the request to get something from the grocery store is the specific task
assigned by the spouse, or CPU, of the household, which is now synonymous
with the server or PC.This example
is much the same as an IT manager telling the computer to go out and retrieve
a specific transaction record from a database, which is the grocery store
in this case.With system bus technology,
this action of going to the grocery store must be carried out almost exactly
as the name implies…on a bus.Here
the spouse, who can now be considered a packet containing the grocery store
trip information, exits the door of the household, or Network Interface
Card, and walks down to the street that just so happens to be a bus route,
or system bus.They then board the
bus either to find many other spouses, or packets, with specific assignments
and destinations, or to be the only one onboard at that time.The
bus then continues on with its route, making various stops to let people
off at their destinations.However,
at many of these stops, more people with completely different assignments
board the bus, forcing the bus to make more frequent stops.Finally,
after going well out of their way, and tolerating a long wait, the spouse
arrives at his or her destination, the grocery store.Then,
once the item is purchased from the store, they must again climb aboard,
and ride the bus while enduring more stops, until the bus route takes them
back around to their house.
With InfiniBand technology,
consider the previous example, only the spouse exits the door, or in this
case the Host Channel Adapter, and steps into his or her own car.They
proceed to drive down the driveway, where they are faced with a large selection
of one-way streets that travel directly to the grocery store, with no intersections
or junctions in the streets whatsoever, which can be considered the InfiniBand
switched fabric.However, before
they left the house, their spouse, or CPU, which now has InfiniBand technology,
tells them which street to take.He
or she then follows that path, and reaches the grocery store without any
delays, and enters the grocery store entrance, or Target Channel Adapter
of the database.Again, once the
item is purchase, or record obtained, the spouse gets back into their car
and travels down another one-way street that heads directly to his or her
house.
InfiniBand technology isn’t
quite this simple, but its basic idea is all referenced in this example.A
detailed description of how InfiniBand works is discussed later.
Methodology
Research
for this project involved various email newsletters that I have received
regarding this technology, as well as several keyword searches using the
Internet.I felt that since InfiniBand
Technology is fairly new and still in development, few textbooks would
contain enough relevant information to thoroughly conduct my research.The
Internet proved to be the best method for this.Simply
searching for “InfiniBand” provided hundreds of hits regarding the technology
and related topics.It was then
just a matter of choosing which hits to include in my research.
Gathering Data
I primarily used Yahoo
for my search engine, although I used Alta Vista and Excite as well.One
search on “InfiniBand” using Yahoo, gave me 11,600 with information pertaining
to the technology.Many of these
were from the websites of the companies that are developing InfiniBand,
such as sunworld.com and intel.com.Others
were Information Technology news websites like cnet.com and zdnet.com,
which provided the newsletter that led me to find out about InfiniBand
as an emerging technology.There
was also a link to the InfiniBand Trade Association website were I was
able to find plenty of information on InfiniBand Architecture in articles
contained in the website, as well as downloadable PDF files.
Analysis Techniques
I
chose to print most of the information I found at each site that I looked
at just to be safe, and not overlook anything that would be helpful in
my research.Many of the PDF files
that I printed were of presentations given on InfiniBand, and the information
was in the form of brief, bulleted points that did not provide any detailed
information.Therefore these were
used very little, if it at all, in my research.Also,
much of the information was the same throughout most of the printouts,
and I chose only to use the ones that describe InfiniBand completely.
Findings
Explanation
of InfiniBand
“The market’s
constant demand for greater system scalability, reliability and performance
has been fueled by explosive Internet growth and the industry transition
to an increasingly distributed computing model.Using
InfiniBand architecture provides a standards-based input/output (I/O) platform
that meets those needs.” [4]InfiniBand
replaces the PCI system bus with a high bandwidth, switched-fabric network
architecture, and shifts input/output control from processors to intelligent
I/O engines called channel adaptors.This
channel-based architecture allows devices to be placed up to 17 meters
away from the server using copper wire links, as much as 300 meters using
multimode fiber-optic links, or even 10 kilometers away with single-mode
fiber-optic links.An InfiniBand
network primarily consists of an InfiniBand Host Channel Adapter (HCA),
which connects servers to the InfiniBand “Fabric”, an InfiniBand Target
Channel Adapter (TCA), which provides the connection between storage and/or
networking devices within an InfiniBand Switch, and an InfiniBand Switch,
which connects servers to remote storage and networking devices.
An HCA
is installed within each server or workstation serving as a host system,
while a TCA is installed into devices or device groups such as printers,
remote storage equipment, and many other network peripherals.These
adapters are connected with serial links either directly or through an
InfiniBand switch, forming a channel between the two connections.The
combination of these links, switches, and adapters, make up the InfiniBand
“fabric” Architecture concept.
A single
InfiniBand link operates at 250 Mbps, point-to-point in a single direction.InfiniBand
Architecture, however, uses a pair of links per connection, with one link
dedicated to input and the other dedicated to output, totaling 500 Mbps
of throughput.These bi-directional
links can be combined together to create larger pipes of 2 Gbps, with four
bi-directional links, or 6 Gbps with 12.These
links are easily scalable by simply adding a larger link between connections.Higher
combinations of links will be possible in the future.
Each link
between the HCA and TCA is subdivided into a series of virtual lanes in
order to provide flow control for communicating. Within each virtual lane
the flow of packets can have different priorities and services that are
maintained across the link.There
can be up to 16 virtual lanes per link, but there needs to be at least
one for the data stream and one for fabric management packets. With different
virtual lanes, communication between devices or groups of devices can be
separated to prevent them from directly interfering with each other's data
transfers. For example, a 300 Mbps link could have 15 separate virtual
lanes of 20 Mbps each going between 15 different pairs of devices – none
of which interrupt the bandwidth going across other lanes.
InfiniBand
uses the new Internet Protocol addressing system called IPv6, which gives
a unique 128-bit address to each device connected to the network.Messages
that are transmitted contain the source address from the HCA, and the destination
address from the TCA.The InfiniBand
Global Route Header then uses these addresses to allow InfiniBand switches
to immediately route the messages to their correct destinations.The
basic unit of communication is a message. Messages can be sent within packets,
or small chunks of a message, which can be combined to make up a single
message.InfiniBand can communicate
by using either this packet-based method or a continuous stream-based method.
Each connected
network device’s IPv6 address is called a globally unique identifier (GUID).Each
channel adapter connected to these devices has a GUID that is separate
from the global identifier (GID).Therefore,
a server may have several HCAs for redundancy or connection to different
fabrics, each with its own GID.In
addition to that, each channel adapter may have several ports with its
own local identifiers (LIDs), which are unique to its subnet and switch.From
a network point of view, the GID is the unique 128-bit IPv6 address, and
the LID is a Transmission Control Protocol or User Datagram Protocol port
at that address.
The InfiniBand
Channel Adapters are intelligent devices that are capable of handling all
I/O functions without having to interrupt a server’s CPU.When
the InfiniBand fabric is turned on, it automatically discovers all HCAs
and TCAs in the fabric, and assigns them their IPv6 addresses.It
keeps track of all devices connected to the network by using a polling
algorithm that lets it find new channel adapters as they are added to the
fabric.
Implementation Issues
One of the biggest advantages
of InfiniBand technology is its improved usability.It
will be much easier to add, remove, or upgrade components of the network
than it is with today’s network architecture.“The
physical modularity of InfiniBand Technology will avoid the need for customers
to buy excess capacity up-front in anticipation of future growth.Instead,
they will be able to buy what they need at the outset and ‘pay as they
grow’ to add capacity without impacting operations or installed systems.” [2]InfiniBand
technology is also expected to reduce the size of servers by as much as
60%, which will benefit crowded data centers and free up more room for
other equipment.
Another benefit of this
technology is that instead of having to specify a certain number of PCI
slots when ordering a server, most InfiniBand systems will come out of
their boxes with a single InfiniBand Host Channel Adapter.That
HCA will connect the server to an InfiniBand switch that will automatically
link it to mass storage systems, routers and other network peripherals
containing InfiniBand Target Channel Adapters.Connectors
that are very similar to the familiar RJ45 Ethernet connectors will make
the connections between these devices.
The whole InfiniBand network
is virtually Plug-and-Play, or Hot-Plug, meaning that the InfiniBand technology
automatically detects and configures new hardware once it is attached.This
will be less time consuming and much easier compared to today’s method
where if a device must be connected to server that has run out of PCI slots,
the server must be brought down, a PCI expansion board added, the new device
connected and configured with the network operating system, and then the
server rebooted.With InfiniBand,
the new device can simply be connected to the InfiniBand switch by its
TCA, and the intelligence of the channel adaptors does the rest without
any downtime for the server.
Conclusion
InfiniBand
will eventually replace the Peripheral Component Interconnect (PCI) bus
for servers.It offers much more
performance capabilities and user-friendly features than PCI, and even
with the release PCI-X, InfiniBand’s channel-based switched-fabric design
provides many more opportunities.With
all the support that this technology is receiving from big-name companies,
it’s hard to imagine InfiniBand not being successful.It
should take a few years for InfiniBand to really catch on and become the
new standard in computing, much like it took PCI a few years to establish
itself enough to replace ISA as a standard during the early 1990’s.Yet
once it does, it will be very interesting to see how it develops and the
possible reality of the “Petabit Ethernet.” [9]
References
Appendix
Common Bus Architecture:
InfiniBand Architecture: |
|
 |
