Voltaire 40 Gb InfiniBand Switch Module for IBM BladeCenter

IBM Redbooks Product Guide

Abstract

Clients want to increase the use of their existing servers to reduce overall cost as well as the IT footprint within the organization. IBM BladeCenter offers several choices, including servers, chassis, and the type of interconnect fabric. This solution offers end-to-end 40 Gb (4X QDR) InfiniBand connectivity. Clients can take advantage of the resiliency of BladeCenter as well as the performance of 4X QDR InfiniBand in a single package. The InfiniBand switch module is based on proven Voltaire design and offers 14 40 Gb ports, one to each server, and 16 ports out of the chassis per switch. This solution offers a no compromise, congestion free solution to meet even the most performance hungry applications.

Contents


Clients want to increase the use of their existing servers to reduce overall cost as well as the IT footprint within the organization. IBM BladeCenter offers several choices, including servers, chassis, and the types of interconnect fabric. This solution offers end-to-end 40 Gb (4X QDR) InfiniBand connectivity. Clients can take advantage of the resiliency of BladeCenter as well as the performance of 4X QDR InfiniBand in a single package.

The Voltaire 40 Gb InfiniBand Switch Module for BladeCenter provides InfiniBand QDR connectivity between the server platform's high performance blade servers and external InfiniBand fabrics in non-blocking designs, all on a single device. Voltaire's high speed module also accommodates performance-optimized fabric designs utilizing a single BladeCenter chassis or stacking multiple BladeCenter chassis without requiring an external InfiniBand switch.

The InfiniBand switch module offers 14 Gb ports, one to each server, and 16 ports out of the chassis per switch. This solution offers a no compromise, congestion free solution to meet even the most performance hungry applications.

Figure 1 shows the switch module.

Voltaire 40 Gb InfiniBand Switch Module for IBM BladeCenter
Figure 1. Voltaire 40 Gb InfiniBand Switch Module for IBM BladeCenter

Did you know?

The module's HyperScale architecture also provides a unique inter-switch link or mesh capability to form highly scalable, cost effective, and low latency fabrics. Since this switch has 16 uplink ports, they can create a meshed architecture and still have unblocked access to data using the 14 uplink ports. This solution can scale from 14 to 126 nodes and offer latency less than 200 nanosecond allowing applications to operate at maximum efficiency.

Part number information

Table 1 shows the part numbers to order the module and the supported cables.

Table 1. Part number and feature code for ordering
DescriptionPart numberFeature codes
Voltaire 40 Gb InfiniBand Switch Module for IBM BladeCenter46M60050057
3 m Copper QDR InfiniBand QSFP Cable49Y9980 3866

The Voltaire 40 Gb InfiniBand Switch Module part number includes the following items:
  • One Voltaire 40 Gb InfiniBand Switch Module
  • Documentation CD-ROM
  • Important Notices flyer


Features and specifications

The Voltaire 40 Gb InfiniBand Switch Module includes the following features and functions:
  • Form-factor
    • Double-height high-speed switch module
  • Internal ports
    • 14 internal 4X QDR/DDR/SDR InfiniBand to the server blades
    • Two internal 4X InfiniBand ports to the bridge modules
  • External ports
    • Up to 16 auto-sensing 4X QDR/DDR/SDR InfiniBand QSFP ports (40, 20, or 10 Gbps auto-negotiate)
  • Characteristics
    • Full QDR rate InfiniBand switching
    • Based on the Infiniscale-IV device
    • Up to 40 Gbps performance for clusters and grids (bidirectional)
    • Hot swap support, with soft start and current limiting
    • Bay address and presence support
    • High temperature monitoring
    • Full BladeCenter H AMM interface via I2C
    • Fully non-blocking QDR blade
    • QSFP front panel connectors
    • On board CPU for chassis management capabilities
    • Ultra-low latency: under 100 nanoseconds blade to blade connectivity
    • Built-in high availability
    • Port status controlability

Supported BladeCenter chassis and expansion cards

The Voltaire 40 Gb InfiniBand Switch Module is supported only in the IBM BladeCenter H chassis as listed in Table 2.

Table 2. IBM BladeCenter chassis that support the Voltaire 40 Gb InfiniBand Switch Module

I/O module

Part number

BladeCenter S

BladeCenter E

BladeCenter H

BladeCenter T

BladeCenter HT

MSIM

MSIM-HT

Voltaire 40 Gb InfiniBand Switch Module

46M6005

N

N

Y

N

N

N

N

The Voltaire 40 Gb InfiniBand Switch Module support the expansion cards listed in Table 3. This table also lists the chassis bays in which the switch module must be installed when used with each expansion card.

Table 3. Voltaire 40 Gb InfiniBand Switch Module and BladeCenter chassis I/O bays support

Description

Part number

Bay 1 (Standard)

Bay 2 (Standard)

Bay 3 (Standard)

Bay 4 (Standard)

Bay 5 (Bridge)

Bay 6 (Bridge)

Bay 7 (High-speed)

Bay 8 (High-speed)

Bay 9 (High-speed)

Bay 10 (High-speed)

2-Port 40Gb InfiniBand Expansion Card (CFFh)

46M6001

N

N

N

N

N

N

Y

Y

Y

Y

InfiniBand 4X DDR Expansion Card (CFFh)

43W4423

N

N

N

N

N

N

Y

Y

Y

Y

Voltaire 4X InfiniBand DDR (CFFh)

43W4420

N

N

N

N

N

N

Y

Y

Y

Y

Note: The Voltaire 40 Gb InfiniBand Switch Module is a double-high switch module and occupies two adjacent high-speed bays (7 and 8 or 9 and 10).

The BladeCenter H chassis has four standard I/O bays (1, 2, 3, and 4), two bridge bays (5 and 6), and four high-speed bays (7, 8, 9, and 10). The Voltaire 40 Gb InfiniBand Switch Module is supported only in the BladeCenter H chassis and occupies two adjacent high-speed I/O bay (bays 7 and 8, or bays 9 and 10).

Popular configurations

Figure 2 shows the use of Voltaire 40 Gb InfiniBand Switch Module to route two 4X InfiniBand ports from 2-Port 40Gb InfiniBand Expansion Card (CFFh) installed into each server. Two Voltaire 40 Gb InfiniBand Switch Modules are installed in bays 7/8 and bays 9/10 of the BladeCenter H chassis. All connections between the expansion cards and the switch modules are internal to the chassis. No cabling is needed.

A 40 Gb solution using two Voltaire 40 Gb InfiniBand Switch Modules
Figure 2. A 40 Gb solution using two Voltaire 40 Gb InfiniBand Switch Modules

Table 4 lists the components that this configuration uses.

Table 4. Components used when connecting 2-Port 40 Gb InfiniBand Expansion Card (CFFh) to two Voltaire 40 Gb InfiniBand Switch Modules

Diagram referencePart number/machine typeDescriptionQuantity
1VariesIBM BladeCenter HS22 or other supported server1 to 14
246M60012-Port 40 Gb InfiniBand Expansion Card (CFFh)1 per server
38852BladeCenter H1
446M6005Voltaire 40 Gb InfiniBand Switch Module2
549Y99803 m Copper QDR InfiniBand QSFP CableUp to 32*
* The Voltaire 40 Gb InfiniBand Switch Module has 16 external ports. To communicate outside of the chassis, you must have QSFP cables connected. You have the flexibility to expand bandwidth using from one to 16 connections per switch.

Connectors and LEDs

Figure 3 shows the left side of the Voltaire 40 Gb InfiniBand Switch Module front panel.

Left side of front panel of the Voltaire 40 Gb InfiniBand Switch Module
Figure 3. Left side of front panel of the Voltaire 40 Gb InfiniBand Switch Module*

* The port LEDs on the right side of front panel of the Voltaire 40 Gb InfiniBand Switch Module are the same as on the left side.

The front panel contains the following components:
  • Three status LEDs display the status of the switch module and the network:
    • OK (indicating that the switch module has passed the power-on self-test (POST) with no critical faults and is operational)
    • Switch-module error (indicating that the switch module has failed the POST or detected an operational fault)
    • SM (reserved for future use)
  • 16 QSFP port connectors to attach QSFP cables.
    • Each external QSFP port on the switch module contains IB physical link and IB logical link LEDs

Network cabling requirements

The Voltaire 40 Gb InfiniBand Switch Module supports the following cable:
  • 3 m Copper QDR InfiniBand QSFP Cable, part number 49Y9980


Related publications
For more information, see the following documents:

Special Notices

This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.

Profile

Publish Date
30 June 2009

Last Update
04 December 2009


Rating: Not yet rated


Author(s)

IBM Form Number
TIPS0696