IBM Flex System EN4132 2-port 10Gb RoCE Adapter

IBM Redbooks Product Guide


Abstract

The IBM Flex System™ EN4132 2-port 10Gb RoCE Adapter for Power Systems compute nodes delivers high bandwidth and provides RDMA over Converged Ethernet (RoCE) for low latency application requirements. Clustered DB2 databases, web infrastructure, and high frequency trading are just a few applications that achieve significant throughput and latency improvements, resulting in faster access, real-time response, and more users per server. This adapter improves network performance by increasing available bandwidth while decreasing the associated transport load on the processor.

Tip: Do not confuse this adapter with the EN4132 2-port 10Gb Ethernet Adapter. These are two separate adapters.

Changes in the October 8 update:
* Added new switches to support table

Contents


The IBM Flex System™ EN4132 2-port 10Gb RoCE Adapter for Power Systems compute nodes delivers high bandwidth and provides RDMA over Converged Ethernet (RoCE) for low latency application requirements. Clustered DB2 databases, web infrastructure, and high frequency trading are just a few applications that achieve significant throughput and latency improvements, resulting in faster access, real-time response, and more users per server. This adapter improves network performance by increasing available bandwidth while decreasing the associated transport load on the processor.

The following figure shows the IBM Flex System EN4132 2-port 10Gb RoCE Adapter.

Tip: Do not confuse this adapter with the EN4132 2-port 10Gb Ethernet Adapter. These are two separate adapters.

IBM Flex System EN4132 2-port 10Gb RoCE Adapter
Figure 1. IBM Flex System EN4132 2-port 10Gb RoCE Adapter


Did you know?

Mellanox networking adapters deliver industry-leading bandwidth with ultra low, sub-microsecond latency for performance-driven server clustering applications. By combining these adapters with the IBM Flex System Fabric EN4093R 10Gb Scalable Switch, your organization can achieve efficient computing by offloading the processor protocol processing and data movement overhead, such as RDMA and Send/Receive semantics, allowing more processor power for the application.


Part number information

The following table shows the part number to order this card.

Table 1. Part number and feature code for ordering
DescriptionPart numberFeature code
(x-config)
Feature code
(e-config)
IBM Flex System EN4132 2-port 10Gb RoCE AdapterNone*None*EC26
* This adapter is only available through the Power Systems® sales channel. It is not available through the System x® sales channel.


Features

The IBM Flex System EN4132 2-port 10Gb RoCE Adapter has the following features:
  • RDMA over Converged Ethernet (RoCE)

    EN4132 2-port 10Gb RoCE Adapter, based on Mellanox ConnectX-2 technology, uses the InfiniBand Trade Association's RDMA over Converged Ethernet (RoCE) technology to deliver similar low latency and high performance over Ethernet networks. Leveraging Data Center Bridging capabilities, RoCE provides efficient low-latency RDMA services over Layer 2 Ethernet. The RoCE software stack maintains existing and future compatibility with bandwidth and latency-sensitive applications. With link-level interoperability in the existing Ethernet infrastructure, network administrators can use existing data center fabric management solutions.

  • Sockets acceleration

    Applications utilizing TCP/UDP/IP transport can achieve industry leading throughput over InfiniBand or 10 GbE adapters. The hardware-based stateless offload engines in ConnectX-2 reduce the CPU impact of IP packet transport, allowing more processor cycles to work on the application.

  • I/O virtualization

    ConnectX-2 with Virtual Intelligent Queuing (Virtual-IQ) technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server. I/O virtualization with ConnectX-2 gives data center managers better server utilization while reducing cost, power, and cable complexity.



Specifications

The IBM Flex System EN4132 2-port 10Gb RoCE Adapter has the following specifications:
  • Based on Mellanox Connect-X2 technology
  • PCI Express 2.0 (1.1 compatible) through an x8 edge connector up to 5GT/s
  • 10 Gbps Ethernet
  • CPU offload of transport operations
  • CORE-Direct® application offload
  • GPUDirect application offload
  • RDMA over Converged Ethernet (RoCE)
  • End-to-end QoS and congestion control
  • Hardware-based I/O virtualization
  • TCP/UDP/IP stateless off-load
  • Ethernet encapsulation (EoIB)
  • RoHS-6 compliant

Ethernet specifications:
  • IEEE 802.3ae 10 Gigabit Ethernet
  • IEEE 802.3ad Link Aggregation and Failover
  • IEEE 802.3az Energy Efficient Ethernet
  • IEEE 802.1Q, .1p VLAN tags and priority
  • IEEE P802.1Qbb D1.0 Priority-based Flow Control
  • IEEE 1588 Precision Clock Synchronization
  • Jumbo frame support (10 KB)
  • 128 MAC/VLAN addresses per port


Supported servers

The following table lists the IBM Flex System compute nodes that support the EN4132 2-port 10Gb RoCE Adapter.

Table 2. Supported servers
DescriptionFeature code
x220 (7906)
x222 (7916)
x240 (8737)
x440 (7917)
p24L (1457)
p260 (7895)
p270 (7954)
p460 (7895)
IBM Flex System EN4132 2-port 10Gb RoCE AdapterEC26
N
N
N
N
Y
Y
Y
Y

The two-socket servers support the installation of one EN4132 2-port 10Gb RoCE Adapter. The four-socket servers support up to three EN4132 2-port 10Gb RoCE Adapters.

See IBM ServerProven® at the following web address for the latest information about the expansion cards that are supported by each blade server type:
http://ibm.com/servers/eserver/serverproven/compat/us/

I/O adapter cards are installed in the slot in supported servers, such as the p260, as highlighted in the following figure. The EN4132 2-port 10Gb RoCE Adapter is installed in slot 2.

Location of the I/O adapter slots in the p260 Compute Node
Figure 2. Location of the I/O adapter slots in the IBM Flex System p260 Compute Node


Supported I/O modules

The EN4132 2-port 10Gb RoCE Adapter supports the I/O modules listed in the following table. One or two compatible switches must be installed in the corresponding I/O bays in the chassis. Installing two switches means that both ports of the adapter are enabled.

Table 3. I/O modules supported with the EN4132 2-port 10Gb RoCE Adapter
DescriptionFeature code
(e-config)
Supports the EN4132 adapter
1 Gb switches
IBM Flex System EN2092 1Gb Ethernet Scalable Switch 3598No
10 Gb switches
IBM Flex System EN4091 10Gb Ethernet Pass-thru3700Yes
IBM Flex System EN4023 10Gb Scalable SwitchESWDYes
IBM Flex System Fabric CN4093 10Gb Converged Scalable SwitchESW2No
IBM Flex System Fabric EN4093 10Gb Scalable Switch 3593Yes
IBM Flex System Fabric EN4093R 10Gb Scalable Switch ESW7Yes
IBM Flex System Fabric SI4093 System Interconnect ModuleESWAYes
Cisco Nexus B22 Fabric Extender for IBM Flex SystemESWBYes
40 Gb switches
IBM Flex System EN6131 40Gb Ethernet SwitchESW6Yes

The following table shows the connections between adapters installed in the compute nodes to the switch bays in the chassis.

Table 4. Adapter to I/O bay correspondence
I/O adapter slot in the serverPort on the adapterCorresponding I/O module bay in the chassis
Slot 1 (Not supported*)Port 1Module bay 1
Port 2Module bay 2
Slot 2Port 1Module bay 3
Port 2Module bay 4
Slot 3 (p460 only)Port 1Module bay 1
Port 2Module bay 2
Slot 4 (p460 only)Port 1Module bay 3
Port 2Module bay 4
* The EN4132 2-port 10Gb RoCE Adapter is not supported in slot 1

The connections between the adapters installed in the compute nodes to the switch bays in the chassis are shown in the following figure.

Logical layout of the interconnects between I/O adapters and I/O modules
Figure 3. Logical layout of the interconnects between I/O adapters and I/O modules


Supported operating systems

The EN4132 2-port 10Gb RoCE Adapter supports the following operating systems:
  • AIX V7.1 with the 7100-02 Technology Level, or later
  • AIX V6.1 with the 6100-08 Technology Level, or later
  • SUSE Linux Enterprise Server 11 Service Pack 2 for POWER, with current maintenance updates available from SUSE to enable all planned functionality
  • Red Hat Enterprise Linux for POWER 6.3, or later

Mellanox OFED software packages are support for SUSE Linux and Red Hat Linux. Refer to http://mellanox.com/support for information.

Users should also update their systems with the latest Linux for POWER service and productivity tools from the following IBM website:
http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/home.html

Note: VIOS is not supported. Network install and network boot are not supported.


Regulatory compliance

The adapter conforms to the following standards:
  • United States FCC 47 CFR Part 15, Subpart B, ANSI C63.4 (2003), Class A
  • United States UL 60950-1, Second Edition
  • IEC/EN 60950-1, Second Edition
  • FCC - Verified to comply with Part 15 of the FCC Rules, Class A
  • Canada ICES-003, issue 4, Class A
  • UL/IEC 60950-1
  • CSA C22.2 No. 60950-1-03
  • Japan VCCI, Class A
  • Australia/New Zealand AS/NZS CISPR 22:2006, Class A
  • IEC 60950-1(CB Certificate and CB Test Report)
  • Taiwan BSMI CNS13438, Class A
  • Korea KN22, Class A; KN24
  • Russia/GOST ME01, IEC-60950-1, GOST R 51318.22-99, GOST R 51318.24-99, GOST R 51317.3.2-2006, GOST R 51317.3.3-99
  • IEC 60950-1 (CB Certificate and CB Test Report)
  • CE Mark (EN55022 Class A, EN60950-1, EN55024, EN61000-3-2, EN61000-3-3)
  • CISPR 22, Class A


Physical specifications

The dimensions and weight of the adapter are as follows:
  • Width: 100 mm (3.9 in.)
  • Depth: 80 mm (3.1 in.)
  • Weight: 13 g (0.3 lb)

Shipping dimensions and weight (approximate):
  • Height: 58 mm (2.3 in.)
  • Width: 229 mm (9.0 in.)
  • Depth: 208 mm (8.2 in.)
  • Weight: 0.4 kg (0.89 lb)


Popular configurations

The EN4132 2-port 10Gb RoCE Adapter is designed to be used with the 10 Gb Ethernet switches listed in Table 3. The following figure shows one adapter installed in slot 2 of a p260 Compute Node, which in turn is installed in the chassis. Two IBM Flex System Fabric EN4093R 10Gb Scalable Switches are installed in I/O bays 3 and 4.

Example configuration
Figure 4. Example configuration

The following table lists the parts that are used in the configuration.

Table 5. Components used when connecting the EN4132 2-port 10Gb RoCE Adapter to the EN4093 10Gb Scalable Switch
MTM / Feature codeDescriptionQuantity
7895-23XIBM Flex System p260 Compute Node1 to 14
EC26EN4132 2-port 10Gb RoCE Adapter 1 per server
7893-92XIBM Flex System Enterprise Chassis1
ESW7IBM Flex System Fabric EN4093R 10Gb Scalable Switch1 or 2


Related publications

For more information, see the following resources:

Special Notices

This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.

Profile

Publish Date
13 November 2012

Last Update
08 October 2013


Rating: Not yet rated


Author(s)

IBM Form Number
TIPS0913