Abstract
The IBM Flex System™ EN4132 2-port 10Gb RoCE Adapter for Power Systems compute nodes delivers high bandwidth and provides RDMA over Converged Ethernet (RoCE) for low latency application requirements. Clustered DB2 databases, web infrastructure, and high frequency trading are just a few applications that achieve significant throughput and latency improvements, resulting in faster access, real-time response, and more users per server. This adapter improves network performance by increasing available bandwidth while decreasing the associated transport load on the processor.
Tip: Do not confuse this adapter with the EN4132 2-port 10Gb Ethernet Adapter. These are two separate adapters.
Changes in the October 23 update:
* Added the x240 M5 compute node to Table 2
Contents
The IBM Flex System™ EN4132 2-port 10Gb RoCE Adapter for Power Systems compute nodes delivers high bandwidth and provides RDMA over Converged Ethernet (RoCE) for low latency application requirements. Clustered DB2 databases, web infrastructure, and high frequency trading are just a few applications that achieve significant throughput and latency improvements, resulting in faster access, real-time response, and more users per server. This adapter improves network performance by increasing available bandwidth while decreasing the associated transport load on the processor.
The following figure shows the IBM Flex System EN4132 2-port 10Gb RoCE Adapter.
Tip: Do not confuse this adapter with the EN4132 2-port 10Gb Ethernet Adapter. These are two separate adapters.
Figure 1. IBM Flex System EN4132 2-port 10Gb RoCE Adapter
Did you know?
Mellanox networking adapters deliver industry-leading bandwidth with ultra low, sub-microsecond latency for performance-driven server clustering applications. By combining these adapters with the IBM Flex System Fabric EN4093R 10Gb Scalable Switch, your organization can achieve efficient computing by offloading the processor protocol processing and data movement overhead, such as RDMA and Send/Receive semantics, allowing more processor power for the application.
Part number information
The following table shows the part number to order this card.
Table 1. Part number and feature code for ordering
| Description | Part number | Feature code (x-config) | Feature code (e-config) |
| IBM Flex System EN4132 2-port 10Gb RoCE Adapter | None* | None* | EC26 |
Features
The IBM Flex System EN4132 2-port 10Gb RoCE Adapter has the following features:
- RDMA over Converged Ethernet (RoCE)
- Sockets acceleration
- I/O virtualization
EN4132 2-port 10Gb RoCE Adapter, based on Mellanox ConnectX-2 technology, uses the InfiniBand Trade Association's RDMA over Converged Ethernet (RoCE) technology to deliver similar low latency and high performance over Ethernet networks. Leveraging Data Center Bridging capabilities, RoCE provides efficient low-latency RDMA services over Layer 2 Ethernet. The RoCE software stack maintains existing and future compatibility with bandwidth and latency-sensitive applications. With link-level interoperability in the existing Ethernet infrastructure, network administrators can use existing data center fabric management solutions.
Applications utilizing TCP/UDP/IP transport can achieve industry leading throughput over InfiniBand or 10 GbE adapters. The hardware-based stateless offload engines in ConnectX-2 reduce the CPU impact of IP packet transport, allowing more processor cycles to work on the application.
ConnectX-2 with Virtual Intelligent Queuing (Virtual-IQ) technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server. I/O virtualization with ConnectX-2 gives data center managers better server utilization while reducing cost, power, and cable complexity.
Specifications
The IBM Flex System EN4132 2-port 10Gb RoCE Adapter has the following specifications:
- Based on Mellanox Connect-X2 technology
- PCI Express 2.0 (1.1 compatible) through an x8 edge connector up to 5GT/s
- 10 Gbps Ethernet
- CPU offload of transport operations
- CORE-Direct® application offload
- GPUDirect application offload
- RDMA over Converged Ethernet (RoCE)
- End-to-end QoS and congestion control
- Hardware-based I/O virtualization
- TCP/UDP/IP stateless off-load
- Ethernet encapsulation (EoIB)
- RoHS-6 compliant
Ethernet specifications:
- IEEE 802.3ae 10 Gigabit Ethernet
- IEEE 802.3ad Link Aggregation and Failover
- IEEE 802.3az Energy Efficient Ethernet
- IEEE 802.1Q, .1p VLAN tags and priority
- IEEE P802.1Qbb D1.0 Priority-based Flow Control
- IEEE 1588 Precision Clock Synchronization
- Jumbo frame support (10 KB)
- 128 MAC/VLAN addresses per port
Supported servers
The following table lists the IBM Flex System compute nodes that support the EN4132 2-port 10Gb RoCE Adapter.
Table 2. Supported servers
| Description | Feature code |  |  |  |  |  |  |  |  |  |  |  |
| IBM Flex System EN4132 2-port 10Gb RoCE Adapter | EC26 | N | N | N | N | N | N | N | Y | Y | Y | Y |
The two-socket servers support the installation of one EN4132 2-port 10Gb RoCE Adapter. The four-socket servers support up to three EN4132 2-port 10Gb RoCE Adapters.
See IBM ServerProven® at the following web address for the latest information about the expansion cards that are supported by each blade server type:
http://ibm.com/servers/eserver/serverproven/compat/us/
I/O adapter cards are installed in the slot in supported servers, such as the p260, as highlighted in the following figure. The EN4132 2-port 10Gb RoCE Adapter is installed in slot 2.
Figure 2. Location of the I/O adapter slots in the IBM Flex System p260 Compute Node
Supported I/O modules
The EN4132 2-port 10Gb RoCE Adapter supports the I/O modules listed in the following table. One or two compatible switches must be installed in the corresponding I/O bays in the chassis. Installing two switches means that both ports of the adapter are enabled.
Table 3. I/O modules supported with the EN4132 2-port 10Gb RoCE Adapter
| Description | Feature code (e-config) | Supports the EN4132 adapter |
| 1 Gb switches | ||
| IBM Flex System EN2092 1Gb Ethernet Scalable Switch | 3598 | No |
| 10 Gb switches | ||
| IBM Flex System EN4091 10Gb Ethernet Pass-thru | 3700 | Yes |
| IBM Flex System EN4023 10Gb Scalable Switch | ESWD | Yes |
| IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch | ESW2 | No |
| IBM Flex System Fabric EN4093 10Gb Scalable Switch | 3593 | Yes |
| IBM Flex System Fabric EN4093R 10Gb Scalable Switch | ESW7 | Yes |
| IBM Flex System Fabric SI4093 System Interconnect Module | ESWA | Yes |
| Cisco Nexus B22 Fabric Extender for IBM Flex System | ESWB | Yes |
| 40 Gb switches | ||
| IBM Flex System EN6131 40Gb Ethernet Switch | ESW6 | Yes |
The following table shows the connections between adapters installed in the compute nodes to the switch bays in the chassis.
Table 4. Adapter to I/O bay correspondence
| I/O adapter slot in the server | Port on the adapter | Corresponding I/O module bay in the chassis |
| Slot 1 (Not supported*) | Port 1 | Module bay 1 |
| Port 2 | Module bay 2 | |
| Slot 2 | Port 1 | Module bay 3 |
| Port 2 | Module bay 4 | |
| Slot 3 (p460 only) | Port 1 | Module bay 1 |
| Port 2 | Module bay 2 | |
| Slot 4 (p460 only) | Port 1 | Module bay 3 |
| Port 2 | Module bay 4 |
The connections between the adapters installed in the compute nodes to the switch bays in the chassis are shown in the following figure.
Figure 3. Logical layout of the interconnects between I/O adapters and I/O modules
Supported operating systems
The EN4132 2-port 10Gb RoCE Adapter supports the following operating systems:
- AIX Version 6.1
- AIX Version 7.1
- Red Hat Enterprise Linux 6 for IBM POWER
- SUSE LINUX Enterprise Server 11 for IBM POWER
Mellanox OFED software packages are support for SUSE Linux and Red Hat Linux. Refer to http://mellanox.com/support for information.
Users should also update their systems with the latest Linux for POWER service and productivity tools from the following IBM website:
http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/home.html
Note: VIOS is not supported. Network install and network boot are not supported.
Regulatory compliance
The adapter conforms to the following standards:
- United States FCC 47 CFR Part 15, Subpart B, ANSI C63.4 (2003), Class A
- United States UL 60950-1, Second Edition
- IEC/EN 60950-1, Second Edition
- FCC - Verified to comply with Part 15 of the FCC Rules, Class A
- Canada ICES-003, issue 4, Class A
- UL/IEC 60950-1
- CSA C22.2 No. 60950-1-03
- Japan VCCI, Class A
- Australia/New Zealand AS/NZS CISPR 22:2006, Class A
- IEC 60950-1(CB Certificate and CB Test Report)
- Taiwan BSMI CNS13438, Class A
- Korea KN22, Class A; KN24
- Russia/GOST ME01, IEC-60950-1, GOST R 51318.22-99, GOST R 51318.24-99, GOST R 51317.3.2-2006, GOST R 51317.3.3-99
- IEC 60950-1 (CB Certificate and CB Test Report)
- CE Mark (EN55022 Class A, EN60950-1, EN55024, EN61000-3-2, EN61000-3-3)
- CISPR 22, Class A
Physical specifications
The dimensions and weight of the adapter are as follows:
- Width: 100 mm (3.9 in.)
- Depth: 80 mm (3.1 in.)
- Weight: 13 g (0.3 lb)
Shipping dimensions and weight (approximate):
- Height: 58 mm (2.3 in.)
- Width: 229 mm (9.0 in.)
- Depth: 208 mm (8.2 in.)
- Weight: 0.4 kg (0.89 lb)
Popular configurations
The EN4132 2-port 10Gb RoCE Adapter is designed to be used with the 10 Gb Ethernet switches listed in Table 3. The following figure shows one adapter installed in slot 2 of a p260 Compute Node, which in turn is installed in the chassis. Two IBM Flex System Fabric EN4093R 10Gb Scalable Switches are installed in I/O bays 3 and 4.
Figure 4. Example configuration
The following table lists the parts that are used in the configuration.
Table 5. Components used when connecting the EN4132 2-port 10Gb RoCE Adapter to the EN4093 10Gb Scalable Switch
| MTM / Feature code | Description | Quantity |
| 7895-23X | IBM Flex System p260 Compute Node | 1 to 14 |
| EC26 | EN4132 2-port 10Gb RoCE Adapter | 1 per server |
| 7893-92X | IBM Flex System Enterprise Chassis | 1 |
| ESW7 | IBM Flex System Fabric EN4093R 10Gb Scalable Switch | 1 or 2 |
Related publications
For more information, see the following resources:
- IBM U.S. Announcement Letter
http://ibm.com/common/ssi/cgi-bin/ssialias?infotype=dd&subtype=ca&&htmlfid=897/ENUS112-169 - IBM Flex System Fabric EN4093 10Gb Scalable Switch Product Guide
http://www.redbooks.ibm.com/abstracts/tips0864.html - IBM Flex System EN4091 10Gb Ethernet Pass-thru Product Guide
http://www.redbooks.ibm.com/abstracts/tips0865.html - IBM Flex System p260 and p460 Compute Node Product Guide
http://www.redbooks.ibm.com/abstracts/tips0880.html - IBM Flex System Information Center (User's Guides for servers and options)
http://publib.boulder.ibm.com/infocenter/flexsys/information - IBM Flex System Interoperability Guide
http://www.redbooks.ibm.com/fsig - IBM Redbooks® publication IBM Flex System Products and Technology, SG24-7984
http://www.redbooks.ibm.com/abstracts/sg247984.html - IBM Redbooks Product Guides for IBM Flex System servers and options
http://www.redbooks.ibm.com/Redbooks.nsf/portals/PureSystems?Open&page=pgbycat - IBM Configurator for e-business (e-config)
http://ibm.com/services/econfig/ - ServerProven for IBM Flex System
http://ibm.com/systems/info/x86servers/serverproven/compat/us/flexsystems.html
Special Notices
The material included in this document is in DRAFT form and is provided 'as is' without warranty of any kind. IBM is not responsible for the accuracy or completeness of the material, and may update the document at any time. The final, published document may not include any, or all, of the material included herein. Client assumes all risks associated with Client's use of this document.